Nicolas Pictet, president of the Swiss Private Bankers Association (SPBA), highlighted "an increasing difficulty in accessing markets and a rise in financial protectionism owing to ever-stricter rules," at a press conference in Bern.
Pictet, partner at the bank of the same name, also pointed the finger at "a rise in infrastructure costs due to the weight of regulations."
The financial hubs of Zurich and Geneva ranked eighth and thirteenth in the world last year after occupying sixth and ninth position in 2009, SPBA vice-president Christoph Gloor told the meeting.
"Where market access is concerned, we are regularly confronted by the problem of our relations with the European Union," Gloor, from Basel-based La Roche & Co added.
The banker said the Swiss financial sector must be careful it does not find itself "offside" where EU rules are concerned.
Swiss banks have in recent years had to bow to the pressure of Europe's key players and the United States in their crusade against tax havens.
They relaxed their tradition of banking secrecy in 2009 when agreeing to share the details of clients suspected of tax evasion.
"Swiss banks have realised that it is not enough to act in line with Swiss law but you must also respect foreign law," Pictet said.
Switzerland has signed agreements with Britain and Germany in order to address tax evasion which allow Swiss banks better access to their markets.
In Britain's case, Gloor said however that "the Swiss negotiators obtained a clarification of regulatory requirements, but no real improved market access."
Measures facilitating market access in Germany have yet to be finalised by authorities in both countries, he said.