Swiss bankers, who have put much energy into convincing Berlin to sign the deal, said they were "disappointed" at the vote in the upper house of the German parliament, the Bundesrat, while officials in Bern glumly said they had "taken note" of the outcome.
The rejected deal, which had been set to take effect on January 1st, called for Swiss banks to deduct taxes from German clients and transfer the tax revenues to Berlin, allowing the clients to remain anonymous.
The Swiss parliament and Germany's lower house had already approved the agreement, but members of the opposition in the Bundesrat had long threatened to block it since they consider it too soft on tax-dodgers.
Switzerland's attempts to reach a tax arrangement with Germany and other countries follows mounting objections to Swiss banking secrecy practices that have long made it possible for foreigners to stash undeclared funds in the country's banks.
Switzerland has signed deals similar to the one rejected Friday with Britain and Austria and is in talks to do the same with Italy and Greece.
However, the agreement with Switzerland's top trading partner Germany was to be the first to go into practice and was considered crucial given the close ties between the two countries.
Despite the setback, Swiss President Eveline Widmer-Schlumpf said on Friday that "Switzerland remains prepared to bring the ratification process with Germany to a successful conclusion."
The German government also expressed its disappointment at the rejection of a deal it has said could net the country around 10 billion euros ($12.8 billion) in additional tax payments.
"I am disappointed," Finance Minister Wolfgang Schaeuble told reporters in Berlin.
"There is no serious argument against this deal."
He said he would recommend that Berlin summon a parliamentary conciliation committee next week to seek a solution to the matter.
German finance ministry spokesman Martin Kotthaus also told reporters the nixed deal was "worth fighting for — we should try to make it work."
The Swiss Banking Association, which had been pushing for the deal, agreed, insisting the Bundesrat had "missed an excellent opportunity to adopt a fair solution" for both countries that would have "put an end to the fiscal dispute" between them.
Swiss critics of the so-called "Rubik" accords however said the German parliamentary rejection showed it was time to give up the bank secrecy practices and adopt a more transparent system based on the automatic transfer of tax details between countries.
"Eight million Swiss people cannot impose their will on 500 million Europeans," Socialist parliamentarian Andreas Gross told public broadcaster RTS.
"It's a question of values."
Switzerland's largest party, the populist right-wing Swiss People's Party (SVP), meanwhile called for the door to be slammed in Berlin's face, insisting that Bern make "no additional concessions to Germany."
With no agreement, Berlin will be stuck with the current long-winded system forcing it to make individual requests to Swiss courts for each suspected tax-dodger, the SVP pointed out.
According to German media, between 130 and 180 billion euros ($168 billion to $232 billion) in German assets are deposited in Switzerland.
If ratified, the tax deal would have entailed taxation rates of between 21 and 41 percent on such assets.
Swiss banks last year managed a total of 5.27 trillion Swiss francs (4.39 trillion euros, $5.66 trillion), half of which belonged to foreign clients.