The deal was thrown out late Wednesday because it was judged too lenient for tax evaders.
Under the terms of the deal, which would have come into effect on January 1st, Germans who have parked undeclared assets in Swiss bank accounts would have been able to regulariize their holdings while retaining their anonymity.
The deal negotiated by Berlin and Bern would have resulted in Swiss banks deducting taxes from German clients and transferring the tax revenues to Berlin, allowing the clients to remain anonymous.
But the deal had to be ratified by both parliaments and while the lower house or Bundestag approved it, it was rejected by the upper house or Bundesrat in November.
Members of the opposition Social Democrats and Greens in the Bundesrat argued the deal was too soft on tax-dodgers.
The government subsequently tried to save the deal by putting it to a mediation committee made up of members of both the Bundestag and the Bundesrat.
However, the committee voted against the deal late Wednesday.
Switzerland expressed disappointment at the collapse of the agreement.
"We regret that Germany has not ratified the agreement with Switzerland," said Swiss president Eveline Widmer-Schlumpf, complaining that the current "unsatisfactory" situation would be allowed to persist.
Berlin and Bern have been embroiled in a spat over tax since 2010 when German authorities raided branches of Credit Suisse bank in 13 German cities after buying data on suspected tax frauds.
Switzerland reacted angrily, saying the data were stolen in violation of its banking secrecy laws.
As much as 180 billion euros ($222 billion) in German assets are hidden in Switzerland, according to unconfirmed media reports.