The lower house of parliament, or National Council, shot down the controversial "Lex USA" accord, which would have temporarily lifted Switzerland's long-sacrosanct banking secrecy and allow banks to draw a line under past wrongdoing.
A total of 123 members voted against giving emergency approval to the deal, amid anger over Washington insisting it was non-negotiable, had to be force by July 1st, and that no details would be revealed until then.
Sixty-three members were in favour, and four abstained.
It was not clear immediately how Washington would react, while Swiss Finance Minister Eveline Widmer-Schlumpf, who backed the accord, said the government was "exploring all options to find a solution".
Opponents from across the political spectrum lined up to blast the deal, saying they would not be railroaded into approving it, and that it was up to the banks to deal with the consequences of their actions.
"We must not cave in to pressure from other countries," said Christoph Blocher, a leading figure in the right-wing Swiss People's Party, the country's largest political force and staunchest defender of banking secrecy.
"We must avoid creating a precedent, because other countries would want to follow the US lead," Blocher said.
"It would be the end of Switzerland," he said.
"It's up to the banks to find a solution."
Susanne Leutenegger Oberholzer, of the centre-left Socialists, described the accord as a "blank cheque" and said parliament could not be expected to approve it without knowing the contents.
"The banks know exactly why we're in this situation and who's to blame," she said.
"They violated another country's law and should face the consequences."
The vote came a day after G8 leaders agreed to chase tax cheats and crack down on corporate schemes to reduce tax bills, and just hours after Switzerland's upper house, the Council of States, approved the deal.
The lower house had the final word on the issue, however.
Swiss banks are believed to hold billions of dollars belonging to American citizens who have not declared these assets to US tax authorities.
The take-it-or-leave-it settlement proposed by Washington was approved by the Swiss government at the end of May.
But parliamentary approval was needed for a one-year waiver of Swiss banking secrecy rules, and lawmakers insisted they would not be railroaded into making a hasty decision.
Without an accord, the banks risk breaking Swiss law if they give US authorities the names of American clients and of employees that handled their assets.
The deal has been seen as crucial if Swiss banks are to escape the threat of a raft of lawsuits in the United States on charges of abetting tax evasion that could end in massive financial penalties as well as being barred from the big and profitable American market.
The banking sector has said it wants to settle an issue which has poisoned ties with the United States since 2009, with the economic crisis having put the issue of tax evasion squarely in the global spotlight.