In all, 53.4 percent of voters balked at releasing the 3.1 billion francs ($3.5 billion) needed to purchase the 22 planes from Sweden's Saab, according to official referendum results.
Polls ahead of the referendum predicted that voters would turn down the government plan, which called for the new fighter jets to replace the Swiss Air Force's ageing fleet of 54 F-5 Tiger aircraft to defend Switzerland's air space.
Citizens from French-speaking Switzerland were the biggest opponents of the deal.
Voters in Neuchâtel, for example, voted 69 percent against, while those from Geneva, 67 percent.
Almost 55 percent voted against the Swedish jets in the Italian-speaking canton of Ticino.
Support for the planes was strongest in German-speaking cantons, but a majority opposed their purchase in Zurich (51.4 percent) and Basel City (67.7 percent).
Critics argued Switzerland does not need new fighter jets and warned they could end up costing far more than the country bargained for.
The government, meanwhile, has for months been playing up the importance of the new planes for Swiss security.
"This decision will create a security gap," warned Swiss Defence Minister Ueli Maurer, the deal's biggest champion.
He told reporters an in depth analysis of the new situation was needed, but acknowledged: "We don't have a Plan B."
Maurer, who came under criticism for not piloting the government plan successfully, said the "no" vote was "was a defeat for the federal government and parliament — for me also."
The Swiss People's Party politician, however, said he could very easily live with the referendum result.
"In my life, I've experienced more defeats than victories," he told Swiss media.
Sweden's Saab also voiced disappointment, but not surprise.
"When you do business with Switzerland, you know exactly what you have in store," company spokesman Peter Larsson told the TT news agency.
"You have to respect their decision making model, and I think the company management included that in its calculations," he said.