The final results showed 52.7 percent of voters said no to changes to the current system which would have raised women's retirement age by a year to 65 as well as hiking VAT to help fund benefits.
The result of the tight vote is a setback for the government, which said the changes would help avoid deficits in a social security scheme facing pressure from an ageing population, with baby boomers entering retirement.
Rising life expectancy, now 83 years, has added further strain.
But critics argued that the changes were too modest to rescue a retirement system in financial peril.
Only 47 percent of Swiss voters cast ballots in favour of the reform, which was approved by parliament in April.
The Swiss did however vote to put the issue of food security in the constitution, after it was requested by the agricultural industry.
The vote is the latest in Switzerland's direct democracy system, which includes four referenda a year on major national issues.
Irrational voting behaviour?
Backed by leftwing and centrist parties, the reform proposal also includes more retirement age flexibility and increased contributions from employers and workers.
The plan, officially known as Pension Reform 2020, would only have come into force if voters also approved the corresponding VAT increase.
Ahead of the vote, opinion polls suggested the result was too close to call, with the pro-reform side slipping marginally in recent weeks.
Support for the plan stood at 53 percent in mid-August but had fallen to 51 percent days ahead of the vote, while just 50 percent of respondents said they backed the VAT increase.
Rightwing parties campaigning against the move said it does not guarantee the survival of the pension system after 2027 and that beyond that date, future generations will be forced to come up with yet another rescue package.
Because the VAT increase requires a change of Swiss constitutional law, it needed double approval to pass, including majority support among voters and majority support in more than half of the country's 26 cantons.