A full 54 percent of voters in the canton came out against paying up to 100 million Swiss francs ($102 million, 86 million euros) to host the games.
With Valais out of the picture, six cities remain candidates for the 2026 Olympics, including Stockholm, Calgary and Graz, with the winning bid set to be announced in September 2019.
The vote marks the latest in a series of failed attempts by Valais authorities over the past half century to convince people they should host the Olympics.
The question was put to a vote in 1968, 1976, 2002 and 2006, but was turned down each time.
Switzerland, home to the International Olympic Committee, has not hosted the Games since 1948.