Preliminary results of a referendum showed on Sunday that 53.35 percent of Tyrol residents opposed the bid.
An IOC spokesman said it would have liked to continue its preliminary talks with Innsbruck, which is the state capital of Tyrol.
"The IOC shares the disappointment for the Austrian Olympic Committee, the many supporting athletes and the promoters of the project who worked on it with so much energy and committment," the spokesman said.
Opponents in Tyrol were worried about the high financial cost of hosting the Winter Games, the third in Innsbruck after 1964 and 1976, as well as the environmental impact.
Fresh talks with interested cities from America, Asia and Europe will now go ahead, including Sion in Switzerland.
"We are certain that an excellent host city for the Olympic Winter Games 2026 will emerge from this process," the spokesman said.
"The exploratory talks will also allow the IOC to better communicate about the benefits of the new candidature process for host cities, highlight our reforms that make it possible to have a sustainable, feasible and cost-effective Olympic Winter Games."
Austria's Olympic Committee had said no extra major infrastructure was needed and that the 1.2-billion-euro ($1.4 billion) cost would be recouped by revenues.
Two previous referendums among residents of Innsbruck -- but not the whole state -- in 1993 and 1997 came out against previous bids as well.
The final results of the latest referendum are due later on Monday, although the outcome is not expected to change.
Hosting the Olympics, either in winter or summer, is by no means universally popular.
In 2015 the German city of Hamburg voted against bidding to host the 2024 Summer Games, while Rome and Budapest also withdrew due to public pressure.
Despite the success of its athletes, Austria has not hosted the Winter Olympics since 1976. A bid by Salzburg for 2014 lost out to Sochi in Russia.
The next Winter Olympics will be held in Pyeongchang, South Korea in 2018 and in Beijing in 2022.
The official bidding process for 2026 starts next year with the IOC due to make a decision in 2019.
Only Sion, plus Calgary in Canada, have expressed an interest so far.
However the Swiss government has not yet given Sion approval to go ahead and bid.
In February this year residents of Graubunden voted against the idea of the canton hosting the 2026 Games.