A total of 18 objections have been filed by property owners and organizations over plans costing around 80 million francs to relocate the museum next to the railway station on the site of former locomotive sheds.
The opposition emerged after a period of consultation ended on Monday.
Barcelona architects Fabrizio Barozzi and Alberto Veiga won a competition last year to design the museum complex, which would also eventually house the Elysée photography and applied arts and design museums.
City and cantonal officials believed there would be little opposition to the complex hailed as a cultural benefit.
“The stumbling blocks are small,” Pascal Broulis, president of the cantonal government, said last month.
But the opposition now puts at risk a timetable that would have seen the cantonal government approve financing next year for the complex, which was to have opened in 2016.
Fourteen individuals and four organizations raised concerns ranging from property compensation to mobility in the neighbourhood.
Worries about the evolution of development around the train station and the protection of heritage buildings, were also cited.
Among the organizations contesting the project are the association for transport and the environment (ATE) and Patrimoine Suisse.
A spokesman for the Vaud department of training, youth and culture said the government did not want to comment on the objections until a deep analysis of the concerns had been made.
This will take three months to do.
Possible legal action could involve appeals up the supreme court.
An earlier project to relocate the cantonal fine art museum to a new building by the shore of Lake Geneva collapsed a few years ago in the face of opposition.
The museum, the second oldest of its kind in Switzerland, is unable to display much of its collection because of lack of space at the Palais de Rumine, its current 19th century home at Place de la Riponne.