Delegates from the party on Saturday gave the green light for a referendum on overturning the agreement.
The initiative will be launched by the end of 2017, the party said in a statement, though there are two possible variants of the vote.
Either it will call simply for the termination of Switzerland's agreement of free movement with the EU, or it will call for immigration to be a national matter - effectively putting an end to any future agreements as well as the current one.
SVP National Councillor Marco Chiesa said party members should "not to afraid to be categorized as 'populist'" and added: "They will try to portray us in a bad light, to scare the population."
He said that his native region, Italian-speaking Ticino, had suffered from an "emergency in the ruined labour market" due to free movement of people. Tensions have risen in the region between locals and cross-border workers, and in September 2016 an SVP initiative called for companies to give Swiss workers preference over foreigners.
The vote will be planned by the SVP together with anti-immigration body, the Association for an Independent and Neutral Switzerland (AUNS), whose members voted unanimously in May to develop an initiative aimed at ending Swiss-EU free movement.
A survey carried out by Le Matin in mid-May showed that only 37 percent of those polled would support an end to free movement. While the vast majority of SVP voters were in favour, supporters of the Socialists, the Greens, the Christian Democrats and the Liberal-Radicals would largely reject the initiative, that survey showed.