Solidar Switzerland said it was time to "blow the whistle" on FIFA boss Sepp Blatter.
"Evictions, human rights violations and exploitation are overshadowing preparations for the world championship," the group said.
"The people affected are those already living in poverty."
FIFA, based in the Swiss city of Zurich, has insisted the protest wave in Brazil is a domestic issue.
But Solidar Switzerland said FIFA should face up to its responsibility as it owns the rights to the World Cup, its main money-spinners.
It demanded that FIFA speak out against an image conscious clean-up of Brazil's working class "favela" districts, charging that more than 150,000 people will have been driven from their homes by the time the tournament kicks off in June 2014.
It also slammed restrictions on some 300,000 street vendors, inspired by a desire to protect World Cup branding rights, saying it would deprive the poorest of their livelihoods.
Nor should FIFA should stay silent about labour exploitation on World Cup construction sites and in factories making tournament souvenirs for fans, it said.
Solidar Switzerland also blasted tax breaks for both FIFA and World Cup sponsors such as Adidas, Coca-Cola and Budweiser.
They "refuse to allow their billions of profits to be taxed in Brazil," the group said.
"But the Brazilian population must bear the cost of the World Cup," it said.
"It is clear where this leads: Brazil will be left with a mountain of debt and social benefits will come under pressure.
"In the end, the 50 million poor of Brazil will pay for the World Cup," unless those who benefit are taxed.
Solidar Switzerland underlined that 2010 World Cup host South Africa was left with debts of almost $3 billion, while FIFA's profits hit $3.2 billion.
Polls show that three-quarters of Brazilians back the protests but that two-thirds are overall in favour of the World Cup, despite anger at the billions of dollars the event is costing, money many feel could be better spent on education, transport and housing.