The controversy first broke last week when federal migration authorities said the small northern town of Bremgarten, with 6,500 residents, had been permitted to deny residents of a new asylum centre access to certain public spaces.
Initial reports that the asylum seekers would be barred from the public pool, gyms and even the town library and churches sparked outrage and charges of segregation and discrimination from rights activists.
Swiss migration authorities maintain the reports were based on a misunderstanding, insisting the asylum seekers will only have restricted access to so-called "sensitive areas" where access is also restricted to the Swiss public, like schools and sports facilities during school hours.
The rules were merely aimed to help "organize the cohabitation between the asylum seekers and the town population", Federal Migration Office spokeswoman Gaby Szoelloesy told AFP.
Denise Graf of Amnesty International's Swiss section was unconvinced, maintaining that the rules, which require asylum seekers to among other things request permission from the town before accessing the pool, "are clearly discriminatory".
And Bremgarten is not the only place where asylum seekers face such discrimination, she said, accusing Bern of making any number of concessions to avoid protests from the communities chosen to house a growing number of temporary asylum centres.
Residents in one centre near the central city of Lucerne had for instance been barred from taking the shortest route to the train station, while other centres impose strict curfews, she said.
Police meanwhile moved in this week to remove ten asylum seekers who had been camped out for days at the Solothurn train station in northwestern Switzerland to protest their living conditions in a subterranean bunker they described as "unworthy of a human being".
Switzerland is one of the countries in Europe that welcomes the most asylum seekers in proportion to its population, with some 48,000 people currently in the process of applying for asylum in the small Alpine nation, including 28,631 who arrived in 2012 -- the highest number since 1999.
Amid the recent spike in refugees, Switzerland has been rushing to open a slew of temporary asylum centres.
But the Swiss public, which in June overwhelmingly voted to tighten the country's asylum laws, often resist the creation of such centres in their neighbourhoods.
Szoelloesy acknowledged that four of the ten communities asked to host new centres since last year had been granted the right to set up "sensitive areas", like Bremgarten, to help avoid "bad feelings" towards the asylum seekers.
She echoed Justice Minister Simonetta Sommaruga's claim last week that in Switzerland, "fundamental rights are not negotiable", and insisted: "our asylum system is fair and humane".
Gerry Simpson, senior refugee researcher for Human Rights Watch, welcomed Sommaruga's statement, saying he hoped it meant the rules agreed upon in Bremgarten "may now be revised, and that they won't be repeated elsewhere in Switzerland".
"Ninety-nine percent of (asylum seekers) are law-abiding people who just want to live their lives . . . so I don't think (there is a need) to assuage fears with special measures," he said.
"They just need to treat them lawfully and fairly."
The ten Solothurn demonstrators, who spent five days protesting their living conditions in a windowless military shelter with insufficient ventilation in the nearby village of Kestenholz, do not feel they have been treated fairly in Switzerland.
"There's no air, no windows and 30 people sleeping together . . . that's not the way it should be," Turkish Kurd Abdullah Ochalan complained to public Swiss radio RTS before the protesters were cleared out of the train station early Tuesday.
Graf decried the increased use of such shelters.
They are "underground, they stink, there's no air, no light, and its always noisy", she said.
"For people who have been traumatised especially it is horrific."
Not everyone shares her sympathy.
"To tell you the truth, I think that if they are not satisfied with the laws in the country that is housing them they would be better off returning to their countries or going elsewhere," young Solothurn local Maria told RTS.