The supreme court found in favour of a police officer who had used the slurs when he arrested an Algerian suspected thief.
The incident took place at a trade fair in the northern city of Basel in April 2007, where the Algerian was detained for allegedly snatching a Russian man's bag.
After checking the suspect's identity papers, the policeman discovered that he was an asylum-seeker and proceed to insult him.
As a result, the officer received a suspended fine for breaking the country's anti-racism laws.
After the penalty was overturned by another court, the case worked its way up to the top of the Swiss justice system.
The Lausanne-based federal court said that while such terms were clearly insulting, they were too broad to fall foul of anti-racism rules because they did not target a particular ethnic group, race or religion.
It also said calling someone "dirty" — even if the individual's nationality was mentioned — was not against the anti-racism law.
The court ruling was published as the UN Committee for the Elimination of Racial Discrimination (CERD) in Geneva issued its findings from a review of Switzerland's record, a regular procedure for signatory states.
CERD member Anastasia Crickley, who was in charge of the review, said of the court ruling: "It sounds to me like a very good case for the recommendation that we have made, that what's needed in Switzerland is a clear definition of direct and indirect racial discrimination and legislation."
She said the CERD was also concerned about referendums in Switzerland on toughening immigration and asylum rules and banning the construction of Muslim minarets.
On February 9th, Swiss voters narrowly approved scrapping rules that gave European Union citizens free access to their labour market after campaigners argued the neutral country was being "swamped" with immigrants.
"Migration laws are needed, but we're concerned at the increasingly protectionist way in which these are being encapsulated and presented, and the xenophobic tone overall that tends to be associated with them," Crickley said.