In response to French finance minister Michel Sapin's charge, authorities in Geneva insisted they had cleaned up their act and suggested he come and check things out for himself.
"Had (the comments) been made several years ago, we could have taken them seriously. Today, they are simply unacceptable," Pierre Maudet, economics chief in Geneva's cantonal government, told Switzerland's One FM.
Maudet's chief of staff, Patrick Baud-Lavigne, added that "a letter has been sent to Mr. Sapin inviting him to come make an on-site inspection."
Sapin's supposed insult was made at a G20 meeting earlier this month.
He called free ports a "weak link" in countering terrorist financing as they helped Islamic State group (IS) sympathisers traffic artistic and archaeological treasures stolen from Iraq and Syria.
"These free ports exist in certain countries. I am referring in particular to Switzerland," he said.
A place to hide loot?
The original concept behind free ports was to provide places to store goods in transit, protecting them from excessive tariffs and bureaucracy.
Perhaps not surprisingly, highly secured warehouses where basically anything could be stashed tax free with few questions asked became increasingly attractive to nefarious operators.
Switzerland became a free port hub, mirroring the country's banking system which has long been known as a place to store enormous wealth with little scrutiny on the origins of funds.
The Geneva free ports, established in 1854, now include two separate heavily guarded facilities where endless rows of valuable wine are kept alongside thousands of works of art including Picassos, Van Goghs and Monets -- assets likely worth billions of dollars.
In recent years, questions have been raised about some of the goods held at Geneva's secretive warehouses.
In April, port authorities sequestered Amadeo Modigliani's painting "Seated Man With a Cane", which had allegedly been looted from a Jewish art dealer by the Nazis and has been valued at $25 million (23 million euros).
Questions about the piece emerged after the Panama Papers leaks uncovered the identity of its owner.
There has, however, been no public confirmation of Sapin's suggestion that treasures looted by jihadists had been kept in Geneva.
Baud-Lavigne said "there is a little bit of fantasy and misunderstanding around the free ports."
But officials have conceded the Geneva facilities had previously been used to traffic archaeological goods and have implemented a series of transparency measures.
Geneva free ports chief Alain Decrausaz told AFP these new rules should ease concerns about the facility being exploited for terrorist funding.
Federal government guidelines approved earlier this year require the nameand address of merchandise owners be mentioned on inventory lists, he said.
And last month, new systems for the "control of archaeological antiquities "were adopted, requiring anyone transporting such goods be approved before arriving at the port.
If there any doubts as to the provenance of any item, international auditing giant KPMG can ask for more information or conduct a "complete inspection" of the cargo, Decrausaz explained.
One unresolved issue is customs agents.
Decrausaz said he has repeatedly asked for more, but the federal government claims that with 1.3 million migrants and refugees pouring into Europe over the last two years, Switzerland can't afford to take staff away from border controls to beef up customs at the port.