The figure, reported by newspaper Neue Luzerner Zeitung on Wednesday, has been on the rise over the last few years.
It includes drivers of cars and motorbikes who have never passed a test and those whose permit has been taken away by police.
The statistics show that those under 25 are more likely to take the risk of driving without a permit.
The ASTRA figures only include those who have been caught, so the true number of unlicensed drivers could be much higher, writes the newspaper.
Neither does it include the estimated 8,000 new drivers with invalid temporary permits, a problem reported by The Local in January.
In Switzerland's two-tier permit system new drivers must complete two mandatory courses within two years of passing the practical test, but many fail to do so.
Nevertheless, the number of unlicensed drivers in Switzerland is nothing on neighbouring France, where official figures estimate between 300,000 and 450,000 people are driving without a permit.
One specialist told Neue Luzerner Zeitung that the true figure for unlicensed drivers in l’Hexagone could be as high as 2.7 million.