Local residents complained that the stilted wooden playhouse and slide in the grounds of the property in Lenzerheide-Valbella, in the canton of Graubünden, impaired their view of the mountains, according to media reports.
Authorities upheld the complaint, judging that the playhouse breeched planning permission rules that prohibit construction in that part of the property’s grounds.
Andrea Bianchi, lawyer for the complainants, said his clients had nothing against the play area in principle but that it should not have been constructed within the building exclusion zone, reports newspaper Blick.
Neighbours are also objecting to the presence of a two metre-high wire fence around the property and the planting of tall trees in the garden, for which Federer was granted permission.
Federer and his neighbours are trying to find an amicable resolution to the ongoing dispute, said Bianchi.
The tennis star and his wife Mirka bought the 8,000m2 piece of land in 2008 before demolishing the house and building two villas.
According to Blick, Federer, from Basel, has had a long association with Valbella, having spent his childhood skiing in the area.
The couple have two sets of twins. Girls Myla and Charlene were born in 2009, while twin boys Leo and Lennart arrived in May this year.