The incident happened in Mastrils near Chur at around 2pm on Thursday, Graubünden police said in a statement.
In starting work on the shelter the man dug a hole and disturbed a wasps' nest.
The wasps promptly started to attack him, so he lit a flaming torch to try and fumigate the nest.
Unfortunately he didn't put the torch out properly and it soon set fire to the hay stored in the shelter.
Some 15 firefighters attended the scene. Due to the inaccessibility of the area, a helicopter was also used to help extinguish the flames.
Photo: Graubunden police
However the haybarn could not be saved and was completely destroyed in the flames, along with two adjacent trees.
The man was uninjured by the flames but was treated by a doctor for wasp stings, said police.
The risk of forest fires is currently a worry in parts of Switzerland, where a prolonged period of dry weather has increased the risk.
Last week the canton of Valais raised its forest fire threat alert to 5 for some areas in the region – the highest level on the scale, denoting a ‘very high' risk.