The incident occurred at around 2.35pm in Mathod, a village of 556 people in the Jura-Nord Vaudois region, west of Yverdon-Les-Bains.
Vaud cantonal police said an investigation has been launched into why the wall fell down on the man.
Despite the rapid intervention of ambulance attendants and a doctor from the local hospital, the worker, a resident of Mathod, died on the spot, police said.
Inspectors from SUVA, the accident insurance company, and the service for the prevention of accidents in agriculture (SPAA) arrived on the scene to investigate.
The local prosecutor has opened a criminal investigation into the incident.
Gendarmes from Yverdon-les-Bains and police inspectors are probing the case.