The IOC, also based in Lausanne, took the decision after the new analysis of their dope tests taken at the Athens Games revealed traces of steroids.
Ukraine's Yuri Belonog was stripped of his shot put gold medal, Belarus' Ivan Tikhon of his silver in the hammer, and Irina Yatchenko of Belarus and Russian Svetlana Krivelyova lost their bronze medals in the women's discus and shot put respectively.
It is the second Olympic medal Tikhon has been stripped of, having had his bronze from the 2008 Games taken away also for a doping offence.
There is a fifth medallist from the 2004 Games who is at risk of being disqualified, but whose name will be revealed later by the IOC because of a delay in the procedure of the athlete's test.
Since the Athens Games all dope tests taken at the Olympics have been stored in a large deep freeze in the basement of the anti-doping laboratory in
Lausanne for a duration of eight years.
The IOC had asked the laboratory to reanalyse 105 tests taken at the Games in Athens, which holds the unwanted record for the most failed dope tests in