The man's 55-year-old wife, their two adult daughters and the 65-year-old Swiss pilot were hurt when the balloon fell about 50 metres (160 feet) near Montbovon, cantonal police said.
The family from California were vacationing in the region of the mountain resort of Gstaad, in the canton of Bern, the SDA news service reported.
The balloon had taken off from Château d'Oex, which is well known for its annual balloon festival.
At 8.35am, shortly before it was due to land near Monbovon, it hit a power line, police spokesman Pierre-André Waeber said.
The American man died on the spot despite rescuers' attempts at resuscitation.
The Malibu Times said he was a descendant of Malibu's founding family.
His wife and two daughters aged 20 and 24 were transported by helicopter to hospitals in Bern and Lausanne, and the pilot was taken away in an ambulance.
Air accident investigators have launched an investigation.
The balloon belonged to the five-star Grand Hôtel Park in Gstaad.
The pilot was an "experienced and recognized balloonist," the hotel said in a statement.
"All our thoughts go to the family and the friends of the victims of this terrible accident."
The SDA news service said the site where the accident occurred is considered safe by balloon pilots.
There were several power lines but these were clearly visible, pilot Raphael Zuccollo said.
Zuccollo said conditions on Tuesday morning had been perfect and visibility had been good.