Shortly before 5.30pm, several swimmers alerted police that a lifeless body was floating in the water, Zurich police said in a statement.
Police and security personnel quickly pulled the man out of the water, and attempted to resuscitate him, but to no avail.
"Unfortunately for the as yet unidentified man, help came too late," the statement said.
Police were working to determine the circumstances surrounding the drowning, it said.
A record 10,700 people took part in the 29th annual swim spanning 1.5 kilometres (0.9 miles) across Lake Zurich from Mythenquai to Tiefenbrunnen.