A body was found a few hundred metres (yards) from the scene, an AFP photographer said, but it was unclear if it was connected to the shooting, which took place inside the city's Islamic Centre, near the central station.
At around 5:30pm, an unknown person entered the Islamic Centre, where several worshippers were gathered, and began shooting, Zurich police said in a statement.
He "fired several shots at the worshippers," police said.
"Three men, aged 30, 35 and 56, were injured, some seriously. The suspect then escaped from the mosque in the direction of Central Station," the statement said.
Witnesses had described the shooter as a man aged around 30 wearing dark clothes and a dark woollen cap, police said.
"The city and cantonal police immediately launched a major search for the unknown perpetrator," the statement said.
Swiss media said the three wounded people were found in the street where the prayer hall is located.
Police have urged witnesses who were in the area around the time of the shooting to come forward.
They did not immediately provide any details about the suspected motive for the shooting.
Around a dozen people were inside the prayer hall at the time of shooting, the ATS news agency reported, citing a number of people on site, adding that a prayer service had been scheduled for 4:45pm.
The worshippers were mainly from North Africa, Somalia and Eritrea, ATS reported.
Police have sealed off the area and were late Monday scouring the streets for the shooter, carrying lamps and using sniffer dogs.
They have blocked of Eisgasse, the street where the Islamic Centre is located, as well as a nearby bridge, where the body was found, the AFP photographer said.
The body was discovered on the river bank, underneath the bridge, and had been draped with a white sheet.
A number of Swiss mosques, including one near Zurich and the main one in Geneva, have in recent months been accused in media of allowing or encouraging the radicalisation of their worshippers, especially the younger members of their congregations.
Switzerland, a country of some eight million people, reportedly has around 450,000 Muslims.