Confirming a report by local media, the museum said that the painting was an 1887 portrait called Madame La Suire by Swiss painter Albert von Keller, who was popular in Berlin and Vienna at the beginning of the 20th century.
The painting was acquired by the Sommerguths, a rich Jewish couple from Berlin who had a substantial collection of 106 paintings, including Renaissance masterpieces as well as works by Camille Pissaro.
But after the Nazis came to power in Germany, they were forced to give up the collection, which was sold during an auction in 1939.
Alfred Sommerguth, who acquired his fortune as co-director of the German tobacco manufacturer Loeser & Wolff, managed to flee to Cuba in 1941 at the age of 82, before reaching New York where he died a destitute in 1950.
His wife Gertrude died four years later.
The painting was found by chance, during an exhibition on von Keller organised by the Zurich museum.
"We had received a heritage donation of 350 von Keller paintings from the widow of a rich Zurich collector, on condition that we organise an exhibition of the paintings," said a museum spokesman.
In New York, the authorities charged with finding paintings stolen or confiscated from Holocaust victims noticed the portrait and sought explanations from Zurich.
It turned out that the donation to the museum was made in 2006, after the death of the widow of Oskar Mueller, a von Keller collector.
After the origins of the painting were verified, the Sommerguth heirs decided to leave the painting with the Zurich museum. However, they asked for a sign be put up to indicate its origins and the fact that it was part of Nazi spoils.
According to the museum's spokesman, the painting is worth an estimated 10,000 francs ($10,982) today.
An investigation is ongoing regarding the origins of two other paintings from the Müller collection, said the spokesman.
It is not the first time that a painting from the Sommerguth collection was found. In 2008, another painting was found on a Sotheby's auction catalogue.
Titled "Scene of a forest with a castle, on the water front," by Karl Blechen, the work was then taken off the sale and returned to the family's heirs.