Japan's foreign ministry said the two were both men in their 60s, but declined to supply further details because their families had not given consent.
Two ran into bad weather on their descent from the summit of the Matterhorn, police in the canton of Valais said on Tuesday.
The two alpinists scaled the iconic 4,478-metre-high peak on Saturday but as they came down they were unable to make their way to the Solvay hut, a shelter at 4,003 metres, because of the weather, cantonal police said in a news release.
On Sunday evening, a Polish climber spotted the body of one of the Japanese men on the Hörnli ridge, above the hut.
He immediately contacted Air Zermatt, the emergency helicopter service, but bad weather prevented rescue workers from recuperating the lifeless body until Monday, police said.
The victim likely died from the cold, according to investigators.
The formal identification of the dead man is under way.
Police said a search was ongoing for his colleague, but the Tokyo official said the body was later discovered.
The Matterhorn is a popular mountain for climbers.
The 150th anniversary this year of the first ascent of the mountain, straddling the Italian border and overlooking the resort town of Zermatt, makes it even more of a draw.
But climbing it without a guide remains a risky business that has claimed more than 500 lives since 1865.
Last week, police said DNA tests on human remains found on the Matterhorn last autumn proved to be those of another pair of Japanese climbers, men aged 20 and 21, who perished on the mountain in August 1970.