Second seed Federer, who had just returned to the tour after a two-month break, joins world number one Novak Djokovic in being eliminated early in the Spanish capital and his lack of match practice showed.
One break was enough for Nishikori to take the opening set 6-4, but Federer roared back in the second to take it 6-1.
However, with the momentum seeming to be with the 17-time Grand Slam champion, it was Nishikori who edged ahead when he broke to move 3-1 in the third.
And after comfortably holding serve in his next two service games, the 23-year-old broke again to seal a first career win over Federer.
"He was my idol actually, so to beat him, that was one of my goals for my tennis career," Nishikori told a press conference.
"That it happens on clay is amazing because, you know, it's not my really favourite surface.
"I feel good, of course. I wasn't really expecting this, it is not easy, 6-2 in the third, that's not easy against him."
Three-time champion Federer, who beat Nishikori in straight sets in their only other previous clash in Basel in 2011, admitted Nishikori was a deserving winner.
"I'm disappointed with my match. Today Nishikori played better and the best man won," said Federer.
Rafael Nadal eased into the last eight with a 6-2, 6-3 win over Mikhail Youzhny and will next face either Tommy Haas of Germany or Spanish compatriot David Ferrer.
Meanwhile, Swiss player Stanislas Wawrinka has reached the quarterfinals in Madrid, after beating Grigor Dimitrov in three sets (3:6, 6:4, 6:1).
The Bulgarian player had pulled off a major feat on Tuesday, eliminating world number one Novak Djokovic.
He proved less of a challenge for Wawrinka, who dispensed with his opponent in 103 minutes.