In a fourth-round match at Roland Garros that seesawed back and forth, Gulbis, 25, finally prevailed 6-7, 7-6, 6-2, 4-6, 6-3.
The 17th seed overpowered Federer, 32, with serves regularly clocking in at more than 215 kilometres an hour.
Fourth-ranked Federer, winner of the French Open in 2009, reached the fourth round for a record-breaking 12th time on Friday after beating Russian Dmitry Tursunov in four sets.
But the Basel native was unable to follow through on Sunday against Gulbis who made few mistakes.
Federer's loss marks the first time he failed to make the quarterfinals at Paris in 10 years.
Gulbis's win, meantime, puts him in the quarterfinals of a Grand Slam tournament for the first time since he achieved the trick at Roland Garros six years ago.
Before the match with Federer, the Latvian said he had a "game plan" to beat the 17-time Grand Slam winner.
"I know how I'm going to beat him."
He sparked controversy when he described the sport's big four of Federer, Rafael Nadal, Novak Djokovic and Andy Murray as boring.
"He has a superb image of the perfect Swiss gentleman," Gulbis said.
"I respect Federer but I don’t like it that young players try to imitate him."