Grigor Dimitrov earlier stunned seventh seed Wawrinka 6-1, 6-2 in the third round.
The exit of 2014 Australian Open champion Wawrinka came a year after he won his first and only Masters 1000 title here.
The Lausanne native was well off his game, out in 54 minutes with 41 unforced errors and just four winners against ninth seed Dimitrov.
The ninth-ranked is now in danger of losing his top 10 ranking place.
World number two Federer missed the chance to reach the last eight in the principality, weighed down by 38 unforced errors against Monfils.
After losing a break for 3-1 in the opening set and standing 5-3 in the second-set tiebreak, 17-time Grand Slam winner Federer could not make much progress against the flashy French opponent who beat him in a Davis Cup rubber last November.
But the Swiss is far from giving up hope only a few days into the spring clay campaign leading to the Roland Garros start on May 24th.
"It was a good week for me anyway to come here and practice with the best, play a couple of matches," Federer said.
"At least gives me some information, if I'm trying to be a bit positive right now," he said.
"I'm going to go back to Switzerland and practice really hard, you know, from whatever the plan's going to be from tonight on until Istanbul."
Fourth seed Milos Raonic beat Spain's Tommy Robredo 6-3, 3-6, 6-3, while sixth seed Tomas Berdych defeated Roberto Bautista Agut 7-6 (9/7), 6-4.
Federer reached the final last year for the fourth time in his career — his loss is a setback just six weeks before the second Grand Slam of the season in Roland Garros.
Federer also missed his chance to win 200 ATP matches on both hard and clay courts.
The Swiss, who has a 633-129 record on hard courts, now holds a 199-63 record on clay.
World number one Novak Djokovic and Rafael Nadal meanwhile later posted wins to head into the quarterfinals.
Top seed Djokovic dispatched Andreas Haider-Maurer of Austria 6-4, 6-0 to line up on Friday against US Open champion Marin Cilic, the number eight, who defeated French 11th seed Jo-Wilfried Tsonga 6-3, 7-6 (7/5).
Djokovic, perhaps rushing to get home a few hundred metres to his flat to help with the evening bath of new son Stefan, needed only 56 minutes to win.
Nadal fought off a challenge from John Isner to earn a 7-6 (8/6), 4-6, 6-3 win and said the success marked a big step in his comeback effort.