Federer, desperate to recapture his best form, went into the match boasting an overwhelming 6-1 record against the flamboyant Frenchman but although he showed flashes of inspiration, he went down 6-4, 6-7 (5/7), 6-3.
Switzerland's eighth seed Stanislas Wawrinka, meanwhile, beat Canada's Milos Raonic, the 10th seed, 7-6 (7/2), 6-4.
Federer's defeat follows a fourth-round loss at the US Open and a second-round exit at Wimbledon to players he would have brushed aside at his peak, results that have contributed to a slide to number seven in the rankings.
The 17-time Grand Slam champion described the third-round defeat to Monfils as "a tough loss" but said the margin between success and failure was small.
Federer still believes he can compete at the sharp end of the men's game, saying he can turn his season around as he bids to qualify for next month's season-ending World Tour Finals in London, featuring the top eight players of the season.
"I can still finish strong. I believe that," the Basel native said.
"There's not much time left," he acknowledged.
"But if I do qualify for London, that gives me an extra shot there," Federer said.
"I usually play well indoors, so I hope this year's going to be one of those years again," he said.
"It's just important not to worry too much, to be honest — it's important to keep on doing what I'm doing," added the 32-year-old.
"Obviously I might get tougher draws as we move along with my ranking not being in the top four anymore," he said.
"But that's OK, I don't really care that much about the rankings if I'm not world number one."
Monfils said Federer still inspired fear in opponents.
"When he's on the court, he's still Roger," he said.
"He can rip it from anywhere," Monfils said.
"He can do stuff that not many players can do."
The fact Federer has had a few losses helps, he said.
"So it's less fear, but still, you know, you're not going, 'OK, I play Federer, it's an easy draw'. No, definitely not."
Playing his first tournament since the US Open, Federer started in the worst possible way, limply losing his first service game to immediately hand the initiative to Monfils, who went on to take the opening set.
Monfils, clad in bright orange, put a straightforward forehand long when he had a chance to engineer two match points in the second set tie-break, allowing Federer back into the set.
But with both players committed to attack, Monfils grabbed the initiative in the third set, breaking to take a 3-1 lead and keeping his nerve to see out the match despite vocal support from a partisan crowd for Federer.
On a day of shocks in Shanghai, third seed David Ferrer lost 6-4, 6-3 to Germany's Florian Mayer and Tomas Berdych, seeded fourth, saw his campaign ended by Spain's Nicolas Almagro in three sets.
But top seed Novak Djokovic had few problems in disposing of Italy's Fabio Fognini 6-3, 6-3 and world number one Rafael Nadal, chasing a record sixth Masters title in a single season, saw off Carlos Berlocq of Argentina 6-1, 7-6 (7/5)
Djokovic, who plays Federer's conqueror Monfils in the quarter-finals, said of the Frenchman: "I'm sure he was very motivated to beat Roger today. He succeeded in that. I don't think he's going to lack this kind of motivation for our match ."
Seventh seed Jo-Wilfried Tsonga brushed aside Kei Nishikori 7-6 (7/5), 6-0 and Juan Martin del Potro progressed with a walkover after German veteran Tommy Haas withdrew with a back injury.