Fourth-ranked Wawrinka, despite lapses of concentration, managed to overcome Troicki 6-4, 7-5.
The French Open champion will now face Spaniard Rafael Nadal in the quarterfinals.
Nadal saved a match point in the second set tie-breaker against South African Kevin Anderson before prevailing 4-6, 7-6 (8/6), 6-2.
World number one and defending champion Novak Djokovic also reached the quarterfinals with a 6-3, 7-5 victory over Gilles Simon that took his unbeaten run to 19 matches.
It was far from being the world number one's most commanding performance of late as he struggled with his serve, but it meant he has not dropped a set since the US Open final against Federer in early September.
Federer was shocked to find himself leaving for home without once having dropped his serve as Isner typically won on two tie-breaks 7-6 (7/3), 3-6, 7-6 (7/5), hammering down 27 aces in the process.
The Swiss legend agreed that Isner had served superbly on the big points.
"He's got the size, got the power, got the angles," the 34-year-old said.
"I thought he did very well today when he needed it," he said.
"The breakers, he served great. Those are the ones he needed. Yeah, that was the difference.
"It's tough to get out of the tournament not having lost your serve, you know, but that's how it can go against John."
Federer, who won his hometown tournament in Basel last weekend, will now switch his focus to London and the ATP World Tour Finals.
Meanwhile, Djokovic has now won 26 sets in a row, taking him past his own previous best of 24 sets which he achieved in early 2014.
The Serb, who has won three of the four Grand Slam titles and five of the eight Masters 1000 series titles so far contested this year, dropped serve five times against the wily Frenchman, four coming in the second set.
But on each occasion he immediately struck back to prevent Simon from gathering any momentum.
Djokovic will next play the winner of the tie between Jo-Wilfried Tsonga and Tomas Berdych for a place in the semi-finals of a tournament he has won three times previously, including the last two years.
The 10-time major tournament winner said his struggles on serve had been frustrating.
"I can assure you, losing four service games in a set, I don't think that has happened to me," he said.
"I'm not a serve specialist, but I think I have a solid serve and it hasn't happened for a long time.
"It wasn't pleasant, but I knew that I have a good return. I was feeling the ball very well from the back of the court.
"So that kind of was a positive to that, knowing that I can break him most of the time. That was the kind of mindset."
Isner will play David Ferrer for a semi-final spot after the Spanish eighth seed outlasted Grigor Dimitrov of Bulgaria 6-7 (4/7), 6-1, 6-4.
In earlier action, British second seed Andy Murray set up a quarterfinal clash with Richard Gasquet of France by pounding Belgium's David Goffin 6-1, 6-0 in just 53 minutes.
The match had extra significance in that the two will lead their respective countries at the Davis Cup final in Ghent from November 27th-29th.
Murray certainly chalked up some psychological points in what was just the second career meeting between the two men.
"Obviously with the result, that's a positive. I mean, mentally for me it's a positive win," Murray said.
Gasquet's win over Kei Nishikori came when the Japanese star abandoned due to back pain while trailing 7-6 (7/3), 4-1.
The Frenchman, whose best showing in Paris Masters was a run into the semifinals in 2007, said that Murray would present the toughest of challenges.
"He's one of the best players in the world," Gasquet said.
"He's extremely difficult to beat every time because he makes few unforced errors."