World number one Novak Djokovic, meanwhile, slumped to a lacklustre defeat to France's Jo-Wilfried Tsonga.
Two-time Canadian winner Federer was forced to the limit by Croatian Marin Cilic, with the Swiss second seed finally claiming a 7-6 (7/5), 6-7 (3/7), 6-4 win on his seventh match point.
Cilic ignited the drama in the 10th game of the second set, holding serve in the 19-minute epic and saving six Federer match points on the way to levelling at a set each.
Federer, who turns 33 on Friday, finally regained control late in the third set as he converted after more than two-and-a-half hours on the court.
"It was a thrilling match and I hope to keep it up tomorrow," said the relieved winner, who next faces David Ferrer in the quarter-finals.
"I was unlucky with some shot selections on some of the match points and he hung in there," Federer said.
"It was nice to win this one as I've had some tight losses this season," he said.
"In the third set, his level didn't drop, I just had to keep going and try to create chances."
Australian Open champion Wawrinka, seeded third for the Canadian tourney, lost 7-6 (10/8), 7-5.
"It was an okay match, for sure, but it's tough to lose that one," said Wawrinka, who will now head for the Cincinnati Masters next week, his final tune-up for the US Open which gets under way on August 25th.
"In general he served really well," the Lausanne native said.
"It was tough to return, a little bit windy," Wawrinka said.
"But I had some chances, especially in the first set," he said.
"I made some mistakes that I shouldn't have."
Tsonga beat Djokovic 6-2, 6-2.
The newly-married Serb, who needed almost three hours to get past Gael Monfils in his opening tie this week, looked under-prepared against Tsonga, the powerful French 13th seed.
Wimbledon champion Djokovic finished with 18 unforced errors and dropped serve four times.
Tsonga saved a break point in the final game of the 63-minute rout, with the heavy hitter pounding over a serve which Djokovic could not handle on match point.
Djokovic came to Canada having won five of the last six Masters events he had contested.
Thursday's loss was only his fifth of the season against 38 wins.
Tsonga next plays Andy Murray who made the quarterfinals without hitting a ball when scheduled opponent Richard Gasquet, the French 12th seed, withdrew with an abdominal strain.
"For me today it was great, I played a good match," said Tsonga.