Nishikori's 4-6, 6-4, 6-1 win over alternate David Ferrer earlier on Thursday left Murray facing the daunting task of having to beat Federer in straight sets to go through from Group B.
But the Scotsman was routed in only 56 minutes in a defeat which equalled his worst ever result, a 6-1, 6-0 loss against Novak Djokovic in Miami in 2007.
Federer, bidding for his seventh Tour Finals title, finished top of the group after winning all three of his matches, and he will be joined in the last four by Nishikori on his maiden appearance in the prestigious season-ending event at London's O2 Arena.
"Clearly I'm very happy to play a good match today," Federer said.
"I knew I was qualified so maybe I went in a bit more relaxed," he said.
"It's not the way I thought it was going to go but there's always next year for Andy."
It was a horrible way for Murray to end a disappointing campaign which saw him surrender his Wimbledon title, briefly drop out of the top ten for the first time in six years and fail to win a tournament until the final weeks of the season.
But he could have no complaints after a lacklustre display was brutally punished by Federer, who becomes the ninth player ever to reach the landmark of 250 career indoor wins.
Murray had a solid record against Federer, winning 11 of their 22 previous meetings, but the Swiss had won both their matches this year and all three of their past encounters at the Tour Finals.
Federer was already through to the semifinals for the 12th time in 13 attempts, equalling Ivan Lendl's record for last four appearances, and Murray was the latest to fall victim to the 33-year-old's dominance under the O2 dome.
Murray might have been playing in front of supposed home crowd but, not for the first time on British soil, it was Federer who drew by far the loudest cheers when he emerged onto the court.
If that disappointed Murray, there was worse to come as Federer quickly seized the initiative with a break in the second game when the Scot sent a backhand long.
Murray knew he was out if he dropped the set, but he was powerless to prevent Federer pushing him toward the exit door as the Swiss, dominating the baseline rallies with typical grace and guile, broke again in the fourth game.
Within 24 chastening minutes the set — and Murray's involvement in the tournament — was over as he drove a backhand tamely into the net.
It was the first time Murray had ever lost a set to Federer without winning a single game and he had managed a pitiful eight points in total.
With nothing to play for, Murray was reduced to chuntering angrily as Federer broke again to make it nine games in a row in only 37 minutes.
Shell-shocked Murray eventually stopped the rot after losing 11 successive games, but he was put out of his misery for good when Federer closed out an incredible win moments later.
Earlier, Nishikori discovered just before his match was due to start that he would face Ferrer instead of his scheduled opponent Milos Raonic after the Canadian pulled out with a torn quad.
But the Japanese star wasn't fazed by that late switch and he recovered from a set down to secure his second group win with 41 winners in just under two hours on court.
"The third set was almost perfect," Nishikori said.
"I served well and everything worked well."