Federer, the top seed in the tournament, saved five match points before driving a forehand cross-court wide.
The 20-year-old Kyrgios won on his sixth match point while saving two Federer match-winning chances in their second round clash.
The fast-paced contest took more than two and a half hours and denied Federer any chance at a fifth title at the claycourt Masters event.
Kyrgios, who knocked Rafael Nadal out of Wimbledon last year, sent down 22 aces to Federer's 15 and next faces John Isner of the United States.
The Swiss saved five match points in the deciding tiebreaker while having two of his own annulled by the big-hitting Australian youngster, ranked 35th.
"I couldn't return his first serve, I had a horrible performance on return of serve," said Federer.
"That made life difficult for me," he said.
"I'm disappointed with how I played on returns, and that cost me the match."
Meanwhile, Rafael Nadal returned to his clay-court comfort zone, starting his campaign for a fifth Madrid Open title with a 6-4, 6-3 defeat of Steve Johnson.
The victory in 74 minutes lifted Nadal's clay record over Americans to a perfect 10-0 while the third seed improved to 12-3 on the surface this season.
The nine-time French Open champion has four Madrid titles and has played in six finals at his showcase home event.
He is aiming to extend his streak of winning at least one European ATP clay title leading into Roland Garros to 11 straight years.
Nadal goes on to face Italy's Simone Bolelli for a quarter-final spot.