The match, lasting 89 minutes, was the first for the 34-year-old Swiss since losing a Monte Carlo quarterfinal to Jo-Wilfried Tsonga nearly a month ago.
The 17-time Grand Slam winner was due to play last week in Madrid but withdrew with back pain. He came to Rome early to train, drawing 6,000 fans to a weekend practice session.
He has laid low in recent days, declining pre-match media responsibilities as he waited to let his racket speak for him.
Federer is playing at the Foro Italico for the 16th time as he seeks his first title here. The Swiss has finished runner-up four times, most recently last year against Novak Djokovic.
Rome marks only the fourth tournament of the season for Federer, who underwent knee surgery in early February, the first operation of his career.
Victory over Zverev sent Federer's record in 2016 to 11-3. He next plays promising young Austrian 13th seed Dominic Thiem, who defeated Portugal's Joao Sousa 6-3, 6-2.
Zverev, aged 19 and ranked 44th, could not have asked for a more challenging Rome debut as he faced off against his tennis idol.
His loss left him 0-8 against top 10 opponents.
Federer broke for 4-2 on his way to collecting the opening set in 28 minutes.
The Swiss then earned a break in the third game of the second set but then lost serve for 2-all.
The veteran broke back on his fourth opportunity in the 11th game for a 6-5 lead and finished off the welcome win a game later, a drop shot setting up two match points with Zverev driving wide on the first.