Green was head of HSBC when it allegedly helped many of its wealthy clients evade paying millions of dollars in tax and later served as trade minister in Conservative Prime Minister David Cameron's coalition government.
Green, who has been has been asked to tell a parliamentary committee what he knew of alleged tax dodging strategies on accounts containing tens of billions of pounds, will be replaced as chairman of TheCityUK's advisory council by Gerry Grimstone, according to a statement.
The incoming Grimstone, currently chairman of the group's board, called Green a man "of great personal integrity" who had quit due to the adverse publicity created by the allegations.
"He doesn't want to damage the effectiveness of TheCityUK in promoting good governance and doing the right thing so has decided to step aside from chairing our Advisory Council," Grimstone said. "This is entirely his own decision."
British lawmakers announced on Monday they would open an inquiry into the London-based banking giant, after allegations that the bank helped rich clients dodge taxes emerged from a massive leak of documents.
The BBC said it had seen 106,000 client accounts in 203 countries, leaked by French whistleblower Herve Falciani, a former HSBC employee who downloaded and leaked the so-called SwissLeaks information cache.
It said the documents include details of almost 7,000 clients based in Britain, and that HSBC had helped its wealthy customers evade hundreds of millions of dollars in tax.
As well as serving as a trade minister, Green was also appointed to the House of Lords, the British government's upper house, by Cameron in 2010.