Miliband attacked Cameron as "a dodgy prime minister" and homed in on Stephen Green, the former chief executive and chairman of HSBC who later became a trade minister under Cameron and was made a lord.
"Can the prime minister explain the revolving door between Tory party HQ and the Swiss branch of HSBC?" Miliband asked in heated exchanges during prime minister's question time, a weekly parliament debate.
"Does the prime minister expect us to believe that in the three years Stephen Green was a minister, he never had a conversation with him about what was going on at HSBC?" Labour party Miliband said.
The scandal over leaked files revealing HSBC's Swiss account holders has erupted ahead of a general election in May, sparking a furious blame game between Cameron's Conservatives and Labour.
Cameron has hit back saying that Labour was in power at the time the files were first brought to the attention of French tax authorities in 2007 and accusing Labour of having close ties to Green.
"No government has been tougher than this one in chasing down tax evasion and tax avoidance," the prime minister said, pointing to thousands of tax prosecutions since he came to power in 2010.
Green himself, a Church of England clergyman who published a book on banking ethics has declined to comment but is now under pressure to testify before a parliamentary committee.
Labour's Margaret Hodge, the head of the committee, has promised a full inquiry, saying the revelations show "a secretive global industry serving a wealthy elite".
HSBC's Swiss banking arm has insisted it has undergone a "radical transformation" since the initial revelations and now has "strong compliance controls in place".
'Friends of the tax dodgers'
Miliband also referred to a report in the Guardian saying that seven Conservative donors were among the HSBC Swiss account holders and gave the party more than £5 million ($7.6 million).
"He can't get away from it, he's a dodgy prime minister surrounded by dodgy donors," he said.
The cache of files made public in the so-called SwissLeaks case includes the names of celebrities, alleged arms dealers and politicians — though inclusion on the list does not necessarily imply wrongdoing.
Published at the weekend, the files claim HSBC's Swiss division helped clients in more than 200 countries evade taxes on accounts containing $119 billion (104 billion euros).
The files were used by the French government to track down tax evaders and shared with other states in 2010, leading to a series of prosecutions.
Several countries, though not Britain, have launched investigations into HSBC in recent years following the revelations.
British tax officials say they have recovered £135 million as a result of information gleaned from the thousands of UK account holders listed but there has only been a single prosecution.
Labour has pledged an immediate crackdown on tax dodging if it wins the general election.
The party's finance spokesman Ed Balls said Cameron and finance minister George Osborne had "totally failed" on the issue over the past five years.
Pointing to Miliband and Balls, Cameron hit back saying: "Those two in the Treasury were the friends of the tax dodgers".