The process was carried out in secrecy, with Arafat's grave carefully shielded from the public eye and media kept far away, but Palestinian sources confirmed the remains were removed for testing on Tuesday morning.
"At 5:00 am (0300 GMT), experts began to remove the stones and began opening the grave in an orderly fashion.
The remains were then transferred to a mosque adjacent to the grave for the removal of samples," a Palestinian source told AFP on condition of anonymity.
The source said only a Palestinian doctor would be allowed to directly touch the remains and remove the samples, but that the process was being conducted in front of Swiss, Russian and French experts.
After the samples are removed, the remains are to be reburied in a military ceremony expected to be broadcast live on Palestinian television.
For weeks now, Arafat's grave in a mausoleum on the Muqataa presidential site from which he once governed has been hidden from view by blue tarpaulins.
The mufti of Jerusalem, Mohamed Hussein, arrived at the Muqataa on Tuesday morning and told AFP he would be present at the opening of the tomb.
The samples being collected are to be tested for the radioactive substance polonium as part of a new investigation into whether Arafat was poisoned.
The probe was prompted by an investigation carried out by the Al-Jazeera news channel, which commissioned a Lausanne lab to test personal effects belonging to the late leader that were given to them by his widow Suha.
The tests revealed the presence of the toxic substance polonium, and prompted calls for the exhumation of Arafat's remains for new testing.
Polonium was the substance that killed Russian ex-spy and fierce Kremlin critic Alexander Litvinenko in London in 2006.
France opened a former murder inquiry into Arafat's death in late August at Suha's request, and French judges in charge of the investigation arrived in Ramallah on Sunday to participate in the exhumation process.
Rumours and speculation have surrounded Arafat's death ever since the quick deterioration of his health before he died at the Percy military hospital near Paris in November 2004 at the age of 75.
Doctors were unable at the time to say what killed the Palestinians' first democratically-elected president and an autopsy was never performed, at his widow's request.
But many Palestinians believed he was poisoned by Israel -- a theory that gained ground in July following the Al-Jazeera report.
The samples taken Tuesday will be flown to laboratories in the three countries involved, with results expected within several months.
Some experts, however, have questioned whether anything conclusive will be found because polonium has a short half-life.
Jean-Rene Jourdain, deputy head of human protection at the French Institute for Radiological Protection and Nuclear Safety (IRSN), said it would take weeks of analysis to be sure that the traces were man-made polonium rather than just coincidental contamination by naturally-occurring polonium.
"Even if traces of polonium are found, it doesn't mean that they are man-made," he told AFP on Monday.