The North African nation has been wracked by conflict since the overthrow of dictator Moamar Kadhafi in a 2011 uprising, with rival governments and powerful militias battling for control of key cities and the country's vast oil riches.
UN special envoy to Syria, Bernardino Leon, had warned at the start of the talks that they were a last-ditch effort to prevent all-out chaos.
"The participants agreed after extensive deliberation on an agenda that includes reaching a political agreement to form a consensual national unity government and the necessary security arrangements to end the fighting," a UN statement said.
It said the talks "were constructive and... reflected the participants' sincere commitment to reach common ground."
The participants called on all the players to cease hostilities to create a conducive environment for the dialogue, and "expressed their unequivocal commitment to a united and democratic Libya governed by the rule of law and respect for human rights."
They agreed to work towards the release of abducted people, providing and allowing humanitarian aid to reach affected regions, opening airports and securing land and maritime navigation.
The delegates will return to Geneva for a fresh round of talks next week after consultations.
As news of the agreement came, the UN refugee agency said an upsurge in fighting since the start of this year across several towns in the east, including the second city of Benghazi, had sparked more displacements.
"In Benghazi alone, the local council is reporting that around 90,000 people are unable to return home," it said, adding that number of people displaced nationwide was approximately 400,000.
'Peaceful transfer of power'
Libya's internationally recognised government decamped last summer to the eastern city of Tobruk after an Islamist-backed militia alliance seized the capital Tripoli and set up its own administration.
The alliance known as Fajr Libya (Libya Dawn) also holds the third city, Misrata.
It launched a bloody offensive in December to seize control of key oil terminals but was repelled by the army.
The broad agreement cobbled in Geneva also saw the factions pledge to work towards ensuring the free movement of people across the divided nation.
They also vowed to respect the legitimacy of state institutions, work towards the peaceful transfer of power and reject violence and terrorism.
The agreement came after months of UN efforts to get the opposing sides back to the negotiating table after a single round of talks in September.
A major concern in Libya is the proliferation of Islamist militias in key areas such as Benghazi.
Those militias are led by the Ansar al-Sharia group, blacklisted by the United Nations for its links to Al-Qaeda.
The Islamic State group that has seized large areas in Iraq and Syria is also thought to have gained a foothold in eastern Libya.
Leon had also underscored the threat of Libya becoming a hotbed of Islamist insurgency, saying it menaced North Africa, the Middle East, the Sahel and Europe, which lies on Libya's doorstep.
Jihadists are reported to have set up camps in Libya, including in the remote southern desert, to train militants to fight in Mali, Iraq or Syria.
The head of Libya's recognized government has pleaded for more international help in combating militias by lifting an arms embargo imposed by the UN Security Council at the start of the anti-Kadhafi uprising in 2011.
"In Libya, the government and armed forces are battling these groups alone, without any support from the international community," Prime Minister Abdullah al-Thani told AFP in an interview just before the Geneva talks.