The RadioShack team member, a winner of the event in 2006 and 2010, took the gruelling 254-kilometre "Queen of the Classics" race in a sprint climax ahead of Belgian Sep Vanmarcke.
"At the end it was like roulette," reflected Cancellera on the dash for the line which he claimed by the length of a bike.
The 32-year-old justified his status among the favourites for this 111th edition of the French race.
The top two surged clear from Belgian Stijn Vandenbergh and Czech rider Zdanek Stybar, hampered by spectators, in the 'Carrefour de l'Arbre' section of the race.
In third, half a minute behind, came Dutch champion Niki Terpstra, who edged in clear of Belgian Greg Van Avermaet and France's Damien Gaudin.
Cancellera was following up his win in last Sunday's Tour of Flanders.
The 2008 Olympic time trial champion moved up to within one win of the Roubaix win record held jointly by Belgian duo Roger De Vlaeminck and Tom Boonen, the latter missing from Sunday's renewal after his fall last weekend.
"I've never known a race as hard as this one today," Cancellera added after taking time to get his breath back.
"A couple of times I thought it was over then I said to myself 'Roubaix is never finished, the battle never ends'," he said
"Everything was against us, against our team," Cancellera said.
"I don't know how I managed to do it," he said.
"I gave everything I could, now I've only got one thing on my mind — a holiday!"