Terpstra attacked just over six kilometres from home to stun Cancellera and other favourites by winning the contest after 257 kilometres of racing.
The race had seemed set for a sprint finish among 11 riders but Terpstra took advantage of the marking of favourites Cancellara, Tom Boonen and Peter Sagan to go solo and win in the Roubaix velodrome after five hours 38 minutes and 57 seconds.
German sprinter John Degenkolb won the sprint for second, 20 seconds later, with Cancellara coming third.
The Swiss was heavily favoured to win the gruelling classic across the cobbled roads of northern France.
Having finished fifth two years ago and third last year, 29-year-old Terpstra, who rides for Omega Pharma, finally delivered on his immense potential on the biggest Spring Classics stage of all.
It was an anti-climax to what seemed set to be the most thrilling Paris-Roubaix finish in many years.
Degenkolb, as the best sprinter, was the main dangerman and as the favourites watched each other to see who would make a break for home, Terpstra jumped away with just over six kilometres remaining.
Initially no one reacted and he quickly eked out a lead of over ten seconds, at which point it was clear the rest were racing for second.