The actual stress of being alone, or in a very small number,doesn't usually kill the fish directly (that's why so many people have one neon, or one rummynose that lives on) but it does leave them open to catching stress related illnesses, like fungus, ich and fin rot.
It really is bordering on cruel to keep shoaling fish in very small numbers. They have evolved to live in huge shoals, often consisting of hundreds, if not thousands of individuals, in the wild and so they do need plenty of company.
Being completely honest, even the oft quoted 'at least six' is barely acceptable; it would be great if we could aim for a dozen instead! Certainly, when I was building up my shoal of green neons (I don't know why they're called 'green', when they're quite obviously blue, but there you are!) I noticed a big difference in behaviour when I got past the dozen mark; they suddenly became much more confident.
And aggression within the group is also a big factor, as eagles points out; even neons can be surprisingly aggressive and nippy, if in small numbers.