I've always stuck with "fish are for looking, not touching"
Fish just shouldn't be touched, and while livebearers who nibble (though if you ask me, they bite your fingers cause they think it's food) and large trusting fish like oscars and the like won't mind being pet at, it doesn't necessarily mean they'll like it, like say, a cat or dog. It can be harmful to their slime coat when you have your hands all over them, which can lead to problems with their immunities. Not to mention it might even be a tad stressful for somebody to reach in with grabby hands.
Relate it to frogs, toads, treefrogs, you know.
You know that usually, after a while, they'll trust your hands enough for you to handle them. But what handling frogs does, is due to their skin sucking up moisture, they suck up the oils and salts, sometimes soaps and lotions in your hands, which can cause them to get very sick.
While cats and dogs and small mammals and such find comfort in being pet at and handled by trusting hands, I highly doubt the same goes for fish. I think to them, you're just that big thing that brings the food and changes the water once in a while
Good rule of thumb for a lot of things in life: Just because you can, doesn't mean you should