As long as your hands are clean (and don't have soap on them) and you are not rough, then it should be ok. Potential problems could arise if you remove most of the fishs' slime coat or have something toxic on your hands. I have been catching large and delicate fish by hand without any ill effects for as long as I have kept fish.. for example, in my opinion, it is by far the safest way to move plecos and Corys as they can easily become tangled in a net.