It's recommended not to because you can remove their protective slime coating causing them to be susceptible to bacteria, and other infections. Why when touching fish your hands should always be wet to help reduce wiping off as much coating as possible.
But I won't say to never do it.. here and there if you put your finger in and the fish rubs up against it then it is fine - as they know how hard to rub, etc, that will keep them safe.
Feeding from the fingers should be fine..
I just wouldn't go around picking them up is all lol :)