I don't, to me it's silly, my hands feed me, so they can feed my animals. Plus, my animals could care less.
I DO clean my hands after I touch anything in my reptiles' cages so that includes every time I take something from the fridge and give them their meal. Before I touch the thing I took from the fridge to put it back in the refrigerator, I wash my hands, just due to prevent anything from the cages contaminating any food.
Also, if the cat wet food accidentally gets on my hands, cus it is just gross smelling like fishy cat food :P