If you have a large enough tank, you don't have to take him out to do frequent water changes. You can siphon out the amount of water you want to change, suck out the yucky stuff at the bottom and then add in new water.
My female did that for a while, but she her tank had been knocked accidentally into several times very hard one day. She was skiddish for weeks, but now she's fine. So it's possible that something spooked him and now he associates people with whatever scared him. If that's the case, he may need a few weeks to get over it.
But test your water parameters, etc. just in case something else is going on.