A picture would be great if it is possible.
What size tank is he living in?
He could be pale from stress due to his environment. The PH, Nitrite, Ammonia etc levels may be too high. Do you have a water test kit? My friend had a similar problem with her Betta. High ammonia was to blame (she never tested the water). The fact that you say he regained some of his color after you cleaned the tank leads me to think that something is wrong with your chemistry.
How often do you do a partial water change? I would try chaning a SMALL amount of his water out each day for the next few days.
Does the white around the gill look fuzzy? If so, it is probably a fungus, though I suggest you research "Columnaris" and see if that is what it looks like.
Also, if you have not, add some aquarium salt to your tank. It reduces stress and is good for many diseases among many other benefits. Do not use table salt unless it is non-iodized with ZERO additives. I've heard of people getting away with ignoring this rule, but I wouldn't chance it. Get the real deal, then you will have it!
How old is the Betta?