I think that will be the problem.
Beneficial bacteria, which keep your amonia, nitrite and nitrates down need to build up. By doing a full water change you are allowing the tank to become dry(ish) and so most of the bacteria will be killed off - the environment needs to stay wet for it to survive. May I suggest you do smaller water changes, but more often?
Bettas don't like strong filteration so I use a weak filter on mine. What do you use? Assuming it's not a very strong filter I suggest perhaps 20%-25% water changes twice a week. This will allow the bacteria to build up, and then to survive.
The temperature could be an issue as well depending what temp your tank is at.
A tank upgrade would be greatly appreciated by your betta. A 5g would be perfect for him on his own, or if you wanted to be able to add a few other fish, perhaps a 10-15g. Just remember to cycle it first. There is a sticky at the top of Starting and Maintaining... with a lot of good information on cycling if you feel you want to read it.