Lots of good advice.. most likely it's due to being in cool water and not much mental stimulation. Bettas aren't your typical fish.. most owners who take the time to get to know them will say they are anything but "just fish", as they love interaction with people and learning tricks.
I would recommend adding in a silk plant or two (the softer aquatic ones), something for him to rest on.. a cave or a decoration on the bottom where he can chill out and explore. They really do need mental stimulation to keep them healthy and active.
Weekly water changes are needed for a tank that size - normal recommendation would be 1 50% and 1 100% per week (using water conditioner each time), but if it's too much for your schedule then I would recommend a 100% every 4-5 days.
Keep trying to give him betta pellets as his main staple, freeze dried worms are best as a treat once a week as it's mostly just empty calories.
A heater is ideal as they are tropical fish - you can find small 10 watt heaters that will be fine for cheap.
I personally love the mini ones as they are adjustable and only about 7 inches long.. the 25w is more than enough for your size tank.
Other than that I highly suggest getting to know your betta, place him in a spot close to where you are for the majority of the time you are home. Use your finger to play with him every so often.. let him get to know you. You will be surprised just how much like dogs/puppies they are for the fish world.
I know.. I sound like a nut.. but I'm not kidding about their personalities :)