At this early stage, the tank is probably cycling, so water changes will have to be done more frequently now than they will later once the tank is cycled. Read about the aquarium cycle here:
As the article states, ammonia and nitrite are both very harmful to your fish, so you should be doing water changes to keep these levels as low as possible until those beneficial bacteria colonies are established in your tank. Once it's done cycling, water changes still need to be done to keep your nitrate levels in check. On a 5 gallon tank with just a betta, I would change roughly 40% of the water once a week.
The best way to do water changes is to get a gravel vacuum. They only cost a few dollars for a small one. You just create a siphon with the vacuum and use it to remove water from the aquarium while also vacuuming waste out of the gravel. After you remove the water, fill a bucket with the same amount of water you removed. Adjust the temperature of the tap so that the water you're adding to the tank is as close as possible to the temperature of the water already in the tank. Add the correct amount of water conditioner to the water in the bucket, and gently pour it into your tank. There's no need to remove the fish during this process; doing so will only stress him more.