The water is fine and I do water changes every few weeks
In order to keep ammonia levels down, water needs to be changed weekly - even with a filter.
And the pea thing is more for goldfish when they are constipated. Bettas can go 3 or so weeks without eating.
First stop the melefix or bettafix. Many say its a miracle drug but there are just as many people claiming it has killed their betta. Its main ingredient is a tree oil which can damage the organ the betta uses to breathe air.
I would do a water change and clean the gravel of any debris. I think clean, conditioned water would be the first step.
Fragil fins? What do they look like? Do they have holes or a ragged look to them? If so he may have fin rot, ammonia burns or both. Clean water will help both.