Betta’s stomachs are the size of their eye generally. A bigger Betta may eat more than a smaller one, I usually try and consider how active they are. For example, a lethargic large betta may require equal or less pellets compared to a hyper small betta. This activity and metabolism can be affected by temperature – warmer tank water will generally mean a higher metabolism and active betta.
I have a large and small betta, Slevin and Victor respectively. I do not exceed 8 pellets per day. I usually try and feed morning and afternoon at the same time period, surprisingly they know when it is getting close to food time. Slevin gets 4 pellets per feed and Victor gets 3 per feed. However I may reduce that amount in the afternoon feed if I see their stomachs are still large.
I have only experienced one rejection of food by Victor when he was stressed, I left the pellets in the tank for 15 minutes – didn’t sink and then scooped them out. The only negative to leaving them in a tank to rot is an increase in ammonia levels, so you are doing the right thing by scooping them out before they are eaten and/or sink.
By the sound of it, the pellets may not be to his liking if he is refusing on occasions. Stress or other factors may be playing a part as well. What kind of pellets are you feeding?