While it's not good per say, it won't kill your fish. Many people won't advise letting this go on for too long becase the betta pellets have the proper nutrients for your betta and the flakes for your other fish. Might I ask what sort of tankmate fish you have in there?
If you want to try and stop them from eating the wrong food, you could do what I did. I conditioned my betta to learn to avoid the turkey baster (by chasing him around with it for a tiny bit each day), and then when he started avoiding it, I would put the fish flakes in a bowl of water and suck it up with the turkey baster and slowly squeeze it out in a corner of the tank and let the fish eat it. While the fish eat the falling fish flakes, I gave my betta his 2-3 pellets. After a while, you don't have to chase your betta away with the turkey baster. Instead when he sees it he'll just stay away (unless he's an absolute pig XD)