The problem may be the brand of pellets....Tetra Betta is a very low-quality brand, and based on the reviews I've read a lot of bettas, especially the more picky sort, will just outright refuse them. Its highly likely he got a tasted of it and hated it.
I would highly suggest picking up Omega One Betta Buffet Pellets(found at Petco and Petsmart) or New Life Spectrum Betta Forumula(found at Petco or online), NLS I'd recommend over Omega one if you can get it, and giving one or both of those a try.
And be patient! I know its hard, but don't break down and feed him any bloodworms until you can get him regularly eating a good-quality pellet!