Just a few pellets probably won't hurt anything. I think the fish would produce more ammonia than the sunken pellets. If it were a lot of pellets sinking, you'd likely experience problems with debris and possible ammonia/nitrate spikes or an algae problem.
If possible, removing the pellets sooner rather than later would probably be best, but no, it should not cause a huge problem. The only big problem I've experienced with my blind boy letting pellets sink is the algae issue and then when I siphon the tank, the pellets disintegrate and its hard to get all the tiny little specs out.
However, I have noticed that New Life Spectrum pellets don't dissolve like other pellets do. I know they aren't as good at absorbing meds so maybe they don't break down as fast as others?