Kinda sounds like how my betta's treating me. >_>
I took out all the plants in his tank but the only silk one he's had. (He likes to slither through every tight space he can, so the plastic plants, though temporary, still tore his fins.) I finally got him a living plant (that won't melt X_x), and he loves it. Slithers through hairgrass, uses it as sort of a bed, lol.... But apparently he liked his plastic stemmed plants. Everytime he sees me, if he doesn't see the food jar in my hand, he flares with both his gills and his fins. >>
Even flares like mad when I put a finger in. He doesn't bite... maybe he only does that 'cause he likes the new plant. Dunno.
But in any case, I don't believe fish have short memories. I hear too many stories like yours and mine (well, not specifically like them of course) that prove otherwise.
What if you floated the shrimp in a cup during his feeding time? That way he can't be bothered... much. (Your betta might try to attack the cup or push it around a lot. <<