I would feed him a bit more if the pellets are small. He may just be emaciated and if he has an internal parasite that could be making it worse. If you look at the shape of the stomach on this fish this is more so how he should be shaped: http://1.bp.blogspot.com/_LV2FKtgcUg...A/s1600/b3.jpg
What brand of pellets do you feed him?
I would cut out the Bloodworms altogether as they hold no form of nutrition. You can feed him twice a day, 3-4 pellets at a time. If you can get ahold of some frozen foods like Glass Worms or Brine Shrimp then he may liven up even quicker since feeding (thawed) frozen foods like these are high in protein. He should have a bit of a belly, it's normal and means he's well fed. Malnourished fish very often act lethargic and if they get to a point when they are so bad they may not be able to eat.
You can wait for lvandert and aokashi response based on the filter pad but I'm not sure if putting it back in and using it still is a great idea. But then again i'm paranoid about those types of things and change them whenever a fish of mine becomes ill.