I use the stones from the Dollar Store. I just get a baggy of them (They usually give you a little over a pound for $1) so with two you could definitely cover the bottom. And it looks gorgeous! I just boiled mine mildly before rinsing and putting in. They were in mesh netting so I was worried about them having cleaner or something from the store. But they've never harmed any of my fish!
Kritter Keepers are great! Especially since they're cheap. I love them because water changes are super simple. Take lid off, dump water out, fill back up! ;) I say get "larger" stoned substrate because this made water changes so much easier for me! Basically dump into a strainer, rinse with warm water, and you're good. Gravel tends to stick everywhere and I find marbles and such are harder to rinse out in a swipe or two.
Because they don't have light, I highly suggest using a desk light or putting in perameters of a window. That way he won't get direct sunlight, but will be in a bright portion of the room.
I highly recommend the Tetra Whisper 3i filter. I have 2 of them and never had a problem. (Except one makes bubbles after a day or two of no water changes, LOL.) And their intakes are at the bottom, not on the side like "grated" pieces. Therefore your fish would have to swim directly underneath to even get into the intake current. Plus, if your fish jumped into the filter (which is nearly impossible) there's no real electronic or super strong "sucking" system which will kill them to shreds. It also comes with an airstone so it really helps add oxygen to the water! Not to mention the out take flow was really gentle.
If you're on a budget, look at your local WalMart. They sell a package of 3 silk plants in most of them for $5.
Total I'd estimate you'd be around $30. Filter is usually $12, Kritter Keeper about $10, Substrate $2, Decor $5. Try and get discounts and coupons, those will help you! I know PetCos around me are giving $2 off coupons on receipts if you do a phone survey. And I'll give you a hint... If you press "3" on the first question, it automatically ends the survey and gives you the code! :D (Yes, I've taken this survey about 10 times. No shame.) So if you're able to get this, you could cut off a couple more dollars, which always counts!