When I first started keeping I lived out in the country where there was only one local privately-owned pet store. It was very nice, very well maintained, and the fish section was HUGE....I even worked there for a while...but supplies were a little on the pricey side.
Since moving last year with access to a Petsmart and Petco now, I get most of my supplies from there, and generally Petco(often online moreso then in store)over Petsmart because I like their silk plants better and a few things are cheaper then Petsmart.
I prefer to get my actual bettas from Petsmart though because the prices are better and mine generally has a wider, more interesting selection.
Besides buying used from thrift stores or some place like Craigslist(thats mostly with tanks and hoods and such, though you could maybe find a lightly used heater or filter that could be fine), I don't know of any place cheaper I'm afraid. There are little things you can cut corners on, like caves(Terracotta pots at the craft store for less then a dollar, just stuff the bottoms with some filter sponge...or a well-cleaned coffee mug works well too), used setups and stuff but fish keeping isn't cheap by any means.
Now, generally with heaters what you get is what you pay for so its best to invest in a good heater, but thanks to another member on here I've discovered and use these heaters for my 2-5 gallon tanks.
http://www.ebay.com/itm/-/2710315149...E:L:OU:US:3160
I don't usually recommend them even though he price is great however because they do take about two weeks to get to you, and most people who don't have heaters need one right away.....but if you're in the process of planning a smaller setup, these are definitely surprisingly good little heaters.