Here's info on pet stores in Connecticut:
Badmans Pet Shop Directory, Connecticut pet shop reviews
I've found that site to be very helpful in finding good fish stores.
However, the online stores, like the Drs. Foster and Smith site, are definitely going to be cheaper for supplies, especially if you buy them all at once to save on shipping. If I were you, I would buy a standard 5.5g rectangular glass tank from a local pet store, as well as a glass canopy to prevent the fish from jumping out. Here's the rest of the stuff I'd get from the tank, all from Drs. Foster and Smith:
Water conditioner:
Aquarium Water Quality & Conditioners: Seachem Prime Water Conditioner
Test kit:
Aquarium Water Testing: Aquarium Pharmaceuticals Freshwater Master Test Kit
Heater (you want 5 watts per gallon, so the 25 watt heater for a 5.5g tank should be fine:
Aquarium Heaters: Visi-Therm Stealth Heater
Filter:
Power Filter for Small & Desktop Aquariums: Azoo Palm Filter
Food (these make a good staple, but you could also get frozen bloodworms from your LFS for a treat to feed once or twice a week):
Betta Bio-Gold Fish Food
Then, you have some other options: how do you want to decorate the tank? Do you want live plants? If you like the resin decorations, you can buy those from the LFS or from the Drs. Foster and Smith site. If you want a more natural look, you can buy rocks from the LFS or, for much less money, from a landscaping place. Slate and shale make great choices and will allow you to build hiding places for your fish. You can also buy driftwood at the LFS for decor. If you want fake plants, get silk rather than plastic. If you want live plants, you should research what types of plants you like before buying them, as different plants have different needs in terms of fertilizer and lighting. If you don't want live plants, any sort of lighting should work for the tank, such as a cheap strip fluorescent light from Home Depot. Some live plants require some more expensive, powerful lighting, while others, such as java fern and java moss, will do well under almost any lighting. As for substrate - you can buy that from the LFS, but I recommend getting a natural colored rather than a dyed gravel as it can leech into the water and hurt your fish. You could also use a sand, such as well-washed play sand from Home Depot. You could even use pea gravel sold at Home Depot. Both of those are very inexpensive options. If you want live plants, you may want to consider a substrate like Eco-complete.