My experience with kits has been kinda bad. The small kits give you like a teeny half-gallon bowl and a pokey plastic plants, and the big ones tend to pick out not-as-nice stuff. I kinda think it's more fun (and often cost-effective in the long run) to pick out your own things =) My basic supplies for a betta would be:
* 2.5g glass tank - $10
* Hydor Theo 25w adjustable heater - $20
* Prime water conditioner - $5
* LCD strip thermometer - $2 (*don't* get the submersible traditional kind - they take a long time to adjust to a new water temperature, and they are less accurate - I have two of the same model that register 2 degrees off of each other) <-- Haha, Myates and I have different experiences here =)
* Brine shrimp net - $2 (*don't* get the "quick net" or basic net - bettas have delicate fins, so the fine mesh is better for them)
* You might like a gravel vac, small size, to make partial water changes easier - $6
* Food: pellets (other threads have recommended brands) and frozen bloodworms (Hikari), about $5 each
* And some basic decor - soft gravel, soft plants (the kits always come with super pokey plants, and you don't want anything that can tear your guy's fins). You can use a terra cotta pot (less than $1) for a cave, but you'll need to stop up the hole in the bottom. How much you spend on decor is really up to you!
All these things are one-time purchases - bettas eat so little that whatever you get will last you basically forever.
As suggested, I would not bother with a filter on a 2.5g - just do the weekly 100% changes + 50% change midweek.
If you want to go all out, there's certainly more stuff you can get and/or bigger/nicer versions of the stuff on this list! For example, I have recently discovered how much easier airline tubing can make water changes (my guy *hates* being cupped or netted, so anything that avoids that is helpful).