If you have some inert(I believe you can test the rocks by pouring Vinegar on them...if it fizzles, don't use it)rocks in your back yard you can gather some up, rinse them off really REALLY well many times with some very hot water or a 1/10 part Bleach solution if you're the super cautious type and either stack them VERY carefully or pick up some aquarium-safe silicone and glue them together(which I would recommend). If you don't have rocks available in your yard or you're super nervous about using them, you can pick up some smooth river rocks at your local craft store probably for a few bucks.
One of my FAVORITE pieces of cheap aquarium decor are terracotta/clay pots from the craft store. I've been using these as caves for YEARS with no problems or issues and they cost less then a dollar each. You'll want to knock out the whole bottom, or plug up the little hole with a bit of filter sponge, a rock if you can find one to fit, or seal it up with some silicone though so your little guy/gal doesn't get stuck but they're really great, cheap decor options. They come in a variety of sizes, you can break them up and scatter the pieces around your tank for a different look, and they're fantastic for tying Java moss too if you're into live plants.
Another option I've heard discussed a couple times is Legos. From what I understand they're pretty safe and non-toxic, though a few people have expressed some concern over the sharp edges some have I've seen a couple people who have made some caves out of Legos with no problems. I think if you select you Legos carefully you shouldn't have much of a problem.
And finally, theres also the option of craft store silk plants/flowers like willow suggested. You'll want to be careful in your selection with these, you want to pick out ones with no scents or perfumes and no wires that will rust and cause problems later and you'll want to of course rinse and soak them REALLY well....but if you find the right ones, silk plants from the craft store can be a much cheaper alternative to the ones you buy made for aquariums in the pet store. ;)
Oh! Thats right, I forgot about that...yes, you can use coffee mugs as well. Just make sure you do some heavy rinsing and soaking and scrubbing with hot water and a clean wash cloth or paper towel to make sure you get any residue off if its a used cup, but once clean even used cups can work very well. I used one for a time without any problems. ;)
Heres a list of things I've seen people use in tanks or I have used in tanks!
1. swimming goggles
2. plastic army men
3. coffee mugs
4. terra cotta pots
5. toy figurines with no metal
7. rocks they collected
8. driftwood they collected
9. ping pong balls
10. plastic or wood train tracts
12. silk plants
the dollar store has some cool stuff you can get for your tank! they have all sorts of stuff!
I have PVC pipe in my tank, I think its 3". I have a T and a 45˚ elbow. There are 2 problems with using it. Sometimes they have some VERY sharp spots on them. Its easier to try to find ones without the sharp bits than try to sand it, unless you have VERY strong sand paper (my dad had some so I got him to do it for me). Also, since its white your betta may not feel safe going inside them (some like their hiding spots to be darker). My boy wouldn't go in them at first. He has black gravel in his tank tho, so I put some in the PVC and made it look as tho it was coming out of the gravel, and he LOVES them.