+1 on using PVC pipe
as a cave. You can pick up PVC Coupler pieces at a home improvement store (Home Depot, Lowe's) for about 60 cents. Look in the Plumbing section.
You can also use a ceramic coffee cup
And +1 on using small terra cotta pots
. However, you'll need to either remove the bottoms, or plug up the holes so the fish can't get stuck. I use the smallest size pots available. (They cost 25 cents each at Walmart.)
To remove the bottoms:
1) Put them in hot water and let them soak overnight.
2) Then, chip off the bottom. To do this, hold a screwdriver where you want to make a break. GENTLY tap the top of the screwdriver with a hammer. (Gently!)
3) While the terracotta is still wet/soft, use Fine grained sand paper to sand down all rough and jagged edges.
Or, you can plug up the holes. To do this:
1) Get "aquarium safe" silicone. (Lowe's has the Loctite brand for about $3.50.)
2) Fill the hole with silicone. Let dry for at LEAST 24 hours.
You can also make a floating mesh cave
1) Pick up a sheet of plastic craft mesh at a store like Micheal's.
2) Cut a section of the mesh and roll it into a tube. (Make sure it's large enough that the fish can't get stuck!)
3) Trim off all sharp edges.
4) You can seal the tube by sewing it shut with white thread, or fishing line. Or you can use heat to melt the edges together.
Here's a thread that shows how to make the floating mesh tubes, and may give you some other cave & decoration ideas, too: http://www.bettafish.com/showthread.php?t=141777