I had been looking at ceramics, because I was alarmed by the number of roughly finished plastic sculptures I ran across in stores. When shopping for these, I would feel them inside and out; some of the sculptures that felt ok on the outside were razor sharp on the inside! And these were ornaments the fish were meant to swim through. :(
I did find suitable plastic oranaments, but had looked at the ceramics and read reviews online. Modern ceramics should have glazes that are perfectly safe for fish. The drawback (from the reviews I read, not personal experiance) is that they don't seem to ship well... but to be perfectly fair, I've had other ornaments fall apart in my hands that weren't ceramic!
It's been my experiance that the bettas in my sorority really don't seem to spend much time hiding... they are busy swimming about, looking for fun or trouble. At nighttime, they all find places to sleep in the weeds. The cory cats are the only fish currently who use caves in the sorority tank.
Soooo... I don't think the ceramic caves are a bad idea, but behavoir-wise, I'm not sure that bettas use caves as frequently as other fish.