I bought river rocks at Dollar Store, and soaked them in hot water (letting the water sit overnight), then let them dry. I stacked them and used aquarium-safe silicone to 'cement' them, into swim-throughs or bridges. (Be sure to let dry for at least 24 hrs.)
I've also used silk plants from places like Micheal's. (I made sure they were glitter & fragrance free, with no 'decorations' like fuzzy centers, etc.) I soaked them in hot water (letting the water sit overnight) to make sure they didn't leach dyes, etc. Also, I only used the silk/plastic parts -- making sure there was no metal in the stems.
And I made my own 'hammocks,' using a silk leaf (from one of the above mentioned plants) and a silicone suction cup. I used a push pin to create a hole in the back of the suction cup, then pushed the 'stem' of the leaf into it.
I tried to use glass pebbles and aquarium-safe silicone to create a cave. It didn't come out like I'd wanted, but I'll keep working on it. Maybe a little swim-through would be better.
I've read that Legos are safe, and I'm thinking about getting some.... And I would think that anything plastic that's safe for children/infants should be OK, since the manufacturers (hopefully) wouldn't use any plastics or dyes that aren't safe for children.
And I've used terracotta pots. I boiled them first, then I removed the backs from some, and plugged up the hole with aquarium-safe silicone in others, so that the fish can't get stuck. I'd read that they can change the pH, but I tested it, and the ones that I got do not seem to affect pH.