I use pots like these in both of my tanks. They are probably the best (and most affordable) caves available to the hobbyist. My bettas love them. I like putting a few inches of substrate into the pot and planting plants so try look like they spilled out of the pots.
I love the way you've got all yours arranged. Very whimsical.