Guppy - Pygmy Cories are a great suggestion! I have never seen them locally, unfortunately. I would have to branch out to find some. But I like that idea.
Probably some of the other small species of corys would do perfect, like habrosus. The pygmies are mostly middle swimmers(at least mine) and it's better if you populate the bottom instead, it will utilize the space better and with the shrimp it will be busy enough. They'll also go and exlore upper areas if they feel comfy so they are entertaining enough as well.
I'd suggest more natural plants before you put the shrimp in. Also, the betta may like a cave like a coconut one. Mine really loves going in and out of his. You can plant anubias and moss on top of it like on the picture below. It gives shelter to the shrimp if you get them breeding, looks natural, as easy to keep as moss and they can hide from the betta in between. Smaller tubes with moss on top will be also a good idea to avoid the betta snacking on every single baby shrimplet.