One thing to take note of, if you do put marimo balls in there do not use excell! It will kill them.
This is my kind of tank. I have done a setup almost exactly the same.

Same filter, heater, light bulb. I used sand and a little piece of wood and everything! It worked out fairly nice. I've done a lot of little tanks.
If there is any direct sunlight on the tank at all I would change the location because that is the worst thing for causing algae issues.
Removing the filter will probably make your algae problem worse but not for sure. I couldn't imagine a betta being happy with it because it really does create a lot of current in such a little tank and there is nowhere to get away from it. Although it may be in your best interest to keep it as far as plants go I wouldn't blame you if you took it out. One thing to consider is that you'll end up with more mulm collecting on the sand, especially around the plants and in the corners, and it could affect the betta's health if it is allowed to linger. The still water, especially if it is kept warm with that heater and those lights, is great for harboring the kind of bacteria that would cause fin rot, as well as water mold which is even worse. So I'd suggest adding an airstone if you do remove the filter.
For plants to fill things in I would suggest some java fern, its the hardiest plant I've found and always looks nice. If you can get a hold of some windelov that is even nicer and doesn't get as big. I would suggest a small crypt but I've never had luck with them. Maybe its my water, though. It might be worth a try. For a stem if you want one, I have had good luck with red ludwigia. It grows a little green and semi-slow under lights like yours but it is nice.
I couldn't tell from the pic, but how far do you have the light from the tank? It might affect the bottom plants if it is too far. Also, what color temperature is the bulb and how old is it? If the light spectrum isn't ideal for the plants it can encourage algae growth.