Go to your local home improvement store for gravel. You can buy a huge bag of it for about two dollars. That's one of the cheapest ways. Tanks can be improvised sometimes, but for a betta, because the heater needs to be at a decent temp, you want to avoid plastic. You can usually find inexpensive tanks at second-hand stores, craigslist (or your equivalent), freecycle, etc. Or you can often times find fairly inexpensive set ups which include tank and hoods only. Plants are best ordered online for the best deal (you can usually find free shipping too) or you should try the smaller LFS in your area as their prices will almost always be better than the large chain stores. For caves, get some clay pots. These are inexpensive and safe. You may be able to source wood in your local rivers or lakes, but make sure the wood is not pine or juniper and be sure to bake it to remove any contaminants. For a light, depending on the type of plants you use, a bright desk lamp may be the cheapest way to go.
So if you got what I suggested, your total breakdown will look like this:
Gravel: $2.50
Wood: free
Clay pot: $1.00
Light: $5.00-15.00
Tank: $0.00-20.00
You are looking at a minimum of $7.50 depending on what kind of light you use and how you source the tank. You can also look into alternative tanks. Large heat resistant tubs often work well, though they frequently are not clear so you can't see your fish all the time. You can get large glass bowls (but it would be difficult to fit in your filter in something round). Or you can usually for $20 dollars get a five gallon tank with a hood. Of course prices may different depending on where you live, but these are the average costs for me. I did not include plant prices because this really depends on where you get them.
Good easy plants would be java ferns and java moss, plus anubius species. These are all low light plants so they would be happy with a desk lamp instead of fluorescent lights or similar.
??
