My general rule for mollies is that they should have 30 gallons not only for water volume but also for swimming space and then an additional 10 gallons for every individual fish after 3. So 1-3 mollies would be alright in a 30 gallon but 4 mollies would do better in a 40 gallon, 5 in 50 etc. it's a rough guideline and other people's opinions vary but it works for me.
For the fry. My style of breeding is philosophized as "natural is better" so I like to move fish around as least as possible, this is my belief for the sake of both the mother and the fry. Some people like to move the mother into a grow out tank until she births, and them move her back after she finishes so that the fry have their own tank to grow safely away from larger fish. I like to keep the fry with the mother as this way nobody is being moved around and stressed out and if I lose a few fry then I just figure it's natural collateral (ironic because there in glass boxes, but still...)
I believe this strategy works well with species tanks but not so much in community tanks, I also am a big believer in heavily planted tanks so my fry naturally have densely planted areas to hide in and don't need breeder boxes, etc.
Some things to note if you are going with the separate breeding tank is that you want a filter that isn't going to suck the fry up into it. Also, sometimes moving pregnant fish around can become so stressful that the mother will abort or even reabsorb the fry, not giving birth at all, and even death of the mother is not unheard of, rare, but not unheard of. In general, however, live bearers are some of the easiest and most unforgiving beginner breeding fish to go with so whatever you decide to do, you will eventually get fry, and most likely you will eventually have so many that you get sick of them.
Good luck
