If you can't live without him, get him!
Just get some tank dividers or a cool home for him.
Keeping Betta females in groups smaller than 5 is NOT recommended because they WILL fight at some point and the winner may kill the looser. If the girls are in a group (sorority) of 3 the 2 most dominant will gang up on the weakest and kill her. In a group of 5...one will rule as queen and she will be so busy bullying the others that HOPEFULLY no one will suffer too much of her wrath, because she will constantly be distracted. And don't forget to give LOTS of places for the girls to hide... They should have at least 2 or 3 designated hiding spots PER fish!
The male should live by himself, or he will kill the the girls.
They are an anti-social fish, they aren't called "Siamese FIGHTING fish" for nothing! There is even a "hobby" of fighting them like roosters, which is STILL very active today in parts of Asia (and elsewhere)
The Betta fish is so anti social because they are territorial, of they feel their space is being invaded they will attack the intruder! The male ONLY likes the females for a VERY short time, when he has set up a nest for her to breed in, after she has laid eggs in it for him, he wants nothing more to do with her, and will kill her if she lingers in his teritory tio long after laying eggs for him.
You also left out one very important piece of information; how big is the tank?
Bettas dont need a whole lot of room, but they each need their own space!

Good luck