Well you just so happen be speaking to a balloon mollie owner!
I have 5 balloon mollies currently. They are WONDERFUL. Mine are not messy, I've never understood why people say that. I have noticed they defecate a lot, but with my filter and weekly to bi-weekly water changes everything is great.
The only reason they need a larger tank is because they are HUGE schooling fish. Plus, if you have a male, it's better to get 2-3 females per male to spread the "bullying" around between the girls. If you got some females, I think you could do a 10 gallon tank with four of them and they'd be happy. I've never tried, so I wouldn't know!
The males can be huge bullies, but it depends on their personality. You know for betta males they can live in the same tank with enough space and ample hiding spaces? Kind of the same way for male mollies. While usually they won't fight to the death, your tank will be much more peaceful with a lot of space and with nice hiding spaces.
Personally, I have 2 males and 3 females in a 20 gallon right now with a GlowFish. I want to add 2 more Glows and maybe 1 more mollie female, we'll see! A lot of times people will cram a whole load of these guys together but that will quickly ruin their structure and they simply won't be happy. No fish likes an inch of swimming room!
Be warned, mollies WILL reproduce pretty rapidly if you have good quality of water. Good news? They will be sushi for the other fish if you don't want 'em! They are in fact live-bearers so you won't have any eggs. If you'd like to raise the fry, having a breeding cup or a small separate tank to place the pregnant female in will easily keep them separated and free from harm.
It is SUPER simple to tell if they are male and female:
Males have a pointed anal fin, females have an actual fin.
If you'd like to know anything else you can message me, I'm happy to share!