I find mollies should be in brackish water... Plus they are bullies. They'd have to be in a group of 3 females, or 3 females to one male...
Actually, I recommend having the females in a 20 rather than 10 to even out space and territory. When you add, you can remove her to a cup and float her with the new ones (please PLEASE quarantine everyone first off) and add together, after a water change

It is normal for torn fins to be here and there... It's to say "hey! I'M boss! ranking

there has to be an alpha.. and they gotta figure it out!
I have 10 in a 40, and there were torn fins, a booboo here and there... and there, done

You also have to pack the tank full of silk, fabric, live plants and hidey holes... >< So they do not see each other 24/7.
Try with a low number, like 4, after you stock that tank full o' plants

you shouldn't see one side of the tank from the other, basically.
Petsmart I think has a 7-14 day policy for live fish...