Is the baby a female? 4 females is the bare minimum and would actually recommend one or two more depending upon if you have other fish in the tank.
To bring the females out more from hiding, make sure the tank is very densely planted will help build their confidence and security.
Males should never be with females unless you are breeding, in which case it's only for the mating process, and even then they will fight and a lot of times one will die. They may not be flaring or doing much right now, but that is because they know there is a barrier.. adding the male in will ensure that one or most likely more, will be harmed and/or killed. They may not fight immediately, but they will fight in a short time.
The only way males and females have been able to be successful (and by that there are still issues) is going a few generations of keeping them together in a very large tank.. it takes multiple generations and a very long time to attempt it.. as well as large tanks of 75 gallons and up- at which point you may be able to keep a male in with females.. even then it's a high risk.
Highly recommended to not put them together, as in the wild neither of the sexes want to live near another.. we can just force it upon the females sometimes, and even those tend to have issues and failed sororities at some given point.