I would still be extremely cautious. I had a sorority of females that I purchased from a young age and there was a male in there who lived very peacefully alongside them as he matured.
Then one day without warning I found him savagely assaulted, with all his fins (even his beard) torn off and basically dead. I nursed him for a week but he passed not long after the attack.
I never saw that coming at all, and it just reaffirmed how unpredictable these fish can be in terms of aggression.
So while things may seem to be working fine now, just be aware that there could be a day when you wake up to one or more maimed or dead fish.