Even so, it is very rare for females and males to get along in any tank, none-the-less a 5 gallon. A 5 gallon tank is only big enough for one betta or two betta's divided. You need at least a 10 gallon for a sorority. (I'm surprised you didn't have any babies.)
IMO you should change your tanks water more often and never put males and females together again. I think MSG is right; the new betta you introduced brought in a disease. But for the name of the disease I don't know.