Around four days is the time you're supposed to take the male out (is what I've read.)
And you're feeding the fry? I believe you should siphon the bottom of the tank every day to get rid of dead fry and gunk.
A sponge filter set on low would help too.
... I have an idea, but I'm really not sure and I don't want to say anything wrong... Other members on the forum can help you.
Don't quote me on this, but maybe small water changes every day or every other day. I'm just familiar with ten gallons.