I would keep the filter on by turning it off you might affect the good bacteria in the tank. You can baffle the filter to slow the flow down. Put a piece of aquarium sponge where the water pours out of the filter. Use a rubber band to hold it in.
Be very careful keeping the male with females you mights wake up one day and have a very beat up Betta or a dead Betta. I would honestly take hime out of there. How many females do you have? What size is your tank?