Males and females, and males and males, should never be kept together unless divided securely, with the divider high enough (3 or more inches) to avoid jumping over and ending up together!
There are even very few community fish that are okay with bettas for a few reasons... some male bettas (and even females!) are very vicious. Other fish find the flowy fins fun to chase and nip. Other fish are aggressive. Or other fish stress the betta which can cause a sickness...
chances are the group of male and female were in a bigger tank right? Planted or something? They were all siblings, more than likely. In larger groups that are already in order of who is alpha, can live peacefully. the alpha (your male) is attacking the only other betta there... your female.
Goldfish are cold water fish and should never be put with bettas, as bettas are tropical - warm temperature. Please don't put them together :) I suggest you separate the male and female... very very few people on here have even had their male and females get along - and they are the very experienced people. If it were possible, I'd love having a male in my sorority... But I won't.
Betta Splendens are called Siamese Fighting Fish for a reason... Males kill each other, and females can get very very hurt from the males and sometimes each other. Even breeding, the females can get killed.