Ehh, I'm just going to say that your tank is overstocked but only the goldfish makes it that way. NOT going to bash you for it, I had 2, 1/2" guys in my 26 for a few months with some tropical species too, till I was able to re-home them, and with enough filtration and water changes, which I'm sure shaina performs, her's will be fine till they are re-homed!
As for the original issue, shredded fins, it's to be expected at first, how long have they been together? If its been over a week then you should probably go get more decorations and hiding spots, but if not then they are probably still trying to sort out dominance. Goodluck with the tank!! And -- when you rehome the goldfish I would suggest adding a couple more females to the sorority just to even things out a little better
as for the 3-4 in a 5 gallon, thats just not enough space for them to live without constant fights eventually. I wouldnt risk it. 5 gallons just doesnt give much room for them to swim and hide from somebody higher on the totem pole.