Goldfish are cold water fish.
Betta fish are tropical, and as such should never be kept in cold water unless you want a very lethargic, and possibly stressed (leading to sickness and/or death) betta fish.
I would never recommend having goldfish and betta fish together because they are completely different for environmental needs, and dietary needs as well. The goldfish also will grow to be a lot larger than what he is now, while the betta fish will probably barely grow anymore than he is already.
The size difference is the least of your problem - the amount of ammonia goldfish put out is scary, and in anything under 20-30 gallons can prove to be potentially lethal. Those with smaller tanks and goldfish usually have 60+ gallon filters, or double filters which then would irritate the betta fish.
Either way, they may or may not bug each other mainly because they would have to be dealing with stress, sickness and potential death