The 20 gallon would be best, without a doubt. Three small goldfish in a reasonably maintained 5 gallon would stay healthy until the switch to a pond. Naturally, you need to keep an eye on water quality. Ammonia can build up quickly in a goldfish tank. Feed carefully and do your water changes. Goldfish will get red veins in their fins when ammonia levels are dangerous.
If you wanted to permanently house your goldfish in an aquarium, the bigger, the better. Be observant, do some research and you will gain an intuitive sense as to what you need to do. The main thing is to enjoy your hobby with mutual consideration for yourself and your dependent creatures.