Welcome.. and don't be too hard on yourself, most of us have made mistakes when starting out, and who knew that we couldn't always trust what a pet store employee tells us?
Most likely he is unsure what to do with himself. Born and soon after placed into a jar at the breeders, then into a cup to be shipped and moved around, then into a vase. Being an older betta and living in a vase, and smaller all his life, he is not used to large spaces.
As mentioned before, more plants would benefit him. Whether real or silk (fake), having a fair amount in there, and you already have a cave.. He may feel more secure in knowing there are hiding places in the plants. It also will give him some stimulation mentally- him swimming around and checking each thing, looking for food, etc is stimulating for them, just like when a dog chews a bone.
If you feel he is stressing out too much, and you will know since you know him, then I would suggest possibly giving him less... I know, I'm sorry.. you've done great trying to help him and give him a great home, but sometimes some just can't handle a larger tank. One of my boys can't do anything bigger then a 1 gallon without withdrawing, not swimming and not eating.
I would suggest going to PetCo and picking up one of their kritter keepers
, the medium and large are great for them.. you can heat them safely as well. It gives him a smaller space, but still enough room to heat and decorate.
But at this point it's up to you- give him some time and some more plants to see if that helps and that he will eventually settle down- or if he is just way too stressed then move him to something smaller. I personally would wait and see how he does when you get some taller plants in there- the bottom looks fine, but I would try to fill up the middle and top a little with plants.. a lot of times they love to sleep on leaves close to the surface, so that will give you some idea of what to keep an eye out for.
Good luck, you are doing great.. he just needs some more security, and your neons would love it too :)