Eventually you will want silk or live plants.
I'm assuming that plant is plastic plant he has which can be bad for fish with long flowing fins such as bettas (torn fins leads to infections). I wouldn't get rid of it right away though because it would probably stress him out too much if he felt like he had no shelter.
Did you clean/rinse the gravel? (I don't know how well (if they did) the previous owners cared for him or his tank).
Also add a thermometer that sticks to the inside of the tank, to the list.
And if you have a very clean
plastic cup (don't use soap/detergent), you could lay it on its side with some gravel in it to weigh it down so he has a makeshift cave/shelter.