Changing The decor around is not upsetting in itself, no. but the frequent ness of putting your hand in and moving things around would probably freak them out if done too often. Do it when you do large or 100% changes, when you need to take the fish out.
Rearranging, like said, helps ward off boredom somewhat and can also help curb excess aggression- the betta thinks its new territory for like a week.
Make sure I'd and when you move the decor, nothing can harm your betta by either having something that would squish him or that he can wedge into, like between a cave and threw tank wall.