Well, I used to have a really small bowl to keep my betta in, and I used to take him out because I always had to do 100% water changes. I would actually take him out
and then dump all of the water out. This included the gravel. I would rinse the gravel to make sure that it was clean. I've had him for 8 months, and he never seemd to react badly to this. I would put him back in almost immediately after changing the water because the label on the water conditioner said that this was ok.
How big is your tank? You don't always have to take all of the water out at once, AND a turkey baster is pretty good to suck up extra food and waste. If you have a one gallon tank or more, you don't have to change 100% of the water all of the time.
So yes, it is stressful, but a lot of it depends on the amount of stress your fish can handle.