What's already been said sums it up nicely. I personally don't find my fish stressful. What I find stressful is my two stupid cats who must inspect every thing I put down on the counter where they're not supposed to be, from a cup with water, a fish net, water test tubes, etc. They seem to think I put things down just for them to knock over. They also like to throw their food all over the kitchen floor.
So, in a nutshell, my fish are far less stressful than my felines and are actually a bit of an escape. :P But I love my stinkers and they remind me of it when they curl up on my keyboard or nuzzle under my arm and it all seems worth it.