I would recommend getting a filter for your tank no matter what route you go, if you are willing to spend $30ish for one. I had a 2.5g and used AquaClear 20, which has an adjustable flow rate. I kept it on the minimum flow and it was perfect for the tank. Having a filter just helps tremendously in keeping the water clear and clean. You still have to do water changes, and I would recommend a 20-30% change every 4 days for a 2.5g. But a filter just makes things more pleasant and easier.
As far as decorations and the like, unless your fish utilize a cave, I would not put one in there. There just isn't very much space, and if they don't seek shelter in the cave, then it could be removed for more swimming room or perhaps a small live plant. Dwarf sags are a good small plant that resemble grass that would look well in a 2.5.