I have 3 males, each in a 3 gallon tank- 2 are great with the tetra's in with them- they flare at them sometimes (It's rare) but it seems to be more out of boredom than anything. (They did chase them a little on the first 2 days, but I think they were curious, it's been years now and they seem so fine with each other) The tetra's can even steal the food out of their mouths and they don't seem to care, they just look for another piece. The other male though, he started to eat the tetra's eyes within a month (gross) so he lost his tank mate "privileges." I also had a male in a large community tank, with colorful guppies and platies even, he was fine. Very non-aggressive. I introduce a sick fish to that tank though and killed most of it off (learned to use QT tanks after that, argh)
I've also had a large group of female betta's in a community tank, and they were extremely non-aggressive towards the other fish. Then I got a new female betta, named Hecate, and she was oddly aggressive and also started going for the eyes. Killed quite a few things before I figured out who it was. She ended up living in a breeder net in that tank for quite a while. She's older now and seems to be far less aggressive. She's back living with a new group of girls.
So, from my experience, I'd say- for the most part you are fine, but you do have the chance of getting a pissy one. Pretty much what the person before me said :)
I would go for it if I were you ;p You're already setting it up, I'd start off with a few additions and would bet it would be fine. You can always get more tank mates later too ;D