Errr.... Let's make sure we are swishing it in the right place ;)
When doing a water change, the old water you are going to dump out, you want to rinse the filter in THAT. Not the actual tank (otherwise you have more debris in there lol). However you don't HAVE to do it weekly. I do it when my filter starts looking "grungy", which is at least once a month.
it depends on hoe dirty it gets. For my tanks with only 1 or 2 bettas..I only swish once every 2-4 weeks but for my 20 gallon which has some rather huge snails, clown pelcos and bettas - I have to rinse out the filter media and the inflow tube Every week or it gets so clogged up with crap. literally. I dont have cartridges in there though..I stuffed it with aquaclear filter foam and havent had to replace it yet - been about 2 years now.