There was a lot of filter media but when I noticed that my betta wasnt feeling so well but there weren't any problems with the water in the tank I started investigating and took apart the filter. That was when I noticed a bunch of brown sludge in there and cloudy water stuck in the bottom of the filter with brown sludge floating around. So I got really worried that there could be bad bacteria in there as well, so I threw everything out and decided to start over. I probably could have kept the filter media and just swished it around in the bucket, but it was before I had posted on here and I just wanted to play it safe. I think I waited too long without even doing a small cleaning inside the filter, but this is my first tank with a filter, and I didn't want to kill the beneficial bacteria so I just didn't clean inside there at all.
I am learning proper filter maintenance now
On a good note, my fish is doing much better, much more active he's gotten his color back so I think we are doing really well now! All I have to do now is re cycle the tank.