Yeah, that's it!
But after treating with aquarium salt and getting my "good" bacteria in balance, it's gone away!
I went to another fish forum, and they told me I was over cleaning my tank. And I know what you all think, 2-3 days for 100% water changes isn't too much, but it was HOW I was cleaning.
I would kinda ummm... scrub the aquarium with vinegar water every 2 days to kill what I thought was "bad bacteria". I would rinse out the gravel multiple times with hot vinegar water, scrub the glass with a sponge, rinse all the plants with hot water repetitively.... Well, you can see where this is going
So now that I've left the good bacteria grow a bit, he's been doing much better!
And I'm sure the aquarium salt helped too