If it's new gravel, especially painted Top Fin gravel, if you don't rinse it off first, it will have dust/debris for a little while, which will eventually settle and then get removed as you do gravel vacuums. This is normal. Even with rinsing, there will likely still be a little dust.
If the particles are from snails (poop) and fish (also poop and leftover food), then there's going to be stuff in the gravel despite vacuuming unless you go over every inch! If it really bothers you, try not to stir up the water so much, or perhaps consider switching to a bare-bottom tank. Gravel is not a necessity for some fish.