Ok, here's how I got rid of it:
-By hand, remove as much of the solid/intact pieces, as gently as possible.
-The above step didn't help much, so lots of soggy pieces where left behind, so next, I left the pieces settle. Then, I took a turkey baster and sucked up as much as possible.
-Let it settle again and then cleaned off the filter intake.
-Rinse and repeat the turkey-baster/ settle/ clean the intake process, over the course of about 5 days, doing it once a day.
There was a few bits left, and I'm sure if I really agitated my gravel, it would stir up more, but it's no noticeable anymore. Mollies and stuff might eat it too.