Sometimes it's the sandpaper dust, try using a brush on it, something like a tooth brush. If it's not that it could be the really fine sandpaper leaving scuff marks from pushing too hard on the sandpaper, if it is then that can be sanded out by going back with a lower grit paper. Hope this helps 👍
