In our experience, Goof Off is safe to use on Toyota paint with caution. Any solvent is going to soften the paint in the area you use it, so it is more likely to cause scratching. Be gentle, and similar to using a clay bar, allow the product to work with light pressure applied to the microfiber.
@sethsphoto You usually don't want to use Goof Off on interior surfaces and definitely not on interior plastic. The white residue you see is actually the result of a chemical process that basically "melts" the plastic leaving a damaged surface. You may be able to reduce the effect with a magic eraser and some interior cleaner, fingers crossed!
