Your interpretation of attack speed for skull Master is not entirely accurate; attack speed will increase speed of the skulls spinning around him whether or not you're casting. So it does help you when you're running around, and mathematically you have higher dps output with increased AS over increased damage, demon pact is pretty equal if not a little better for output
Attack speed has 0 effect on skull spinning while moving. I would bet money you are experiencing the rotation effect. If you run around an enemy the same direction as your skulls spin, your damage will be horrible. But if you run the opposite way, it's much much higher. I'll be showing this in my hard mode video for skull master.
