What was the set screw for that you had to drill out? I don't remember seeing that before. Also, not sure where you are, but would it be better to use a frost proof valve?
There's a set screw on the side of the vacuum breaker that keeps it from coming off. If your bibb is old then you probably can't back it out. I'm in Alabama so the frost proof is not a huge deal but I'm going to put one in when I replace my other faucet.
