When a vill is getting attacked you can force it to keep building by spam clicking whereas if a vill is getting converted it will walk away for a second no matter what so clicking through each enemy vill with a monk is a good way to defend against castle drops.
I think the vills return to work normally. The AI pulls them back towards the TC. You can see one or two vills go back to work after having been targeted and de-targeted.
