Five Steps to Take to Leave a Church Well 

Sorry this happened, that's awful, but that's not the biblical process. Bring formal charges and a second witness, do it quickly, and if the rest of the elder board doesn't take action, bring it to the church publically. Depends if you're church is elder ruled or congregational, or appeal to your presbytery if you have one, but if your pastor is that obviously disqualified it will be easy to rally people, to your cause. You don't want to abandon the rest of the congregation to a sexual predator. But need to follow Matthee 18, have some faith in the rest of the elders, and if you're congregational, at least go for a vote of no confidence first. Better to remove a wolf from a church than leave the church entirely. Shouldn't make such hasty assumptions of the rest of the church that they won't do anything about it.