So how about this: Any priest can create a schism which triggers a kind of a vote where every member of the religion votes. If a certain threshold of votes is gained, a completely separate religion is formed that gets the name that the wayward priest entered at the time of declaring the schism, and the priest becomes the "prophet". Those who voted for him switch religions, and the peasants of the lords doing the switching swap religions as well, as do temples in their regions. If the threshold is not reached, the initiator is kicked out of the religion and reverts back to warrior.
Alternatively an elder can declare a priest heretic. This triggers a schism, which prompts the "heretic" to write a name for the potential new religion. If he gains enough support, he forms a new religion with the mechanism mentioned above. If he does not gain enough support, he is driven out as a heretic.
So if a priest actually enjoys some support (at least one lord with a temple in his region + others) there is a true danger he will take some of the religion with him. But if he is just annoying, he's going to get kicked out.
Also, when the schism is going on, he "heretic" would try to convince as many followers to his side as he can, and vice versa. This creates interesting interaction. Perhaps also the timing of the split would not be set in stone, but would be somewhat random and would depend on the members choosing their sides. Perhaps the schism would last as long as some percentage of the followers have made up their mind (default would be to follow the old creed). While the schism is going on, there are drops in peasant following, and perhaps violent outbursts among the peasants, and perhaps even some temples being destroyed in the process.
The larger the split and the longer it takes, the more there would be these nasty consequences. Would that not work?