Everyone is assuming the realm is united against him. That isn't necessarily the case.
Of course if the realm is united its easy to cause enough trouble to make him leave. I'm talking about the possibility of being removed from normal in game plotting or efforts.
