Remind me of the time when Cleatus was removed from Sovonoval region lordship due to his old age slow recovery from serious wound after his religion preaching. He lost the lordship but was able to regain it back
Trying infiltrator attack on that Royal Duke and make him seriously wounded. Coupled with his old age factor and he unable to perform any work on the city, he will be remove from his post. If your Judge and your own realm infiltrator cannot do the trick, try put a high bounty on that Royal Duke head. Sooner or later other realms infiltrators will come to attempt the bounty
Exile option, I only saw it worked once in the past on Colonies island. When Spearhead was exiled in the past due to his Royal status. The person who done it of course lost a lot of h/p as long as Spearhead remained in the realm without leaving.