A potential fix for this, would be that if your lord will not release you, the duke can go over his head to do it.
If the duke wont do it, the King/ruler can.
Also, it shouldn't prevent you turning to a new realm, only changing within yours.
Turn coats happened, they could take a hit to their honour, but if they're willing to turn their back on a realm then the oath means very little.
