but as a lord, you are switching oath from 1 duke to another. that it happens to be a different realm is neither here nor there. your oath is to the duke, not the realm.
If there were a region-ban for knights, or a duchy-ban for lords, I would advocate for it to be applied in these cases, too, and leave it up to the lords and dukes whether or not to let those autobans stand.
However, there is no such thing in BattleMaster right now, so there's no way to punish oathbreaking at the sub-realm level.
If you break an oath in such a way that you leave the realm, however, there is something already in place to deal with it: a ban.
If you want to argue that the realm shouldn't care about regions and knights leaving it, you're welcome to try, but I think that ship has sailed.