We're not commoners, and the lord of the region can go swear to whatever duchy he pleases. Who knows how "realistic" this is, but it would sure as heck make a lot of people mad that they either must get banned or stay with an enemy realm in case something weird causes their region to join an enemy realm while they stay on as lord. This has happened occasionally in the past.
