It would be historically accurate, IMHO. Nobles tended to have contacts keeping tabs on other nobles, and with the idea of feuds, hatred, political machinations, etc. it only makes sense to me that any Landed Lord be able to control who can come in and out of his region.
It also would make an interesting dynamic for personal politics, that I feel is missing from the game. So far, feuds tend to either end in blood or explosive words (I'm thinking about thalmarkin in recent weeks), this would allow for another option to be pursued, with a lasting effect.