It is that Lord who has been exiled and the locals won't associate that anyone associated with that Lord, such as the Stewart, in fear of punishment.
Historical example is Sir Thomas Moore. After he was exiled, most of his staff quit because they were too afraid to be associated with him.
