Make it a carrot instead of a stick. All Dukes could receive a temporary +1 prestige for every Lord sworn to him. This works like unique items: when the Lord leaves, the +1 disappears, but the Duke can never get below his original prestige level.
This also neatly avoid all griefers issues.
I've grown increasingly fond of methods like this of applying both bonuses and penalties in recent months...if we do make some kind of incentive for Dukes to keep Lords in their duchies, I think it should look a lot like this.
