Make it a carrot instead of a stick. All Dukes could receive a temporary +1 prestige for every Lord sworn to him. This works like unique items: when the Lord leaves, the +1 disappears, but the Duke can never get below his original prestige level.
This also neatly avoid all griefers issues.
That would also work if when they first become Duke they receive a temporary +1 for every region already in the duchy.
