Downside
This would be another factor that reduces communication between a Liege and his vassal(s) further hollowing out whatever remains of a feudal hierarchy.
Look at how taking estates freely has changed stuff: I can't tell how many knights have taken up estates without as much as a notification to their new Lord. Tbh I kick them out regularly in hopes that they will do better in the future (and obviously take them back if they open a dialogue)