Yup, the downside is pretty much why it was removed.
10 years ago, a realm of 5 regions had 20 nobles, most of them young, almost all of the participating in the mobile army. Say, 75%.
If we extrapolate with how things were when the penalties were removed, and the current density... Today, that same realm would have like 10 nobles total, 7 of which would get like 5 hours per turn, at least 5 of those would forever squat safe regions, and your mobile army would be about 3 active nobles and 2 who always lag behind. Good luck waging any kind of war with that.
