That's not an issue in my experience. It's more like, some regions require you to have so many knights at a minimum, and sometimes, even that minimum can't be reached. For bigger realms, downsizing is not so much an issue. For smaller realms, it means they never get past the stage where they are a little puppy in a forest filled with wolves.
