My suggestion would be to have two takeover options.
We used to have standard takeovers and colony takeovers.
In the same way, we could have standard takeovers and hinterland takeovers.
The first would simply not be an option if we are over the limit.
If this is how it gets handled, we would need a way to switch regions between core and hinterland. Either a ruler or a duke I would say could do that. Like handing a region to another realm though, I would say the one doing it would need to be in region. And I would suggest that a region with a lord cannot be converted into a hinterland region. A random chance for it to work would be interesting, modified by whether the region lord agreed or disagreed with doing it, but the easiest way to code it would be with a hard yes or no depending on if there is a region lord.
