Those 81 men are Special Forces. Yes, there are stricter limits on how many Special Forces you can recruit than there are on how many Infantry.
Tom can come in and contradict me if he wants, but as far as I'm concerned, that's never going to change.
Special Forces are elite, highly trained troops. Infantry are basically just commoners off the street trained to handle a sword and shield. Want to have a staggeringly massive unit? Recruit the el cheapo infantry.