I would say the same in your position. Although, alone you would not have been able to take on Averoth, Caerwyn and the rogue forces. The rogue forces aren't going away, so you still have to defend your north-western border. While Caerwyn has few nobles compared to its size, so they can easily maintain another 40 nobles. It will increase their army, and their home regions will be better maintained to provide more resources to fight with.
After the "Golden Farrow will be next" talk, I think you might have underestimated your opponents.
