towns are very expensive to build up - which kingdom are you in? if you are in mine i'll help you out
I'm in Staberinde.
It is pretty expensive... not just in gold, but in having to range around to get clay and stone and iron and wood. But since I now have 10 characters, I think the Barracks is a good idea, that way they can get XP even though they're not in the party.