But then they would take forever to get anywhere since bigger units are slower and older characters get less hours. It could work with cavalry though.
Which aren't as good as others against walls. And as it's the cities that would get more gold, they'd likely also have more militia, and thus, cities would be much harder to assault directly.