For cities I'd say the militia is mostly fine. Cities should be hard to take, after all. But I agree that seeing 20,000 CS worth of militia in a rural backwater is pretty ridiculous.
Perhaps there could be a chance for every unit that it simply does not show up for the battle? Militia is basically leaderless, it wouldn't be a stretch to have some of them to get cold feet and simply not be present.