Training is most effective for getting your unit to 100% cohesion, not training. The chances you increase their training is actually pretty low, it seems.
They're more likely to increase training in battle; I don't think it's tied to a specific stat. Given enough time and gold, you could theoretically train any unit to 100% with any noble... it's harder to keep them alive that long.
I picture it like this: Running drills together and practicing makes them work better together (cohesion stat). They aren't actually learning anything new, for the most part. In battle, they get real experience and might be more attuned to battlefield conditions. This includes practical choices made on the battlefield (training stat), but not necessarily taught in drills.
There may also be a connection between the chances of increasing training stat with higher Leadership skill, but that's just a wild guess on my part.