I agree, it would be a whole lot more fun if people who actively participate in battles could beat that guy who's been wasting gold with solitary academy training in a duel.
It could perhaps also be cause to decrease the randomness of duels a bit, since it'd decrease the gap between characters' skills.