Sorry, I took you meant the warrior class when you mentioned warrior.
What if P/H could fight personally, e.g. in duels? Or can they already? In that case, I think it would keep the balance of having priests unable to have a unit, but still keeping a "warrior" feel to them.
Priests and Infils cannot duel.