I could possibly live with temporary nerfs, but not permanent.
Permanent could lead to a judge being feared so much, that nobody would want to take any risks-- boring for both parties.
Infiltrators already have a random chance to lose some skill when tortured, don't they?
The other side of the coin though-- Judges do very often get nerfed for their acts, in the form of honour and prestige loss (once had 5 prestige loss in one single hit).