No.
Infiltrators have to go through many checks. When you don't have a unit, you are an easier target but if you are without a unit with high swordfighting and high infiltration then you are no easy target. If you just have high swordfighting, you are more vulnerable but still a tough target.
