In my experience, infiltrator skill deteriorates faster, and it seems harder to regain the skills with training sessions (more success with sword training, but that may be random). Plus only 6 hours per turn at his age...
Yeah, the hours thing is annoying.
Interesting about the skills. I wonder if oratory degrades too?