That means someone who is sitting at 100% fame has gotten his fancy swordfighting item back and pushed the upper bound even higher. It is not that you lost your skill level. It is just that someone gained more. So relatively speaking the gap between you and the best has widened.