No, you have misunderstood.
The oath a Lord swears is to his people. Everyone in the region. It is an oath to protect them. Stepping down means that you are abrogating that oath.
No one can release you from it. Once you accept a Lordship, stepping down is, and should be, seen by its people as an admission that you cannot or will not protect them.
I've never heard of it being like that, I've always been told that it was an oath to the Duke that you would ensure the people and region itself were protected and taken care of. A slight, but significant, difference. That is why lords are part of a duchy after all, otherwise there's no reason a lord couldn't just be attached to the realm without any duke over him.
