Lords should definitely be able to claim the land back. But the church is going to hate that, and any local peasantry of that faith will be very upset; the trick is to make it sufficiently hard for the lord to claim it back, without making it impossible.
I love this idea. It creates another mechanism for conflict. It gives religion a way to gain power, to create an incentive for people to be involved. And, bonus, it's historical roots are absolutely authentic.
