Lord is never, impolite, especially if you use it as "My Lord". It conveys a general feeling of the other character as being of very high rank, without being specific about which piece of land this refers to.
If one of my characters is called "Lord" by another noble and my character happens to be a Duke or higher, he will take insult for it. The only nobles able to address my characters as "my lord" and get away with it are my direct vassals.
