I think allowing priests to build level 1 temples would be fantastic, as would allowing lords to upgrade them.
I do think though that if a priest builds a level 1 temple, they cannot upgrade it for a set amount of time. Giving the lord more than enough time to destroy it if they don't want it there.
