Temples are not so easily destroyed, so much as looted and reduced in size. A level 6 temple would take three full days of looting twice a day to destroy, assuming you are successful every time, and the temple is not guarded. Given that they can be built instantly, I don't think that making you take three days to tear one down is all that unreasonable.
You are correct, I think, in that looting temples doesn't cause unrest among the peasantry. Perhaps it should, based on the number of followers. Looting and burning temples should probably cause the same effects as arresting priests. After all, sacking and closing temples using the lord's options does cause unrest. Quite a lot of it.
