Lots of things factorin. Other religions, preaching frequency and skill, number and size of temples, region stats (?), some other modifyers that are unknown to the player base maybe... but in general, you should always maintain your following, especially when other major religions are nearby.