There is, and I'm still baffled as to how we still have some fame points that get accidentally given twice, so they have to be taken away.
I think I have fewer fame points than I'm supposed to based on my reading of the wiki page. According to my offices, events and attributes, I should have 10 fame points, but I only have 7.
http://wiki.battlemaster.org/wiki/Buffalkill_Family#Fame