I'm still uncertain as to whether there should be different sized starting families or not. I don't really see any benefit to starting with fewer people - yes you eat less food but it takes longer to collect resources and build/upgrade defences and I don't think that offset balances.
This is one of the things I want to balance out during beta testing.
I like the idea of different family sizes. It just needs to be balanced properly. Maybe I can tie it to plots - the inside plots could have smaller families, the outside larger, or something.