I currently have:
17 Fame
6 Trust Medals
1 Fun Medal
I do not donate and started almost a year to the day after Silverfire family.
I can have
4 characters active, 3 noble, 1 adventurer.
So you need to get one or two more medals I reckon, as seen from both my experience and Sacha's. Once I get more fame, I will likely be able to have 4 nobles active. That shouldn't be a problem for you though Silver
