what good reason is there not to ban them?
Because you should play as you would play a board game with friends. When I play cards and someone cannot follow the game because they're busy with their kids, then I just play without them. I don't kick them out of the room and shut the door.