At one point, KB and PF were close. And honestly, we don't know what KB thinks about her family, her upbringing, her childhood. She might have shared details about her early life with PF that scared him. Whether true or false memories, this might be the reason he does not want his daughter to see his maternal GM. He might be thinking, "anything but that family".
As I have mentioned, it might have been a false memory, or a tiny episode that stuck in KB's head, and yet it might be scary for FP, and the sole reason for not allowing his daughter to see MGM.