After the stories about the expelled citizens spread in the territory, all the survivors in Black Haven realized that the Guild leader wasn't someone you could take advantage of. Even more, he wasn't as kind and nonchalant like how he usually acts.
Black Haven was neutral territory. It wasn't a place where only good or bad people gather. The rules it established for everyone was fair and reasonable, albeit siding with those with strength. Still, strong survivors were people who worked hard for themselves. In a sense, Black Haven promotes actively improving yourself.
The National Police left with most of its citizens but dozens from the Volunteer group changed their minds after getting expelled. They thought they could retain their rights to use Black Haven facilities even after they left but unfortunately, they weren't allowed to do so.