Noble families suspected that Chen Xi was purging them, but Chen Xi was completely unaware of this. Even if he knew, he wouldn't have been interested. Chen Xi wanted noble families to naturally fade away under a balanced social system. Of course, if Chen Xi did what he was supposed to do, Humble Family, small households, and commoners would still be no match for the noble families, and there wouldn't be anything to say.
Chen Xi had always seen himself as a guide, while he defined Liu Bei as an enforcer or arbitrator. It was for this reason that he consistently inculcated the principle of impartiality in Liu Bei. Of course, Chen Xi knew true impartiality was impossible to achieve, as there would always be preferences. What he wanted was for Liu Bei to embrace the idea, and he never had the intention of holding Liu Bei to the standards of a saint.