Xavier understood that Leila needed people around her that she could trust.
While Leila had been travelling to the outside world for a long time, she did not have her people. Nirvana village was her real home; she did not need other people when she could have everything from Nirvana.
The people in Nirvana maintained the principle of not interfering with the real world. This was a requirement for the deal they made when they were all forced to retire and become part of the mundane world.
Leila was the only connection they had in the real world.
It was a burden for them to have endless knowledge and ideas but only one person knew them. This was the reason everyone in the village was obsessed with teaching Leila. She was someone no one in the village thought could exist.