Only very few people knew of the early stages of Xander's life, and fewer knew about the complete story of his life.
Xander was the product of an affair with 2 married members of both the Karius and Ulrik families. When he was delivered none of the families wanted to accept the infant, not only was his birth disgraceful, but they were both rival families.
His parents had 2 options; raising him and getting kicked out of their respective families, or they found a way to dispose of him.
They could bear the shame and punishments that would surely come to them, but they couldn't abandon all the perks that came from being members of 2 of the top 3 families in the empire, so they easily chose the second option.
Since the only legal way to get rid of a child was when it was still in the womb, his parents decided to abandon him at an orphanage. Usually, this was the start of a good story, but things slowly got worse.