Back when our blind little boy met our little Ela.
Ela had told him her narrative of how she had been separated from her family and how she had teleported extremely far away from the [Holy Kingdom]. And, from how conveniently her narrative unfolded from a perspective which was hers, Ken could tell that there might have been an internal conflict within the [Holy Kingdom] and Ela was just somebody who had been dragged into the ploys of the elderly and was thrown away after either fulfilling her need in their ploys, or was just outright abandoned.
And from his times as a mercenary, he had memorized the entire map of Miorus and he was sure that he hadn't ever learned of a kingdom named [Holy Kingdom], hence a possibility of something known to none opened up for him.