He had no recollection of his childhood and parents after the accident. After it, he took a lot of time to return normal. His grandparents alone took care of Sigma and had very little time to spend with him.
He didn't remember anybody being affectionate to him other than Olivia. But seeing her busy all the time, he used to stay alone and work hard not to make her worry.
When he got to know David sent Sky to the Barton family, he was happy for her. Because living with a single parent who was working on building a company won't be able to give as much love and care as a complete family could give her.
Even after knowing she was heiress of Harley, she didn't let it affect her relationship with Bartons. That showed how much she cared about relationships. Knowing she was even ready to leave everything to be with the Barton family, it showed she didn't care about social status or money.