The current problem was that Ye Xiao was too weak. He could only hinder the others in the fight. He totally couldn't make any positive contribution. What was most likely to happen was that somebody would have to stay around and protect him because he was too weak. That would not be a good thing for the future.
One more thing, with or without Shuang and Han, Ye Xiao's people were still not strong enough to destroy the three factions.
The three factions had dozens of thousand years of history. There were countless superior cultivators in the three sects. Every one of the three factions had tens of cultivators at the top of level nine of Dao Origin Stage.
That was after the war against Xiao Monarch and Fierce Blade. There would be much more if they didn't lose a lot of men in the fights.
Each of the three factions had thousands of Dao Origin Stage cultivators. After all, they were thoundands of years old sects. Ye Xiao and Li Wuliang were just two unaided individuals!