Everyone felt there's more to it why the Heavenly Void backed away. They were one of the strongest lineages in Nine Heavens, how could they be afraid of those two others? The void had produced three monarchs and three slayers at the same time. With this powerful lineage, not even Disaster Sea King would belittle them.
The void had been waiting for the will to appear, and a few in the know was aware of this fact. Yet, they were not participating? Who or what might be able to hold them back? Remember, there's only one will in this generation. Those who wouldn't be lucky enough to win had to wait for another generation for it to appear.
Many lineages, which were hiding, felt something was off. They were here to participate, also. However, they wouldn't show themselves unless it's necessary. After all, they were still no match against the sea king and buddhist temple. Not to mention, they might earn the ire of these two behemoths.