Take the Brilliance Shrine as an example. Putting the pope and the four archbishops aside, the cardinals alone were at least Legendary powerhouses. There were also the Brilliance Knights who were responsible for guarding, and were led by a Sword Sage. Actually, there were nearly 100 Legendary powerhouses in the Brilliance Shrine. Such a force was not to be trifled with at all.
As for the Tower of Dusk, there were barely any such powerhouses apart from the Legendary-level President and his Legendary Undead servants, at least in the eyes of others. Even if the other mages already reached the peak of level-19, they would still be unable to pose any threat to a real Legendary powerhouse unless they took the last step.