"We have to leave."
The person that made this statement wasn't the Prince, rather, it was one of the other four Peak Elemental Venerables.
They didn't feel confident in being able to keep Grey at bay, so they believed the best thing to do was to leave the woods and return to the castle. If possible, they didn't mind leaving the secret realm entirely.
Grey's figure appeared close to them, and without hesitation, he attacked. His fire domain spread out, covering a large area, he didn't use the normal flames, rather, he used his icy flames to slow them down.
The icy flames had the ability to slow down the thought process of anyone caught in it, so it was very effective against such a group. Especially given their current mental state.
Grey took advantage of this and attacked immediately he used his icy domain.