The part of the Hyloh state that was annexed was quite large but far too deep. Despite that, it would be a risky move to ignore the soldiers camped in the dungeon town. Unless they left some garrison along the way to cause them trouble in case they decided to attack any of the other cities and towns under the enemies control, even if they do that, there would be any point since they would destroy the lands and kill the people that were being controlled.
In any case, Jean and the guild masters couldn't see the enemy's target. At that point in time, it was clear that they split up their forces. Billy had no answer to that question either. However, after thinking for a while in the enemy's place, he would target either Jean, Aura, or Rosalie. Those were the closest targets they could try to kill using an army.
"... We have to send a message to Aura to make her army return, she might be targeted," Jean said.