The entire area around the demon castle was dreary and desolate. There was not a single grain of vegetation growing around there.
The huge monolith of Demon Castle is supported by auxiliary buildings that form an impenetrable fortress that would even make things difficult for an Overlord realm expert.
About being attacked or invaded, there was no need to worry about that as any army marching over their lands would soon run out of supplies. The distance between the cities is vast and the terrain made it hard to walk around. One had to go through hell to step in here.
That's why even if a few cities and fortresses fall, Agnes was not worried about it as he could easily conquer them later but who would think that this human, a puny human would get the courage to belittle him and step into the Tiger Den alone without any shred of fear?