Getting lost is also terrifying. You're in a strange place with no help in sight. Like when you're hunting and you stray too far from your marked path. You might run out of supplies and be completely alone in a vast wilderness. There's also the fear of the supernatural. Strange noises that can't be explained, like the sound of footsteps when there's no one around, can send shivers down your spine.
The unknown is the scariest part. In the wild, you don't know what could be out there. It could be a wild animal that's much more dangerous than you expect. For example, a cougar that might be stalking you without you knowing.
For me, the isolation in camping and hunting areas is really scary. You're away from civilization and any form of immediate help. If something bad happens, like getting injured or encountering a really aggressive animal, it could be a long time before anyone finds you. And the darkness at night just adds to the overall sense of fear, as it hides all the potential dangers.