Damien's training period lasted for over a month, and though it was a short time in the grand scheme of things, it was quite a frustrating predicament for his enemies.
After all, Hidden Death Valley wasn't friendly to conflict outside of established duels. While anything could happen within them, external battles would be heavily punished regardless of one's status.
Hidden Death Valley was a land ruled by multiple Demigods, and through their strength and influence, had no need to bow before any outside influence. This was half the reason why it gained its current reputation.
Damien's enemies could only bite their nails and wait until he entered the battle arena. When that time finally came, they could get rid of him.
As for Damien himself, he paid no mind to these raging voices, despite being fully aware of their existence. In fact, he made particularly sure to flaunt his daily routine to rile these people up and fuel their anger.