It was just like how the Watcher had a taboo against colonizing the Holy Land Planet.
The lost empires in the Milky Way more or less had their own taboos, and violators had to be prepared to receive punishment from them.
It was the same for the Silent. As one of the Lost Empires with the least communications with the Milky Way, almost everything about them was unknown to the younger civilizations.
They didn't issue missions like the Observer, nor did they improve the native civilizations like the Watcher and the Voyager. They seemed to be forever immersed in their own world, paying no attention to the outside world.
However, even such a lost empire had something that annoyed them, and that was a violation of their borders.
The Silent paid special attention to their territory and would never tolerate any border contact.