The emperor had been tolerating their careless words for a time now and decided to say that. Though the empire itself had an emperor he is not one who would worry about trivial things such as their emotional stability and so forth, those things are split between the high noble ministers after a basic discussion which will conclude once their points are solidified among each other.
Perhaps the emperor would allow them to decide among themselves that they had grown bold.
The emperor had found his rule is stable and he should dote more on his empress.
His two sons were old enough to shoulder some responsibility.
Why should he bother?