Well, Doctor Strange outsmarts Dormammu in the comic. He uses his knowledge of the multiverse and time manipulation to create an endless cycle that Dormammu can't escape from. Eventually, Dormammu gets so frustrated that he agrees to Doctor Strange's terms and backs off.
Strange was able to defeat Dormammu in the comic by manipulating time with the Time Stone. He made it so that the same events kept repeating, which eventually wore down Dormammu's patience and forced him to back off. Strange's knowledge of magic and his willingness to take bold risks were key factors in this win.
Sure, Doctor Strange has been part of the Avengers in the comic universe. His magical expertise added a unique dimension to the team's capabilities and strategies.
I wouldn't say Doctor Strange comics are niche. They offer unique storylines and magical elements that attract a specific group of fans. But they haven't achieved the same level of mass popularity as the most well-known comic franchises.