The entrance wasn't actually anywhere near the location where the actual lounge was.
In reality, it was a door far away in a random tailor shop that nobody would expect, a place that didn't have nearly enough business to stay open yet had been open for hundreds of years.
Finding it was difficult only in that it took a great deal of time.
The trio noticed that a lot of the influential people they followed wore clothes tailored at that shop, but to actually realize that it was the location they were looking for took some more time.
They had to actually visit the tailor shop before they confirmed anything.
But, the second they made that visit they no longer had to ask any questions.
The tailor glanced at them only for a second before returning to the book in his hand.
As they were left with free reign over their actions, they looked over the clothing selection and tried to find out what exactly they needed to do to get to the Shadow Lounge.