The Tower of the Magi wasn't exactly a tower, but it was called that because it was the best nomenclature they could find to describe a place with limited space and floors to go up and down. In reality, the Wizard's Tower or Olaynore, as the locals called it, was shaped more like a funnel than a tower.
The Forest of the Forgotten, the second floor, or the last depending on your point of view, was the most decayed and also the largest of all. This place of dense rainforest was so large that it could easily be compared to a small Midgard kingdom, which is why it had so many different tribes and villages, and also why it was so difficult to find the Library of the Magi.