Sharman's guards helped pave the way for the merchant convoy, they were all light cavalrymen. Each person in the convoy had a horse. Even though the servants were not riding on warhorses their speed was not any slower too.
When the bandits, who were observing from afar, realized that their target had charged out instead of guarding the town, they were in a dilemma.
The person who hired the bandits had asked them to entrap the merchant convoy in Telensu Town, but before the bandits could recruit sufficient manpower their target had left the town instead of defending it as thought. The bandits had no choice but to mount an offensive.