The Reaper had detected the timeline shift the moment Leo ambushed Vann and successfully killed him.
Since it happened just within 15 minutes after the Azzy's battle with Vann and timeline shift can only be done by both of the time travelers, the Reaper wondered whether it is due to Barbara or Azzy.
If it's the latter, then, the reason for the timeline shift is the duel that's just happened. Thinking that it's unlikely, the Reaper concluded that Barbara might be the reason.
However, he didn't know the reason. Too bad that he had already given the control back to Azzy.
Back then, Azzy was a little displeased with how the Reaper spoiled the match with one move. Respecting his host's feelings, the Reaper never tried to take control of Azzy's body even while he's sleeping.
But, now that Leo's secret is out and the worse that he tried to prevent is going to happen, the Reaper left the choice to Azzy.