Usually, when Victor went back in time, he had a good amount of time before the main events that led to his death took place. However, this time, every time he opened his eyes, there was the leader of the Night Guardians, standing on stage giving a speech.
Victor had tried to get out onto the balcony, he had tried to fight, he had tried to talk and he had tried to let everything happen as the man who could stop time wished. Nothing changed, one or more people always ended up dying, including him.
Besides, his strength and sword skills were no use against the disfigured man who was currently his greatest obstacle.