Well, in some comics, the invincible characters might die if it serves a greater narrative purpose or to shock the readers. But in others, they might always remain unbeatable. It really varies.
The possibility of an invincible character dying in comics is there. It could be due to a powerful foe, a self-sacrifice, or a change in the story's direction. However, it's not a given and often comes as a surprise when it happens.