One great mysterious science fiction book is 'Annihilation' by Jeff VanderMeer. It takes readers into a strange, otherworldly area known as Area X, filled with mysteries and things that seem to defy the laws of nature.
A classic in this category is 'Frankenstein' by Mary Shelley. While it's often considered a horror novel, it's also a work of science fiction. The mystery of creating life from non - living matter and the consequences that follow are still thought - provoking today. It's a story that explores the boundaries of science and morality.
I highly recommend 'Solaris' by Stanisław Lem. The planet Solaris is a huge enigma. The ocean on it behaves in ways that are completely baffling, and the story delves deep into the psychological impact of encountering such a mysterious and alien world. It makes you question what we know about intelligence and the nature of reality.