There are indeed a few works that touch on this. In some older, more obscure horror literature, the idea of sex with monsters was sometimes used as a device to explore the darker side of human nature. It was also a way to create a sense of the truly alien and terrifying. But these works are not widely known or promoted due to their often disturbing nature.
There are some fictional stories that involve such themes in certain types of fantasy or horror literature. For example, in some ancient mythologies, there are tales that could be loosely interpreted in this way. However, these are often more about exploring the boundaries of the unknown and the strange, rather than explicit sexual content in a modern sense.
Yes, in some cult classic horror and fantasy books, you might find elements related to this. But they are not common in mainstream literature. One example could be some of the more esoteric works where the author is trying to create a sense of the truly otherworldly by introducing the idea of relationships with non - human entities. But it's important to note that these are often metaphorical or used for shock value.