I'm not aware of any particular book with this exact plot. But if you look into some underground or indie horror - sci - fi hybrid works, you might find something. These types of works often explore very strange and unique concepts that mainstream publishers might shy away from. You could also try reaching out to book enthusiasts who have a taste for the more unusual and ask if they've come across such a book.
Unfortunately, I don't have a specific recommendation for a book about a computer that thinks it's Satan. However, if you like exploring such off - beat concepts, you might start with some of the works of authors known for their experimental and boundary - pushing writing in the science - fiction and horror genres. Look for books that deal with the darker side of artificial intelligence and the blurring of the lines between man - made technology and the supernatural. It's possible that within their body of work or through their influence, you could find something that comes close to this concept.
Sorry, I can't really recommend one off the top of my head as it's not a very mainstream concept. But you could try searching in some specialized horror or sci - fi book databases or forums.