All parts of lilies are toxic for cats. Even just licking the pollen that might've gotten stuck on their fur or drinking the vase water can lead to fatal kidney failure.
If you have a cat, it would be a very good idea to at least inform your neighbors on how deadly lilies are (for cats). Otherwise you'll eventually end up with a dead cat.