The Kaf HaChaim quotes the Kabbalists who say that since Yom Kippur is such a holy day, the holiness that permeates the world stifles the strength of the evil spirits. Thus, even though we do not wash our hands, we are still not harmed. In a similar vein, Tisha BeAv is the day when national calamities occurred (and continue to occur) e.g. the destruction of both Temples, the exile from Spain in 1492, the beginning of WW1 etc. The evil spirit finds great contentment on Tisha BeAv and relies on the sins of Israel to bring heavenly retribution, without its own intervention.(Based on Kaf HaChaim 4.14)