Is it simply because they trigger memories of The Exorcist, The Shining, The Ring, etc.?* Or do those films themselves draw at some more ancient well of horror? Either way, if that had been a six-year-old boy I don't believe people would have been quite as freaked.
* None of which I've seen, by the way: just reading the Wiki entry for The Ring spooked me for days.
