They're lookin' at your butt.
No, are you near a wall or a clock or something? Maybe they aren't staring directly at you just around you, and when you look at them they see that you are in their field of vision, and they stop staring because they don't want you to think that they are staring at you. WHOORUNONFTW
I stare at stuff all the time, and people in my field of vision give me the most awful looks, but I'm not staring at them.