There are several things going on when you see someone looking at you, all of which happen very quickly.
This applies to actually seeing someone looking at you, not "sensing it" from behind or in the periphery.
Primates (including humans) are unique in the degree to which the eyeball can move around in the eye socket. This allows visual attention to be shifted quickly without physically moving the head.
Primates and certain other mammals can tell when another animal is looking at them, but humans are particularly good at doing this from a distance. In fact, humans have the added ability to be able to tell where someone is looking, even when it is not at them.
It is easy to see why this skill confers an evolutionary advantage: By being able to do this, you can essentially "read out" the location of another animal's attention. If you are a social animal, and the one looking at you is a superior, you'd better behave. Or if it is an inferior, you are being challenged and need to respond so you don't lose your place in the status hierarchy. For humans, knowing where another human is looking allows you to read their mind regarding what they are thinking about. This is invaluable when trying to learn language, since it allows you to pair particular words with particular objects in the environment. Pointing is also effective for this.
So, how do we do it?
Detecting the direction of gaze has to do with noticing the relative location of the dark spot of the eye (the pupil and iris) in the context of the whites of the eye. The differential size and location of the white region shows where the eye is pointed. And if the pupil is exactly in the middle with equal white regions on each side, then the eyes are looking at you. We can see this from across the room. Head direction also provides a cue, which is primarily determined by where the region of the two eyes and the nose are relative to oval face region, with hair as another reference marker. When the head is turned, the brain has to do some geometry to determine gaze direction from both head angle and relative eye angle.
Figure: Ratio of dark to light region of eye reveals direction of gaze. Bottom row: Location of facial features relative to head reveals head orientation. The visual system combines head orientation and eye orientation to calculate direction of gaze.
There is an additional effect that happens when "eyes meet". When you look at someone and they look back, you have the feeling that your gaze was met. This can feel uncomfortable, and the person who was "caught" often quickly looks away. This effect is caused by a feedback loop. The second person to make eye contact sees immediately that the first person is looking at them. The first person realizes they were "discovered" and responds often according to perceived relative status or confidence. There is also the mutual knowing that eyes met, which becomes a shared event establishing a transient relationship.
The meeting of gaze helps people recognize each other. You may think you recognize someone, but if they seem to think they recognize you too by not looking away, then the odds are greater that you are both correct. The visual systems of both individuals thus collaborate to establish mutual recognition. This happens quickly and subconsciously, allowing the social exchange to move forward toward acknowledging each other. If one person doesn't acknowledge back, it becomes an awkward case of mistaken identity.
Public speakers use the illusion of eye contact to create emotional intimacy with the audience. When people learn public speaking, they are told to glance around the room as they talk. This creates the illusion of intermittent eye contact with as many people in the room as possible, which allows the audience to feel that the speaker is talking to them personally, creating a feeling of intimacy with the speaker.
When TV newscasters deliver the news, they want the audience to have the impression they are talking to them. To accomplish this, they talk to the camera lens as if it was a person. In movies, actors avoid looking at the camera so that the audience never experiences mutual eye contact with them, preserving the feeling that the viewer is invisible. To look at the camera is called "breaking the fourth wall."