Well, if it makes any difference, anthropologists have long recognized that among most primate species (of which we are one), direct eye contact is usually considered an "invitation" to combat, especially with males. And not surprisingly in prison "culture", it's the same thing. So even if we "crave" it, recognize that it is instinctively "invasive", and needs to be approached with care, or at least in small "doses" 'til you know them better.
And BTW, it's just these little social "courtesies" and unspoken cultural "rules" (or the lack thereof) that often marks the difference in such "socially-challenged" behavioral conditions as Autism and Asperger's... they just don't "get" these kinds of things.
But all that said, when you've both reached a proper level of "trust", I'd agree there's no better form of "communication" between people.