I people-watch quite a bit. Whenever I am around people, actually. I believe it is not too difficult to pick up on relationships between people and nuances of intimacy within groups. I think picking up on things one relates to can be quite simple, as well, such as an introvert understanding another introvert's introspective tendencies. And I think I usually have an idea of how people are feeling, not that I always understand why they are feeling that way.
I think you have to be careful, though. Usually, when people-watching, you are observing your 'subjects' in a single environment. People behave differently in different environments. You're also limited because, as a spectator, you cannot significantly influence your 'subject' or the environment. You can only observe and make inferences based on what information is made available to you, and you have no way of knowing if you're making the right inferences. (e.g. There's a lot that affects a person's behavior that you'll never know about.)
Besides, I think you can 'know' a person for a really long time and never truly know them. It's all relative.