I'll take a crack at this.
First of all, Fe demands that we keep the peace and not rock the boat. INFJ's are highly aware of how different we are, so we're naturally cautious to be different around others. It really has to be someone we trust. A lot. Someone who won't be too upset, and won't upset us with their response.
Now, like Von Hase said, it has to be really strong inside before coming out. Think of a hurricane in slow motion. When people want to look inside us, understand our emotions, they're asking us to open the door to the hurricane. They want to be able to pin down, take apart, and understand it. But it's too ethereal - it's here, no it's there, no it's all around me! They ask what size and shape it is, but that's always changing. I can tell how powerful it is, and generally what direction it's going, but you just can't nicely define a hurricane. You can't define or contain the INFJ emotions. I don't want to say "stop trying" - just please understand that it's very nebulous, but quite powerful. Some people can't withstand the force of it, others will want to confront it. We'd rather just keep it to ourselves.
INFJ's have a great lense on other people, like a telescope. But what happens when you look through the other end? Everything gets really distant and hard to examine. Thus, "I don't know" is an extremely valid, though unhelpful, response.