I used to think I was INFJ. It came down to Fe vs Fi for me.
I feel I relate to all of the infj description with the exception of Fe. I'm just not the warm and fuzzy type, despite feeling very passionate about people and ideals, I often keep it all under the surface. I'm very private with my emotions. I'd rather work things out for myself.
I can handle group disharmony and conflict (so long as it's not violent or abusive) it doesn't really bother me. I may see it as necessary in some instances to actually work things out and achieve greater harmony. I put much more stock into my own internal code than what the group agrees on.
Fi tends to get a bad wrap but I think that is because people don't understand it. It's inwardly focused. Fi is OK with going against the grain if their code dictates it. I'm INFP and empathic. I don't favor Fe yet still manage to absorb the emotions of others as well as have the ability to discern what is felt in someone without it being said.
I'm not an expert, but I think that we are not limited to using just four functions, however. I think we can use and develop them all.. it's just a matter of preference. So maybe narrow it down to what seems more comfortable or natural for you?