Thank you all for your replies. I have learned a few things since starting this thread, of course. I need to do a whole thread about INFJ men!!
This guy, the one I loved turned out to have actual mental health issues. I didn't want to see it at the time, because I loved him. His love was always self destructive, because what he wanted from a partner was not something which actually exists. He had an image in his head of a perfect woman, who was an angel and a goddess and the Virgin Mary, but he couldn't find it. He liked to strike up relationships with facebook profiles who looked like what he was after, but in reality, he couldn't deal with human beings who get tired, angry, ill or who piss shit and get periods. In fact, anyone who gets close to him will end up getting the door slammed on them, because no one will ever be magical and sinless. I think a lot of INFJs are like that, myself included. We probably disappointed each other. I thought that he was somebody he is not, and he wanted me to be something which doesn't exist. Basically, he lives in a little world of his own, and he's happy, because they know him there.