That's the thing. I have friends asking me to hang out on certain dates, events that are upcoming within these two weeks, and I haven't replied them. I know I should give a definite answer, but I have conflicting desires - part of me wants to hang out with them (know I should probably hang out with them) but part of me just wants to stay where I am now and not see people. It's easy for me to schedule one or two engagements with friends per month and not want more than that. For me, without tact, my personal reasons would be, "I don't want to hang out right now or make the effort to socialize." Which is a little too brutal for me to say, so I tend to delay my replies in hopes that it will pass or perhaps maybe I would feel like emerging from my shell that week and reply with a yes instead of a "No, I don't want to see you."
So, why don't I say no? Partly because I want to, yet I also don't want to. It's often a case of "I don't feel like socializing, but I probably should," and "should" is usually not the best motivation to meet people. Partly because I hate rejecting people, so I just don't. I don't reply; I don't do anything about it.
Again, this is from my perspective. But if your INFJ really wants to go (if he's made a decision and puts his mind to it), you can be sure that's what he'll do. If he's hemming and hawing, then maybe you should consider he may not be all that enthusiastic about it. Or just ask him if there's another event he'd like to attend or if he just wants to chill at a cafe with a cup of coffee, etc. I don't know. But if you've asked twice, and he still hasn't given a reply...