Maybe both are uneasy with each other.
I think no matter what, we can take the initiative by
putting ourselves at ease first. ie not to presume or
guess what the other party is feeling. As what you've said, it
could be a misconception from our side, so it's no point making the
situation more tense, as the other party may also felt our
"uneasiness", which in turn may make him behave likewise,
endless...
And if it is really something bothering him, then the more we
should calm down the situation by observing ourselves instead in
term of our speech or action, as we may have provoke his emotion
unknowing. If it can be solved, good, if not, I think we should
make our leave for the time being.