Well Neon, it does look like you're working on a fair bit of
conjecture here. He may not actually be gay, and may just be, shall
we say, genteel. Even if he was gay, what's to say that he's
interested in a relationship with you?
The simplest thing to do is to lay your cards on the table: "Look,
you're a good bloke and all, but I have to be honest and say that
I'm no boxer, and I don't fancy being punched in the ring", or
something to that effect. 
If he's cool with that, well and good. If he's not, then you've got
the grounds to say that his intentions are making you really
uncomfortable, and because of that you would no longer wish to
maintain contact with him.
