Always try to use his own phrase/statement, to counter his
possible advancement.
I suspect you said leave him alone - which doesn't say much about
rejecting him directly. You know some people only give up hearing
the actual way of rejecting from the actual person involve.
He said he is not interfering me nor forcing to do things that you
don't want - make use of this statement. If he wanted advance,
appear aloof and remain distance - he get pissed off and show
temper - always remember never to react to his doings - he is doing
all that to get your attentions. The killer combo comes with the
ability to totally don't even give a sh|t about this temper and yet
still able to make merry with your other colleagues. 
If he has the face to confront you - use his own statement that
seemed approriate against him.
Key example: Am I your gf? *Boss has no stand or position,
therefore he can say nothing.* If he say says that he likes you and
dislike the way you treat him, etc, use that statement - he is
forcing/interfering with the way you are leading your life,
therefore result?
LEAVE YOU ALONE! 
