There's nothing much you can do to stop that guy from doing so.
You can't possibly send someone to beat him up, can you? What you
can do now is to have a good talk with your girlfriend, and tell
her how you feel about this situation, and that "uninvited"
intrusion makes you feel nervous and irritated, not to mention
insecurity. If she loves you and respects you, I'm sure both of you
can work out a solution to deter that guy.
What I perceive is that that guy must be a sweet talker, and he
somehow managed to get your girlfriend's attention even though she
claimed to be "irritated". Beware of this person. A good heart to
heart talk is definately a must, and this can be a test for the
love between you and her.
If she understands how you feel, by all means she will ignore that
man's pursuit, even to the extent of changing mobile number. On
your side, you must have the confidence in this relationship even
though you are feeling uneasy. Never be shaken. Jealousy sometimes
in a relationship is healthy, but don't extend to the point of
being childish and immatured.
Good luck.