Kinda
tricky, but I would do my best to allow you to have a decent
picture to analysis your own situations.
In any relationship, the yin and yang dynamic of it would be giving
and receiving. To give is often certain degree of sacrifice, while
to receive is merely to receive the goodness of what your
partner/relationship has to offer. Regardless of the type of
principles involved, an unbendable rule (e.g. against pre-marital
sex) clashing against one who desire what you are against, jammed
up this system and causes continual conflicts.
When your boyfriend's clarity returned, he probably would feel that
'he would try to respect your wishes.' This is a form of sacrifice
on his part and would term as 'giving' in it's own meaning.

***
I am kinda flabbergasted to learn that he said this
to you:
He says that next time if he strays, it is my
fault.
This goes beyond the theory of giving and receiving - it's
intimating and demanding. And a complete lame excuse, stated in
advance, to 'protect' his own self-absorbed interest when this
actual occasion arises. He had previous sexual activities with his
'sexy' ex-gf back then and because they played him, he prefer
someone more down to earth like yourself currently. I don't know if
you have already made known to him that you are conversative by
nature, but because you demonstrate the kinda of quality he is
looking for NOW: what comes together, comes in
a package.
He wants to settle down with a "proper" girl and
"gentle" girl.
You want
a proper girl, yet you don't want the 'proper' girl to be proper.
(I am not saying people who indulge in pre-marital sex are not
proper, rather, it's a similar expression of saying a man who
demanded a virgin wife, yet wants her to be a sex goddess). Kinda
wtf.
So what if the relationship is four years? So what if you are
already in your late twenties? Basically, all these figures don't
really matter much without your personal 'love investment'
consideration for this relationship. You are with a bf who
introduces a third party into the relationship, causing serious
upheaval, JUST because he wants to change your mindset about
pre-marital sex. Let me ask you: would things be very different if
you were to change your mindset back then?
What will change? In fact, nothing much.
It's no longer about the sex issue - it has bring forth new
factors that has almost eliminated the pre-marital sex problem in
your relationship: The empathy, trust, foundation and
everything else in the relationship have shook violently, leaving a
trail of smoky dust, bearing pure evidences of how weak your
relationship is.
Let me tell you, ANY one girl would have the ability to destroy
your relationship. Even if they don't, you wouldn't have that power
to endure affairs-of-your-man for long.
You need more than just communication - its almost like rules
stating. You present your personal proposal on what you expect in a
BGR, against what he sees in one. Learn to suck-it-up and
compromise or die in the process. If certain critical clauses are
left in a stalemate, congrats him on his future relationship and
bid him farewell.
P.S: If you two are not looking in the same direction, its kinda
difficult to substain your relationship ultimately, even if the
relationship continues to run. Same direction as in having good
knowledge of how your OWN relationship works and work around it. If
you have to end a relationship, dragging it served as a burden for
everyone involved. That may not be a product of love - more like
insecurity and fear to move on.
You got to know the difference. 
Cheers