Some
people just highly subjected to 'environment feelings' - they are
easily disturbed by the aura people exude from their feelings and
it clings and lingers onto them easily (especially so if he's a
Pisces - next comes Cancer and Scorpio).
This is a clash of emotional management, rudely brought forth into
the private space of your relationship.
The theory of equilibrium will tell you that if you are subjected
to negativity and if your influence do not seek to overcome the
source of negativity, you 'lower' yourself to be on 'par' to this
very source, in which I reckon that your man neither has the life
force nor positive condition to take on such choking influences
from people around him.
It's only natural for wanting a quiet space when you spend some
quality time with your boyfriend. Your work is probably drudgery
and toilsome - with all that deadlines, distraction, noise,
unhappiness, anger, frustrations and all sort of unpleasant moods
and situation shrouding you like some malicious spirit, attempting
to break your spirit.
Does he knows about your frustrations at work? Or do you choose to
avoid sharing as you prefer to keep your time together as musically
sweet as possible?
From his POV he's likely to think along this line:
'I want to share with my
girlfriend some issues in life, but she seemed to be irritated with
the notion of me sharing with her. Yes, it's true, these are
people's problem, but I am somewhat implicated because these are my
love ones too. Sigh. Perhaps, next time, I should just keep
everything to myself.'
And of course, you are only but a human - you can only take so much
stress before you collapse emotionally.
These realities are realities of a love relationship in an urban
city; stress comes in several form of dimensions and they will seek
for the 'line of least resistance' - hoping to crash at the spot
where your vulnerability lies and crumble your love through
external means.
But does that mean that this is a sure fail? Not necessary.
Since you two are unable to control individual constrains in life,
communicate and decide the best way you could compromise and work
out something that you could keep the life of your relationship
flowing. (If you two share the same religion together, both of you
could seek for solace in your faith through prayers. Or speak with
your religious leaders if possible).
You see, what happens is that both of you are facing challenges in
life; it's just that each of you were hoping that your other half
could provide you the avenue to alleviate this frustration from
your state of life and the condition it brings. Therefore, how
would it be possible for anyone of you to afford the emotional
luxury to tender those sensitive needs when in your perception,
both your visions have been blotted out by all these immerse
problems, acting as a barrier to enjoy your relationship?
Sometimes when we are so overwhelmed by all these distractions in
life - we tend to lose focus and forget that the basis laws to
happiness in love is not about the tangibles - it's about being in
our real self and communicating likewise in this form. The next
time when he starts this again, cuts him off with a momentary kiss.
Or put a finger to his lips and embrace him.
There are always no definite answers to problems and frustrations
in life; but surely, there's strength in support. 
Cheers
