I get alot of that as well from people who do know. And for us, "relaxing" is probably not going to work, b/c DH's count is going to be low whether he is relaxed or not.
From people who don't know our situation, I am VERY tired of THE question. And people always say it teasingly, like, "You'd better hurry up and have another one.. *hehe*". Which makes me want to scream b/c:
a)BFing was the only form of birth control we ever used and we started trying when I weaned at 10 months old....it's not like we waited too long to start trying for #2 or something.
b)How, exactly are we supposed to hurry this up? Considering we've been trying for a year already, and my DH has a very low sperm count. Not sure what we are supposed to do to "hurry this up"
c) Gee, do you think it hasn't occured to us to try to have another baby? Do you think we are just too dumb to figure out on our own that our child is getting older, and that it's about time to have another one? Good grief.
Sorry for the vent

. Most of my friends and aquaintances are so fertile that seriously, I don't think infertility even registers on their radar screen. And since we have a child already, they are even less likely to consider the possibility that maybe, just maybe, we are heartbroken that we aren't pregnant yet...and their words just make everything harder.
Even my friends who have dealt with infertility aren't always that sensitive about the subject. One friend (who doesn't know our situation) tried for over a year for her first before conceiving on Clomid and then had her 2nd when her first child was only 12 1/2 months old. And now she talks about babies like they are something you can just order up on Amazon and have delivered to your front door 9 months later. Sometimes I want to remind her how painful infertility was...but it's like now she's so busy with her 2 kids that she can't even remember.
One thing that this experience has taught me is to always be sensitive to others...you just never know what someone is going through.