I dunno, I would if I were you I would take it as a compliment too!
I have 3 biological and a stepDD, people always ask if they are all mine and "oh geeze you have your hands full"... but really I don't! (and I find offensive... so I always say NOPE they're AWESOME kids!)
4 kids doesn't seem like "allot" to me, they're good kids and maybe because my father is 1 of 11 or my mom 1 of 5. DH and I do not have any together, we would like to have a babe together and he's even mentioned more than one. We'll see. I wouldn't have even considered having a baby with DH if "it was too much" to handle KWIM