We have come to the conclusion that we will always have debt (if it's not my student loans, it will be a morgage payment or something), and so our goal is to get it mostly paid off before having a baby. But we also have other things we would like to do before we have a baby (and most of those things we plan to do in the next 6-12 months).
IMHO, you and your DH need to really decide if its time or not. If you think about it the way we have, it makes it a little easier to deal with.
My DH and I compromised in our plan. Originally we were going to wait 3-4 years after our wedding to start ttc, but we have both discovered we want to start a family long before then.
Hope that helps you somehow. I know that since we set a date to start trying, I haven't been saying "I want a baby" quite so often.
Christina