I"ll try to write out my story here sometime soon, but long story short DH has had sperm counts as low as 1.3million/ml w/ 36% motility...never higher than 12.8million/ml with 50% motility. Low morphlogy as well. And we've concived naturally three times (one m/c) and have 2 beautiful boys. So don't give up hope...I really believe the fact that it happened once is a great sign for you! We didn't even know we had fertility issues the first time we were TTC...it took about 6 months.
He did have varicocele surgery while we were TTC #2 but honestly I think his vitamin regimine may have helped more. He took a ton of vitamins but the biggies were 4 grams of L-arginine (I really think that improved the quality of his motility/speed) and 4 grams of L-carnitine and 1 gram Acetyl L-carnitine. Also Vitamin C and a good multivitamin. It was spendy but Vitacost.com is a little more reasonable. It took us 13 cycles to conceive our angel baby and another 11 to conceive our 3-month old baby...so 24 long cycle and 26 months but it was worth it.
I hope this gives you a little hope