Friendly lurker/TTCAL grad here... but I had to reply because I work in environmental risk assessment, and yes, a lot of toxins can affect reproduction. The 2 major types of toxicity that tend to be studied (besides acute poisoning reactions) are carcinogenicity (causes cancer) or teratogenicity (causes birth defects). They don't generally do studies to test effects on egg quality or fertility rates, and such studies would be especially tricky to interpret because it would be even harder to extrapolate from lab mice or other animals to humans. But some chemicals do have known health effects just from unfortunate accidental "experiments" (like women who have workplace exposures to X chemical had Y increase in miscarriage rates).
That's pretty general, but for the pollutants you mentioned: asbestos is a carcinogen, causes lung cancer and mesothelioma when in its airborne, breathable form. Roofing project fumes I assume would be mainly coal tar residues, which are also carcinogens, mainly for lung cancer. Those maybe aren't clear culprits. HOWEVER, benzene (which has LOTS of avenues of toxicity) IS associated with reproductive system effects and could quite conceivably have affected your fertility. It could also do so secondarily, via causing anemia.
CDC | Facts About Benzene
I'm glad you aren't working there anymore!!! And even after you've removed yourself from the exposures, it could still take some time for your body to process all the metabolites. Exercise, drinking lots of water, eating well and sleeping well will be the best way to support your body's natural removal mechanisms. And I would make sure you aren't anemic too, if you don't know that - if you are, you need to supplement your iron, B12, and folic acid.
Good luck and hope this helps - sorry it's so long!!!