Ohh yikes. Sounds like a tough situation! I, like Heather, only have control issues with MIL and FIL. But I also have a very laid back DH, like you. He doesn't want to cause ripples in the water with his family. We have talked about this soo many times.
I know now that when you have problems with the in-laws it is particularly hard on the DH. With my DH anyway, he explained it like this:
He doesn't want to be caught in the middle. Even if you can't stand his family, they are still his family and he loves them. You can't ask him to abandon his family when they are all he knew growing up. And if you ask him to choose either you or them it will cause problems in your marriage (resentment, etc.). At least that is how it was in our situation. Your situation may be different and your DH's opinion might be completely different.
I would honestly just sit down with your husband and have a candid conversation about it with him. Tell him how his family makes you feel, tell him that you don't really want your child in that environment, and compromise on HOW many visits you will make/when you will visit.
That's the only advice I have. But I am no marriage expert!!
Good luck with your situation!!! ((((HUGS))))