I would suggest seeing an RE and having them do all the testing you both first. See where his count is and then go from there. There are things that can be done to help increase his count and give you guys a better chance if possible. It really depends on where exactly his count is at.
As for me: We are a young couple (both 25) and tried for 3 years to get pg with my son. We tried clomid, femara, IUI with Follistim, a Fresh IVF cycle, and then the miracle FET cycle. This time around we are going straight back to IVF. I guess I am kind of at the point that I am done messing around with the little stuff cause it didnt work for us before.
Either way I think it is completely a personal decision. If you want to take the time to try the less invasive stuff then by all means I think you should give it a shot!