I have had the same issue.
Have you tried block feeding at all? That has helped for me. Also, if I am super engorged then it is really painful when Tatum latches on and she also pulls away b/c the flow is too fast. What has helped me is manually expressing some milk into a nursing pad before I latch her on. It only takes a couple of minutes and it makes a big difference. Eventually your body will regulate the supply and block feeding should help.
If none of those methods work, you can try cabbage leaves. I think there is some medicine that lowers supply too. Google over supply and check out
KellyMom : keeping you informed about breastfeeding and parenting for some good info.
Hope it gets better soon!