Which size backstrap do you use? I just switched to Small (#1) on my XDm 9mm and I think that's what I'll use going forward.
That has been my experience as well.I am more accurate with the largest strap. I think it tries to turn in my hand a little with the smaller ones.
Take it to the range and try them one at a time and see which you shoot best with.