It's VERY normal for people to anticipate the recoil and pull the barrel down as they fire. Often they also pull it to the left. Have a buddy load between 0 and 3 snap caps randomly in your mags so you don't know where they are. Then see what happens when you get to them. If you are like the majority of people, the gun will be jerked down as you anticipate. The goal we all strive for is the gun not moving at all when we pull the trigger (except for actual recoil).
A good practice drill is to (with an EMPTY GUN) stand an empty casing on the front sight post, then dry fire -->pull the trigger. Any twitching will knock it off.
If you have a spare 30-50 bucks, you can get a cheap airsoft laser and dry fire practice while watching the dot to make sure it holds dead steady.
I own one like this. It's really good for a little $ This one is 20 bucks including shippping.
Low Profile Red Laser Sight Scope for XDM XD 9 40 45 | eBay