Like many here (and elsewhere) have said...I believe more in shot placement than caliber. Now...I do fully understand that comment has its limits (AKA .22LR-vs-.44 Mag) but you get what I'm saying. I'd rather go up against a complete novice with a .40 S&W than an expert with a 9mm.
With that said 9mm is cheaper than .40...often up to 33% cheaper depending on where you get it. Cheaper ammo = more range time and more range time = better weapon handling, better weapon handling = better shot placement under stress (if need be).
The barrel swap is an ok idea but I have an OCD when ti comes to weapon consistency. Same gun at the range/same gun in the home/same gun during carry...same/same/same...including the caliber.
Contrary to popular belief, there is a difference between 9mm and .40 when it comes to report and recoil...darn near the same difference between .40 and .45 auto. Shooting 9mm in a gun at the range all of the time, you get used to that recoil. You learn to manage the gun under that exact recoil to control tactics like double tap. Even switching from a 115gr FMJ 9mm to a hot 124gr +P 9mm will change your shooting performance. Trust me, I've seen it with myself and others for years. heck changing gun manufacturers in one shoot, even if the same caliber (Glock-to-Sig-to-SA-to-etc) will mess with a shooter.
If you don't believe me...rent a 9mm XDm AND a .40 XDm...take them to the range and double tap with both...I guarantee your your results are different.
Long story short...if you practice with 9mm, defend yourself with 9mm. If .40 then stick with that as well.