I have both. I took a few shots of both side by side (they are in the XDS section) and did a little review.
The XDM is not much bigger, but it's where it's bigger that counts. IMO CC'ing is about about carrying the smallest package - comfortably. I think it's going to vary from person to person on what they prefer.
For me, well, my XDM is for sale right now in the FS section.......
Nothing wrong with it (or them) at all, just MY personal preference is the smaller XDS. Round count doesn't mean much to me as shots need to count, not the ol' "spray and pray"
Worried about round count? Carry a spare mag