Yep...like already said sooss well, it almost sounds like you have a take down pin detent spring installed instead of the proper selector detent spring.
The proper spring is heavier and longer and is a snug fit into both the pistol grip and the hole in the lower. In this location, a heavier spring is needed.
We still need more info. Did you do your own build, or did some one else or did you buy the M4 complete from the factory? Sometimes kits are incomplete.
And...never start a paragraph with a conjunction. No, wait...sometimes the wrong springs are in the build kits. What you need is the proper spring.
Also... do you have a new selector? (safety) The tiny detent can be either silver or brass colored and must correctly fit into the notches on the safety.
Sounds like either the problem is the wrong spring....OR the detent does NOT fit snugly and correctly into the appropriate knotches on the safety.
Finally again, and this may be related to your problem, when squeezed on "SAFE", the trigger must have very little or no creep. HB of CJ (old coot)