It sounds like the motor is "winding up" from drivetrain deflection. Lego parts "give" a little when stressed and therefore store some energy like a spring. This is likely being caused by the high speed of the PF "M" motor. A picture of the assembly would be very useful in determining a solution. Without seeing it, the simplest fix would be to gear down the motor. Given the fairly high speed of the "M" motor output, I would suggest doing this with a worm gear assembly. A worm gear will greatly reduce your output ratio to the driving ring while also giving you more precise control that is less susceptible to back-tracking from deflection wind-up.
Here is a photo demonstrating how a worm gear can greatly reduce output ratio in very little space:
Another potential solution, if you have the part, is to affix a linear actuator to the "M" motor. It would have the same benefits as the standard worm gear:
One last thing... Which version of part# 6538 are you using? 6538a & 6538b have ridges that allow the drive ring to "click' into place. If you're using part# 6538c then swapping it with one of the others may be all you need to fix your problem.
A picture of your build would be appreciated to help reduce speculation. It would also be nice to see this just for the "neat-o" factor :)