I am sure that I read somewhere that the resolution of the motor sensor was 2 degrees, but I can find no mention of this in the NXT Help file. What's more, the motor control block allows the Duration to be set to an odd number of degrees. Can anyone please clarify this for me?
I found the resolution of the EV3 motors by attaching a 90° connection to the motor, and a long axle with a pointer at the end so I could see changes in small angles. By running a loop a large number of times with settings of 360°, then with 360.5°, then 360.75°, and watching for when the changes trigger actual accumulation of ending point, I found that there was no difference between a rotation of 360° and 360.75°, but at 361° the difference appeared.
I'll try the same setup with an NXT motor and let you know the results.
So it's like a step function with an interval of 1°.
I have building instructions and the program that I'll but on Rebrickable sometime this week.
Hope this helps.
I found a partial answer myself in the Mindstorm Education NXT User Guide, page 31.
It says the rotation sensor has an accuracy of +/- 1 degree. Not quite the same thing, but good enough to work with. If it turns out that I don't get the results I'm looking for, I will have to use some gear wheels.