The button in the battery compartment is NOT the reset button - it's a switch meant to be depressed by the NXT rechargeable battery. The NXT reset button is inside one of the technic connector holes on the underside of the brick - you can push it with an uncoiled paperclip or a pencil lead.
The reset button only toggles the power, it doesn't actually "reset" anything. The best way to reset the brick is to reinstall the firmware (this can be done from inside the Lego NXT-G programming software or from a 3rd-party utility like BricxCC). However, doing this requires that the brick has power and is able to turn on.
Try charging your batteries (if they're rechargeable) or put in new ones (if they're not). Your symptoms sound like the NXT was on the brink of the low-power threshold - putting load on the motors drained the batteries below a critical level and now the brick won't start.
Be sure to report back if fresh batteries don't work for you.