If you want to stick to your current design (despite its obvious flaws), then here's another idea. Let's go!

As you can see, I tried to rebuild the crucial part of your framework.

Alright. Now two things I've noticed. You're using axles with stop. And there's some play between the bush and the wheel.
Okay, firstly on axles with stop.

Here's two pairs of axles. The two axles with stop are on top, the bottom ones are without stop bit. See that the top ones don't touch each other? See the gap? That's because of the stop and the fact that the wheel has no counterbore. Can you spot the inner rim on the liftarm (blue arrow)? That's what the wheel lacks.
So what about it? Well, here's the problem. Let's take a yellow connector and make them one long axle. But, remember the gap between them?

Once you push them into the connector -all the way in till they 'hit the ground'- you will flex the red and yellow liftarms inwards. Maybe just a tad, but still. They will sure flex back, but because of the stop on the axle the wheel can't pop off. So the axle gets pulled out of the connector instead.
Next about the connector. You chose a connector with a pinhole. This piece is three studs long. So your wheel is now off by one stud. And I think you've therefore adjusted the wheel. That's why we can see this play.