Hope this helps dude.
My rear end used to be out of sync with the front of the car, felt like it was drifting at the back like you say. I particularly noticed when driving at 40-50mph, and if there was a ridge or bump in the road, then the back of the car would almost skip up and across, which didn't inspire confidence. Lots of understeer was present too.
Main culprit was the rear axle carrier bushes (which were split), which Lexus only sell as part of new rear hubs (a small fortune to replace). Luckily, you can obtain aftermarket bushes from Ozzi's Automotive in Atlanta, which is where I got mine ($48 for both sides). They are polyurethane versus factory original rubber, so ride will not be as pillow soft as new, but I did save myself the equivalent of $1,500!! The back end is much tighter now, but not perfect. Next to change are the rear upper control arms, in which the bushes are worn too. However, they only cost about $100 each.
If you search the forums for "ozzi", you should find his contact details.
Best to get a mechanic to check the components I have listed above. In fact, ask them to check the entire rear end.
How many miles has you car got?