Yea the clunk I have is also intermitent, maybe 6 times in the past year, but never had the grind with it. Today it doesn't pull as much, but still a little. I checked the motor mounts by hitting the gas with my foot on the breaks, D and R, and it didn't jump so those are still good, for now. I do have a sort of sway when I hit the breaks, like the back of the car is floating left to right. I had an old '71 Impala convertable that did that after I snapped the frame on it doing a neutral drop at about 6 thousand RPM.
Eh, I was young. I gotta check the carrier arm bushings in the rear after this rain stops. After searching, I think that's deffinately the sway, all the symptoms are there.