This seems to happen only when I am in gear with the A/C off and not all the time. The engine does not shake or feel rough but the RPM does seem to be low. What do you think? It almost looks like it is idling at about 450 RPM's.

 

If it happened automatically, it could be your IAC sensor, or MAF.
If it happened slowly and surly there is crap somewhere clogging the IAC or MAF, and also the throttle body.

 

I think this is very common with our cars.
Mine idles at the exact same place and I have replaced or cleaned the iac maf and trottle body.

My best guess is that it is ecu problem from the factory. The same problem that causes the throttle shock around 45-55mph. I believe toyota issueds a service bulletin on the problem awhile back and came out with new settings for our ecu.
If everything else is good and this is your problem, you can pull your ecu, and have someone from that auction site flash it to the corrected settings for under $200. I wouldnt worry about it though...

 

If it's idling at "D", that looks normal. Try cleaning MAF sensor and TB.

 

mine does the same when in D stop at red light and A/C off.