As I was driving today, I noticed that for my car to go in a straight line, the steering wheel had to be turned slightly to the left.
When I straighten the wheel out to what i see as straight, the car steers to the right.
Any way to adjust the steering wheel so it's straight?

 

If it is very slight then the issue is most likely either tires or alignment. When you have to keep the wheel slightly to the left, you are correcting for a very slight drift to the right. If the alignment is fine, then the cause is likely the tires (radial pull).

One quick solution may be to swap the left and the right wheels to inverse the pull and see what happens.

 

If it was a problem with the tires, wouldn't swapping the tires just cause the car to pull to the other side?

Also, the rear right tire was replaced about 500 miles ago while the other three already had ~8k on them (normal driving). Would that cause the car to pull left even though it's the rear tires?

 

Yes swapping tires left to right would cause car to pull left instead of right. But radial pull is very slight, and since most roads camber to right, you would less notice the pull if you reversed it.

Not sure about the right rear tire. It can cause some handling issues, but I'm not sure it would cause pull. Normally you should replace tires in pairs.

In Atlanta, I have taken my car to a place called Gran Turismo. They tend to be quite good at solving alignment and tire pull issues.

 

if letting go of the steering wheel and the car go straight without any pull, then an alignment shop can very easily adjust how the steering wheel points

 

You need to have your alignment redone every time your change your tire or after every 10k-15k miles. It is just part of normal car maintenance.