Sounds like some kind of loud spark or click. Might be a starter issue. If it was the battery, you wouldn't be able to eventually start it, unless you're jumping the battery every time after it sits.
I don't know where the starter is on the GS300, but if you can, next time when it acts like this, try to tap the starter lightly with a long tool that can reach it. Then try to start it again. That might free up the solenoid plunger that *might* be getting stuck and possibly causing that loud clicking noise. In any case, this would point to a starter issue. If that's the case then you can either get a new starter, or save some cash and get a rebuild kit which basically replaces the worn parts if you're able to do it yourself.
Aside from that, check your battery terminals for any loose wiring/connection. Check the wires running down to the starter to make sure they are attached to the starter tight (if they are bolted in or whatever). I am not sure if it's the sound quality or if it's really what the sound is when it's clicking, but that's the first time I've heard something that loud related to a starting issue.
"Rexus" - '98 GS 400 - Daily Driver - Alpine Silver - 249,000 miles - Megan Racing EZ Street Coil-overs - Muffler delete - LED inner tails - stock 17" chromes
"Rando Kuruza"- '91 Land Cruiser or The Zombie Apocalypse Getaway Vehicle - Work in Progress - Silver/Black - 230,000 miles - EMS Powered Exhaust - 31" Falken Rocky Mountain ATs - Lots of character