The new setup was having a few issues with the Bluetooth signal strength, I guess being behind all that metal doesn't help. So I cracked it open to see what could be done!
This is the Bluetooth module, and I found the built in antenna.
I sanded back the antenna trace and soldered a 30cm wire to it. This has greatly improved the range, I can now have the phone a couple meters away from the car. I also don't have any more stutter issues!