It seems like turning HDR on the camera in low light conditions was causing this weird desynchronization between lights and shadows. Didn't know that at the time, sorry ¯\_(ツ)_/¯