Does ‘Do Differentiable Simulators Give Better Policy Gradients?’ Give Better Policy Gradients?

In policy-gradient reinforcement learning, differentiable models enable first-order gradient estimation, but nonsmooth dynamics can introduce bias. This work studies when estimator switching helps and shows that careful variance control can dominate in practical differentiable robotics tasks.

Paper