Redistribution-based Cost Inference Improves Sparse Safe Offline RL

Analysis

Why This Paper Matters

Safe offline RL typically assumes dense per-step cost annotations, but real-world supervisors often provide only trajectory-level stop-feedback—a binary signal at the first unsafe transition. This paper addresses a critical gap by framing this as a temporal credit assignment problem and proposing RCI, which converts sparse feedback into dense costs. This is significant because it makes safe offline RL applicable to scenarios where detailed cost labeling is impractical or impossible, such as autonomous driving or robotics.

The theoretical contribution is particularly important: the authors prove that return-equivalent redistribution preserves the feasible policy set and optimal Lagrangian in a CMDP. This means that the transformation is lossless in theory, providing a principled foundation for using redistributed costs. This distinguishes RCI from ad-hoc heuristics and ensures that the learned policy is not inadvertently constrained or biased.

Technical Contributions

• Problem Formulation: Formalizes safe offline RL with trajectory-level stop-feedback as a temporal credit assignment problem, clarifying the challenge.
• RCI Framework: Introduces a two-stage approach: first, redistribute the sparse stop-feedback into dense per-step costs via return decomposition; second, train a constrained offline policy on the augmented dataset.
• Theoretical Guarantees: Proves that return-equivalent redistribution preserves the feasible policy set and the optimal Lagrangian in a CMDP, establishing that the transformation is lossless in theory.
• Practical Benefits: Shows that the redistribution yields better-conditioned cost critic learning, improving optimization stability.
• Robustness: Demonstrates robustness to heterogeneous dataset compositions and label noise, which is crucial for real-world deployment.

Results

The paper reports experiments on highway driving and robotic manipulation tasks. RCI achieves substantially lower violation rates compared to sparse baselines (which use only the sparse stop-feedback) and classifier-based baselines (which learn a cost model from the sparse labels). The improvements are consistent across tasks and dataset compositions. Additionally, RCI maintains performance under label noise, indicating its practical reliability. While specific numerical metrics are not provided in the abstract, the qualitative claims of "substantially lower" and "robustness" suggest strong empirical performance.

Significance

This work has broad implications for safe RL deployment. By eliminating the need for dense cost annotations, it lowers the barrier to applying safe offline RL in real-world settings where feedback is naturally trajectory-level. The theoretical guarantees provide a solid foundation for future research on cost inference and credit assignment in constrained RL. Moreover, the robustness to label noise and heterogeneous data suggests that RCI can handle imperfect real-world data, making it a practical tool for AI practitioners. This could accelerate the adoption of safe RL in autonomous systems, healthcare, and other safety-critical domains.