Inference-Time Policy Alignment for Fair Reinforcement Learning

When a reinforcement‑learning agent is released into the real world, its behavior is frozen by the reward function it was trained to maximize. Yet stakeholders often discover new concerns—such as equity, safety, or regulatory compliance—after deployment, forcing costly retraining or even abandonment of the model. This paper proposes a lightweight way to steer an already‑trained policy toward fairness objectives at inference time, borrowing the idea of inference‑time alignment used in large language models.

What You Need to Know

The authors frame fairness as a welfare‑based objective that can be expressed as an additional scalar term added to the original reward. Instead of retraining the policy network, they introduce a small, trainable alignment module that operates on the policy’s action distribution during inference. The module learns to reweight actions so that the expected cumulative welfare improves while keeping deviation from the original policy bounded.

Experiments are run on standard continuous‑control benchmarks (HalfCheetah, Walker2d) and a resource‑allocation environment. The alignment task (GridWorld with fairness constraints). A single alignment network, updated via a few gradient steps on a validation set that encodes stakeholder preferences, yields measurable gains in fairness metrics—such as reduced disparity in reward across sub‑populations—without a significant drop in the original task performance.

Why It Matters

Deployed RL systems often encounter shifting societal or regulatory expectations after they have been frozen. Retraining from scratch can be prohibitively expensive in terms of compute, data, and downtime, especially for large‑scale robotic or recommendation systems. An inference‑time adjustment offers a practical path to adapt behavior quickly and safely, akin to fine‑tuning a language model with a few prompt‑based examples.

Beyond fairness, the same mechanism could accommodate other stakeholder‑defined criteria—risk aversion, interpretability, or multi‑objective trade‑offs—by simply swapping the auxiliary objective. This flexibility makes the approach a promising tool for responsible AI deployment, where models must remain aligned with evolving human values without incurring the full cost of retraining.

Key Details

• Alignment module: a lightweight neural network (≈1 % of policy size) that takes the current state and outputs a multiplicative factor for each action logit.
• Training objective: minimize a Lagrangian that balances expected welfare gain against a KL‑divergence penalty to keep the adjusted policy close to the original.
• Data requirement: only a small validation set (few hundred trajectories) labeled with stakeholder welfare preferences is needed to learn the alignment weights.
• Performance impact: in HalfCheetah, welfare improved by 18 % with <2 % loss in original return; in GridWorld fairness disparity dropped from 0.34 to 0.09 while task success stayed above 95 %.
• Computation overhead: inference adds less than 1 ms per step on a GPU, making it suitable for real‑time control loops.
• Generalization: the same alignment module, trained on one environment, transferred to related tasks (e.g., different terrain in Walker2d) with minimal re‑tuning.

What’s Next

Future work could explore scaling the alignment approach to high‑dimensional visual policies, integrating multi‑criteria welfare functions, and providing formal guarantees on constraint satisfaction. Additionally, extending the method to online settings—where the alignment module updates continuously as new stakeholder feedback arrives—would make it a versatile tool for continually aligning deployed RL agents with evolving societal values.

📌 Source: Arxiv Ml