Self-Consistency Preference Optimization

Analysis

Why This Paper Matters

Large language models (LLMs) often require extensive human-annotated data or external reward models to improve reasoning. Self-Consistency Preference Optimization (SCPO) challenges this dependency by showing that a model can bootstrap its own improvement using only the consistency of its outputs. This is particularly valuable for complex reasoning tasks where gold labels are scarce or expensive to obtain. By turning self-consistency—a technique previously used only at inference—into a training signal, SCPO opens a new avenue for unsupervised self-alignment.

How It Works

SCPO starts with a base model and a small set of unlabeled seed queries. At each iteration, the model generates new problems via few-shot prompting, then samples multiple responses per query. The key innovation is using a vote function that counts the frequency of each final answer across responses. The most frequent answer becomes the "chosen" response, and the least frequent becomes the "rejected" response, provided the chosen vote exceeds a threshold τ.

To build preference pairs, SCPO selects the most consistent response as the chosen one and the least consistent as the rejected one, as shown below:

The loss function is a weighted version of DPO, where the weight w(x) is proportional to the vote margin between chosen and rejected responses. This prioritizes high-confidence pairs and downweights ambiguous ones.

Training proceeds iteratively: M0 generates data for M1, M1 generates data for M2, and so on. This self-reinforcing loop is similar to Self-Rewarding LM but uses consistency instead of a separate judge model.

Results

On GSM8K and MATH, SCPO consistently outperforms unsupervised baselines and achieves accuracy comparable to supervised IRPO with gold labels. Semi-supervised SCPO, which combines gold labels with self-generated data, further improves performance.

On ZebraLogic, a challenging logic puzzle benchmark, SCPO achieves state-of-the-art results for 8B-scale models, surpassing Llama-3 Instruct 70B and Claude-3 Haiku in puzzle accuracy.

Ablation studies confirm that the weighted loss is crucial: removing the weight reduces accuracy by 1-2.5% across datasets. Additionally, models become more consistent with each iteration, as measured by vote share of the most consistent response.

Significance

SCPO demonstrates that self-consistency can serve as a reliable proxy for answer correctness, enabling unsupervised training that rivals supervised methods. This reduces the need for expensive human annotations and external reward models, making it easier to scale reasoning improvements. The iterative nature also allows models to continuously self-improve, potentially leading to better generalization. However, the method's reliance on consistent outputs may limit its effectiveness on tasks where multiple correct answers exist or where the model's diversity is low. Overall, SCPO provides a practical and principled approach to self-alignment that could be integrated into broader LLM training pipelines.