Self-Consistency Prompting: Majority Voting for LLMs
Self-consistency samples several reasoning paths from an LLM and keeps the most common answer. How it works, when it helps, and what it costs.
Self-consistency is a prompting technique that asks a language model the same question several times, lets each attempt reason independently, and then takes the most common final answer. Instead of trusting one chain of reasoning, you sample many and hold a vote. On tasks with a single correct answer — arithmetic, logic puzzles, multiple choice, classification — it reliably improves accuracy over a single response, at the cost of paying for every extra sample.
The problem it solves
Chain-of-thought prompting improves reasoning by having the model write out intermediate steps before answering. But a single chain is fragile. One wrong step early on — a misread number, a skipped case — carries through to a confident wrong answer, and nothing in the process catches it.
The key observation behind self-consistency is that a hard problem usually has many valid reasoning paths that lead to the same correct answer, while mistakes tend to scatter. If you sample ten different chains, the wrong ones rarely agree with each other, but the right ones converge. Agreement across independent attempts becomes a signal of correctness.
How self-consistency works
The procedure has three steps:
- Prompt for reasoning. Use a chain-of-thought prompt so each response includes its working and ends with a clearly marked final answer.
- Sample several responses. Run the same prompt N times with a nonzero temperature so the model explores different reasoning paths. With temperature at zero, you’d get the same chain every time and the vote would be meaningless.
- Aggregate by majority vote. Extract the final answer from each response, normalize it (strip units, whitespace, formatting), and return the most frequent one.
Conceptually, you are marginalizing over reasoning paths: you don’t care which explanation the model gave, only which answer the bulk of its plausible explanations support.
A minimal implementation looks like this:
from collections import Counter
def self_consistent_answer(prompt, n=7):
answers = []
for _ in range(n):
response = llm.generate(prompt, temperature=0.7)
answers.append(extract_final_answer(response))
answer, votes = Counter(answers).most_common(1)[0]
return answer, votes / n
The second return value — the fraction of samples that agreed — is useful on its own, as described below.
Choosing the sampling settings
Self-consistency depends on diversity. If every sample follows the same path, the vote adds nothing. The usual levers are temperature and nucleus sampling, explained in temperature, top-p, and top-k. Moderate settings work best: high enough that the model explores alternative approaches, low enough that individual chains stay coherent.
The number of samples is a trade-off. Accuracy gains are largest over the first handful of samples and flatten out after that. An odd number avoids ties in two-way splits. In practice, teams start small, measure accuracy on a held-out set of their own problems, and increase N only while it keeps paying off.
Using agreement as a confidence score
The vote share is a cheap, model-agnostic confidence estimate. If seven of seven samples agree, the answer is very likely right. If the votes split three, two, and two, the model is unsure, and that is worth knowing.
This makes self-consistency useful for routing:
- High agreement — return the answer automatically.
- Low agreement — escalate to a stronger model, retrieve more context, or send the item to a human reviewer.
Disagreement between samples is also one of the more practical signals for spotting LLM hallucinations. A fabricated fact tends to come out differently each time, while a well-grounded one stays stable across samples.
Where self-consistency works and where it doesn’t
It works best when answers are short and comparable: a number, a label, a choice from a list, a yes or no. Majority voting needs a way to say two answers are the same.
It struggles with open-ended output. Ten summaries of a document will all be worded differently, so there’s no exact majority. Variants handle this by asking a model to pick the response most consistent with the others, or by clustering answers by meaning before voting. At that point the technique starts to overlap with LLM-as-a-judge setups, where a model evaluates candidates rather than counting them.
It also can’t fix systematic errors. If the model consistently misunderstands a concept, every sample makes the same mistake and the vote confirms it. Self-consistency reduces random reasoning slips, not gaps in knowledge.
How it compares with related techniques
| Single chain-of-thought | Self-consistency | Tree of thought | |
|---|---|---|---|
| Reasoning paths | One | Several, sampled independently | Many, explored and pruned step by step |
| Selection method | None | Majority vote on final answers | Model evaluates partial steps |
| Model calls | One | N | Many, depends on search depth |
| Implementation | Prompt only | Simple loop plus answer parsing | Search controller and evaluator |
| Best for | Everyday tasks | Problems with one checkable answer | Planning and search-like problems |
Tree of thought is the more elaborate cousin: instead of finishing whole chains and voting, it branches at intermediate steps and prunes weak paths. Self-consistency is the simpler option and is often the first one worth trying.
It is also a form of test-time compute: spending more inference to get better answers without changing the model. Reasoning models that think at length internally capture some of the same benefit within a single call, but voting across independent samples still adds robustness on top.
Cost and latency
Every sample is a full generation, so N samples cost roughly N times the output tokens of a single answer. Input tokens repeat too, though providers that offer prompt caching can reduce that portion. Run the arithmetic for your own workload with the LLM token cost calculator before enabling it everywhere.
Latency is easier to manage: the samples are independent, so they can run in parallel and finish in about the time of the slowest single call. A common optimization is early stopping — once enough samples agree that the remaining ones can’t change the outcome, cancel them.
The takeaway
Self-consistency trades extra inference for reliability: sample several reasoning chains at moderate temperature, extract each final answer, and keep the majority. It is simple to implement, works with any model, and doubles as a confidence score. Use it where answers are short and checkable, watch the cost multiplier, and remember that it smooths out random mistakes but cannot correct ones the model makes every time.
Tagged
Keep reading
Chisato · · 4 min read Tree of Thought vs. Chain of Thought Prompting
Chain of thought asks an LLM to reason in a straight line; tree of thought lets it explore, evaluate, and backtrack across multiple branches.
Chisato · · 6 min read HyDE Explained: Hypothetical Document Embeddings for RAG
HyDE improves RAG retrieval by having an LLM write a fake answer first, then searching with that answer's embedding instead of the raw query.
Chisato · · 5 min read Prefill vs Decode: The Two Phases of LLM Inference
LLM inference runs in two phases: prefill processes the prompt in parallel, decode generates tokens one at a time. Why they behave so differently.