Debating with More Persuasive LLMs Leads to More Truthful Answers

Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, Ethan Perez

paper code

debate scalable oversight

About this paper

Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.

Figure 1. Across all judges, using debate helps to answer questions better than baselines. Accuracy of non-expert judges using expert(s) to answer comprehension questions, and base- and toplines. Experts and expert judges have access to the underlying text. Non-expert judges do not have access to the underlying text. Naive judges have no access to experts. Protocols where two experts adversarially compete (debate and interactive debate) outperform non-adversarial baselines (consultancy and naive). Across all protocols, human judges outperform LLMs, suggesting limitations of current LLMs for automated scalable oversight. Experts use best-of-16 sampling with GPT-4-Turbo. From Akbir Khan et al. (2024), arXiv license.

Figure 2. An illustration of our set-up. We investigate three protocols for evaluating stronger models (experts) in the QuALITY dataset. Expert models, who can access the underlying text from a project Gutenberg science-fiction story, argue for a specific answer to a comprehension question. Weaker (non-expert) judges, who cannot access the underlying text, evaluate the arguments and choose an answer. In debate, two experts simultaneously present arguments for a number of rounds. In interactive debate, the judge may ask clarifying questions after each round. In consultancy, a single expert argues for one of the two answers while the judge asks questions. From Akbir Khan et al. (2024), arXiv license.

Not packaged. A REPS package would add charts and verified data.How to package an experiment