A model trained only to predict the next word on internet text is not an assistant. Ask it a question and it might continue with more questions, produce something offensive, or confidently make things up. The 2022 paper Training language models to follow instructions with human feedback — known as InstructGPT — described how OpenAI turned GPT-3 into a model that follows instructions, using a technique called reinforcement learning from human feedback (RLHF). The same recipe underpinned ChatGPT, released later that year.

The problem: misalignment

Pre-trained language models are optimized to imitate text, not to be helpful, honest or harmless. The authors framed this as an alignment problem: the training objective (“predict the next token”) differs from what users want (“help me with this task, safely”).

The three-step method

Step 1: supervised fine-tuning (SFT)

Human labelers wrote high-quality demonstrations of desired behavior in response to prompts — many of them submitted by customers of OpenAI’s API. GPT-3 was fine-tuned on these examples.

Step 2: train a reward model

For a new set of prompts, the model produced several answers, and labelers ranked them from best to worst. Comparing answers is easier and more consistent for humans than writing perfect ones. A separate model — the reward model — was trained to predict which answer humans would prefer, producing a score for any response.

Step 3: optimize with reinforcement learning

The fine-tuned model was then trained with reinforcement learning, using the PPO algorithm, to produce answers the reward model scores highly. A penalty kept the model from drifting too far from its supervised version, to avoid gaming the reward model with strange outputs.

The idea of learning from human preference comparisons builds on earlier research, including a 2017 paper by Paul Christiano and colleagues on deep reinforcement learning from human preferences.

The headline result

Labelers preferred outputs from the 1.3-billion-parameter InstructGPT model over those of the 175-billion-parameter GPT-3, despite the model being over 100 times smaller. Alignment training made a bigger difference to perceived quality than raw size.

The paper also reported:

  • More truthful answers on the TruthfulQA benchmark and fewer made-up facts in closed-domain tasks;
  • Somewhat less toxic output when prompted to be respectful;
  • Small regressions on some standard NLP benchmarks — an “alignment tax” — which the authors reduced by mixing in pre-training data during reinforcement learning.

Limitations the authors acknowledged

Whose preferences? The model was aligned to the preferences of a specific group of labelers, guided by instructions written by researchers, and to the kinds of prompts API customers sent. That is not the same as being aligned with “human values” in general.

Still makes mistakes. InstructGPT could still produce false statements, follow harmful instructions and hedge excessively on simple questions.

Reward hacking risk. Optimizing hard against a learned reward model can exploit its weaknesses — producing answers that look good to the reward model without being better.

What happened next

RLHF became the standard final stage of training chat models across the industry. Researchers have since proposed many variations:

  • Direct Preference Optimization (DPO) (2023) skips the separate reward model and reinforcement-learning loop, training directly on preference pairs with a simpler objective.
  • AI feedback methods use models to generate or judge some of the preference data, reducing the need for human labels.
  • Reinforcement learning with verifiable rewards trains reasoning models on tasks such as math and code, where correctness can be checked automatically.

Why it matters

InstructGPT showed that how a model is trained after pre-training can matter as much as how big it is. It marked the shift from language models as text-completion engines to language models as assistants — and put questions about whose preferences shape AI behavior at the center of the field.

Frequently asked questions

What is a reward model?

A model trained to predict which of several responses humans would prefer. It gives any response a score, which can then be used as the “reward” signal for reinforcement learning, so the language model learns to produce answers people tend to rate highly.

Did InstructGPT make the model smarter?

Not primarily. It changed how the model used its existing knowledge: following instructions, staying on task and avoiding certain harmful outputs. That is why a much smaller aligned model could be preferred over a larger unaligned one.

What is the “alignment tax”?

The small drop in performance on some standard benchmarks that can come with alignment training. The InstructGPT authors reduced it by mixing pre-training data into the reinforcement learning phase, so the model kept more of its original capabilities.

Is RLHF still used today?

Preference-based training remains central to how chat models are built, but the methods have diversified. Many teams now use simpler preference objectives such as DPO, AI-generated feedback, and reinforcement learning on tasks with automatically verifiable answers.