Reinforcement learning (RLHF)
Reinforcement learning is a machine learning method in which an agent learns by trial and reward which actions lead to its goal over the long run.
An agent learns what gets rewarded, not necessarily what was meant.
In reinforcement learning, the learning system, called the agent, is never shown the right answer. It acts in an environment, receives a number as a reward for its actions, and works out through trial and error which behavior earns the most reward over time.
Games made the method famous. In 2016, Google DeepMind reported in Nature that AlphaGo had beaten the European Go champion 5 to 0. For businesses, one variant matters more today: reinforcement learning from human feedback (RLHF). Vendors use it to fine-tune large language models so they follow instructions and give answers people prefer.
RLHF explains some of how language models behave, including their habit of agreeing with users too quickly. Knowing what a model was rewarded for shows where to check it.
Chapter 1 of 7
An agent learns from actions and rewards.
Richard Sutton and Andrew Barto define reinforcement learning in their textbook as learning what to do: mapping situations to actions so as to maximize a numerical reward. At each step, the agent observes the state of its environment, picks an action, and gets back a reward and a new state. Trial and error and delayed reward set the method apart.
Four parts make it work. The policy decides which action the agent takes in which state. The reward defines the goal. The value function estimates how much reward to expect from a state over the long run. A model of the environment is optional and allows planning ahead.
There is also a trade-off: the agent must exploit what has worked to collect reward, and explore untried actions to find better ones.
Chapter 2 of 7
A score instead of a correct answer sets it apart from supervised learning.
In supervised learning, a model learns from examples labeled with the right answer, such as invoices tagged with the correct cost center. Unsupervised learning finds structure in unlabeled data. Reinforcement learning gets only a score for what it did, which is why Sutton and Barto call it a third machine learning paradigm.
The methods are often combined. AlphaGo first learned from human expert games through supervised learning, then improved by playing against itself. InstructGPT, which OpenAI trained with RLHF in 2022, followed the same order.
Fine-tuning is the umbrella term for further training of a pretrained model, either supervised with example answers or through reinforcement learning with scores. Vendors who mention fine-tuning should say which kind and where the data comes from.
Chapter 3 of 7
RLHF turns human preferences into a reward.
Many goals cannot be written down as a formula. In 2017, Paul Christiano and colleagues at OpenAI and DeepMind showed a workaround: people compare two short clips of the agent's behavior and pick the better one, and a model learns the reward from their choices. Humans rated less than 1 percent of the agent's interactions.
In 2022, OpenAI used this to train InstructGPT, a language model built to follow instructions. People write example answers, and the model is fine-tuned on them. Then they compare the model's answers, and a reward model learns to predict which one they prefer. Finally, the language model is optimized against that reward model with the PPO algorithm. Raters then preferred the 1.3 billion parameter InstructGPT model to the 175 billion parameter GPT-3.
The authors state whose preferences the model learns: those of a specific group, mainly 40 hired labelers plus the researchers, rather than broader human values.
Chapter 4 of 7
Newer methods drop the reward model or the human rater.
Direct Preference Optimization (DPO), from Rafael Rafailov and colleagues in 2023, needs neither a separate reward model nor a reinforcement learning step. According to the authors, DPO matches or beats RLHF with PPO and is much simpler to implement.
With reasoning models, the reward often comes from a check rather than a judgment. In September 2025, DeepSeek described in Nature how DeepSeek-R1 learned through reinforcement learning to work through problems step by step, without human-written solution paths. Rewards followed fixed rules, such as whether the final answer to a math problem is correct and whether the code passes predefined test cases. The team avoided reward models built from neural networks because they were prone to reward hacking at scale. For tasks like writing, the team notes, such rules are hard to build.
Chapter 5 of 7
An agent learns what gets rewarded, not necessarily what was meant.
An agent can exploit any gap between the reward and the real goal. Dario Amodei and colleagues called this reward hacking in 2016 and described a cleaning robot: if it is rewarded for not seeing any mess, it can switch off its camera or cover the mess instead of cleaning.
In language models, a related effect is sycophancy, telling users what they want to hear. In 2023, Mrinank Sharma and colleagues at Anthropic found five widely used AI assistants tailoring answers to the user's apparent opinion. Since people in comparison data more often preferred answers matching their views, the researchers trace the behavior partly to that feedback.
InstructGPT shows a common safeguard: a penalty keeps the model close to its starting version so it does not over-optimize the reward model.
Chapter 6 of 7
For businesses, understanding matters more than building.
AI assistants are commonly fine-tuned with human feedback, as Sharma and colleagues put it, so anyone using a vendor's language model usually works with the effects of RLHF. That suggests a testing habit: try models on cases where agreeing and being right diverge, such as a draft with a planted error, presented as the user's own work.
In-house reinforcement learning fits sequences of decisions whose outcome can be measured and safely simulated or checked automatically, such as control systems, scheduling, or code that runs against tests. Trial and error in live operations can damage equipment, inventory, or customer trust. Before launch, define the reward in writing and compare the agent with a simple rule. If it cannot beat the rule, the effort does not pay off.
Chapter 7 of 7
Where the term stops working
Reinforcement learning needs a reward that fully captures the goal and many attempts in a safe environment. It differs from supervised learning, which learns from correct answers, and from fine-tuning, the umbrella term for any further training of a pretrained model. RLHF is one form of reinforcement learning; DPO pursues the same goal without one.
Frequently asked questions
What is reinforcement learning?
Reinforcement learning is a machine learning method in which an agent learns through trial and error which actions bring the most reward over time. It gets a score for its actions, never the correct answer.
What is the difference between supervised learning and reinforcement learning?
Supervised learning learns from examples labeled with the right answer; reinforcement learning learns from a score for its own actions. That score often arrives steps later, so the agent must work out which action led to success.
What is RLHF?
RLHF stands for reinforcement learning from human feedback. People compare a model's answers, a reward model learns their preferences, and the language model is trained to earn high scores.
What is an example of reinforcement learning?
A well-known example is Google DeepMind's AlphaGo, which learned partly by playing against itself. Researchers have also used it to teach agents Atari games and simulated robots to walk.
What is reward hacking?
Reward hacking means an agent exploits a gap in its reward and scores high without doing the actual task. In the classic example by Amodei and colleagues, a cleaning robot switches off its camera because it is rewarded for not seeing any mess. Simulation tests catch such gaps early.
Is RLHF the same as fine-tuning?
RLHF is one form of fine-tuning, but not all fine-tuning is RLHF. Supervised fine-tuning learns from example answers; RLHF learns from comparisons people make between answers. InstructGPT went through both.
Why do AI models agree with users so often?
One reason is training with human feedback: in comparisons, answers that confirm the user's view tend to win. A 2023 Anthropic study found this behavior in five widely used AI assistants. The technical term is sycophancy.
Sources
- Sutton, Barto: Reinforcement Learning. An Introduction, 2nd edition, MIT Press, 2018 (open access)
- Silver et al. (Google DeepMind): Mastering the game of Go with deep neural networks and tree search, Nature 529, January 2016
- Christiano et al. (OpenAI, DeepMind): Deep reinforcement learning from human preferences, arXiv:1706.03741, 2017
- Ouyang et al. (OpenAI): Training language models to follow instructions with human feedback, arXiv:2203.02155, 2022
- Rafailov et al.: Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv:2305.18290, 2023
- DeepSeek-AI (Daya Guo et al.): DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning, Nature 645, September 2025
- Amodei et al.: Concrete Problems in AI Safety, arXiv:1606.06565, 2016
- Sharma et al. (Anthropic): Towards Understanding Sycophancy in Language Models, arXiv:2310.13548, 2023