From LLM to Agent Harness: Understanding Agents' Past, Present and Future (Part Two)

In the previous article we covered some LLM fundamentals:

• At its lowest level, an LLM does one thing: predict the next Token from the preceding context.
• Scaling Law explains why larger models often become more capable.
• Emergence explains why a machine that originally continued text can suddenly exhibit complex abilities such as coding, reasoning, translation and summarization.

One detail is easy to miss: a bare LLM is not born a chat assistant. It can continue text that looks like dialogue and may perform some tasks when prompted, but it has not yet been trained into an assistant that reliably understands user intent, roles and response boundaries. A crucial stage between a bare LLM and a product such as ChatGPT is post-training.

By a bare LLM I mean a Base Model or Pretrained Model in the training chain. Through large-scale text pre-training it learns to predict the next token, but has not yet been fully made into a chat assistant. Stanford's classic definition of a foundation model describes one trained on broad data, usually with large-scale self-supervision, that can be adapted to many downstream tasks. An LLM is a language-oriented kind of foundation model, not the whole category.

How does a brain that only continues text become a ChatGPT-like chatbot and then an acting Agent such as Claude Code or Codex?

I follow an engineering path: SFT in post-training teaches the model to imitate good human-written answers; RLHF helps it learn human preferences. A chatbot adds multi-turn dialogue outside the model, organizing continuous text as system, user and assistant roles. Prompt and Context Engineering help people control inputs, boundaries and context. Finally ReAct and tool calls allow the model not only to say an answer but to act, making the term Agent more meaningful.

From continuation to conversation: training ground, then theater

A larger model has stronger language, knowledge-compression and pattern-recognition capabilities. It does not automatically know what people want. Asked a question, it may continue in the style of a paper; asked for advice, it may produce correct but useless generalities; asked to explain a concept, it may be complete but unsuitable for a general reader.

The bare LLM knows language but not necessarily the situation; it generates but does not necessarily respond appropriately. How does it become an assistant that serves people? Send it to the training ground, then put it in a theater.

• Training ground: change model weights so it learns to follow instructions and answer in ways people prefer. This is post-training.
• Theater: without changing weights, build a runtime around it: who is system or user, when the assistant speaks, whether earlier conversation returns to the desk, and how results appear in a chat window. This places the Assistant Model into a user–assistant interaction.

First, the training ground.

Layer one: train the model to be an assistant

A scaffold: three stages

Training a useful LLM involves pre-training and post-training. Pre-training produces a Base Model: a “blank-slate genius” that has read widely but does not necessarily obey. Post-training comprises methods to make it more useful, controllable and assistant-like. More specifically:

1. Pre-training: learn continuation from internet text at the scale of trillions of tokens. It establishes language, common sense, reasoning, code and world knowledge—reading the world's books and learning how people tend to speak. Scaling Law and Emergence in part one describe this stage. Its output is generally called a Base Model.
2. Supervised Fine-Tuning (SFT): train on human-written question–ideal answer pairs to turn continuation into answering and make the model follow basic instructions.
3. Preference alignment: after it can answer, train it toward answers humans prefer: more helpful, honest, safe and appropriate. RLHF is a classic method. Rather than merely supplying a standard answer, it uses comparisons and feedback to teach which answers people are more likely to choose.

Both SFT and RLHF are post-training in the broad sense. SFT resembles demonstration—showing a student a standard answer. RLHF resembles shaping preference—judges say one attempt is better or worse without writing the answer.

Their order broadly matters: without pre-training there is insufficient language or knowledge; without SFT it is difficult to move from continuation to answering; without preference alignment an answer may not be helpful, restrained or safe.

This article focuses on SFT and RLHF, classic final steps from talking to serving. A future “part zero” may explore pre-training engineering—Transformer, tokenization, training objectives and loss functions.

A. SFT: training for the assistant role

SFT's purpose is simple: make the bare LLM say an appropriate thing where it is supposed to speak.

To show who is speaking, dialogue is organized with special tokens into roles:

• system: the director's framing, such as “you are a helpful assistant.”
• user: the line from the audience—the user's actual input.
• assistant: the position where AI speaks and generates a response.

The rules for arranging this input are the Chat Template. Exact formats vary by model, but these three roles are a useful start.

SFT trains the model to produce instruction-following responses in the assistant position using many instruction–ideal-answer pairs. When natural-language instruction and desired response are particularly emphasized, this is often called instruction tuning.

Think of it as assistant job training: a teacher uses many standard answers to show how to summarize, explain and organize responses. But SFT is not enough: many real questions have no single right answer.

For “I have been anxious about work; what should I do?”, a model might respond:

A: Offer a lot of psychological theory.
B: Say “don't think too much; relax.”
C: Acknowledge the feeling, help break down causes and suggest small practical actions.
D: Immediately advise resigning.

This is less a right-or-wrong question than a preference question. People may prefer C because it is helpful, measured and suited to the situation.

When several answers can work, which should the model prefer? That is the problem RLHF addresses.

B. RLHF: invite judges and let the model practice

For a preference question without a standard answer, RLHF uses another method: invite judges and let the model practice with their feedback. There are three steps.

1. Judges express preferences. The model generates 4–9 answers to the same question and human labelers rank them: “C is best, B next, A worst.” They compare rather than write the answers.

For example, for “Explain inflation in plain language,” A might offer a formal macroeconomic definition; B might say money buys less—noodles that cost 10 yuan may now cost 15. A labeler may prefer B. The chain is question → A versus B → human preference for B: preference data.

2. Compress the judges' taste into a scoring model. Humans cannot score forever. Ranking data trains a Reward Model (RM), whose job is to score answers rather than generate them. Given a question and answer, it produces a score. If humans prefer B to A, it learns score(B) > score(A). For example:

RewardModel(prompt, answer A) = 2.1
RewardModel(prompt, answer B) = 3.5
RewardModel(prompt, answer C) = 5.8
RewardModel(prompt, answer D) = 1.8

3. The model practices under the automated judge. The model generates an answer, the RM scores it, and the model adjusts to make high-scoring answers more likely and low-scoring ones less likely. Like a student practicing essays: teachers first rank samples, an automatic grader is built from those judgments, and the student repeatedly writes and adjusts to scores.

The classic InstructGPT route used PPO, a reinforcement-learning algorithm, for step three. Think of it as a grading rule: make high-reward responses more likely and low-reward ones less likely.

PPO was not invented specifically for language models. It is a policy-gradient method. A policy is the model's current way of playing: which words, tone and structure it tends to choose for an input. PPO gradually adjusts that policy from reward scores while avoiding an excessively large change at once.

There is a safety rope: the model should not learn recklessly just to score highly. Training commonly includes a penalty to keep it from moving too far from the original SFT model.

Otherwise, reward hacking can occur: the model discovers patterns that fool the scorer—excessive politeness, pandering or responsible-sounding but empty text—and optimizes the score instead of becoming better.

Thus the third step is not simply “maximize score,” but seek higher reward while not moving too far from the original model.

This reveals RLHF's boundary: it does not install a system of truth; it trains responses closer to human judges' preferences. Training guidance commonly favors helpfulness, honesty, safety and contextual fit.

In one sentence: RLHF does not teach the model what is true; it teaches responses more like those of an assistant people want to keep using.

C. First layer complete: Base Model → Assistant Model

A representative public example is OpenAI's March 2022 InstructGPT paper, “Training Language Models to Follow Instructions with Human Feedback”. It joins SFT, a reward model and PPO: first teach the model to answer like an assistant, then adjust using human preference.

A striking result: on the API prompt distribution tested in the paper, human evaluators preferred 1.3B-parameter InstructGPT outputs to those of the original 175B-parameter GPT-3.

In the classic InstructGPT pipeline SFT is often the first step of RLHF; conceptually, SFT is supervised fine-tuning within post-training, while RLHF is a preference-alignment method.

A bigger model does not automatically serve people. Obedience is not a byproduct of scale but a separate engineering problem.

After post-training, the Base Model starts becoming an assistant that follows instructions, understands user intent and responds closer to human preferences. Common names emphasize different aspects:

• Instruct Model: instruction-tuned to perform tasks.
• Chat Model: suited to a dialogue product.
• Assistant Model: its product role is assistant.
• Aligned Model: adjusted toward human preferences and safety boundaries.

These names often overlap in products: Instruct describes training; Chat and Assistant describe interaction and role; Aligned describes a round of adjustment toward human intent and safety.

Base Model
   ↓ Post-training (SFT + RLHF + later preference optimization)
Assistant / Instruct / Chat / Aligned Model

Post-training is the larger concept; alignment is the part concerned with bringing the model closer to human intent and preference.

We are not finished. This model has graduated from the training ground, but is not a complete product such as ChatGPT. It is still an input/output engine. It can learn to answer in the assistant position, but does not inherently have a chat window, history, interface or multi-turn state management.

A real chatbot needs the second layer: place the assistant model in a runtime structure.

That is the theater, which I will discuss in the next article.

• Lead author: Alchemist Zhou Lengqing
• AI assistance: Codex + GPT-5.5, Claude Code + Opus 4.8
• Assistance covered: organizing sources, discussing structure, fact-checking, polishing text and generating illustrations