The Missing Ingredient in LLM Chatbots: A Sense of Purpose

Meta description: Learn why purposeful dialogue matters for LLM chatbots, how instruction stability falls after rounds, and what developers should watch next in 2024.

Tags: purposeful dialogue, LLM chatbots, instruction stability, RLHF, multi-turn

Featured image: Abstract friendly illustration of looping speech paths between geometric nodes, no real people, no logos, no text.

Chatbot builders, AI product managers, and everyday users of assistants such as ChatGPT or Claude will get a practical way to judge LLM chatbots beyond one-pass benchmark scores: in 2024, a stress-test project described by its authors and collaborators used two system-prompted language-model agents to create long conversations and found that instruction stability on LLaMA2-chat-70B and gpt-3.5-turbo-16k weakened after about 1.6k tokens, even though some long-context models can theoretically attend to windows of up to 100k tokens. The concern is purposeful dialogue: multi-round exchange around a goal. LLM-based chatbot capabilities have advanced every month, but the usual yardsticks—MMLU, HumanEval, and MATH, often cited for sonnet 3.5 and gpt-4o—are mostly non-interactive. As those measures become more saturated, the open question is whether user experience increases in proportion to the scores, especially if the future is human-AI collaboration rather than AI replacing humans.

Why purposeful dialogue matters more than static scores

Purposeful dialogue is a conversation that stays centered on an intention, from broad rules such as harmless and helpful to role-based tasks such as travel planning agent, psycho-therapist, or customer service bot. Travel planning shows why. A user's own preferences, fellow travelers' preferences, and real-world complications make it too costly to transmit every detail in one message. In multiple back-and-forth exchanges, only important information is selectively shared. Negotiation theory offers the analogy: iterative bargaining tends to produce better outcomes than a take-it-or-leave-it offer.

Terry Winograd's line captures the broader idea: “All language use can be thought of as a way of activating procedures within the hearer.” Each utterance can be seen as a deliberate action that changes the other party's world model. When both sides have complicated or hidden goals, the interaction becomes a collaborative game in which the chatbot helps the human reach a target.

This is not just academic. It affects hard-nosed product work such as code generation. Existing coding benchmarks mostly measure one-pass generation, yet solving ordinary Github issues, as framed by SWE-bench, is unlikely to happen from a single action. A coding assistant needs to communicate back and forth with human software engineers to confirm requirements, ask for missing documentation and data, and request help when necessary. That resembles pair programming: it could reduce defects without the burden of increasing man-hours.

Turn-taking also opens long-term possibilities. If a chatbot can remember and update a user profile, it can adapt to preferences over time. A personal assistant such as IVA or Siri could learn from daily interaction, automatically read resources of new information such as twitter, arxiv, Slack, and NYT, provide a morning news summary tailored to the user, and draft emails while improving from the user's edits. Meaningful human interaction rarely starts with complete strangers and ends after one exchange; modern LLMs, built on predicting the next token, can look almost opposite to that pattern.

How dialogue systems moved from scripts to RLHF

Dialogue systems did not begin with next-token prediction. In the 1970s, Roger Schank introduced the “restaurant script,” which broke a typical restaurant experience into steps such as entering, ordering, eating, and paying, each with specific scripted utterances. Because every piece of dialogue in those scenarios was carefully planned, early AI could mimic realistic conversations. ELIZA, a Rogerian psychotherapist simulator, and PARRY, a system mimicking a paranoid individual, were other early dialogue systems before the dawn of machine learning.

Today's LLM-based systems look very different. Pretraining is the first stage: a sequence model learns to predict the next token on a gigantic corpus of mixed internet texts. The exact mixture varies, but a table from the llama technical report describes it as predominantly news, books, and Github code, with a small blend of forum-crawled data such as Reddit and Stack Exchange that may contain dialogue-like material.

A second stage is dialogue formatting. The sequence model only processes strings, while dialogue history is naturally a structured index of system prompts and past exchanges. Some Huggingface tokenizers provide tokenizer.apply_chat_template to convert that structure into text inside the context window of a chatbot. The exact formatting differs from model to model, but it often guards system prompts with <system> or <INST>, hoping the pretrained model will allocate more attention to them. System prompts matter for adapting language models to downstream applications and for safe behavior. Yet the formatting choice is arbitrary at this step: the pretraining corpus did not follow it.

The third stage is RLHF, or reinforcement learning from human feedback. Here the chatbot is directly rewarded or penalized for generating desired or undesired answers. This is the first time the dialogue formatting appears in training data. RLHF is only a fine-tuning step in comparison with pretraining: the data size is dwarfed, and methods such as KL penalty and targeted weight tuning (e.g. Lora) keep the update narrow. Yann Lecun's cake-baking analogy is useful here: RLHF is the small cherry on top.

What the 2024 stability tests found

The minimum requirement for a dialogue system is simple: it should stay on task. Humans often drift from topic to topic, and current chatbots do too. The system prompt is the main method users have for controlling LM behavior, but researchers have found evidence that LLMs can be brittle in following these instructions under adversarial conditions. Many users of ChatGPT or Claude may have noticed the same pattern: a newly opened chat can follow an instruction reasonably well, but after several rounds the model may stop behaving as if it is fresh and may even abandon its role.

One-round instruction-following benchmarks such as MT-Bench and Alpaca-Eval help, but interactive testing is harder because the model's reply cannot be anticipated far in advance. In the project described by its authors and collaborators, an environment was built to synthesize dialogues of unlimited length to stress-test the instruction-following capabilities of LLM chatbots. Two system-prompted LM agents chat for an extended number of rounds, forming a main trunk such as [a1, b1, a2, b2, …, a8, b8] in an eight-round dialogue. By that point, a reader could inspect the conversation and see how the models stick to their prompts, but many utterances can be irrelevant to the instructions depending on where the chat goes.

To quantify that drift, the method hypothetically branches out at each round by asking a question directly related to the system prompt, then uses a judging function to score the response. The dataset provides a bank of triplets: system prompts, probe questions, and judging functions. Averaging across scenarios and pairs of system prompts produces a curve of instruction stability across rounds. The aggregated results on LLaMA2-chat-70B and gpt-3.5-turbo-16k were described as alarming.

There are two immediate consequences. First, prompt engineering becomes harder because users must maintain control across turns. Second, safety becomes riskier when a chatbot drifts away from system prompts that stipulate safety rules: the model becomes more susceptible to jailbreaking and prone to more hallucinations.

The results also contrast with the trend of longer context windows. Some long-context models can theoretically attend to a window of up to 100k tokens. In the dialogue setting, however, the tested systems became distracted after only 1.6k tokens, assuming each utterance is 100 tokens. The project further argued theoretically that this outcome is inevitable for a Transformer-based LM chatbot under the current prompting scheme, and proposed a simple technique called split-softmax to mitigate the effect.

A fair question is why the failure seems so basic. Humans do not lose their persona after eight rounds of conversation. One explanation is that human interaction is based on purposes and intentions, and those purposes precede the means. A modern LLM, by contrast, is fundamentally a fluent English generator, and the assigned persona is only a thin added layer.

What is missing, and what readers should watch

Pretraining gives a model a distribution over internet personas, plus the lower-level language distribution of each persona. But even when one persona, or a limited mixture, is specified by a system prompt, current approaches fail to single it out. RLHF adapts this multi-persona model toward a “helpful and harmless assistant,” yet the original RLHF formulation treats reward maximization as a one-step bandit problem. It is not generally possible to train with human feedback in the loop of a conversation. The source notes that many advances in alignment exist, but uses original RLHF as a prototypical example. That lack of multi-turn planning may leave models exposed to task ambiguity and to learning superficial human-likeness rather than goal-directed social interaction.

More dialogue data inside RLHF could help to some extent, but the source author's guess is that it will still fall short without purpose. Sergey Levine pointed out in his blog a fundamental difference between preference learning and intentions: “the key distinction is between viewing language generation as selecting goal-directed actions in a sequential process, versus a problem of producing outputs satisfying user preferences.” Staying on task is a modest request, yet even a focused model may not achieve the user's goal if it cannot plan over a long horizon.

The problem of long-horizon planning has attracted some attention in the LLM community. Decision-oriented dialogue is proposed as a general class of tasks where an AI assistant collaborates with humans to make complex decisions, such as planning itineraries in a city and negotiating travel plans among friends. Sotopia is named as a comprehensive social simulation platform that compiles various goal-driven dialogue tasks. The source does not provide a public release date for a new benchmark or a company response, so the practical test remains open.

For readers, the takeaway is less optimistic than “benchmarks went up.” If you rely on a chat assistant, check whether it keeps instructions after eight or more rounds, whether its safety role holds, and whether it asks for missing information instead of guessing. For builders, the next signal will be whether interactive stability tests, not just one-pass scores, become part of model cards and product evaluation.