Teach LLMs to Learn During Chat, Not Just Answer It

SML's core move is concrete: turn static problems into asymmetric, multi‑turn dialogues. During training the student sees the original prompt, produces an answer, and then a teacher—either a stronger model or a frozen peer with privileged information—replies with corrective feedback. Because the teacher's messages depend on hidden ground truth, the student must learn to ask, interpret, and incorporate those teacher utterances. The paper formalizes this as a multi‑turn MDP with sparse terminal rewards (correct vs incorrect). That lets RL methods assign credit over entire conversational trajectories rather than optimize single‑turn likelihoods, producing different in‑context dynamics than standard SFT. (https://www.researchgate.net/publication/400929964_Learning_to_Learn_from_Language_Feedback_with_Social_Meta-Learning)

DeepMind evaluates two practical pipelines. The offline SML pipeline generates dialogues, filters for successful trajectories, and supervised‑finetunes the student on the student turns. The online pipeline uses grouped‑policy rollout optimization (GRPO). For each prompt the model samples a group of conversational rollouts, computes trajectory‑level binary rewards, normalizes group rewards into advantages, and updates the policy toward rollouts that solve the dialogue. Implementation choices were pragmatic: GRPO group size 8, no KL regularization, conversational reward discount ≈0.7, and training rollouts capped at 4 turns. Evaluation includes 10‑turn test dialogues to probe generalization. (https://www.researchgate.net/publication/400929964_Learning_to_Learn_from_Language_Feedback_with_Social_Meta-Learning) Empirically, GRPO yields substantially larger improvements in using corrective feedback than the offline filtered SFT pipeline. The paper controls for compute by inflating group sizes on single‑turn RL baselines and still finds multi‑turn GRPO wins. In short: multi‑turn credit assignment, not just more samples, drives the gain. Training on 4‑turn rollouts generalizes to improved performance in 10‑turn evaluations.

Models that can use feedback still tend to guess when the task is underspecified. Q‑priming fixes this with a short SFT stage that injects clarifying questions into the dataset. Concretely, whenever the student’s reply would be incorrect, the dataset replaces that reply with a generated clarifying question derived from the teacher's private information. The injection probability decays across turns to encourage early exploration. (https://www.researchgate.net/publication/400929964_Learning_to_Learn_from_Language_Feedback_with_Social_Meta-Learning) On the Lost‑in‑Conversation benchmark—designed for under‑specified, sharded problems—Q‑primed models ask clarifying questions over five times more often and make far fewer premature answer attempts than baselines. The practical pattern is tidy: cheaply teach the habit with targeted SFT, then let RL consolidate the habit into a reward‑earning strategy.

The learned skill transfers. Training on math problems (Omni‑MATH) improved feedback‑utilization on coding tasks (LiveCodeBench). A stronger teacher at inference gives richer feedback but is unnecessary during training: peer‑level teachers suffice for the student to acquire the learning‑from‑feedback capability. That implies a modular workflow: you can add SML to a product pipeline without training giant teacher models. (https://www.researchgate.net/publication/400929964_Learning_to_Learn_from_Language_Feedback_with_Social_Meta-Learning) There are practical caveats. Experiments focus on verifiable domains with sparse binary rewards and static goals; open‑ended subjective tasks need richer reward models. GRPO still requires many conversational rollouts, so sample efficiency is nontrivial. And when a model can change how it solicits information, safety auditing becomes more complex. From an engineering perspective the takeaway is simple: change the training objective and data geometry and you can teach LLMs to treat conversational corrections as immediate supervision rather than background noise. That is the sort of reproducible, actually‑working progress that makes chat agents behave like collaborators instead of one‑shot autocompletes.