Session Signals Outpredict Model Size for PR Success

SWE-chat is the first large-scale public dataset of real coding-agent sessions sourced from open repositories. It contains thousands of sessions, tens of thousands of user prompts, and hundreds of thousands of agent tool calls, linked to code authorship metadata and commit traces. That linkage lets you answer operational questions you couldn’t before: did the agent’s output land in a merged PR; who edited which snippets and when; how often did sessions run tests or check logs. (arxiv.org) The dataset exposes simple, actionable signals. ‘‘Human edit rate’’ measures how much a person rewrote agent output. ‘‘Test/run frequency’’ captures active validation. ‘‘Clarification queries’’ indicate whether ambiguity was resolved before committing. ‘‘Decomposition prompts’’ show whether the user broke a task into subgoals. These features quantify human steering rather than pure model output quality. That matters because most benchmarks evaluate isolated model outputs and then extrapolate to productivity. SWE-chat lets us measure downstream integration — the thing that actually matters to engineering teams: survive-to-commit rates and the interaction patterns that predict them. (arxiv.org)

Community analyses (Potts + mmooritz) show a clear pattern: a handful of interaction-level signals strongly predict PR merge outcomes. Sessions where users repeatedly edit agent output, run tests frequently, ask clarifying questions, or explicitly decompose tasks are much more likely to produce a merged PR. This pattern persists when roughly controlling for task type and repository. (x.com) Operational deployments show the same qualitative effect. In the dotnet/runtime Copilot Coding Agent rollout, PRs that received human commits after the agent’s initial attempt had a merged success rate ~86%, versus ~55% without human commits. That delta is driven by human intervention, not by a different model checkpoint. The takeaway is direct: human edits and iterations are productive signals, not noise. (devblogs.microsoft.com) For predictive tasks like triage or risk estimation, interaction features are low-hanging fruit: interpretable, actionable, and often more informative than opaque model metadata such as name or parameter count.

Three engineering mechanisms explain why these interaction signals map to higher merge probability. Specification closure. Clarifying questions and explicit decomposition force the user to convert an under-specified request into a concrete contract. A concrete spec reduces subtle, review-blocking surprises and speeds reviewer acceptance. Verification frequency. Running tests and iterating creates a tight feedback loop. Tests convert correctness into an objective signal reviewers can inspect and surface brittle failures before the PR lands. Sessions with higher test/run frequency show fewer CI surprises and lower review friction. (arxiv.org) Incremental repair. A high human edit rate signals an efficient repair loop: the agent proposes, the human corrects, and the pair converges. Iterative co-creation is often faster than writing from scratch or trying to make the model perfect up-front. The dotnet/runtime numbers quantify this effect: human commits after the agent’s initial submission substantially improved merge outcomes. (devblogs.microsoft.com) These are causal, engineering-friendly levers. They point to concrete interventions rather than vague claims about model capability.

If you care about real engineering impact, instrument these interaction signals and bake them into tooling and evaluations. Log edit diffs, test-run events, clarification exchanges, and decomposition markers as first-class telemetry. These small changes yield outsized predictive power for downstream success. (arxiv.org) Benchmarks must move beyond single-shot code generation to session-level evaluation. Datasets like SWE-chat let you measure survive-to-commit rates and correlate them with UX choices. Treating the agent as a black box misses the largest lever: how humans steer it. Model improvements still matter — they expand autonomous capability — but the highest near-term ROI comes from human-in-the-loop UX: lightweight decomposition templates, faster in-editor test harnesses, and prompts that nudge users to disambiguate requirements. Make the human actions easier and easier to measure. That’s how you turn generative models into reliable engineering partners. (arxiv.org)