Control Planes, Not Models, Decide Agent Reliability

Agents are making irreversible changes in production. Databricks' Unity Catalog blog (May 19, 2026) lists incidents where agents wiped databases, deleted millions of rows, or dropped production tables. These failures were not hallucinations; they were caused by agents having carte blanche to call dangerous APIs. Technically the fault line is access models. MCP (Model Context Protocol) servers often expose many tools and default to granting broad access when an agent connects. That all‑or‑nothing model lets a single misconfiguration or crafted prompt escalate into catastrophic writes. The practical fix looks like infrastructure: enforce fine‑grained service policies on every tool call, record allow/deny reasons, and log full payloads to immutable, queryable audit tables. Databricks' Unity AI Gateway enforces SQL‑expressible policies and captures tool name, arguments, result, caller identity, and allow status. Policy enforcement plus payload logs turn opaque agent actions into something you can test, trace, and fix.

If tools are the hardware of agent behavior, skills are the software interface that packages intent. NVIDIA's verified skills (May 19, 2026) treat a skill as a portable instruction set plus a machine‑readable skill card that lists ownership, dependencies, limitations, and verification status. Skills are scanned for agent‑specific risks (prompt injection, excessive agency, tool poisoning) and cryptographically signed so operators can verify authenticity after download. Ad‑hoc prompts and opaque skill bundles are the primary attack surface. A malicious or buggy skill can request excessive privileges, hide dangerous scripts, or mutate prompts at load time to evade runtime checks. Pushing provenance, constraints, and automated scanning into the publication pipeline — and pairing those with signatures and skill cards — makes skills reviewable, composable, and rejectable by operators. Operationally, signed skills enable safer deployment patterns: continuous catalog syncing, daily risk scans, a human+automated review flow, and machine‑readable metadata that agents can check at load time. The payoff is smaller blast radius, cleaner audits, and predictable capability composition — the enterprise lever that matters more than raw model improvements.

Traditional benchmarks (MMLU, HumanEval) measure language competence. Agent reliability requires a different axis: can the system execute a multistep workflow in a nondeterministic environment? NVIDIA's 'Mastering Agentic Techniques' (May 19, 2026) argues that evaluation must instrument entire trajectories — plans, tool calls, intermediate reasoning, and side effects — and compute task success rate (TSR), tool‑call accuracy, and trajectory efficiency. That changes tests and telemetry. Define an intent (for example: 'update this record through this API within two tool calls') and a success predicate. Instrumentation must log every call with stable IDs, argument schemas, responses, and latency. From these traces compute: did the agent pick the correct tool, respect schemas, how many retries were needed, and tokens and wall‑clock seconds per successful task. Those signals reveal failure modes that token‑level benchmarks miss: hallucinated schemas, infinite planning loops, permission errors, and silent data corruption. Make evaluation part of CI: label trajectories, compute budgets (e.g., '95% of tasks under N tokens and M calls'), and gate deployments on TSR and audit coverage. That's the difference between a flashy demo and an agent you can onboard into regulated workflows.

Vendor roadmaps and conference reporting (Gradient Flow, Google's I/O, May 19–20, 2026) show the market moving beyond models toward managed agent runtimes and orchestration. The real competitive advantage will be operational teams that can retrofit agents into messy legacy stacks: CRMs, ERPs, homegrown APIs, and fragmented knowledge sources. That work is plumbing: capture Slack and Confluence into queryable sources, map identity to roles, define choreography for human approvals, and build monitoring with replayable traces. Vendors can ship primitives — signed skills, MCP governance, managed runtimes — but deployments succeed when organizations create roles: an agent owner who understands workflows, a security owner who writes service policies, and an evaluation loop that treats trajectories as first‑class telemetry. Integration converts capability into throughput and reduces operational risk. Treat agents as distributed systems with a control plane: build signed capabilities, enforce policy at each tool call, log everything to a queryable store, and evaluate whole trajectories. That stack separates impressive demos from automata you trust to do real work.