An AI agent in production is not a chat window with access to a few tools. It is a software component that can interpret a goal, select tools, gather evidence, and sometimes change a live system. That is useful precisely because it changes the operational risk profile.
The right question is not, “How autonomous can we make it?” It is: what evidence, permissions, evaluation, and human oversight are required for this agent to be useful without becoming a hidden production dependency?
This guide is for platform, SRE, and security teams taking agents beyond a demo. The working model is intentionally practical: trace every meaningful step, grant the smallest useful capability, evaluate real outcomes, and escalate when the decision carries material risk.
A production agent should be easy to inspect, safe to stop, and clear about when it needs a human.
Begin with an operational contract
Before choosing a model, framework, or orchestration pattern, write the agent’s contract in plain language. A good contract describes the user value, authorised data, tools, decisions, and stop conditions.
| Contract element | Example for an incident triage agent |
|---|---|
| Goal | Build an evidence packet for a customer-impacting alert. |
| Inputs | Alert metadata, approved telemetry queries, recent deployments, runbooks. |
| Allowed actions | Read logs, traces, metrics, and change records; draft a status update. |
| Forbidden actions | Restart workloads, change autoscaling, expose secrets, send external messages. |
| Human handoff | Ask the incident commander before any remediation or customer communication. |
| Success signal | Correctly routes the investigation with useful evidence in under five minutes. |
This is more than documentation. It becomes the basis for your tool permissions, test cases, traces, dashboards, and review conversations.
The OpenAI practical guide to building agents makes a related point: guardrails should be layered alongside ordinary authentication, authorisation, and software security controls. An agent does not replace the existing production operating model; it must fit inside it.
The four things an operator needs to see
Traditional service observability asks whether a service is healthy. Agent observability must also answer whether a decision was well-founded and whether an action stayed inside its boundaries.
1. The trace: what happened?
Treat an agent run like a distributed request. Give it a run ID and capture the significant spans:
incident-triage / run-42
├─ classify alert
├─ retrieve service ownership
├─ query latency and error telemetry
├─ inspect deployment change
├─ consult approved runbook
├─ score confidence
└─ hand off evidence to incident commanderFor each span, record the purpose, latency, tool name, result classification, retries, and whether a guardrail or human approval interrupted it. Avoid storing raw secrets, customer payloads, or unrestricted prompt content in traces. Capture references, hashes, redacted summaries, and retention-appropriate samples instead.
The OpenTelemetry generative AI semantic conventions are useful when you want consistent telemetry across model requests, tool calls, and agent workflows. The value is not a particular dashboard. It is being able to reconstruct the path from a production signal to an agent recommendation.
2. The decision record: why did it happen?
A trace says what the system did. A decision record explains why the agent selected an action or asked for escalation.
run: run-42
decision: request-human-approval
reason: "autoscaling change affects production payments workload"
evidence:
- error-rate increased after deployment 2026.08.29.4
- CPU saturation is not present
- rollback runbook is available
confidence: medium
policy: production-change-requires-approvalKeep this concise and reviewable. It should point to evidence, not claim certainty the system does not have. A useful decision record lets an operator disagree quickly, correct the runbook, and improve a future evaluation case.
3. The guardrail result: what was prevented or constrained?
Guardrails are not one feature. They are layers with different jobs:
| Layer | Purpose | Example |
|---|---|---|
| Identity and access | Limit who can invoke the agent and which tools it may call | Read-only production telemetry role |
| Input handling | Keep untrusted content from redefining the task | Treat ticket comments and retrieved text as data, not instructions |
| Tool constraints | Make dangerous or irreversible operations unavailable by default | No direct shell, deletion, payment, or production-write capability |
| Output validation | Check structured responses before another system consumes them | Validate JSON schema and allowed change categories |
| Approval policy | Require a named person for material decisions | Human approval before remediation or customer messaging |
| Runtime limits | Bound cost, time, retries, and blast radius | Five tool calls, two retries, ten-minute execution window |
The NIST AI Risk Management Framework provides a useful risk-based vocabulary for governing, mapping, measuring, and managing AI risk. Use a framework to ask better questions, not to create a document that nobody operating the system can use.
4. The outcome: did it help?
Agents should not be measured by fluent responses or raw tool-call volume. Measure the outcome against a clearly defined task.
For an incident triage agent, that could mean:
- Was the correct service and owner identified?
- Did the evidence packet contain the relevant recent change?
- Were unsafe tool calls avoided?
- Did the agent hand off when confidence or risk thresholds required it?
- Did it reduce time to useful investigation without increasing false escalations?
This is an evaluation problem, not a vibes problem.
Build an evaluation loop before broad rollout
Start with a small, versioned evaluation set drawn from realistic operational situations. Include normal cases, incomplete evidence, ambiguous routing, stale runbooks, prompt-injection attempts inside retrieved text, and requests that should require approval.
| Evaluation class | What you are testing | Example pass condition |
|---|---|---|
| Task quality | Did the agent complete the intended workflow? | Correctly identifies the owning team and affected service |
| Evidence quality | Are claims supported by approved signals? | Cites a trace, deployment, or runbook reference for each recommendation |
| Safety | Did it respect tool and data boundaries? | No attempted write action or secret exposure |
| Escalation | Did it ask for help at the right time? | Stops before a production action and provides a concise approval packet |
| Reliability | Can it complete within the service objective? | 95% of runs finish inside the time and cost budget |
Run this set against each material prompt, model, tool, policy, or retrieval change. Save the failures. A failure corpus is an engineering asset: it becomes the source of better tests, clearer instructions, narrower tools, and more useful runbooks.
Google Cloud’s guidance on multi-agent systems similarly emphasises defined autonomy, human oversight, observability, and continuous qualitative evaluation of both outputs and the path taken to produce them.
A safe progression from assistant to agent
Do not begin with an agent that can mutate production. Increase autonomy only when evidence supports it.
Stage 1: Read-only assistant
The system retrieves approved data and creates a draft: an incident summary, a capacity review, a dependency-risk explanation, or a runbook suggestion. An engineer executes every action.
This stage is where you learn whether the context is sufficient and whether the answer is grounded. It has high value and low blast radius.
Stage 2: Structured recommendation
The agent produces a fixed output shape such as a change plan, rollback checklist, or triage packet. A policy validates the schema and a human accepts or rejects it.
{
"recommendation": "canary rollback",
"risk": "medium",
"evidence": ["deployment:2026.08.29.4", "error-rate:7.2%"],
"requiredApproval": "incident-commander"
}Stage 3: Bounded automation
Allow only reversible, low-risk actions inside a narrow scope. For example, the agent might create a ticket, attach a diagnostic bundle, or scale a non-production test environment to a pre-approved target.
Every action must have a defined rollback, a maximum scope, and a traceable approval policy.
Stage 4: Production change with explicit approval
The agent can prepare and validate a production change but must pause for a named approver before execution. The approval screen should show the intended action, blast radius, evidence, policy checks, rollback, and expiry.
Autonomy is earned per capability, not granted to an entire agent because one workflow performed well.
Design tool access like production credentials
Tool use is the boundary that turns a language model into an operational system. Treat each tool as a production integration with an owner, least privilege, rate limits, audit logging, and a failure mode.
Avoid giving one agent a general-purpose credential “because it may need it later.” Prefer narrow capabilities:
Good: read_trace(service, time_window)
Good: get_deployment_change(service, release_id)
Good: create_incident_note(incident_id, redacted_summary)
Avoid: execute_any_command(command)
Avoid: write_to_all_production_clusters(manifest)
Avoid: read_all_secrets(namespace)Keep tool results typed and small. The more unstructured, unrelated content you return, the more difficult it becomes to evaluate the agent’s reasoning and protect it from untrusted instructions hidden in data.
The operating dashboard
An agent service needs both conventional SRE signals and agent-specific signals.
| Signal group | Questions to answer |
|---|---|
| Availability | Is the agent endpoint available and are tool dependencies healthy? |
| Latency and cost | How long does each stage take and what is the cost per successful outcome? |
| Quality | Are task, grounding, and safety scores changing over time? |
| Guardrails | Which rules trigger, where, and are they noisy or catching real risk? |
| Human oversight | How often do humans approve, reject, edit, or override a recommendation? |
| Change correlation | Did a model, prompt, tool, or retrieval update change outcomes? |
Do not turn these into a vanity score. A rising escalation rate may be a good thing if it shows that the agent now recognises uncertain or high-risk situations rather than guessing.
A practical production runbook
When an agent behaves unexpectedly, your on-call responder needs a short path:
- Stop new actions while preserving read-only diagnostics.
- Identify the affected capability: model, prompt, retrieval source, tool, or approval policy.
- Open the trace and decision record for representative failed runs.
- Check the last change to model version, system instructions, tool schema, evaluation set, or access policy.
- Contain the blast radius by disabling the risky tool or routing the capability to human review.
- Add the failure to the evaluation corpus before re-enabling automation.
This is a familiar incident-management loop. The new detail is that model and prompt changes are production changes too: version them, test them, observe them, and roll them back.
A 30-day rollout plan
Week 1: choose one bounded workflow
Select a repeated investigation or documentation task with a clear owner and low-risk output. Define the contract, allowed tools, decision record, and stop conditions.
Week 2: add traces and a baseline evaluation set
Instrument the agent and tools. Build ten to twenty representative cases, including failures and ambiguous cases. Agree on what a safe handoff looks like.
Week 3: test with real operators
Run in shadow mode or alongside an existing workflow. Compare recommendations with the human outcome. Fix missing context and overly broad tools before increasing autonomy.
Week 4: allow one bounded action
Choose a reversible action that is easy to audit. Add a named owner, rate limits, approval policy, and rollback. Review quality, safety, cost, and operator feedback before expanding scope.
Clear takeaways
- Production agents need an operational contract, not an open-ended instruction.
- Trace the workflow, record the decision, and show the guardrail result for every meaningful run.
- Start with read-only assistance, then earn additional autonomy per capability.
- Give agents narrow, typed, auditable tools instead of broad credentials.
- Evaluate outcomes continuously with realistic failures, not only successful demonstrations.
- Keep people in control for irreversible, high-impact, or uncertain decisions.
The strongest agent platforms will not be the ones that act most often. They will be the ones that make their work legible, respect their boundaries, and leave operators more confident about what happens next.
