← All posts

AI Agents · 9 min read

AI Agents Leave the Demo Stage. Production Is Less Forgiving

Pilot chatbots impress boards. Production agents fail on permissions, latency, and audit trails. Here is what Asian operators are learning the hard way.

AI agentsproductionLLM opsautomation

The demo always works. A product manager types a polite request, the agent books a meeting, updates a CRM field, and the room applauds. Three months later the same agent is stuck in a retry loop because a downstream API returned a 429 and nobody owned the failure path. The transcript looks confident. The customer record looks wrong. Support is already on the phone. Across Singapore, Bangalore, and Jakarta, Tech Corp Asia has watched the same pattern: teams treat agents as chat UIs with tools bolted on, then discover that production needs contracts, budgets, and kill switches. An agent is not a smarter chatbot. It is a distributed system that happens to speak English, and distributed systems fail in partial, awkward ways that demos never show.

What breaks first

Permission sprawl arrives early. An agent given broad SaaS tokens will eventually do something a human would have hesitated to do. One regional retailer gave an inventory agent write access to the warehouse system. It corrected stock counts overnight. It also overwrote a manual freeze put in place for a recall. The model was not malicious. It was obedient and under-specified. The post-incident review found three overlapping admin roles and no change ticket for the freeze, which meant the agent had no signal that the write was forbidden. Latency is the second surprise. Multi-step tool calls look fine in a notebook. On a mobile checkout path, two seconds of model thinking plus three sequential API hops feels broken. Users abandon. Support tickets spike. The business concludes that "AI does not work," when the real issue was architecture: synchronous orchestration on a latency-sensitive journey. Parallelizing safe reads and caching tool results for the session often recovers more user trust than swapping to a larger model. Audit trails come third. Regulators and internal risk teams ask a simple question: who approved this action? If the answer is a probabilistic model with no signed plan and no human checkpoint for high-impact moves, the project stalls even when the demos look great. Finance wants a trail that survives an audit. Legal wants a story that does not collapse under discovery. Engineering wants logs that are searchable without dumping every prompt into a spreadsheet.

A concrete pattern that holds

A logistics firm in Peninsular Malaysia built an agent that proposes route changes but does not apply them. The agent drafts a plan, attaches cost and ETA deltas, and posts it to a Slack channel owned by dispatch leads. A human accepts or rejects. Accepted plans write to the TMS through a narrow API with idempotency keys. Rejected plans feed a weekly review so the team can see where the model is systematically optimistic about traffic or dock times.

That design is slower than full autonomy. It also survived peak season. The team measured suggestion acceptance at 61 percent and cut planner overtime without inventing a new incident class. The caution is obvious: if you skip the human gate before the agent has proven calibrated, you trade demo speed for operational debt. Autonomy is a privilege earned by measured accuracy, not a default setting in a vendor console.

Guardrails that actually matter

  • Bound tool scopes per environment; production tokens should not match staging.
  • Cap spend and step counts per session; runaway loops are a cost and safety issue.
  • Require structured plans before irreversible writes.
  • Log prompts, tool calls, and outcomes in a searchable store with retention rules.
  • Rehearse kill switches the way you rehearse incident response.

None of this is glamorous. All of it separates a press release from a system you can sleep next to. Teams that treat agents as workflow participants with narrow powers keep shipping. Teams that chase cinematic autonomy keep explaining outages to executives who still remember the applause from the demo day.

Cost, evals, and the unglamorous middle

Token spend looks abstract until finance asks why the agent budget doubled during a promotion weekend. Cap per-session spend, and separate exploratory sandboxes from production keys. Evaluation sets should include the ugly cases: empty CRM fields, bilingual instructions, and partial tool failures. A weekly eval dashboard that tracks suggestion acceptance, human override reasons, and cost per completed task tells you whether the agent is improving or merely talking more. Without that middle layer of measurement, teams argue from anecdotes and the loudest stakeholder wins.

Change management is part of the system. Dispatch leads will not trust a bot that occasionally invents dock codes. Train them on what the agent can and cannot do, and give them a one-tap reject with a reason code. Those reason codes are gold for the next model update. Treat the human reviewers as co-authors of the system, not as temporary scaffolding you plan to remove next quarter without evidence.

Takeaway

Ship agents as workflow participants with narrow powers, measurable acceptance rates, and clear rollback. Instrument suggestion quality the same way you instrument conversion. If your roadmap still says "fully autonomous by Q2," rewrite the roadmap. Production prefers boring competence over cinematic autonomy, and Asia's operators who learned that early are already quieter on pager channels.

More from the desk