“Agent” is the most overloaded word in software right now. It gets applied to chatbots, to cron jobs that call an LLM, to browser extensions, and to genuinely autonomous systems that write and ship code. The marketing fog is thick enough that plenty of engineers have quietly concluded the whole category is vapor. It is not — but the real thing is much more specific, and much more mundane, than the pitch decks suggest.
Here is the entire concept: an agent is a language model running in a loop with access to tools. The model proposes an action, a piece of ordinary software executes it, the result gets fed back into the model’s context, and the loop repeats until the model decides it is done. Everything else — planning, memory, “reasoning,” multi-agent orchestration — is elaboration on that loop.
By the end of this post you will know exactly what that loop looks like at the API level, why the boring orchestration code around it is most of the actual work, when an agent beats a single prompt or a scripted workflow (and when it loses), and where agents genuinely fail today. No vendor cheerleading, no doom — just the mechanics.
An agent is a model in a loop with tools
Strip away the branding and every agent product runs the same cycle:
- You give the model a goal and a list of tools it may call.
- The model responds with either an answer or a tool call — a structured request like “run this shell command” or “fetch this URL.”
- Your code (not the model) executes the call and appends the result to the conversation.
- The model sees the result and decides what to do next. Repeat until it stops requesting tools.
One iteration looks roughly like this on the wire — the exact field names vary by vendor, but the shape is universal:
// iteration N: the model proposes an action
{
"type": "tool_call",
"name": "run_tests",
"arguments": { "path": "tests/auth.test.ts" }
}
// your harness executes it and appends the result
{
"type": "tool_result",
"tool": "run_tests",
"content": "FAIL: expected 302, received 500\n at line 41..."
}
// the model reads the failure and proposes the next action
{
"type": "tool_call",
"name": "read_file",
"arguments": { "path": "src/app/login/route.ts" }
}That is the whole trick. The model never executes anything itself — it emits JSON describing what it wants, and deterministic code you wrote decides whether and how to do it. What makes this feel qualitatively different from a chatbot is the feedback: the model gets to see the consequences of its actions and correct course. A single prompt gets one shot. An agent gets to run the tests, read the failure, and try again.
Tools are just function calling
A “tool” is a function signature you describe to the model: a name, a JSON schema for the arguments, and a plain-language description of what it does. All the major APIs support this natively — you pass the tool definitions alongside the prompt, and the model is trained to emit well-formed calls when a tool would help. Anything you can wrap in a function can be a tool: a database query, a search API, a shell, a headless browser, an email sender.
The description matters more than people expect. The model chooses tools the way it does everything else — by reading text — so a vague description produces vague usage. Writing good tool definitions is closer to writing good documentation than to writing good code.
For years every product wired up its integrations bespoke, which meant the same Postgres or GitHub connector got rebuilt for every agent. The ecosystem has largely converged on the Model Context Protocol as the standard way to expose tools to any client — I cover how that works in What Is MCP? — but MCP is plumbing, not magic. It standardizes how tools are described and called; the loop is unchanged.
The harness is the unglamorous 80 percent
The loop itself is maybe twenty lines of code. The reason agent products are real engineering efforts is everything wrapped around it, usually called the harness or orchestration layer:
- Permissions. Which tool calls run automatically, and which stop and ask a human? “The model wants to run
rm -rf” is a product decision, not a model capability. - Sandboxing. Serious harnesses execute commands in containers or restricted environments so a bad tool call has a small blast radius.
- Error handling and retries. Tools time out, APIs rate-limit, the model emits malformed arguments. The harness has to feed failures back in a way the model can recover from.
- Context management. Every iteration appends more text. Long tasks fill even large context windows, so harnesses summarize old turns, truncate tool output, and decide what the model actually needs to see.
- Stop conditions and budgets. Without a cap on iterations, wall-clock time, and spend, a confused agent will loop until your invoice notices.
The economics deserve a sentence, because the loop structure makes them unusual: each iteration resends the entire conversation so far, so input token usage grows with roughly the square of the conversation length. Prompt caching is what makes this affordable — on the Claude API, reading a cached prefix costs roughly a tenth of the base input price as of August 2026 (see the official pricing page for current numbers, since these change). If you want the full token math, I walk through it in LLM API Pricing, Explained.
Single prompt, workflow, or agent: when each wins
Anthropic’s Building Effective Agents draws the line worth internalizing: a workflow orchestrates LLM calls along code paths you predefined; an agent lets the model decide its own path at runtime. Add the degenerate case — a single prompt, one call, no orchestration — and you have the full menu.
The way I choose between them is cost-of-error framing. Two questions: is the task open-ended (you cannot enumerate the steps in advance), and are errors catchable (there is a cheap way to detect that the output is wrong before it does damage)?
| Task shape | Errors catchable? | Best fit |
|---|---|---|
| Well-defined, same steps every time | Either way | Single prompt or workflow |
| Open-ended, path unknown upfront | Yes — tests, compilers, review | Agent |
| Open-ended, errors silent or irreversible | No | Human, or a workflow with checkpoints |
Agents shine in exactly one quadrant: open-ended tasks where errors are catchable. That is why coding is the breakout use case — a coding agent can be wrong repeatedly and cheaply, because the compiler, the test suite, and code review catch mistakes before they ship. It is also why “agent books my flights” demos keep underwhelming: booking is closed-ended enough that a workflow does it more reliably, and the errors (wrong ticket, real money) are exactly the uncatchable kind.
The corollary: if you can write the steps down, write a workflow. It will be cheaper, faster, and debuggable. Reach for an agent when enumerating the paths is the hard part.
Where agents still fail
Compounding errors over long horizons
A per-step success rate that sounds great multiplies into a per-task rate that does not: many small chances of going wrong, compounded over dozens of steps, add up fast, and one bad early decision can poison everything downstream. METR measures this directly as the length of task (in human-expert time) that a model completes with 50 percent reliability. That horizon has grown at a startling, roughly exponential pace — but note what the metric says: at the frontier of what a model can attempt, it still fails about half the time. Agents are dramatically better at hour-scale tasks than they were two years ago, and still unreliable at the edge of their range.
Context limits and context rot
Million-token context windows sound infinite until an agent spends a long session reading files and tool output. Long contexts also degrade subtly: models weight recent tokens more heavily and lose track of details from hundreds of thousands of tokens ago. Harnesses compact and summarize to cope, and every compaction is lossy — the agent that “forgot” a constraint from an hour ago usually lost it to summarization, not stupidity.
Prompt injection, the unsolved one
This is the failure mode I would weight most heavily. Models follow instructions in their context, and they cannot reliably distinguish your instructions from instructions embedded in a web page, an email, or a README the agent happened to read. Simon Willison calls the dangerous combination the lethal trifecta: access to private data, exposure to untrusted content, and the ability to communicate externally. An agent with all three can be turned into a data-exfiltration tool by text on a page it was asked to summarize. There is no robust model-level fix as of mid-2026 — real deployments mitigate architecturally, by making sure no single agent holds all three capabilities at once.
What actually works in August 2026
Three categories have crossed from demo to daily use, and the pattern behind which ones made it is instructive.
Coding agents are the clear leader. Terminal-based tools like Claude Code and OpenAI’s Codex run the loop against your repo — read files, edit, run tests, iterate — while GitHub-integrated agents such as Copilot’s coding agent take an issue and come back with a pull request. Coding won first because it sits squarely in the good quadrant: open-ended tasks with compilers and test suites as free error-catchers.
Deep research products — OpenAI’s Deep Research, Gemini Deep Research, Claude’s research mode — run search-and-read loops for minutes at a time and synthesize cited reports. Errors here are semi-catchable: the citations let a human spot-check, which is exactly why the products emphasize them.
Computer use — agents driving a real screen with clicks and keystrokes — remains the weakest of the three, though benchmark scores have climbed steeply since the first public betas in late 2024. GUIs are a hostile interface for models: no structure, no error messages, and every mis-click is silent. Where an API or MCP server exists, the loop works far better through it than through pixels.
About the hype
Two things are true at once. First, most products marketed as agents are workflows — a fixed pipeline with LLM calls inside. That is not a scam; workflows are often the right design, and the relabeling is mostly harmless. Second, the underlying capability curve is real and steep. The honest failure of the hype is not that agents are fake — it is that demos are selected from the half of attempts that worked, and the marketing treats autonomy as a binary you unlock rather than a dial you turn up as error-catching improves. “Fully autonomous” is not a milestone anyone ships; supervised autonomy with widening scope is what actual progress looks like.
Bottom line
- An agent is a model in a loop: propose a tool call, execute it outside the model, feed the result back, repeat.
- The harness — permissions, sandboxing, retries, context management, budgets — is most of the engineering. Budget for it.
- Use a single prompt or workflow when you can enumerate the steps. Use an agent when the task is open-ended and errors are cheap to catch. Keep humans in the loop everywhere else.
- Respect the failure modes: compounding errors, lossy context, and — above all — prompt injection whenever an agent touches untrusted content and real systems at the same time.
If you are deciding which model to put inside the loop, capability and cost trade off differently for agents than for chat — long contexts and caching matter more than raw benchmark scores. I compare the current options in Claude, GPT, or Gemini: How to Choose an LLM in 2026.