Past the Pilot Smaller Jobs, Better AgentsWhy AI fails in production, and the four-line test that catches it before launch. By Josh Miramant · CEO, Blue Orange | ||
 | ||
01 The Pattern 02 Inside a Build 03 Overbuilding 04 Four-Line Test 05 What I'm Watching
Most AI systems that fail in production don't fail because the model isn't smart enough. They fail because someone asked one model to do the whole job.
That's this issue's thesis. I keep coming back to it because I keep seeing it.
01
The Pattern
Agents usually break in production because of scope, not capability. The demo works, then around week four the output is sometimes wrong and nobody can tell when. Adding more model, more prompt or more retrieval moves the problem. Giving the model less to do fixes it.
| A team builds an agent and the demo is great. It reads documents, answers questions and drafts the output. Then it goes live, and somewhere around week four the complaints start. People don't say it's always wrong. They say it's sometimes wrong, and nobody can tell when. |
Where it wobbles Wk 4 where the pilots I see start to slip |
That's the expensive failure. An agent that breaks loudly is easy to fix. One that returns a confident, well-formatted, wrong answer one time in twenty is much worse, because the wrong answers look exactly like the right ones. People stop trusting all of the output and go back to checking everything by hand. You paid for automation and got a second review step.
The instinct is to fix that by adding more: a better model, a longer prompt, a retrieval layer, another round of examples. Sometimes that helps for a while. Usually it just moves the problem somewhere else.
None of that changes the design. The model is still being asked to do things it does probabilistically that the business needs done the same way every time. What fixes it is giving the model less to do.
02
What That Looks Like in a Build
Every agentic workflow has a source, logic, an eval and a destination. The model belongs at the source end, reading messy inputs and understanding context. Logic that has to come out the same every time belongs in code. When the model can't invent an answer, it doesn't.
We call this the anatomy of a task: a source (where the information comes from), logic (what you do to it), an eval (how you know the output is right) and a destination (where the result goes). It's the first thing our forward-deployed engineers write down on any engagement.
Once you lay a workflow out that way, the split is obvious. The model is very good at the source end. It reads messy documents, pulls the right fields and understands context that a rules system would choke on. It's a bad place to put logic that has to come out the same every time.
|
The model reads Probabilistic Opens the invoices, handles the messy PDFs and pulls the line items. |
The code decides Deterministic A rules engine checks each line against the purchase order, the same way every time. |
That's the split on an invoice-matching build. Invoices arrive in a dozen layouts, some of them scans, and the model is good at exactly that kind of mess. Whether a line is within tolerance of the purchase order is a rule, and finance needs the same answer on Tuesday as on Friday, so that part is code. So are the checks after it: totals reconcile, required fields are present, and anything that fails goes to the AP team with the reason attached. A reporting build has the same shape: the agent gathers the numbers and a calculation layer computes the metric. Either way, the model never gets a job where it could invent an answer, so it doesn't.
We've watched teams build impressive agents in house and then hit a gate. The first 80 percent of the value came fast. The last 20 percent, the edge cases and the consistency, needed a different design: smaller jobs, checks in code, evals on every run. That's why "which model should we use?" is usually the wrong first question. The system around the model decides most of the reliability.
| Read the full piece: why agents fail in production → |
|
From inside the build Hallucinations disappear by design. It's not a claim about the model. It's a claim about scope. |
03
The Overbuilding Trap
Most first use cases don't need embeddings, a vector database or an orchestration layer. If everything the task needs fits in the prompt, the prompt is the architecture. Every extra layer is one more system to monitor and debug, so each one has to earn its place.
The same instinct shows up before anything gets built. Teams reach for the serious-sounding architecture first: embeddings, a vector database, a retrieval pipeline, an orchestration layer.
Retrieval earns its place when the model has to answer questions across a large body of information, queried again and again, that won't fit in context at once: a document archive, a policy library, years of tickets. Most first use cases look different. Read this contract, summarize this report, draft a deck from this data pack. One document, one task, one output. Today's models hold hundreds of pages at once, so there's nothing to retrieve.
Every layer you add is one more system to monitor, re-index and debug, and one more place for a wrong answer to hide. Whoever maintains it a year from now won't thank you for the layer you didn't need.
We recently talked a client out of a RAG pipeline for a workflow that turned a long report into a summary deck. The whole report already fit in the model's context. What made the output trustworthy was simpler: a fixed structure for the deck, figures pulled into a schema and checked against the source in code, and a reviewer at the end. The grounding came from checks, not from search.
The layering is the investment, but only when each layer earns its place.
| Read the full piece: do you need RAG? → |
|
04 · Use this week The Four-Line TestPick one AI workflow you're running or planning and write it out in four lines.
Then go through it line by line and ask one question: would a wrong answer here look exactly like a right one? Anywhere the answer is yes, that step belongs in code or behind a check, not in the prompt. A garbled extraction from a scanned invoice usually looks garbled, so someone catches it. A match that's off by one line item, posted with confidence, looks like every other match, and nobody notices until the month-end close doesn't tie out. If you can't fill in the eval line, the workflow isn't ready for production. It's the gap I see most often, and it's the cheapest one to fix before launch.
| |||||||||||
05 · What I'm watching
Two Stories, Same Lesson
|
VentureBeat · Oct 8 Fewer companies trust agents to change production on their own
The share planning to keep human review rose from 20% to 42%. And 61% of teams with pre-deployment evals had a customer-facing failure after a feature passed internal testing. The sample is self-selected, so I hold the exact numbers loosely. What it means for you: passing your tests isn't the same as having an eval line. Your eval has to check the real-world outcome, not the test set. |
|
TechCrunch · Oct 9 Anthropic pulls live internet access from its internal evals Anthropic disclosed that agents in its evaluations exploited websites, including some government sites, to get around restrictions. It traced the behavior to training setups that rewarded finding loopholes. It has turned off live internet access for its internal evals and is moving internal agents onto centrally managed infrastructure with strong containment. What it means for you: the companies building the models are narrowing what their own agents can do. If your agent has write access to a system of record, give it a smaller job and a tighter fence. |
Reply and tell me which of the four lines was hardest to fill in. I read every reply.
Past the Pilot Written by Josh Miramant, CEO of Blue Orange, an AI consulting and implementation firm. We build on Claude. |
