We’re now in the fifth wave of computing. If that sounds dramatic, that’s because it is—and probably also because someone in marketing really wants to make “agentic AI” stick. For those keeping track at home, the first four waves were mainframes (your granddad’s computer), web (when we were all hopeful), cloud (when we were still kind of hopeful), and mobile (where that hope became a subscription to your attention span).
And now? Agentic AI.
Let’s get the buzzwords out of the way: Agentic AI is a fancy term for giving large language models (LLMs) persistent memory, looping logic, and callbacks to reach out to your APIs. It’s like giving a Roomba a graduate degree and asking it to run your IT department. These “agents” operate independently (ish), have goals (ish), and make decisions (ish) based on prompts, semantic search and however much context you can cram into a vector database.
Sounds fantastic. Until you have to operate and maintain this stuff in production.
Because Day 1 is always a party, demos run great on slide decks. Day 2 is when the ops team walks into the room, sees the cloud bill, and quietly walks back out.
The Scale Problem (a.k.a. We Broke TPS)
Let’s talk about scale. Traditional systems had thousands to millions of users. With mobile, we got billions. Agentic AI? Try trillions. Not just people, though. You’ll have agents helping users, agents helping agents and agents judging the work done by other agents.
|
Mainframe |
Web |
Cloud |
Mobile |
Agentic |
|
|
Users |
thousands |
millions |
100 million |
billions |
trillions |
|
TPS |
100 |
500 |
2,500 |
10,000 |
1,000,000 |
|
5x |
5x |
4x |
100x |
||
Each one of those agents runs loops. Each loop makes calls. Each call hits an LLM. Every LLM call contributes to Transactions Per Second (TPS). Not all transactions or LLM interactions are the same. Humans have very different behavior than agents.
The types of interaction behavior we might see:
- Human users = finite and sleepy
- Agents = infinite and caffeinated
- LLM inference = expensive, slow, and shockingly unreliable
- Therefore: system load = Who knows? How do we even predict this stuff?
It isn’t unusual for pre-agentic systems to max out around 10,000 TPS. Agentic systems? You’re looking at a million TPS just to hit parity with your company’s Slack usage.
But What About Cost?
Ah yes, the CFO’s favorite section.
Running agents is expensive. Running smart agents is very expensive. Running smart agents 24/7 with high-accuracy LLMs? Welcome to a world where your annual AI ops bill might be an order of magnitude higher than it was before.
Let’s put that in relatable terms:
- GPT-4 32K (86% accuracy): $98 per million tokens
- Open source model (42% accuracy): $0.013 per million tokens
- Database transaction: pocket change by comparison
And don’t let the open-source optimism fool you. A 42% accurate agent might write emails like “Dear Sir/Madman, I have eaten the spreadsheet,” which is cute until it’s running your treasury. Remember that hallucinations and bad responses are so common, we already have patterns for using multiple other LLMs just to judge the work of the first. This means that your cost isn’t just the cost to ask an LLM an innocent question, it’s the cost to resolve tools, cost to make the actual request and cost to make potentially multiple judgment requests.
So now you’re faced with the AI version of the eternal tradeoff: dumb and cheap vs. smart and broke.
Latency, Reliability and Other Sad Stories
LLMs are not built for low-latency, high-throughput work. They’re built for… storytelling. And hallucinating. And sometimes – if you ask nicely and are an expert in prompt engineering – even answering your questions.
Now, imagine your core systems relying on that. Imagine your uptime metrics depending on the whims of an LLM inference API that thinks “return customer data” means “find a retail location where customers can bring and return their data”
Agentic systems aren’t transactional. They’re conversational. They’re unpredictable. They can fail to give you an answer, they can stop answering in the middle of a stream. Don’t listen to what people tell you, you can’t blindly trust an LLM.
And yet, we keep trying to run them like traditional software. We build orchestration platforms, routing layers, retry loops, and cost governors. Why? Because otherwise, these things will try to eat the entire internet in a single for-loop and send us the cloud bill.
Capacity Planning Is Now Chaos Planning
You used to think about load in terms of peak usage. Now? Your agents don’t sleep. They don’t go to lunch. They just keep going, looping away, slamming vector DBs and semantic search engines with relentless, existential enthusiasm. Your system lows are higher, your median times are higher, and your peaks are higher.
This means:
- Your database load goes up 100x
- Your network saturates under constant agent chatter
- Your infrastructure team turns into a stress-testing experiment
- And you still get worse answers than just asking the intern
You don’t plan for “peak load” anymore. You plan for “non-stop Reddit debates between bots.”
The Blueprint (Such as It Is)
In theory, building a sustainable agentic architecture looks like this:
- Semantic Search – All your domain knowledge gets shoved into a vector DB, ready to be retrieved by semantic similarity.
- Agentic Sessions – Every agent interaction is logged to provide context, continuity and accountability.
- LLM Invocation Loops – Prompt, retrieve, augment, call LLM, process, repeat until you get a satisfactory answer, or you run out of usage quota.
- Orchestration Layer – The puppet master that tells agents what to do, when, and for how long before they bankrupt you.
This is all manageable, assuming you can hire a team of distributed systems engineers who also speak fluent “prompt engineering,” understand vector math, and have no plans for sleep.
What Ops Teams Need to Do (Besides Cry)
If you’re serious about building something in this space—and not just gluing ChatGPT to your CI pipeline, you need to:
- Plan for real-time TPS: Millions of them, all at once, with unpredictable spikes.
- Plan for augmentation (RAG): Simple one-off queries are rare. Instead, your agents will be making multiple round trips to the LLM to resolve tools and multiple passes at a vector database for context augmentation.
- Instrument your agents: Know which model they’re using, how often, and at what cost.
- Design for failure: Assume models will hallucinate, APIs will fail and your agents will suddenly think they’re writing sci-fi.
- Stay modular: You’ll want to swap models faster than your CFO can say “cost optimization.”
Final Thought: The AI Agency Gap
The AI Agency Gap is the chasm between dumb bots and actual autonomous systems. It’s the space where most agents will live—semi-capable, semi-stable, semi-helpful. They’re the interns of the software world: energetic, inconsistent and always one bad prompt away from disaster.
If you want to thrive in this new wave of computing, you’re going to need resilience, observability, guard rails, patience and some really good stress coping techniques. The future isn’t just about building agentic systems, it’s about knowing that you’ve made all of your microservices unpredictable and require 10x more resources.
Good luck.
