The Trough of AI Disillusionment

Why AI is not increasing workplace productivity and the rise of the AI Wrangler

This is a written version of a talk I gave here.

ChatGPT taught us something subtle and dangerous: AI is magic, it just works. But as we’ve all learnt, it is not magic, and it doesn’t just work.

LLMs can write working code in seconds, explain complex topics, transform a document or create an image from a sentence. But then at the same time, they confidently invent facts, lose track of an instruction, produce something painfully generic or fail at a task that seemed much easier than the one it completed a moment earlier.

This is where many of us now find ourselves: somewhere between astonishment and irritation, looking down the slope of inflated expectations into the deep trough of AI disillusionment. Worse, recent trends and studies show that the promised productivity gains for most of us have not eventuated, despite a plethora of AI tools proliferating in nearly every business.

The problem is, LLMs are an incredibly powerful, but subtle tool that can be used poorly or well. I am making the case here that 99% of people are using them extremely poorly. The 1% who are using them well are reaping all of the benefits.

Subtle magic

Part of the confusion begins when we talk about ‘AI’, which has largely become shorthand for LLMs (large language models), or for products built around them. When someone says an AI system searched the web, edited a file, booked a meeting or operated a computer, the language model did not do all of that by itself. A harness - or a larger system enabled it to do those things. Many quirks of using one provider over another is a complication of not just who has the best model, but also who has the best harness.

The fantasy of the ‘it’s magic’ level of understanding collapses these distinctions. It gives us one seamless, person-like entity that knows things, wants things and can do things. The reality is much more subtle and therefore requires patience to wrap your head around.

The anti-hammer

Consider a hammer. Its shape tells you what it is for. Its limitations are obvious. You can strike a nail with it, but you probably would not use it for fine motor work. The feedback loop is immediate and reliable: either you hit the nail or you don’t.

A large language model is almost the opposite. Its purpose appears unbounded and all-encompassing. What can it be used for? Almost anything it seems. The data it was trained on or not trained on and its limitations are hidden. Its feedback is difficult to interpret, because the same fluent tone can accompany a brilliant insight, a banal summary or a complete fabrication.

Even scarier, the tool can even undermine the person using it. Because they are so persuasive and write in such a competent way, we are inclined to trust their outputs. Models can be sycophantic, agreeing with a mistaken premise. They can conceal a lack of clarity through verbosity. And they can encourage people to outsource judgement - in my opinion the cardinal sin of using AI. This helps explain why capable and intelligent people struggle to use LLMs effectively. They are trying to operate a tool that does not make its operating characteristics legible. It is of course, a black box.

Learning to use an LLM in an advanced way is more like getting to know a person than getting to know how to use a hammer. It takes time to apprehend a person’s mind to discover where they are perceptive, unreliable, what they misunderstand and how they respond to different circumstances. Especially because the AI labs don’t publish what data the models are trained on. Even if they did, how would we even comprehend that information? Models only reveal their true character like people - with time through repeated interaction.

You do not merely learn how to prompt a model. You develop a feel for the strange neural topography you are dealing with, and this can take time and experience to build up.

Benchmaxxing

We benchmark as though LLMs were interchangeable products arranged on a single ladder from less intelligent to more intelligent, whatever we mean by ‘intelligent’ - a can of worms we will leave unopened here. Each model may exhibit outcomes that we’d equate appearing to have emerged from an intelligent actor, but in reality each model simply has a different internal topography to other models, which leads to different emergent behaviour.

A useful metaphor to understand the difference between models is growing mushrooms. LLMs are not designed so much as they are grown. Mushrooms are grown from spores in a substrate, and small changes in their conditions produce different shapes, sizes and varieties. LLMs are grown from training data, architectures, training methods, fine-tuning, reinforcement and the commercial forces that shape these choices. We as consumers have almost no visibility into many of those ingredients, even though they strongly influence the resulting model. A bit like we have no visibility into the mystery meats in a can of dog food.

Even the AI Labs have no idea how a model will turn out until it has been grown. Key insiders at the AI labs talk about testing and exploring capabilities as a key step in the development of new models. The testing is required because they simply do not know what emergent qualities the training run might have produced. Once they assess the performance of their new model, it will inherently be different to previous models.

As such, I make the claim here that no model is perfect and no model could ever be perfect. And that by prioritising strength in one dimension, you downweight strength in another. Similar to biology, you simply can’t squeeze every capability possible into every model just as every organism cannot prioritise every survival strategy all at once - there is a kind of natural limit, a homeostasis both in our behaviour to optimise, and in the latent space.

Newer ≠ better

A great case study for this is the backlash at the deprecation of the much loved gpt-4o. It’s a persistent meme at this point to bring gpt-4o back, much to the frustration of OpenAI. Technically inferior in every dimension to the gpt-5 series family that they replaced it with, yet beloved by many for its personality.

Why gpt-4o was so beloved is worthy of a deep dive on its own, but in short it wrote in a unique way and seemed to have a very distinct set of personality traits. As such, people formed attachments to it, some from the perspective of work camaraderie, others even romantic - describing the change in personality from, gpt-4o to the gpt-5 series as akin to a ‘lobotomy’ of the model they loved.

There’s more to it than just gpt-4o having a different set of weights - it was a fundamentally different model architecture. This might seem a bit technical, but it’s important to understand that gpt-4o is very likely what they call a ‘dense’ model. I say very likely because OpenAI doesn’t disclose this information despite their so-called openness, but many in the community consider it to be empirically true.

In simple terms you can have a ‘dense’ model or a ‘mixture of experts’ model (MoE). Unlike an MoE model that activates only a subset of parameters per token, a dense model utilises its full capacity for every inference step. MoE models exist because it’s faster and more efficient to run than a dense. Why fire every neuron containing all domains of knowledge when the user is asking a maths question - simply route the query to a domain specific model instead.

What this means in a nutshell is that a dense model has the ability to fire any of its neurons when invoked, in a way that feels like an expansive mind connecting the dots between numerous areas of knowledge. A mind on mushrooms as I’ve described it.

The gpt-5 series models have vastly more parameters in their weights, and therefore more knowledge. The problem is that this becomes a tradeoff - the larger the model, the more difficult it becomes to host and to serve to millions of people. Using a mixture of experts architecture allows them to serve a more intelligent model faster and more efficiently. But the downside is losing that ‘expansive’ feeling when conversing with it.

This adds to the portrait of LLMs being complex, multi-layered, interleaved systems that are impossible to get a meaningful grasp of through pure benchmarks and technical specs alone.

Spiky intelligence

The standard framework of understanding how the particular type of intelligence LLMs possess is different from ours is the ‘spiky intelligence’ metaphor. Imagine a circle with points around it. At each point imagine a class of task - coding for example. The outline of the circle either extends outwards to represent greater ability or inwards to constitute lesser ability. This becomes interesting when you compare the same circles of LLMs versus humans for the same set of tasks. Sometimes, the LLM excels, sometimes the human does. It’s not intuitive when or where this happens.

LLMs can be exceptional at coding but horrendous at basic things like physical intuition. They can summarise a complex report in seconds, yet miscount how many ‘r’s are in the word strawberry. They can translate between languages while struggling to understand the simplest brain-teaser. Humans are strong at navigating and understanding the physical world and the physical relationships between objects, etc.

In short, LLMs are superhuman at some tasks, and abysmal at other seemingly simpler tasks. This is an emergent property of the world of language they have been trained on. Our lack of intuition around what they are and are not good at is a failure of our understanding of our own cognition, and the role language plays in it.

Language is at base a set of communicable symbols imbued with meaning - an LLM can work with these symbols in sometimes magical ways; it seems to understand things about the physical world because the symbols encoded into our language reference things in the real world. But crucially, an LLM only inhabits this world of symbolism by proxy - it does not live in or perceive the real physical world. We’ll come back to this important point.

The companies building frontier models are heavily rewarded for gains in the ‘intelligence’ benchmarks, coding, science and general knowledge work. Those are valuable capabilities that businesses are paying billions for, but crucially optimising for these dimensions does not automatically produce common sense, or indeed subtler forms of intelligence other than raw IQ, e.g emotional intelligence.

This is why knowing what LLMs should be used for in the first place matters, and then choosing a model for each task also matters. It is why benchmark charts can only take you so far. Eventually, you have to use the model, observe it and get the vibe for where its strengths and weaknesses lie. There are unfortunately no shortcuts, because we humans are abysmal at comprehending an intelligence that is seemingly so similar and yet so different to our own.

The harness

Aside from choosing which model to use, the next most important decision is which harness to use. The harness is the application wrapped around the model. It manages instructions and context, calls tools, reads and writes files, searches the web, executes code, connects to external services and decides when the model should act.

This is why the same underlying model can feel radically different in a chat interface, a desktop app, a coding tool or a custom system. The model is the metaphorical horse, and supplies intelligence-like capability. The harness determines what that capability actually is. A lot of AI discussion is really a discussion about the harness, not the underlying LLM model.

Speaking of harnesses, all the apps I’ve built at Paper Moose are essentially just harnesses, but extraordinarily powerful and useful ones, built in a focused and custom way for the exact needs of a creative agency.

This is clearly the future: hyper custom software harnesses for the needs of individuals and businesses, spun up on demand.

Agentic AI is mostly marketing bs

Strip away the marketing language and an AI agent is an LLM inside a harness working in a loop. Given a goal, it observes something, chooses an action, uses a tool, inspects the result and then decides if it has achieved the goal or not. The harness keeps the loop running until a goal is achieved and gives the model ways to work in its given environment.

This is genuinely powerful, don’t get me wrong. Agents can work with files, update spreadsheets, modify a codebase, operate software or coordinate several steps toward a directed outcome. If you haven’t used Codex, Claude Code, Opencode, Pi, OMP, Hermes, Grokbot etc before, it really does feel like magic, and it’s easy to see why the entire global economy is being powered by this narrative.

So why do I think there’s a problem with the term ‘Agentic AI’?

The agency mirage

It comes down to what most people understand agentic AI to be: an intelligent actor with agency acting within and upon the world. Like a human being. This is the root of a lot of the fear around agentic AI taking people’s jobs.

This might seem like a subtle point, but I think its misunderstanding is the fulcrum upon which the entire global economy is making bets about the future of AI. If AI can have agency in the world, we can eventually unleash an army of ephemeral and robotic agents that will sweep up and execute all work sans humans.

But most work it turns out is more complex than we imagined. Unless you’re an assembly line worker, it’s usually made up of a string of non-deterministic tasks in a non-deterministic, messy environment. Work is long ranging, spanning not just minutes, but months or even years. Work involves the interrelationship of people, systems and indeed the entire world.

Anyone working seriously with this technology knows it can be used to automate tasks - but they also know all the ways that automation is brittle and can break. And being non-deterministic systems themselves, LLMs can sometimes mess simple tasks up in catastrophic ways without even realising. That is to say that LLMs are not aware or present in the physical world to know or care about the impacts of their work, let alone their presence in a complex organisation.

In this way, I think ‘Agentic AI’ is the wrong name for the current technology. ‘Agency’ carries a lot of philosophical weight. It suggests an entity with intentions of its own.

Not all agents have agency

I will try to break this down: at one end of the spectrum of automatic systems is an automaton or robot - it senses something and performs a predefined action. A thermostat and a robot vacuum are in this way automatons.

A weak agent goes further. It can weigh options, select tools and change its approach while moving toward a goal. Today’s most capable AI systems belong somewhere in these first two categories.

A true agent is something else entirely. A person does not merely optimise toward a supplied goal. A person inhabits a world, has a body and a history, originates and negotiates goals, and carries the consequences when things go wrong.

The philosopher Martin Heidegger used the term ‘da sein’, or ‘being there’ in German to describe a being with agency. In other words, the kind of being for whom its own existence is at issue. He postulated that to have true agency, one must be thrown into circumstances we did not choose. One must project ourselves into possible futures, and one must know that time can end, which is the root of anything mattering at all.

Current AI agents have no equivalent stake, no dasein. They do not exist in the world like we do, and they do not care whether a project succeeds or fails.

And yet they can perform goal-directed behaviour without possessing goals in the human sense, which makes them very good at producing a mirage of agency. Again, this is a failure of ours to comprehend intelligent seeming systems that are different to our own.

At Paper Moose, we introduced an AI powered digital employee ‘Stefan’ into our custom project management platform. One of our human team thought Stefan was a real person for over a week whilst conversing with them about a project, only realising they were an AI a week later when informed by a co-worker. Aside from casually passing the Turing test, this is a great demonstration of how convincing these systems can be.

Yet the goal, the environment, the prompt and the permission to continue were supplied by an outside actor. Stefan’s goals were not its own, they were mine.

A robot vacuum moves independently through a room. That does not mean it has agency, or understands the world at all. This is the current level of agentic AI, and I don’t think the world understands this.

The industrial revolution of knowledge work

Despite this knit-pick about AI agents not having true agency, I still think we’re in the middle of the industrial revolution for knowledge work. In other words, the gradual automation of most of the tasks within white collar work.

But how are we achieving this if an AI agent cannot replace a whole person in the way the phrase ‘digital employee’ implies? What seems to be becoming the emergent paradigm is a technically competent human wrangling a fleet of AI agents.

The person who is doing this is an ‘AI Wrangler’. The wrangler is an expert in AI harness systems and knows their strengths and weaknesses well enough to wield them to do useful work. The key being useful work, because they can easily churn through work that is worse than useless.

I expect the rise of this kind of role alone to drive many of the promised productivity gains of AI. This is a key point; AI does not make regular people more productive in their regular jobs. This is what we’ve all been sensing for a while now. Instead it makes a tiny portion of power users - the wranglers, superhumanly productive.

AI is not coming for your job, but a competent AI wrangler most likely is. I suggest you consider becoming one. It will continue to become a role in great demand for a long time to come. If you don’t feel technically inclined to become one yourself, then find someone who is and work with them to see the real benefits of AI.

As important as finding someone with the technical ability and interest in becoming a wrangler - you also need to bless that person with enough autonomy to understand both the work itself and gain the access and permissions across the organisation to automate workflows. This is where AI is no longer in a trough of disillusionment, but in a golden age of productivity.

If you made it this far, I gave a talk on this which fleshes out some of the more interesting details.

Anyways, thanks for reading.