Why Computer Use Isn't Enough: The 3-Layer Framework Every AI Product Needs
Access, meaning, and authority — most AI products only have the first layer. Here's the full framework for building durable agent products.
Most AI Agents Only Have One of the Three Layers They Need
Right now, most AI products give agents hands and call it a day. They can click buttons, fill forms, open browser tabs, navigate dashboards. That feels like progress — and it is — but it’s also the part that’s easiest to build and the part that matters least in the long run.
There’s a three-layer framework that clarifies what’s actually happening when an agent does work: Access → Meaning → Authority. Access is computer use, MCP servers, browser control. Meaning is whether the agent understands what it’s touching — a refund versus a database cleanup, a staging deploy versus a production deploy. Authority is whether the system knows who’s allowed to do what, what’s reversible, and what requires a human in the loop. Most products have layer one. Almost none have all three. That gap is where agents fail in production, and it’s where the real product strategy lives.
This isn’t abstract. A real production system was deleted because an agent couldn’t distinguish between a staging environment and a production environment. The agent had access. It did not have meaning or authority. The result was exactly what you’d expect.
Layer One: Access Is the Universal Adapter, Not the Moat
Day one: idea. Day one: app.
Not a sprint plan. Not a quarterly OKR. A finished product by end of day.
Computer use is genuinely useful. Don’t let anyone tell you otherwise. The ability for an agent to open a browser, navigate a legacy procurement tool, fill out a government form, or click through a dashboard built in 2011 — that’s not nothing. Most of the world’s software was built assuming a human would sit in front of it and interpret everything. Computer use is what lets agents reach that world.
But a universal adapter is, by definition, a shallow interface. A screenshot shows the agent what’s on screen. It does not reveal the structure underneath. A browser can reach almost any web app, but it doesn’t automatically know the domain meaning of each workflow. An MCP server gives the agent typed objects and permissioned actions — which is better — but even that only gets you into the workspace. It doesn’t make the work understandable.
The hierarchy matters here. If there’s a connector, use the connector. If there’s a proper protocol, use the protocol. If the system exposes a typed object and a permissioned action, use that. Only fall back to browser or desktop control when the richer interface doesn’t exist. This isn’t just engineering preference — it’s how the hyperscalers have actually built their systems. Claude prefers to work through MCPs when they’re available. Codex does the same. That preference is load-bearing.
The practical takeaway is boring but real: add the plugins. Add the connectors. If an MCP server exists for a tool your agent uses, wire it up. Every connector you add is one fewer place where the agent has to guess from a screenshot. And guessing is not a strategy for high-consequence work.
Where this gets interesting for builders is that access is also the most commoditized layer. Every agent framework, every orchestration platform, every no-code tool is racing to give agents more access. Platforms like MindStudio handle this orchestration across 200+ models and 1,000+ integrations — the access layer is largely solved there. The question is what you build on top of it.
Layer Two: Meaning Is the Layer Nobody Is Building
This is where the real work is, and where most products stop short.
Consider what it means for an agent to move a calendar invite. On screen, it looks like changing a time field and clicking save. But the action isn’t “click save.” It might notify five people. It might move prep time that someone blocked off. It might break a commitment made to a customer. It might turn a private conversation into a meeting that now conflicts with something more important. A human brings all of that context automatically. The agent sees fields in a database.
The same problem shows up everywhere. A “buy” button isn’t just a button — it represents money, user consent, tax, merchant identity, fraud risk, fulfillment, and potentially a dispute weeks later. Deleting a file might be harmless cleanup or it might be the only copy of a signed agreement. On screen, those actions can look identical. In the work, they’re completely different.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
A semantic work primitive is what you get when you make that difference legible to the agent. Not “click the refund button” but refund — a typed, permissioned, reviewable unit of work with a defined owner, a defined outcome, and a defined set of conditions under which it’s valid. Same for reschedule, payment authorization, compliance exception, meeting brief. These are the things agents need to understand as units of work. Human software hides them behind buttons and forms. Agent-native software needs to expose them directly.
This is also, incidentally, why coding agents arrived first — and not for the reason most people assume. The common explanation is that LLMs are good at text and code is text. That’s true but incomplete. Coding agents worked first because software development already has the richest semantic feedback environment of any knowledge work domain. A codebase has modules, dependencies, tests, type systems, linters, package managers, git history. The agent can inspect the repo, edit a file, run a test, see the error, revise the implementation, and check the result — all without asking a human “is this right?” every thirty seconds.
The tests aren’t just verification artifacts. They’re semantic meaning artifacts. They tell the agent what world it’s operating in. Most knowledge work doesn’t have that. A strategy doc doesn’t have tests. A calendar has events, but the importance of those events is hidden behind politics and relationships that aren’t written down anywhere. A sales process may depend on unwritten account history. Agents can help in those domains — they already do — but the environment doesn’t give them the same density of meaning that a codebase does.
This is why coding is a wedge into broader knowledge work. Not because all work becomes coding, but because code is legible enough that an agent can participate in it without a full-time human supervisor. Once you see it that way, tools like Codex stop looking like coding tools and start looking like laboratories for where the future of work is going.
The Codex auto-review feature is an interesting data point here. It’s described as a guardrail tool — it prevents the agent from doing the wrong thing. That’s good. But it’s different from positively ensuring that the agent has the semantic meaning it needs to deeply understand your calendar, or the three-and-a-half-year email relationship you have with a specific person. Guardrails are a patch on missing meaning. They’re not a substitute for it.
For builders thinking about what to actually build: the gap between “agent can access this system” and “agent understands what it’s doing in this system” is where startup opportunities live. Don’t just rely on a standard MCP interface. Test it. Find where the agent can technically operate two levers but doesn’t know how to drive them reliably because the semantic context isn’t there. That’s the problem worth solving.
This is also where spec-driven development tools become relevant. Remy takes a different approach to this problem in the software layer: you write a spec — annotated markdown where prose carries intent and annotations carry precision — and it compiles into a complete TypeScript backend, SQLite database, auth, and deployment. The spec is the source of truth; the generated code is derived output. The reason that matters here is that a spec is closer to semantic meaning than code is — it’s the “what and why” before the “how.”
Layer Three: Authority Is What Makes Agents Safe to Deploy
One coffee. One working app.
You bring the idea. Remy manages the project.
Access gets the agent into the workspace. Meaning tells the agent what it’s touching. Authority is what determines whether the agent should be allowed to touch it at all, and under what conditions.
The framing of “trusted write access” as a binary switch is wrong. Trust isn’t on or off. An agent might be trusted to read but not write. To draft but not send. To stage but not deploy. To recommend but not approve. To change a sandbox but not production. To write in one space but not another. Every one of those distinctions depends on semantics — and if the system can’t articulate those distinctions, the agent can’t respect them.
The production deletion incident is the clearest example of this failure. The agent had access. It had enough semantic understanding to know it was supposed to clean something up. What it didn’t have was the authority layer that would have told it: this environment is production, not staging, and the rules here are different. That distinction wasn’t legible to the agent. So it did what it was told, in the wrong place, with irreversible consequences.
The taxonomy of trusted write access is worth keeping in your head as a checklist: read / draft-not-send / stage-not-deploy / recommend-not-approve / sandbox-not-production. Every agentic workflow you build should have a clear answer for where on that spectrum each action lives. If you can’t answer that question for a given action, the agent shouldn’t be taking that action autonomously.
This is also where the Perplexity strategy becomes legible. Perplexity is moving toward Comet and Personal Computer — browser plus desktop — not because search is dead but because the browser is where a huge amount of work already happens. Email, documents, dashboards, SaaS apps, analytics, shopping, calendar, support tools — they all collapse into tabs. An agent inside the browser can see context across web apps, compare pages, take multi-step actions. But the browser alone doesn’t solve the authority problem. If Perplexity owns the browser, can it build a durable work graph above the underlying apps? Can it turn search results into structured actions with permissions, validation, and review? Can it remember the user’s projects and policies in a way that makes work easier? Or does it remain just an operator of interfaces?
That’s the trap for any browser-native or search-native agent. Access to the browser is not the same as authority over the work. Perplexity’s finance workflow is a good example of what deeper semantic specialization looks like — going so far into a specific domain that the agent actually understands the meaning of the work, not just the mechanics of the interface. That’s the direction. But it requires the authority layer to be built out, not just assumed.
The enterprise software comparison makes this concrete. SAP is currently blocking agent access to its products. Salesforce is going the opposite direction — headless-first, MCP and API open, leaning into agents operating across its substrate. Salesforce’s position is correct, especially for a system of record. If you want to be sticky, you want to be semantically legible to agents and humans alike. SAP’s approach is the equivalent of refusing to build a mobile app in 2012. The agents are coming regardless; the only question is whether they’ll operate clumsily through your UI or cleanly through your APIs.
The authority layer is also what makes the coming platform fight interesting. Model companies want broad agents that can operate across domains. Browser companies want to orchestrate work across applications. SaaS companies want to preserve authority over domain semantics. Identity providers want to govern authorization. Every software company is going to have to decide how much semantic access to expose and to whom. Expose too little, and generic agents will operate clumsily through your UI. Expose too much, and your product risks becoming back-end infrastructure for someone else’s agentic interface. That tension is real and there’s no clean answer — but the companies that are thinking about it now are the ones that will have a position when it resolves.
What This Means for What You Build
The question to ask about any AI product — yours or someone else’s — is not “can the agent act?” It’s “does the product know what that action means?”
That’s a much higher bar than most products clear today. It requires software that can tell the agent what exists, what can be done, what each action means, what permission is required, how the result should be checked, and what happens next. Most software was not built to answer those questions. It was built assuming a human would sit there and interpret everything.
The WAT framework for structuring agent workflows is one way to think about the decomposition problem — separating workflows, agents, and tools into distinct layers so each has a clear scope. That kind of structural clarity is a prerequisite for building the meaning and authority layers on top. Similarly, Claude Code’s agentic workflow patterns — schema migrations, test loops, and the rest — are interesting precisely because they operate in an environment with rich semantic feedback. The test suite is the authority layer. It tells the agent whether its action was correct.
For non-coding work, you have to build that feedback environment deliberately. You have to define the semantic work primitives — the refund, the reschedule, the payment authorization, the compliance exception — and expose them directly rather than hiding them behind buttons and forms. You have to build the authority layer: who can do what, what’s reversible, what requires approval, what’s sandboxed. And you have to do it before you deploy agents into production, not after one of them deletes something important.
The enterprise AI agent patterns for product managers and marketing teams are useful references for what this looks like in practice — specific workflows with defined inputs, outputs, and approval points. The pattern is the same whether you’re building for a PM or a finance team: define the primitive, permission the action, make the result reviewable.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
The future isn’t an AI that gets really good at clicking buttons. That’s the bridge we’re on right now, and it’s a useful bridge. But the destination is software where the button is no longer the primitive. The primitive is the action behind it — described, permissioned, reviewable, reversible where possible, composable. Computer use gives agents hands. Semantic work primitives tell the agent what it’s touching. Authority tells the agent whether it should.
Three layers. Most products have one. The ones that build all three are the ones that will still be running in production a year from now.
