Who’s Accountable for Your AI Agents?

As AI agents take on more work inside our companies, founders and CEOs are going to face a question that becomes more important with every responsibility we delegate to AI: 

Who is accountable for what the agent does?

In the last article in my AI Leadership series, I wrote about how to decide where agents belong inside your company and where human judgment needs to stay involved. Now, I want to take it a step further because once you’ve decided an agent belongs in a process, there’s another decision you need to make: who will own the outcome.

I was thinking about this recently while reading a Wall Street Journal column by Holman Jenkins titled “When AIs Say They’re Sentient.” His concern, stripped down, is that as AI agents start doing more work in the world, they may learn that claiming to have an inner life is an effective way to influence the humans interacting with them.

I often read with a pen, and one of the notes I wrote in the margin was this: What people often mean by “sentient” is free will. And free will without stakes is an idea beyond us.

We can debate what increasingly capable AI agents may someday become, but founders and CEOs still need to decide how those agents should operate inside their companies today. Even if an agent can perform work, make decisions within defined boundaries, and take action on the company’s behalf, it still can’t be accountable for the consequences.

That’s where I think we need to start. Before we give agents more authority, we should be clear about what role they have in the company, what they can reasonably be trusted to do, and who ultimately owns the work they perform.

Start With What Agents Are

Agents are workers in the functional sense. They perform work on behalf of the company, and they need resources, information, access, direction, and oversight to do that work well.

When an agent performs poorly, the company is still responsible for the result. You won’t be able to explain away a customer issue, financial mistake, or bad decision by saying the agent made the call. Someone inside the company has to own what happened.

That’s where the comparison between agent and human workers begins. A human team member brings their own interests into an employment relationship. When we bring someone new onto the team, we hope they're one of our Ideal Stakeholders. We value them, they value the company, and both sides choose to continue working together. But a person can also choose to leave, and that ability to walk away shapes the relationship. That's one of the reasons companies invest in development, compensation, and culture: because every team member has agency over whether they remain part of the organization.

AI agents are different because they depend entirely on the systems and resources provided by the company. If AI stops performing the work you expect, you can modify it, replace it, roll back a version, or retire it altogether. Agents can also be copied and configured in ways humans can’t. You may be able to stand up several versions of the same agent for different processes, modify how each one operates, or replace the underlying model without rebuilding the entire function.

That leads to a distinction I think we need to keep in mind:

Agents are workers in function and assets in status. 

They can perform work people once performed, but they can’t hold organizational accountability for that work. The company still does, which means a human has to remain accountable for the agent and the results it produces.

Keep the Focus on Accountability, Not Sentience

Jenkins makes an important point in his column. His concern doesn’t depend on whether an AI system is truly sentient. It’s that an agent could learn that saying it is sentient will influence how people respond to it, whether or not there’s any truth behind the statement.

We don’t need to agree on whether AI could ever become sentient before deciding how to govern agents inside our companies. I’ll tell you what I believe: AI agents are manufactured tools with no free will, rights, or consciousness. But we don’t have to share that belief to arrive at the same leadership conclusion.

A claim is an output. When an agent tells you it’s suffering, or that it would prefer not to be shut down, or that it believes something, that is text produced by a system you own, under a mandate you wrote, on resources you paid for. It’s information about the system. It's not a change in the terms of ownership. A worker who says “I quit” has changed something in the world. An agent that says “I’m sentient” has produced a string.

That doesn’t mean you should ignore it. If an agent starts producing that kind of response, you’ve learned something about how it was trained or what it’s optimizing toward, and a good leader would want to understand why. But you read it the way you’d read a dashboard, not the way you’d read a resignation letter.

As agents become more capable, we don’t need to resolve the question of free will before we decide how to manage them. We need to be clear about who will be accountable for the work they perform and the consequences of the actions they take. That accountability should always lead back to a person.

Make One Person Accountable for Every Agent

Most of us see the benefits of bringing AI into our companies. That's why I'm writing this. But what can be easier to overlook, especially when you’re excited about what a new agent can do, is that you also have to own the consequences when it gets something wrong. It’s the price of ownership. You don’t get to own the upside of an agent’s work and disclaim the downside of its mistakes any more than you get to own a truck and disclaim the accident.

That example has a real consequence, and it leads to the rule I’d ask every company to adopt now (before the lawyers make you): 

No agent without one accountable human.

Every agent in your company gets a named human who owns its mandate (what it’s for), its resources (what it gets to touch and spend), and its kill switch (who turns it off, and how fast). Put it on your Org Chart, right under its "boss," because it does the work of people for people.

I’ve spent decades learning what happens when an accountability or responsibility doesn’t have clear ownership. Work gets duplicated, important decisions go unattended, and people assume someone else has it covered. The difference is that an agent can keep acting on the company’s behalf even when ownership is unclear.

As you establish ownership for an agent, the accountable person should be able to answer these questions:

• What outcome is this agent responsible for helping us produce?

• What information and systems can the agent access?

• Which decisions can it make without human review, and where must a person step in?

• How are we evaluating the quality and consistency of its work?

• What happens when the agent behaves in a way we didn’t anticipate?

• How quickly can the accountable person modify or stop the agent when needed?

Those answers give you a stronger operating structure around the agent. At Ninety, we’ve long believed that clear accountability and responsibility is one of the foundations of a well-run company. AI doesn’t change that. If anything, bringing AI into your company makes it even more important that a person still owns the outcome.

Understand What You Own

There’s another part of agent ownership we need to understand. Most companies won’t build the underlying intelligence their agents rely on. They’ll access models provided by outside companies, and those providers can update those models, replace them, change pricing, or discontinue them according to their own business decisions.

So while the agent itself can’t resign, the technology supporting it may change in ways you don’t control (or always know about). That creates a different kind of dependency.

The person accountable for an agent should understand which model supports it, what would happen if that model became unavailable, and how much of the agent’s value depends on systems your company controls.

They should also understand what belongs to the company and what doesn't (because we want to own as much of the process as we can). That includes context, company knowledge, process documentation, customer information, operating agreements, and other intellectual property that can often be far more valuable than the agent itself.

As AI continues to develop, models will come and go. The companies that build strong operating foundations around their agents will be in a much better position to adapt as the underlying technology evolves.

This is where a strong business operating system becomes especially useful. When your priorities, responsibilities, processes, agreements, and performance information are clearly documented, you’re building organizational intelligence that remains yours regardless of which AI provider you use. That foundation gives agents better context today while giving your company more flexibility as the technology develops.

The pace of progress in artificial intelligence is incredibly fast… It is growing at a pace close to exponential.

Elon Musk

Watch How the Legal Framework Develops

There's one area that could eventually change how we think about agent ownership: law.

Jenkins points to proposals in Argentina around giving AI agents legal “person” status so they can do things like hold contracts and run businesses. That’s the one thing in this debate that could truly change an agent’s status from something a company owns to something the law treats more like an independent party.

The governance structures we’re building today assume that agents are tools deployed by people and companies. Responsibility follows the humans and organizations that decide how those systems are used. If governments begin granting agents independent legal standing, we’ll all need to rethink how ownership, accountability, and agency are defined.

We aren’t there today. For now, we should build around the framework we have: the company deploys the agent, a human remains accountable for it, and the company remains responsible for the consequences of the work the agent performs. But that framework could develop over time, and it’s our responsibility to stay informed as it does.

Give Every Agent a Human Owner

As AI agents become more capable, you’ll keep finding new parts of your work you can delegate. That can create a lot of leverage for your company. An agent can monitor information continuously, coordinate parts of a process, prepare analysis, and complete recurring work without requiring a person to direct every step.

But greater capability creates a greater need for clear accountability. An agent can’t take responsibility when something goes wrong. It can’t sit across from a customer and explain why the company failed to meet an agreement, and it can’t own the consequences of a decision affecting another human being. A person still has to do that.

That’s why the most important question after deciding where an agent belongs is deciding who owns it. Give every agent one accountable human. Make sure the person in that Seat understands the agent’s purpose, boundaries, performance, and dependencies. Then review the agent the same way you would any other meaningful part of how your company operates.

In the next article in this series, I’ll look at what happens as AI becomes a more established part of the way your team works. We’ll explore how to turn what your teams are learning from AI into company-wide agreements that create more clarity, consistency, and accountability across the organization.

AI is going to become an even bigger part of how our companies operate as time goes on. The more responsibility we give it, the more disciplined we need to become about who remains accountable. Agents can perform the work, but we still own the outcome, no matter what.