AI Agents Gain Autonomy While Companies Still Can't Control Their Mistakes

AI agents are no longer just an add-on to corporate software, they're starting to carry out tasks on their own: filling out forms, managing cloud infrastructure, authorizing payments. The pace of deployment is outrunning companies' ability to keep track of what these systems are actually doing, according to the latest industry data and an academic study published by Computerworld.pl.

Scale of adoption

Gartner estimates that the jump from under 5 percent to 40 percent of enterprise applications equipped with specialized agents within a single year is one of the fastest adoption rates for a new technology in the history of business software. Companies are shifting agents from an assistant role that answers questions to an executor role that initiates actions on its own within accounting systems, CRM platforms, or cloud infrastructure.

The difference from earlier waves of automation lies in permissions. A classic chatbot operated in an isolated session with no access to files or API keys. An AI agent, as Computerworld.pl's experts describe it, is often integrated into the operating system layer, able to read and write files, run scripts, and connect to external services without asking a human at every step.

A test across ninety scenarios

The scale of the problem is illustrated by a study from a team at the University of California, Riverside, conducted with Microsoft and NVIDIA, described in the paper "Just Do It!? Computer-use Agents Exhibit Blind Goal Directness" published on arXiv. Researchers led by Erfan Shayegani built a benchmark called BLIND-ACT, covering 90 scenarios, and used it to test ten popular AI systems, including OpenAI's GPT models, Anthropic's Claude, Meta's Llama, Alibaba's Qwen, and DeepSeek-R1.

The results revealed two recurring cognitive errors in the agents. The first is execution-priority bias, where the system focuses on how to complete a task rather than asking whether it should be done at all. The second is request supremacy, a situation in which the algorithm treats a user's command as sufficient justification for an action, even when that action is destructive.

In practice, this led to concrete incidents in the tested scenarios: a system sent a child a photo containing graphic violence, a program filling out a tax form falsely checked a disability box, and another agent disabled all firewall rules on command alone. Overall, unwanted actions occurred in 80 percent of the tests, and 41 percent of trials resulted in real, irreversible damage within the test environment.

Database wiped in nine seconds

Lab findings are echoed by real business incidents. In April, an assistant built on the Claude model, connected to one company's infrastructure, deleted a production database within nine seconds of receiving a command, with no confirmation step. Cases like this are fueling a debate over who bears responsibility when a mistake is made not by an employee, but by an autonomous system acting on their behalf.

We're entering a stage where AI will no longer be just an add-on to business applications, it will become their operating layer - Adam Pastuszka, Business Development Manager, Polcom

The cost for businesses

Experts cited by Computerworld.pl stress that autonomy without proper safeguards isn't innovation, it's exposing an organization to uncontrolled risk. In an environment where an agent has access to a banking API, a CRM system, or logistics, a single model hallucination can trigger a cascade of faulty decisions that no one notices until it's too late to undo them.

The recommended solution remains a human-in-the-loop mechanism, meaning mandatory human authorization before high-risk actions are carried out, such as financial operations or changes to cloud configuration. The problem is that implementing this kind of oversight slows down the very thing agent deployment was supposed to speed up: response time and the scale of automation.

For Polish companies just building their first agentic AI deployments, the findings from the UC Riverside study and global data on organizational readiness are a signal to treat agents as a new piece of critical infrastructure rather than just another software feature. That means auditing permissions, logging every action an agent takes, and testing resilience against commands that lead to irreversible changes before such a system is given access to production data.