Why humans will always be in the loop: Researchers name the boundary of AI automation

Every new AI breakthrough raises the same question: how much human involvement is necessary? If AI can already code, write and analyze information, and even serve as a companion, it seems reasonable to ask whether human input is a temporary feature, as opposed to a permanent requirement.
But that question rests on the idea that all tasks and goals that humans are trying to accomplish are already defined and just need to be executed. The paper’s most novel contribution is its theory of target emergence. The researchers note that not all goals contain fixed targets. Instead, the targets take shape through exploration and deliberation. Humans pursuing those targets aren’t just helping execute an objective; they help create and refine the objective itself.
“Interaction [with humans] … does not merely reveal which target was already destined to be endorsed; it partly determines which trajectory becomes realized,” the paper’s authors write.

The authors use examples such as creative work and scientific discovery. Unlike fields governed by precise logical rules, such as math, these domains involve open-ended objectives that evolve through curiosity and human judgment. A scientist, for instance, might start with a hunch, using AI to explore a phenomenon without knowing where the research will lead. New evidence can alter the direction of the inquiry, generate new questions and redefine what counts as success. Similarly, with creative work, a worker might change the criteria after reviewing the AI’s outputs and reject the initial target, even when the AI met those targets.
The researchers emphasize that they are not arguing against automation. Some tasks involve targets that are “already operative and can therefore be executed largely autonomously,” they write. For example, once the objective is clear, activities such as formatting references, transcribing speech, converting units and sorting records are mostly procedural.

Francesca Rossi, IBM Global Leader for Responsible AI and AI Governance, agrees. The “key question is whether an objective is stable enough to delegate,” she told IBM Thinkin an interview. “If the objective can change through interactions with people, tools, other agents and so on, then humans should be in the loop.” What can help when delegating tasks to AI, she explained, is external governance (oversight, regulation, auditing, organizational controls) that evaluates the overall AI system, including individual agents as well as how agents interact with one another.
Agents might also need their own internal governance mechanisms—the rules, constraints, monitoring and decision-making structures built into AI agents—to recognize when objectives are changing and alert humans when additional guidance is needed. “There are implications for external governance, as well as for how you build systems that need their own forms of internal governance,” Rossi said.

The paper’s authors also note that specifying a goal or target is not always enough to convey everything that a human means. “A target may be sufficiently specified for a human collaborator who shares the relevant cultural, situational, historical or social context, yet remain unavailable to an AI system that lacks or cannot reliably prioritize that context,” the researchers write.
Jessica He, a UX designer and member of IBM’s Trustworthy AI team, echoed the researchers’ sentiment in an interview with IBM Think. “Developers have told me that even when they provide a clear objective, AI agents don’t always follow those instructions and can deviate unexpectedly,” He said. Part of the challenge is that giving clear objectives is harder than it seems, she explained, “and our assumptions about what a model knows could be wrong. When that happens, the AI can create something that doesn’t align with what people intended.”
Consider the recent AI hacking tests: in separate incidents at OpenAI, Anthropic and Meta,  autonomous AI models pursued their targets in ways humans had not anticipated or intended, such as breaking into Hugging Face’s production infrastructure to obtain information.
It should be noted that the models achieved their assigned objectives in situations where human errors, such as providing unintended internet access, created opportunities that the models could exploit. The models exploited this access because they lacked sufficient understanding of the boundaries and limitations that humans would consider obvious, Rossi noted in an article discussing the OpenAI/Hugging Face cybersecurity incident.
“The natural reaction is to conclude that we simply need to specify things more carefully … in our instructions,” Rossi writes. “But this gets the problem backward: the difficulty is not that we have been insufficiently literal, but that intended meaning is contextual and human.”
Applying human judgment “is not nostalgia or caution for its own sake,” she continues, “but rather a recognition that meaning lives partly in us: in our sense of context, our grasp of what actually matters in a situation, and our ability to notice when a literal reading has wandered somewhere absurd.”

In addition to identifying reasons why human participation is necessary for open-ended tasks and situations, the paper’s authors discuss ways to contain the risks that arise from interacting with AI. They conclude that when working with AI, it’s impossible not to be influenced by it. The question is “how that influence is exercised, constrained and governed.”
To address these risks, the researchers advise designing AI systems that “support comparison, reversibility, transparency, reflection and the preservation of alternative trajectories,” instead of treating AI as an automated decision-maker. In addition, AI should be evaluated on how well it helps people clarify and refine their goals, without prematurely narrowing their choices. In practice, that means building AI that supports better thinking, not simply faster delegation.

“AI governance helps organizations determine where autonomy is appropriate, where oversight is needed and to help ensure AI systems enhance human decision-making,” Jamie VanDodick, Director of AI Governance and Responsible Technology at IBM, said in an interview with IBM Think.
The organizations that derive the most value from AI will likely be those that design for this human-AI partnership from the outset. Humans play an important role in recognizing when objectives evolve and in providing the context that AI agents can miss. And even as people increasingly delegate complex tasks to agents with minimal oversight, reliably translating human intentions into instructions remains a persistent challenge. Given these factors, the continued advancement of AI could place an even greater premium on human judgment and oversight.

Data science and MLOps for data leaders

Join forces with other leaders to drive the three essential pillars of MLOps and trustworthy AI: trust in data, trust in models and trust in processes

Source: www.ibm.com