Human Workers Are the Secret Engine Behind Your AI Agents

Marissa Mayer, the former Yahoo chief executive, now leads a startup called Dazzle AI. Her company builds digital agents intended to manage complex life tasks for users. However, Mayer admits the technology often struggles with basic logic. When a user asked the system to coordinate a roof replacement, a window repair, and the purchase of a glass table, the agent confused the materials. It spent resources trying to determine if the tabletop glass would fit into the window frame. The error was caught quickly because Dazzle flagged the task and handed it to a human contractor who resolved it in seconds. Mayer noted that an artificial agent might have consumed a massive number of tokens or failed to find a solution at all.

This manual intervention is becoming a standard feature in the digital assistant industry. Meghan Joyce, the CEO of Duckbill, says current AI models face consistent barriers when tasks require real-world judgment. These barriers typically involve making phone calls, booking doctor appointments, or reserving cleaning services. Joyce describes a model similar to ride-sharing apps. Her company breaks these tasks into small, manageable pieces that an average adult can perform consistently. Companies building personal agents use Duckbill's application programming interface to delegate the work their software is likely to fumble. Joyce explained that they track the outcomes to ensure quality.

Shivani Poddar, who runs Wajo AI, is implementing a similar approach. Her team uses human workers to handle phone calls and even complete CAPTCHA tests that are designed to block bots. Poddar argues that a collaborative model combining AI and human labor is necessary to finish the job. She stated that users will abandon the technology if agents cannot complete assigned tasks. The transparency of this process is a growing concern for privacy advocates.

Meta employees recently raised concerns about the company's internal testing methods. Workers pointed out that human contractors were handling phone calls on behalf of Meta's Muse AI agent. A Meta spokesperson said at the time that this kind of testing is core to the product development process. The company currently does not use humans to complete user requests for Muse, but the practice highlights a gap in current capabilities.

Duckbill and Dazzle are careful about who they hire. Joyce said her startup performs full background checks on its 1099 contractors. She compares them to drivers for Uber or DoorDash, describing them as people in the United States seeking additional income. Duckbill also employs workers from businesses that typically provide contact center labor. Mayer said Dazzle screens for discretion and good judgment, qualities she associates with a personal assistant. She acknowledged that handling user information is a privilege and emphasized the need to compartmentalize that data afterwards.

The industry leaders describe the current market as a transitionary period. Foundation models from companies like OpenAI and Meta are improving, but they still lack common sense for complex web interactions. Mayer said Dazzle is not trying to hide the human involvement. She views the human in the loop as a benefit rather than a flaw. The goal is to deliver a service that the companies can stand behind, even if the software is not yet ready to do it alone.