Media agencies build audit tools to prevent AI agents from overcharging

“It can get out of control very, very quickly,” said Jonathan Whiteside, global evp of technology at Dept.

Related Insights

The Programmatic Marketer

Butler/Till extends agentic media buying tests into audio with iHeartMedia Read More

Keeping an eye on agents

Most companies (56%) are still implementing AI tools without clear usage policies, while marketing leaders specifically are less likely to assign financial controls to their team’s AI usage, according to an April survey of 1,300 senior marketers conducted by Gartner. The consultancy estimates that 60% of organizations using AI will face cost overruns related to the technology caused by lack of usage tracking. Some of the agencies in the forefront of AI agent development are attempting to get ahead of that problem.

Performance media agency Rise, part of the Quad agency group, has been testing AI media buying agents with a supermarket client since June, according to group director Klaudia Smykowska. She declined to share the name of the advertiser.

It’s been using an audit log feature developed by PubMatic — an SSP provider that’s worked with several ad agencies to test AI media buying tools, including Butler/Till and Abovo Maxlead in the Netherlands — to monitor when agents are “drifting” outside pre-set parameters and provide a log of actions taken. 

“I can go and ask, ‘why did you make that change? What was the thought process based on that initial brief?’ I can make sure everything is recorded and that we can reconstruct what happened and why,” said Smykowska.

”Any change [the agent] makes in any environment is time-stamped, stored with all the details of the change, and how that change was made — was it in the UI [for example] — is also recorded” said Harry Tong, director of sales engineering at PubMatic. 

Monitoring the specific actions taken by an agent as it attempts to execute a brief is necessary for companies testing such tools, but it’ll likely become more important as they leave the experimental phase.

Rise isn’t the only firm attempting to track its agent use, or how its agents are meeting the goals staffers set. Brainlabs, for example, tracks the use and development of AI agents by its teams across the company. Whiteside told Digiday that Dept is developing monitoring tools that can produce decision logs and quality control reports in a digestible format.

“Every deliverable has an accountable human,” he said. “It’s not an excuse to say, ‘Oh, AI did it.’ You are still accountable whether AI did it or if it was done by a junior.”

Tracking agentic costs

At the moment, PubMatic’s auditing tool doesn’t tell users what a specific path might have cost in AI tokens. Instead, Rise’s staffers triangulate the cost of its agentic tests by monitoring licensing and token spend, the time saved or spent by employees, and what proportion of a campaign budget ended up as working media. “It’s a very rough math equation at this point,” said George Forge, svp of client technology and product development.

Understanding model selection, however, could illuminate matters further. More powerful AI models use up more computing power, and cost more in tokens. Using the most powerful models by default, therefore, could cost a company by inflating token costs. 

“A lot of the [rising] token consumption costs are because people are literally not choosing the right model to meet the need of the activity,” said Gartner analyst Nicole Greene.

That’s a problem for ad agencies. Many marketing firms have allowed staff to choose the model they feel is appropriate to the task to encourage innovation and individual development. Brainlabs currently grants staff access to AI tokens using a tiered system; when an employee go over, they can be assigned more tokens if their usage merits it, after a review. “We want people to spend. We just want them to do it usefully,” said founder and CEO Daniel Gilbert. Gilbert said such monitoring is not “a nice-to have… it’s existential.”

Now, some agencies are taking a more prescriptive approach. Dept uses an “AI gateway” which removes individual staffers’ ability to choose AI models for a given task. “We’ve had some people burn through 1.5 million tokens in a day,” explained Whiteside. To stay on the right side of the company’s AI budget, staffers send their prompt, or launch their agent, and the choice of model is decided centrally according to commercial — or legal — considerations.

“We can direct, for each request, which model the request goes to,” he said. He noted that some clients were asking the agency to use specific models to ensure their data was not hosted outside their country of operation, a consideration that meant the agency had to use models that could be hosted outside the United States.

PMG developed a tool dubbed “Alli4U” that provides users a daily cap on AI token usage. Staff that routinely hit the cap might be assigned a larger token budget, or receive advice on using different AI models, said Dillon Larberg, consulting and strategy director.

“We can assign token limits or cost limits associated with tokens to users… that provides us a human in the loop moment to meet with teams or meet with individuals,” he said.

There are other ways of keeping agents within specific bounds, such as using “Skills” in ChatGPT or Claude, a means of saving specific documents or sequences and forcing an agent to use them, with the aim of reducing an agent’s capacity to invent its own course of action.

Agencies like PMG use them to “bake best practices” into an agent’s design, said Larberg. The company uses skills to “codify the steps we want the agent to take to ensure that it stays on the rails,” he added.

While agencies must be mindful of hallucination risk, and client-specific data security measures, this is principally a problem regarding economy of design — and particularly model selection. As Larberg put it: “I don’t drive an 18-wheeler to work, and I don’t go cross-country in a hybrid.”