The bots are hungry for content.
The content owners are hungry for monetization.
And the IAB Tech Lab is working on new recommendations that will – hopefully – help everyone get fed.
On Wednesday, the Tech Lab released its guidance for bot management strategies, which is open for public comment until June 26.
The purpose of the public comment period is to invite content owners to provide feedback and ask questions so that the final document can be as helpful as possible.
Because, right now, everyone’s strategy looks a little bit different – and a lot of them aren’t as fruitful as they might sound.
For example, blocking crawlers entirely isn’t an effective strategy, said Hillary Slattery, the Tech Lab’s senior director of product management and programmatic, since doing so blocks everything, including useful bots like ads.txt crawlers that verify which companies are authorized to sell inventory.
Content owners (and this means anyone who owns content on the web, not just publishers, Slattery emphasized) that block indiscriminately eventually see their programmatic demand disappear “because their supply chains couldn’t be validated,” she said.
The problem with bot traffic “is not the existence of crawling,” according to the draft guidelines. Rather, the problem is that many content owners have “little to no understanding” of how much it costs to deliver their content to bots, nor how the content is being “accessed, interpreted or used.”
‘Crawl Me Maybe’
The IAB Tech Lab has spent the better part of the past year fighting the good fight for publisher monetization standards – and it hasn’t been easy.
Slattery joked that her nickname for the Content Monetization Protocols Working Group, which is the body that devised the draft guidelines, is “crawl me maybe,” since every online publisher seems to have a different perspective on how their content should or shouldn’t be accessible to bots.
The new document provides ways for content owners to better understand how their content is being used and what they’re receiving in return.
When asked if the guidance includes any specific suggestions regarding what content owners should and should not permit in terms of scraping, Slattery didn’t mince words.
“Hell no,” she said. “That’s between the content owners and their gods.”
What the IAB Tech Lab does want to drive home to content owners, however, is the importance of understanding how much it costs their businesses to deliver content to bots.
At a minimum, content owners need to know “which bot is crawling your content, how often and how much it’s costing you,” said Slattery – all of which can be answered by a company’s content delivery network (CDN), like Fastly or Akamai. But the caveat – and it’s a big one – is that CDNs aren’t generating those reports unless they’re asked. Their clients need to take the initiative to ask for a report that lists which bots are crawling their content, how often and how much it’s running them, said Slattery.
Because once you understand who’s hitting your site and how often, the next question is how much you’re paying to serve each of those visits.
It costs money for a website or application to deliver content, and an AI crawler doesn’t need to access a full webpage with formatting the same way a human does. One way content owners can save money is by delivering a plain-text file to bots, which is equally effective and cheaper than a page aesthetically built for a human, which might include headers, footers and imagery.
Delivering the full version of a webpage to a bot, rather than a simplified text file, isn’t just a waste of budget, but also of energy, Slattery said. It’s “melting the goddamn planet for no reason.”
Crawl or nothing thinking
Once content owners have a report from their CDN, they need to understand the “value exchange,” said Slattery, as in how much bang they’re getting for their buck when it comes to content scraping.
Claude, for instance, has an extremely stark crawl-to-refer ratio, with roughly 8,800 crawls for every referral it drives as of April – which is pretty bad, but looks great compared to the 65,000:1 ratio Business Insider reported back in January.
But some content owners might not care about referrals, and are fine with getting scraped under any circumstances as long as they get cited or paid or whatever metric matters most to them.
And if referrals are their primary focus, maybe they’d opt to turn away Claude’s crawlers but allow those coming from DuckDuckGo, which has an impressive 1.5:1 crawl-to-refer ratio.
Companies need to find the “strategic trade-off” that best suits their goals and priorities, said Slattery.
“Don’t turn everything off,” she added. “It’s not a light switch; it’s a mixing board.”
