For two years, the 404bot worked unchecked, exploiting a flaw in the ads.txt spec that cost advertisers $15 million in wasted video ads.
The 404bot served 1.5 billion video ads, according to Integral Ad Science, which revealed the scheme Tuesday with a warning for the industry, including for publishers to audit their ads.txt files.
Ads.txt was designed to stop domain spoofing by allowing publishers to list all direct partners and resellers. Advertisers can confirm they are buying inventory from sellers with legitimate access to a publisher’s inventory.
But if publishers add an untrustworthy partner, they can abuse their position as an ads.txt-verified path and spoof the publisher’s inventory.
The few hundred domains where Integral Ad Science found ads.txt files linked to the 404bot all had something in common, said Evgeny Shmelkov, head of the IAS Threat Lab. “Their ads.txt files were huge,” he said. “There were lots of parties freely trusted.”
Once the 404bot was added to a publisher’s ads.txt list, it sold legitimate ads from the publisher and ads at other sites spoofed to look like they came from the publisher’s domain. Since the partner was listed as an approved path to a publisher’s inventory, advertisers had no easy way to determine that the domain was spoofed.
As the name suggests, the 404bot relied on fake URLs. The bot would also create an article page name that didn’t exist on the publisher’s site but existed legitimately elsewhere, such as a story about the week’s highest-grossing movie.
Although some domain spoofing simply puts lipstick on a pig – repackaging human traffic to dating, porn or non-brand safe content sites as higher-value URLs – the 404bot showed the ads to bots, not humans. So publishers’ inventory was not only spoofed and devalued, but their invalid traffic rates would appear higher.
IAS notified the publishers affected by 404bot, Shmelkov said.
Publishers should audit their ads.txt files using best practices outlined by the IAB Tech Lab, he added. By closely monitoring their ads.txt files, they can avoid letting partners onto their sites that could misrepresent their inventory.
And DSPs can track fake URLs in their inventory to root out potential domain spoofing, in addition to buying only from ads.txt-compliant paths to supply.
