Home Daily News Roundup Robots.txt Can’t Fight Generative AI; Streamers On The Margins

Robots.txt Can’t Fight Generative AI; Streamers On The Margins

SHARE:
Comic: Room For More?

Here’s today’s AdExchanger.com news round-up… Want it by email? Sign up here.

Pub Crawling

Generative AI is breaking established internet etiquette to satisfy a bottomless appetite for training data.

For example, Microsoft-backed OpenAI and Amazon-supported Anthropic ignore robots.txt to scrape publisher sites without permission, Business Insider reports.

Robots.txt is the standard by which web publishers designate whether and which parts of their sites can be accessed by search engine indexers and other web crawlers. But there’s no legal standard or entity to enforce compliance. 

So OpenAI and Anthropic are apparently choosing to bypass robots.txt and scrape all parts of a given page, according to TollBit, a startup that facilitates paid licensing between publishers and AI firms. 

Both OpenAI and Anthropic publicly claim to respect robots.txt. But generative AI companies, including OpenAI, have argued to regulators that any publicly accessible content on the internet is open to fair use for training AI models.

If generative AI companies ignore robots.txt, might it affect whether ad tech firms honor the standard, too? Contextual data and brand safety vendors could likewise bypass robots.txt protections.

It’s another example of how little control online publishers have over who makes money off their copyrighted content.

The Streaming Squeeze

Entertainment execs are reconciling how dang hard it is to run a profitable streaming business during the upfront negotiations this year.

Streaming subscriber churn is at an all-time high while more platforms are competing for traditional TV ad budgets. 

The general consensus among executives and analysts is that four streaming services survive in the long run. Netflix is a given – it makes 28% margins on streaming operations, which no competitor comes close to, The New York Times reports. Analysts also expect Prime Video and the triumvirate of Disney, Hulu and ESPN to outlast the pack.

Which leaves Peacock, Paramount+ and Warner Bros. Discovery’s Max to fight over the scraps. 

One tip for these three, ironically, is this: Stop trying to be Netflix.

Peacock, for example, only has 34 million subs, but doesn’t need to be all things to all viewers. It’s limiting operations to North America, where it’s most profitable, for instance, says Comcast Chairman Brian Roberts.

Streaming services should also be more cautious with bundling plans, these TV insiders advise, since adding too many discounted sign-ups can painfully lower the total lifetime value of those subscribers.

There’s No Easy Flix

It isn’t all sunshine for Netflix, though.

Netflix’s next step is advertising. It’s had its chance to acquire the majority of potential subscribers; it’s in practically every global market; it cannot raise subscription prices on and on.

But, while Netflix has rock-solid cultural cache and enviable streaming profit margins, it does not have scale on the advertising side. 

In terms of total video ad spend, Netflix trails Roku, Univision and Amazon, not to mention the category leaders: Fox, NBCU, WBD, Paramount, Disney and YouTube, Bloomberg reports. 

One thing Netflix has considered is a free streaming tier with ads for some markets in Asia and Europe, according to Bloomberg. That would make sense, if it can add new pools of users who see ads and at a higher ad rate. (It’s free, after all.) Because Netflix also has self-imposed limitations on its ads business. It brags about its comparatively low ad rate and reluctance to interrupt content. 

Amazon’s video ad sales are low, too. But Amazon has strong industry inroads and seems likelier to scale that business in a big way and fast. 

Ad execs just don’t seem that urgent to be on Netflix. “There is not a clamoring for them,” says one exec.

But Wait, There’s More!

At Cannes, publishers attempt to quell marketers’ open programmatic apostasy. [Digiday]

How Amazon is using Amazon Outlet to keep shoppers around with discounts. [Business Insider]

How streaming services are faring in their fight against subscriber churn. [Marketing Brew]

Forget privacy, young internet users want to be tracked. [Financial Times]

BuzzFeed wants to sell First We Feast, which makes the YouTube show “Hot Ones,” for $70 million. But it’s struggling to find a buyer. [Bloomberg]

X CEO Linda Yaccarino fired her right-hand man Joe Benarroch, partly for failing to warn clients about the platform changing its adult content policy and partly due to pressure from X owner Elon Musk to cut costs. [Financial Times]

Must Read

Monopoly Man looks on at the DOJ vs. Google ad tech antitrust trial (comic).

2025: The Year Google Lost In Court And Won Anyway

From afar, it looks like Google had a rough year in antitrust court. But zoom in a bit and it becomes clear that the past year went about as well as Google could have hoped for.

Why 2025 Marked The End Of The Data Clean Room Era

A few years ago, “data clean rooms” were all the ad tech trades could talk about. Fast-forward to 2026, and maybe advertisers don’t need to know what a data clean room is after all.

The AI Search Reckoning Is Dismantling Open Web Traffic – And Publishers May Never Recover

Publishers have been losing 20%, 30% and in some cases even as much as 90% of their traffic and revenue over the past year due to the rise of zero-click AI search.

Privacy! Commerce! Connected TV! Read all about it. Subscribe to AdExchanger Newsletters

No Waiting for May – CES Is Where The TV Upfront Season Starts 

If any single event can be considered the jumping-off point for TV upfronts, it’s the Consumer Electronics Showcase (CES), which kicks off this week in Las Vegas, Nevada.

Comic: This Is Our Year

Comic: This Is Our Year

It’s been 15 years since this comic first ran in January 2011, and there’s something both quaint and timeless about it. Here’s to more (and more) transparency in 2026, and happy New Year!

From AI To SPO: The Top 10 AdExchanger Guest Columns Of 2025

The generative AI trend generated endless hot takes this year, but the ad industry also had plenty to say about growing competition between DSPs and SSPs. Here are AdExchanger’s top 10 most popular guest columns of 2025 and why they resonated.