---
title: "Reddit's Paywall Doesn't Control Distribution. It Controls Price."
site: "Silverback Marketing"
domain: "silverbackmarketing.com"
type: "Blog"
description: "Reddit blocks nearly every crawler and charges Google $60M a year. Its content still reaches AI answers through Google. What that means for your strategy."
last_updated: "2026-09-10"
canonical: "https://silverbackmarketing.com/reddit-blocks-ai-crawlers"
markdown: "https://silverbackmarketing.com/reddit-blocks-ai-crawlers.md"
---


Audio narration (15:28): https://200xpimeicclmpf9.public.blob.vercel-storage.com/audio/reddit-blocks-ai-crawlers.65056574c828dad0.mp3


Reddit runs the most aggressive content paywall on the open web. Since July 1, 2024 its robots.txt has [blocked most search and AI crawlers](https://techcrunch.com/2024/06/25/reddits-upcoming-changes-attempt-to-safeguard-the-platform-against-ai-crawlers/), including [Bing, which Microsoft confirmed](https://searchengineland.com/microsoft-confirms-reddit-blocked-bing-search-444385). Google is the notable exception, because Google [signed a licensing deal](https://searchengineland.com/reddit-google-ai-content-licensing-deal-437782) reported at roughly **60 million dollars a year**.

It has not worked the way it looks like it works. In October 2025 Reddit [sued Perplexity, SerpApi, Oxylabs and AWMProxy](https://searchengineland.com/reddit-sues-perplexity-serpapi-scraping-google-463681), alleging they scraped Reddit content out of Google's results rather than from Reddit. Reddit says it planted a test post visible only to Google's crawler, and it surfaced in Perplexity **within hours**.

The clearest proof came from an accident. When Google removed the num=100 parameter around September 10, 2025, Reddit's ChatGPT citation share [fell sharply](https://www.searchenginejournal.com/why-reddits-chatgpt-citation-drop-isnt-fully-explained/586479/). A change inside Google moved Reddit's visibility inside OpenAI's product. That only happens if the content was travelling through Google.

Meanwhile the thing Reddit is charging for is losing value. Q2 2026 revenue rose **61 percent to 805 million dollars** and the stock [still fell](https://www.cnbc.com/2026/07/30/reddit-rddt-q2-2026-earnings-report.html), because search referrals went "choppy." The paywall never controlled who got the data. It set the price. And the buyer is now reconsidering.

## What Reddit actually did

On July 1, 2024, Reddit rewrote its robots.txt to shut out nearly everything. Not a tightening, a wall. Search engines, AI crawlers, commercial scrapers: blocked by default. Reddit carved out an exception for what it called good faith actors, naming the Internet Archive, for non-commercial use.

Microsoft was not amused. Bing's crawler was among the blocked, and the two companies [argued publicly about why](https://www.404media.co/microsoft-and-reddit-are-fighting-about-why-bings-crawler-is-blocked-on-reddit/). Reddit CEO Steve Huffman was unambiguous about the terms, telling [Microsoft and AI search engines to pay](https://searchengineland.com/reddit-ceo-microsoft-ai-search-content-444616) if they wanted access.

This was, on its face, the boldest publisher move of the AI era. Every other site was arguing about whether to block GPTBot. Reddit blocked essentially everyone and put a price list on the door.

It is also the single best real-world test of the strategy half the publishing industry has been debating since, and it comes with unusually good conditions. Reddit had scale, a corpus nobody else owned, a public listing that forced it to report the consequences quarterly, and enough legal budget to sue anyone who went around it. If pay-to-crawl works anywhere, it works here. That is exactly why the results matter to everyone who does not own Reddit.

| When | What happened |
| --- | --- |
| Feb 2024 | Google signs a content licensing deal, reported at roughly 60 million dollars a year |
| Jul 1, 2024 | Reddit's robots.txt blocks most search and AI crawlers, Bing included |
| Jun 2025 | Reddit sues Anthropic over continued scraping |
| Sep 10, 2025 | Google removes the num=100 parameter; Reddit's ChatGPT citation share falls within weeks |
| Oct 2025 | Reddit sues Perplexity, SerpApi, Oxylabs and AWMProxy for scraping Google's results |
| Jul 22, 2026 | Report that Reddit may not renew the Google deal; the stock sinks |
| Jul 30, 2026 | Q2 revenue up 61 percent, net income up 183 percent; the stock falls on "choppy" search referrals |
| Aug 2026 | Promptwatch measures ChatGPT Search citation share dropping from 3.83 to 0.52 percent, cause unexplained |

## The exception that pays

Google kept crawling because Google had already signed. The February 2024 agreement, reported at around 60 million dollars annually, gave Google continued access and the right to train on Reddit content.

The commercial logic was clean. Reddit converts an unpriced input into a licensed one. Google removes uncertainty about a corpus it considers valuable. Both sides get something. And for two years the arrangement looked like proof that content licensing could work at scale.

It also created an asymmetry worth naming. One company crawls; everyone else is locked out unless they pay. That is a competitive structure as much as a commercial one, and it puts Reddit in the unusual position of having made a single search engine the sole legitimate gateway to its content. Reddit and Google have since [reportedly discussed deepening the partnership](https://searchengineland.com/reddit-google-deepen-ai-partnership-report-462136), which would weave Reddit discussions further into Google's AI products.

Notice what that does to Reddit's negotiating position over time. The more thoroughly Reddit's content is embedded in Google's AI surfaces, the more Reddit depends on the counterparty it is trying to charge. Leverage that comes from being indispensable to one buyer is leverage right up until the moment you need to walk away.

## The content escapes anyway

Here is where the strategy meets reality.

In October 2025, Reddit filed suit in the Southern District of New York against four companies. The allegation was not that they crawled Reddit. It was that they took Reddit content out of Google's search results, at what Reddit called industrial scale, then resold or reused it. As [CNBC framed it at the time](https://www.cnbc.com/2025/10/23/reddit-user-data-battle-ai-industry-sues-perplexity-scraping-posts-openai-chatgpt-google-gemini-lawsuit.html), the case expanded Reddit's data rights battle from the companies training on its content to the intermediaries supplying them.

That distinction is the whole story in miniature. Reddit had already solved the crawling problem. What it had not solved, and arguably cannot solve, is the resale problem.

The detail that makes the complaint stick is the trap. Reddit says it created a test post configured to be visible only to Google's crawler. Within hours, according to the filing, that post appeared in Perplexity's results. If accurate, that is not an inference about scraping. It is a tracer dye.

Perplexity rejected the characterisation, calling the suit "a show of force in Reddit's training data negotiations with Google and OpenAI" and arguing that Reddit's position is "the opposite of an open internet." Reddit had already [sued Anthropic in June 2025](https://techcrunch.com/2025/06/04/reddit-sues-anthropic-for-allegedly-not-paying-for-training-data/), alleging its bots kept scraping more than 100,000 times after Anthropic said it had blocked them.

The plumbing is visible from the other end too. Search Engine Land has reported that [SerpApi was or is an OpenAI customer](https://searchengineland.com/openai-chatgpt-serpapi-google-search-results-461226), which helped explain how Google search results turned up inside ChatGPT. Separately, researchers reading ChatGPT's network traffic identified [retrieval pipelines labelled bright and oxylabs](https://searchengineland.com/chatgpt-retrieval-stack-index-cache-pages-485036) and described them as scraped Google results purchased live. We went through that retrieval stack piece by piece in [ChatGPT built its own search index](/chatgpt-search-index).

Be careful with the inference here, because it matters. OpenAI holds its own Reddit licence, so its position is not Perplexity's. And a pipeline name in a network capture is not proof of what any given result contained. What the evidence does establish is the shape of the market: Google's index is a distribution layer, third parties sell access to it, and content inside it travels well beyond the parties who paid the original owner.

## The accident that proved it

The strongest evidence is not the lawsuit. It is something nobody engineered.

Around September 10, 2025, Google [removed the num=100 search parameter](https://www.searchenginejournal.com/google-modifies-search-results-parameter-affecting-seo-tools/556080/), which many SEO tools and data providers used to pull up to 100 results per request. It was a housekeeping change to Google's own product.

Reddit's citation share in ChatGPT collapsed within weeks, and Reddit's stock moved with the coverage. G2 growth advisor Kevin Indig argued, in hedged terms, that the num=100 removal made it harder for outside data providers to reach the deeper results where Reddit threads typically sit, and that this rather than any decision by OpenAI or Reddit was the likely cause.

Sit with that. Google changed a URL parameter. Reddit's visibility changed inside OpenAI's product. Neither Reddit nor OpenAI did anything.

That is only possible if Reddit's presence in ChatGPT was flowing through Google's results. Which means Reddit's robots.txt, the most aggressive on the open web, was not governing where Reddit content ended up. Google's index was.

A second decline followed in August 2026. The GEO analytics firm Promptwatch measured Reddit's ChatGPT Search citation share dropping from an average of 3.83 percent between July 18 and August 7 to 0.52 percent between August 14 and 17, an 86 percent relative fall.

Search Engine Journal's coverage is worth reading for its restraint. The popular explanation was a change on August 8 in how ChatGPT builds its background searches, when queries scoped to a single domain jumped from 0.37 percent to 16.8 percent of all fan-out queries in a day and the average number of fan-outs per response nearly doubled. But Reddit's share did not collapse until six days later, and Promptwatch itself called the magnitude provisional and said it could not rule out a data-collection issue on its own end.

So the second episode is unexplained, which is its own kind of finding. The honest summary is that Reddit's AI visibility has now swung violently twice, that on neither occasion did anyone at Reddit touch anything, and that on the one occasion where a cause was identified it was a housekeeping change at a company Reddit does not control.

## The asset is worth less than it was

While all this was happening, the value of the thing Reddit sells has been quietly eroding.

Reddit's second quarter of 2026 was, on paper, excellent.

| Q2 2026 | Result |
| --- | --- |
| Revenue | 805 million dollars, up 61 percent |
| Net income | 253 million dollars, up 183 percent |
| Guidance | 860 to 870 million dollars, ahead of expectations |
| US daily active uniques | 53.5 million to 53.2 million |
| Stock | Fell sharply anyway |

The cause was one line in Huffman's shareholder letter, [reported by TechCrunch](https://techcrunch.com/2026/07/30/reddit-reports-a-solid-quarter-but-shows-signs-of-ais-impact/):

> Search referrals were choppy in the quarter, and traffic was more volatile later in the quarter, but the bigger picture is unchanged: the commercial business is strong.

On the earnings call an analyst put it bluntly, saying investors sense "a user problem, especially in the U.S.," and spelling out the mechanism: logged-out traffic comes under pressure as search shifts to AI, which makes it harder to convert logged-out visitors into logged-in members. Then came the question that matters for this whole argument, whether Huffman saw "any world where you're not licensing data to Google and OpenAI next year." His answer was not reassuring to anyone modelling that revenue line: "I don't think there's a binary outcome."

It is worth being fair to Reddit's counterargument, because it is not weak. Huffman's position is that people come to Reddit for something a summary cannot deliver, telling CNBC that users "want the human perspective, the multiple perspectives, the lived experience that we provide" and that "people want Reddit, they don't necessarily want a summary of Reddit." The company has been pushing to convert web visitors into direct traffic for exactly this reason. If that works, the licensing revenue becomes a bonus rather than a dependency. The Q2 numbers say the commercial engine is strong. The US user line says the conversion is not finished.

He has been consistent about the underlying complaint. AI Overviews, he told CNBC, has ["yet to make a similar level of positive impact"](https://www.cnbc.com/2026/07/30/reddit-ceo-says-googles-ai-overviews-cant-replace-10-blue-links-.html) to the ten blue links, and he has said flatly that [chatbots are not a traffic driver](https://techcrunch.com/2025/10/31/reddit-ceo-says-chatbots-are-not-a-traffic-driver/).

The industry data backs him. Cloudflare's crawl-to-refer ratios, [reported by Search Engine Land](https://searchengineland.com/cloudflare-pay-per-crawl-seo-geo-458310), and TollBit's referral finding, also [via Search Engine Land](https://searchengineland.com/google-more-visitors-ai-systems-report-462905), tell the same story.

| Platform | Crawls per referred visitor |
| --- | --- |
| Google | Roughly 18 to 1 |
| OpenAI | 1,500 to 1 |
| Anthropic | 60,000 to 1 |

TollBit's version of the same point: Google sends 831 times more visitors than AI systems do.

Then in July 2026 came the report that Reddit [might not renew the Google deal at all](https://www.cnbc.com/2026/07/22/reddit-stock-google-ai-content-deal.html). The stock sank.

So Reddit is in an awkward position. Its leverage over Google was always partly the threat of withdrawal. But withdrawal costs Reddit the referral traffic it still depends on, from the one crawler with a defensible ratio, at a moment when its US user growth has stalled. And if it walks, the content does not become unreachable. It becomes unpriced, reachable by the same indirect routes the lawsuits are about.

## What this means if you are not Reddit

Most sites will never be offered a licensing deal. That does not make this irrelevant, because Reddit ran the experiment everyone else is theorising about, with more leverage than any of us will ever have. We argued a year ago that [Reddit is to GEO what DMOZ was to SEO](/reddit-ai-citations-geo-dmoz-lesson), a proxy signal with an expiry date. This is what the expiry looks like from the inside.

Three findings transfer.

- **Blocking crawlers controls your relationship with a company, not the location of your content.** If your pages are in Google's index and Google's results are commercially scrapeable, your content has a distribution path you do not administer. Robots.txt is a request to a specific bot, not a property boundary.
- **The value of being crawled and the value of being cited have decoupled, and are moving in opposite directions.** Reddit is being fetched more and credited less. Research on ChatGPT's traffic found a conversation in which [84 of 221 retrieval-pool entries were Reddit threads and none were credited](https://www.searchenginejournal.com/chatgpt-rebuilt-its-search-tool-i-read-the-new-language-it-speaks/586710/) in the answer. Content that shapes an answer without appearing in it is the worst of both worlds: you pay the serving costs and get no attribution.
- **Blocking without scarcity is not a strategy.** If a company with Reddit's leverage cannot make pay-to-crawl hold cleanly, a mid-market brand blocking GPTBot is opting out of a surface while its competitors stay in it, and gaining nothing in return. Reddit could block because Reddit had something a search engine would pay 60 million dollars a year to keep. The blocking was never the leverage. The scarcity was.

There is a fourth lesson, and it is the uncomfortable one. Reddit's visibility in AI answers has now moved twice for reasons that originated at Google, not at Reddit and not at the AI companies. If the most valuable user-generated corpus on the internet cannot hold its position in a third party's product against a routine parameter change, then nobody's AI visibility is as durable as their reporting implies. That is not an argument for giving up. It is an argument for building demand that survives a bad quarter on any single surface.

## What we would do first

- **Stop treating Reddit citation share as a KPI.** Two collapses in twelve months, neither triggered by Reddit and one triggered by a Google parameter change, is not a metric. It is weather. If your reporting has a Reddit visibility line, annotate it with the August 2026 measurement caveat and stop making budget decisions on its slope.
- **Keep doing Reddit for the reason it worked before anyone measured citations.** Real participation in communities where your buyers actually are still reaches those buyers, and years of threads feed the training data that shapes which brands a model names before it searches anything. What has become unreliable is the receipt, not the influence.
- **Audit your own crawler permissions with the distinction this story exposes.** Decide separately, per bot, whether you are making a training decision or a visibility decision, and accept that neither controls what happens to content already sitting in Google's index. If that reasoning is new to you, it is the same argument we made in [AEO vs GEO vs SEO](/aeo-vs-geo-vs-seo).
- **Notice the dependency Reddit is currently discovering in public.** A business whose visibility rests on another company's contract renewal has a single point of failure it does not control and cannot see the terms of. That is worth checking for in your own demand model, because the answer is rarely zero.

The useful question is not whether Reddit wins this negotiation. It is what your own plan looks like if the surface you rely on most changes its terms without asking you. Reddit is finding out that a wall only works when the thing behind it cannot be reached another way. Most of us do not have a wall to begin with, which makes the second half of that sentence the part to plan around. If you want that plan pressure-tested, it is the shape of our [SEO and GEO work](/services/seo-and-geo).

A note on sourcing: where a figure originates from a commercial analytics vendor, we credit the vendor by name in the text and link to the independent trade publication that reported it, rather than to the vendor's own page. Reddit's robots.txt is described here from that trade reporting rather than from a direct fetch.

## About the author

### Russ Wittmann

SVP of Technology, Silverback Marketing

Helping brands become the source AI cites. GEO · AIO · Technical SEO · AI search strategy.

[LinkedIn](https://www.linkedin.com/in/russwittmann/)

[X](https://x.com/russwittmann)

FAQ

## Frequently Asked Questions

Common questions about GEO, SEO, and AI-driven search visibility.

### Does Reddit block AI crawlers in its robots.txt?

Yes. Reddit updated its robots.txt on July 1, 2024 to block most search engine and AI crawlers from accessing its content. Microsoft confirmed that Bing's crawler was among those blocked. Reddit said good faith actors such as the Internet Archive would keep access for non-commercial use. CEO Steve Huffman has been explicit that the block stands unless a company signs a content licensing deal, telling Microsoft and AI search engines to pay for Reddit's content.

### Why can Google still crawl Reddit when others cannot?

Because Google pays. Reddit signed a content licensing agreement with Google in February 2024, reported at roughly 60 million dollars a year, which gave Google continued crawl access and the right to train on Reddit content. That arrangement is why Reddit results remained prominent in Google Search while Bing and most AI crawlers were shut out. In July 2026, reports emerged that Reddit might not renew the deal, and Reddit's stock fell on the news.

### If Reddit blocks a crawler, does its content still reach AI systems?

Often yes, indirectly. In October 2025 Reddit sued Perplexity, SerpApi, Oxylabs and AWMProxy in the Southern District of New York, alleging they scraped Reddit data out of Google's search results at industrial scale rather than from Reddit itself. Reddit said it set a trap by creating a test post visible only to Google's crawler, and that the post appeared in Perplexity results within hours. Blocking a crawler at the source does not prevent content that is already in Google's index from being lifted out of it.

### Why did Reddit's ChatGPT citations drop in 2025 and 2026?

The 2025 drop appears to have been caused by a change at Google, not at Reddit or OpenAI. Around September 10, 2025 Google removed the num=100 search parameter that many data providers relied on to retrieve up to 100 results per request, and G2 growth advisor Kevin Indig argued this made it harder to reach the deeper results where Reddit threads often sit. A second decline followed in August 2026, when the GEO analytics firm Promptwatch measured Reddit's ChatGPT Search citation share falling from an average of 3.83 percent to 0.52 percent. Promptwatch called the size of that drop provisional and said it could not rule out a data-collection issue on its own end.

### Is Reddit's data licensing business still growing?

The commercial business is growing fast, but investors are nervous about the traffic underneath it. Reddit reported second quarter 2026 revenue of 805 million dollars, up 61 percent year over year, and net income of 253 million dollars, up 183 percent, beating expectations along with its guidance. The stock still fell sharply after Huffman told shareholders that search referrals were choppy during the quarter, and US daily active uniques slipped from 53.5 million to 53.2 million.

### Should marketers still invest in Reddit for AI visibility?

Treat Reddit as an audience channel first and a citation channel second. Research reading ChatGPT's network traffic found one conversation where 84 of 221 entries in the retrieval pool were Reddit threads and none of them were credited in the answer, meaning Reddit content can shape a response without producing a measurable citation. Genuine community participation still reaches buyers and still feeds the training data that shapes which brands models name. What has become unreliable is the measurable citation payoff, so do not build a visibility model that depends on it.

## Sources

1. [TechCrunch: Reddit's upcoming changes attempt to safeguard the platform against AI crawlers](https://techcrunch.com/2024/06/25/reddits-upcoming-changes-attempt-to-safeguard-the-platform-against-ai-crawlers/)
2. [Search Engine Land: Microsoft confirms Reddit blocked Bing Search](https://searchengineland.com/microsoft-confirms-reddit-blocked-bing-search-444385)
3. [404 Media: Microsoft and Reddit are fighting about why Bing's crawler is blocked on Reddit](https://www.404media.co/microsoft-and-reddit-are-fighting-about-why-bings-crawler-is-blocked-on-reddit/)
4. [Search Engine Land: Reddit CEO to Microsoft and AI search engines, pay for our content](https://searchengineland.com/reddit-ceo-microsoft-ai-search-content-444616)
5. [Search Engine Land: Report, Reddit signs AI content licensing deal with Google](https://searchengineland.com/reddit-google-ai-content-licensing-deal-437782)
6. [Search Engine Land: Reddit sues Perplexity, SerpApi over scraping Google Search data](https://searchengineland.com/reddit-sues-perplexity-serpapi-scraping-google-463681)
7. [CNBC: Reddit accuses Perplexity of stealing user posts, expanding data rights battle with AI industry](https://www.cnbc.com/2025/10/23/reddit-user-data-battle-ai-industry-sues-perplexity-scraping-posts-openai-chatgpt-google-gemini-lawsuit.html)
8. [TechCrunch: Reddit sues Anthropic for allegedly not paying for training data](https://techcrunch.com/2025/06/04/reddit-sues-anthropic-for-allegedly-not-paying-for-training-data/)
9. [Search Engine Land: How Google search results sometimes appeared in ChatGPT](https://searchengineland.com/openai-chatgpt-serpapi-google-search-results-461226)
10. [Search Engine Land: Inside ChatGPT's retrieval stack, the index, cache, and pages it actually reads](https://searchengineland.com/chatgpt-retrieval-stack-index-cache-pages-485036)
11. [Search Engine Journal: Why Reddit's ChatGPT citation drop isn't fully explained](https://www.searchenginejournal.com/why-reddits-chatgpt-citation-drop-isnt-fully-explained/586479/)
12. [Search Engine Journal: Google modifies search results parameter affecting SEO tools](https://www.searchenginejournal.com/google-modifies-search-results-parameter-affecting-seo-tools/556080/)
13. [Search Engine Journal: ChatGPT rebuilt its search tool, I read the new language it speaks](https://www.searchenginejournal.com/chatgpt-rebuilt-its-search-tool-i-read-the-new-language-it-speaks/586710/)
14. [CNBC: Reddit stock sinks on report it may not renew Google AI content deal](https://www.cnbc.com/2026/07/22/reddit-stock-google-ai-content-deal.html)
15. [TechCrunch: Reddit reports a solid quarter but shows signs of AI's impact](https://techcrunch.com/2026/07/30/reddit-reports-a-solid-quarter-but-shows-signs-of-ais-impact/)
16. [CNBC: Reddit shares sink on choppy search referrals even as results blow past estimates](https://www.cnbc.com/2026/07/30/reddit-rddt-q2-2026-earnings-report.html)
17. [CNBC: Reddit CEO says Google's AI Overviews can't replace 10 blue links for referral traffic](https://www.cnbc.com/2026/07/30/reddit-ceo-says-googles-ai-overviews-cant-replace-10-blue-links-.html)
18. [TechCrunch: Reddit CEO says chatbots are not a traffic driver](https://techcrunch.com/2025/10/31/reddit-ceo-says-chatbots-are-not-a-traffic-driver/)
19. [Search Engine Land: Cloudflare pay per crawl and what it means for SEO and GEO](https://searchengineland.com/cloudflare-pay-per-crawl-seo-geo-458310)
20. [Search Engine Land: Google sends 831x more visitors than AI systems, report](https://searchengineland.com/google-more-visitors-ai-systems-report-462905)
21. [Search Engine Land: Reddit and Google in talks to deepen AI partnership, report](https://searchengineland.com/reddit-google-deepen-ai-partnership-report-462136)