What Is llms.txt? Google Says You Don't Need It
— Why I Set One Up Anyway
Hi, I'm Tsubasa.
If you read articles about LLMO, you'll often see the advice "set up an llms.txt file." At the same time, on May 15, 2026, Google explicitly stated in its official documentation that llms.txt is unnecessary. So should you set one up or not? A lot of people are stuck on that question.
This site actually has an llms.txt file, and my articles have been cited by Perplexity. Drawing on that first-hand experience, I've put together Google's official position, how SEO experts reacted, what's actually happening in practice, and then the format, setup steps, and how to decide for your own site.
Contents
Google Officially Stated "llms.txt Is Not Needed" (May 15, 2026)
Start with the primary source. Google Search Central's "Optimizing your website for generative AI features on Google Search," updated on May 15, 2026 (UTC), is unambiguous about llms.txt and other AI-specific files or markup: they are not needed.
To summarize the original text: you do not need to create new machine-readable files, AI text files, markup, or Markdown to appear in Google Search's generative AI features (AI Overviews and AI Mode). Google crawls and indexes many file types beyond HTML, but that does not mean any of them receive special treatment.
G Google Search Central: Optimizing your website for generative AI features on Google Search (updated May 15, 2026) developers.google.comThe same document also addresses the terms "AEO" (Answer Engine Optimization) and "GEO" (Generative Engine Optimization) directly: from Google Search's point of view, those are still just SEO, and there is no need to treat them as separate disciplines. In short, Google has made its position clear — within the world of Google Search, there is no such thing as special optimization for the generative AI era.
John Mueller Explains the Difference Between Discovery and Functionality (May 20, 2026)
Five days after the guide was published, John Mueller of Google's Search Relations team went deeper on where llms.txt fits, in a thread on Bluesky. It was a reply to SEO consultant Lily Ray, who had asked whether it wasn't contradictory for Google to use llms.txt and Markdown on developers.google.com while telling everyone it's unnecessary for SEO.
Mueller's argument breaks down like this. His starting premise is that websites serve many purposes beyond search. From there, he argues that the llms.txt debate should separate two things: "discovery" — being found through search engines — and "functionality" — helping people who have already found the page accomplish their task. llms.txt, in his framing, is not a discovery (SEO) mechanism. It belongs on the functionality side.
To illustrate the distinction, he compared it to a CTA (call to action) on a traditional web page. You don't add a CTA for SEO; you add it to improve conversion for people who are already on the page. Likewise, llms.txt is not a mechanism for getting found — it's a mechanism for improving the experience after someone (or something) finds you. He added a caveat: it only contributes to overall results when the site's discovery rate (SEO) is already strong and the page itself is highly functional.
So where does llms.txt make sense as a functionality tool? Mueller says developer documentation sites are a valid case now that AI coding is widespread. AI tools increasingly consult documentation while generating code, and serving a Markdown version can potentially save tokens. Even then, he tempers it: AI can read HTML just fine, so this is at best a temporary aid.
For everything else, he is flatly negative. For non-developer sites, he said, there's little point in setting up llms.txt even if AI agent traffic grows in the future — making a Markdown version of a shoe's specs won't sell more shoes.
He closed the thread by saying that every site would be better off working on the SEO issues that matter today rather than preparing for an AI-agent-filled future that may or may not arrive.
B John Mueller's Bluesky post (May 20, 2026) bsky.appHow SEO Experts Reacted
Right after Google's official guide came out, reactions from SEO professionals in Japan and overseas split into two camps: "SEO fundamentals were the answer all along" versus "LLMO-style work still has value once you look beyond Google Search." Neither side is simply right or wrong — the conclusion changes depending on what you're targeting.
Note: several of the posts below are in Japanese. Browser auto-translation handles them well, so I've kept the original links.
The "No Hacks, Just Do SEO" Camp
Immediately after the guide was published, many SEO practitioners posted some version of "AEO/GEO turned out to be SEO after all."
Shoichi Hasegawa (@haseshout) shared Google's official guide and listed what you can skip: llms.txt, chunking, fake mentions, and over-investing in structured data. His summary: prioritize effective SEO strategy over AEO/GEO hacks (LLMO) (post).
Koki Okuda (@OkudaKouki) framed AI search optimization as an extension of SEO rather than a separate discipline, and added in a thread that articles written "to be picked up by AI" lose to articles that AI cites because they're useful to humans. The most dangerous move, he says, is mass-producing generic content with AI (post).
okano (@web_e_design) agreed that "AEO and GEO are all SEO," but pointed out that only people who have been doing SEO at a high level can afford to say "so no AI measures needed." He treats AI optimization as a kind of answer key for checking whether your content is genuinely user-first. He also noted that Google and ChatGPT/Claude/Perplexity are different systems, so you can't conclude structured data is useless (post).
Masahiro Miyamoto (@masa_yco) drew up a clear two-column list — do: non-commodity content grounded in first-hand experience, human-centered structure, technical SEO; skip: llms.txt, chunking, AI-specific Markdown, AI-specific writing styles. His conclusion: turn your company's primary information into content and apply SEO to it. The work hasn't changed — it's still steady accumulation (post).
Overseas, Lily Ray (@lilyraynyc) posted that SEO is the foundation, that you should create original people-first content, and that GEO/AEO hacks like chunking and llms.txt can be ignored (post). Search Engine Journal also reported that Google's guide explicitly labeled llms.txt, chunking, and AI-specific rewrites as unnecessary (article).
What these voices have in common: as far as Google Search's AI features are concerned, no special hacks are needed — strengthening conventional SEO is the right answer.
The "LLMO Work Still Has Value" Camp
On the other side, practitioners with real AI-traffic data argue that a blanket dismissal doesn't match reality.
Sanrenkyu (@san_renkyu) wrote that people loudly declaring "LLMO is meaningless" after reading Google's AI search documentation come across as lacking hands-on experience. They're only reading the parts that say there's no secret trick for AI Overviews, llms.txt isn't needed, and the basics are an extension of SEO. His counterpoint from practice: AI-referred traffic is consistently high quality, with conversion rates that are sometimes unusually high. His bottom line — optimizing specifically for AI Overviews is questionable, but making your content easy for LLMs to understand and cite matters a great deal (post).
Tsuyoshi Nakagawa (@gahutti) took the guide as a call to return to SEO fundamentals, but was explicit that cases where LLMO work increases AI-referred traffic actually exist, so denying LLMO outright contradicts observable reality. "I'm in the 'do it' camp," he wrote. The key, in his view, is building non-commodity content — things not just anyone could write — implemented along SEO principles in a form AI can parse easily (post).
Motoharu Sumi (@motoharusumi) pushed back on a viral post claiming Google had "officially declared AEO, GEO, and LLMO don't exist," calling that framing misleading. What Google actually said is that most of the hacks don't work — not that LLMO-style effort itself is pointless. His practical read: in a world where users ask questions in natural language and AI answers them, you simply do SEO adapted to that world (post).
The Two Camps Are Looking at Different Things
Put side by side, the "no hacks" camp is talking primarily about Google Search's AI features (AI Overviews and AI Mode). The "LLMO has value" camp is talking about AI-referred traffic in general, including ChatGPT, Claude, and Perplexity. The scope is different; the positions don't actually contradict each other.
Google's guide is explicit on this point too — it only speaks for Google Search. How other AI services handle llms.txt is outside Google's jurisdiction.
Three Reasons I Keep My llms.txt Anyway
Having read Google's official position, I still keep an llms.txt on this blog. Three reasons.
1. Google's Statement Only Covers Google Search
The official statement is strictly about Google Search's AI features (AI Overviews and AI Mode). Whether ChatGPT, Claude, or Perplexity reads llms.txt is each company's own decision, and Google is in no position to speak for them. Anthropic (Claude's developer) and Perplexity run their own crawlers with their own behavior, separate from Google's.
This site has been cited by Perplexity — an article explaining product-listing workflows for e-commerce. When your site shows up as a cited source, the benefit of explicitly telling AI crawlers which pages matter via llms.txt is hard to dismiss. Google Search is not the only market.
As Sanrenkyu noted above, there's practical data showing AI-referred traffic can convert at unusually high rates. How much weight you give non-Google AI traffic depends on your site's size and goals, but the fact that it's not zero is itself part of the case for keeping the file.
Mueller did say llms.txt has little value for non-developer sites — but that statement, too, lives inside Google's ecosystem. Given that this site has an actual citation record on Perplexity, applying Google's verdict wholesale to phenomena happening outside Google Search doesn't hold up logically. For how AI citations translate into behavior and traffic, see "What Economics Taught Me About AI Search Strategy."
2. In SEO, Official Statements and Reality Often Diverge
If you follow SEO discourse for any length of time, you'll see repeated cases where Google's official statements and actual search results don't line up. Google said for years that content length doesn't affect rankings — yet top results are routinely dominated by long-form articles. "Link quality over quantity" is the official line — yet sites with large raw backlink counts keep winning.
A recent example makes this concrete: the December 2025 core update. Generic "best X" and "X comparison" articles dropped hard, and the SEO industry buzzed about a new era where experience and expertise would finally win. Six months later, look at the search results — nothing has changed. The clearest case is the query "SEO company" (in Japanese, "SEO 会社"): if any query should have been first in line for an experience-and-expertise shakeout, it's this one, and yet the top results are still SEO companies' own media properties and comparison/ranking articles. The industry's excitement outran what actually happened in the SERPs. I covered this pattern in more detail in "Is Google Cracking Down on 'Best Of' Comparison Articles?"
The llms.txt "not needed" statement deserves the same skeptical reading. As Sumi pointed out, stretching Google's statement into "LLMO is officially dead" is misleading. "Google won't treat it specially" and "setting it up is pointless" are not the same claim. At minimum, the side benefits — organizing your site structure, explicitly listing your key pages — remain.
3. The Cost Is Zero and There's No Downside
Setting up llms.txt means creating one text file and uploading it to your root directory. Ten minutes, no money. And there is no penalty for having one. Google says it won't treat the file specially — it has never said the file will hurt your rankings.
When the cost is effectively zero, the expected value can only be positive. From the operator's side, that makes it a default-yes decision.
What llms.txt Is (The Basics)
From here on, this is practical information for once you've decided to set one up. llms.txt is a text file that tells AI crawlers what your site is about and how it's structured. The specification was proposed in September 2024 by Jeremy Howard of Answer.AI and collaborators, and the file goes in your site's root directory (example.com/llms.txt).
The design idea: when AI crawlers — ChatGPT's GPTBot, Anthropic's crawler for Claude, PerplexityBot — visit your site, this file lets them quickly grasp what the site contains and which pages matter. That said, as noted at the top, how much each AI company actually reads llms.txt remains largely undisclosed.
How It Differs From robots.txt
| Item | robots.txt | llms.txt |
|---|---|---|
| Audience | Search engine crawlers such as Googlebot | AI crawlers such as GPTBot and Anthropic's crawler |
| Purpose | Allow or disallow crawling | Describe the site's overview, structure, and key pages |
| Format | Its own syntax (User-agent, Disallow, etc.) | Markdown |
| History | Since 1994; a de facto standard for search engines | Proposed September 2024; adoption still limited and unofficial |
robots.txt says "here's where you may and may not go." llms.txt says "here's what this site contains." Different purposes, so having both is not contradictory. Just make sure the two are consistent — blocking AI crawlers in robots.txt while inviting them in via llms.txt is an incoherent configuration.
How to Write an llms.txt
The format is Markdown, structured like this:
# Site Name > A one- or two-sentence description of the site. ## Key Content - [Page title](URL): A brief description of that page - [Page title](URL): A brief description of that page - [Page title](URL): A brief description of that page ## Content by Category ### Category name - [Article title](URL): Description - [Article title](URL): Description
Points to keep in mind:
- Don't list every page. Limit it to the content you most want AI to read. Dumping every URL like a sitemap makes the file harder to parse, not easier.
- Keep descriptions to one sentence. The goal is for an AI to grasp the whole site at a glance.
- Organize by category. Grouping content by category helps AI understand what your site specializes in.
- Your site's language is fine. A Japanese site can write its llms.txt in Japanese; AI handles multiple languages.
A Real Example (This Site)
On this blog, llms.txt lists articles by category with a short description for each. The structure looks like this:
# Tsubasa's Notes > A working memo blog covering e-commerce operations, product photography, > image editing, retouching outsourcing, and side work. ## E-commerce / Operations - [What is "sasage" work](https://example.com/sasage.html): The photography, measurement, and copywriting tasks behind EC product listings, and where to outsource them - [Background removal pricing](https://example.com/cutout.html): Comparing outsourcing options starting at 39 yen per image ## SEO / LLMO - [SEO and LLMO service pricing](https://example.com/seo-cost.html): Budget options for small businesses - [What is LLMO](https://example.com/llmo.html): How it differs from SEO, GEO, and AIO
How to Set It Up
The short answer: the fastest route for a beginner is to ask Claude or ChatGPT to do it. Tell it "my site is at this URL — please create an llms.txt that matches its content," and it will crawl the site and draft the whole file. Then you adjust and upload. That beats writing from scratch in a text editor, and there's a certain logic to it — llms.txt was designed for AI, so having AI write it fits.
The process is three steps, no special tools:
- Create the text file. Have AI draft it, or write it yourself in any text editor (Notepad, VS Code), following the format above, and name it llms.txt.
- Upload it to your server. It goes in the site's root directory — the same level as index.html. Use an FTP client or your hosting control panel.
- Verify. Open https://yourdomain/llms.txt in a browser. If the contents display, you're done.
On WordPress, use a plugin (such as an llms.txt generator) or place the file in the WordPress root via FTP. On GitHub Pages, add the file to the repository root and push — that's the whole setup.
Why Niche Sites Shouldn't Take Google's Word at Face Value
Google's official documentation is written mostly with large sites and general cases in mind. The same SEO tactic behaves differently on a high-authority major media site versus a personal blog with few backlinks. Anyone following SEO discussions has seen this — applying Google's general guidance directly to a small independent site often doesn't fit.
Take a niche site like this one — a specialist blog for e-commerce practitioners. Competing head-on with major SEO media in Google Search is a losing proposition. The smarter premise is not to fight there, and instead to actively pursue entry points outside Google Search: citations from Perplexity and ChatGPT. In that context, a niche site has a rational reason to invest in exactly the file Google calls unnecessary. If you want to check whether it's working, Bing Webmaster Tools' AI Performance report and Google Search Console's generative AI report are the places to look.
Know Google's official position. Then judge against your own site's size, niche, and target traffic sources. That, I think, is the correct stance for a niche site operator.
L llms.txt official site llmstxt.orgFAQ
Q. How did SEO experts respond to Google's guide?
A. Two broad reactions. One camp — Shoichi Hasegawa, Masahiro Miyamoto, and overseas Lily Ray, among others — says SEO fundamentals are everything and AEO/GEO hacks are unnecessary. The other camp — Tsuyoshi Nakagawa, Sanrenkyu, and others — argues the statement only covers Google Search, and that optimizing for AI in general remains worthwhile, backed by their own traffic data. The disagreement largely comes down to scope: Google Search alone, or AI services in general.
Q. If Google says llms.txt is unnecessary, is there no point in setting one up?
A. Google's "unnecessary" applies only to Google Search's AI features (AI Overviews and AI Mode). Whether ChatGPT, Claude, or Perplexity reads llms.txt is each company's independent decision, outside Google's scope. As part of a broader AI strategy that includes services beyond Google Search, it remains a valid option.
Q. What is llms.txt?
A. A text file that describes your site's overview and structure for AI crawlers (GPTBot, Anthropic's crawler, and others). It sits in the site's root directory, is written in Markdown, and was proposed in September 2024.
Q. What's the difference between llms.txt and robots.txt?
A. robots.txt tells search engine crawlers where they may and may not crawl. llms.txt tells AI crawlers what the site contains — its overview, key content, and structure. Different purposes; both can coexist.
Q. Is setting up llms.txt free?
A. Yes. Create a text file and upload it to your root directory — no tools or plugins required. With essentially no downside, setting one up remains a reasonable call even after Google's dismissal.
Q. How has the SEO industry historically treated Google's official statements?
A. Divergence between what Google says officially and what search results show is not rare — content length and backlink volume are two long-running examples. On llms.txt, many practitioners support setting it up for ChatGPT, Claude, and Perplexity, all of which sit outside the scope of Google's statement.
🇯🇵 この記事の日本語版を読む
よく読まれている記事
