In June 2026, Ahrefs went through the server logs of 137,210 websites. Of the sites that had an llms.txt file, 97% got zero requests for it in an entire month. Not from ChatGPT, not from Google, not even from a curious human. Even so, more sites keep adding an llms.txt file every month. Hostinger offers a one-click generator, and Google’s Chrome team now checks for the file in Lighthouse. So which is it? Is llms.txt essential AI infrastructure, or the meta keywords tag of the AI era? My honest answer, from the GEO side of the fence: both. It depends on who you think is reading the file.
An llms.txt file is a Markdown file at your site’s root that gives AI agents a curated map of your most important pages. It does not improve Google rankings or AI Overview citations, and most AI crawlers ignore it today. Its real value is helping AI agents and coding tools understand your site faster, at almost zero cost.
What is an llms.txt file, exactly?
An llms.txt file is a plain Markdown document published at yourdomain.com/llms.txt. It tells large language models what your site is and which pages matter most. Jeremy Howard of Answer.AI proposed it on September 3, 2024, and the official llms.txt specification is still an open community proposal. No standards body has adopted it.
The format is simple:
- An H1 with your site or company name. This is the only required element.
- A blockquote summary of what you do, in a sentence or two.
- Optional context paragraphs (who you serve, where you operate, what you don’t do).
- H2 sections listing your key pages as Markdown links, each with a short description.
- An “Optional” section for secondary links an AI can skip when its context window is tight.
There’s also a companion file, llms-full.txt, which puts your actual page content into one Markdown file instead of just linking to it.
How is llms.txt different from robots.txt and sitemap.xml?
| File | What it does | Who reads it | Enforced? |
|---|---|---|---|
| robots.txt | Tells crawlers what they may or may not access | Every major search and AI crawler | Respected by convention |
| sitemap.xml | Lists every indexable URL | Search engines | Discovery aid |
| llms.txt | Recommends what an AI should read first, in a format it parses easily | AI agents and coding tools, rarely crawlers | Purely advisory |
The key difference: robots.txt controls access, a sitemap lists everything, and llms.txt is a curated reading list. Nothing forces anyone to read it.
Does Google use the llms.txt file for AI Overviews or rankings?
No. Google has been consistent about this for more than a year.
In June 2025, Google’s John Mueller wrote that no AI system currently uses llms.txt, adding that server logs show the bots don’t even request it. Gary Illyes later confirmed at Search Central Live that Google wasn’t pursuing the format. Then on May 15, 2026, Google published its AI optimization guide, which lists llms.txt alongside content chunking and special schema as tactics you don’t need for AI Overviews or AI Mode. A June 2026 update went further and stated that llms.txt files won’t help or hurt your search rankings.
Google’s reasoning makes sense to me. An llms.txt file is self-reported. You can write anything in it, just like the old meta keywords tag, so a search engine has no reason to trust it as a ranking signal. AI Overviews and AI Mode use the same index and core ranking systems as classic Google Search. If you want to show up there, the work is still SEO.
Do ChatGPT, Claude, or Perplexity actually read llms.txt?
Rarely, and almost never their search crawlers. The data from 2025 to 2026 is consistent across several independent studies:
- Ahrefs (June 2026, 137,210 domains): Ahrefs’ llms.txt server-log study found that 28% of domains publish the file, but 97% of those files got zero requests in May 2026. AI retrieval bots made just 1.1% of all requests. SEO audit tools were the biggest readers at 21.7%. AI agents were the largest AI category at 10.5%, and Claude Code made more requests than almost every AI crawler.
- SE Ranking (November 2025, about 300,000 domains): SE Ranking found no relationship between llms.txt and AI citation frequency. In fact, removing llms.txt as a variable made its prediction model more accurate.
- Search Atlas (2026): Search Atlas compared llms.txt adoption on 13,911 domains with 83,387 LLM visibility records across six AI platforms, including Gemini, Perplexity, and OpenAI. Adoption did not predict citations, and the small effect it first saw turned out to be explained by domain authority.
- EZY.ai (July 2026, 83 sites over 12 weeks): EZY found OpenAI fetched llms.txt 7 times and Anthropic 9 times. Perplexity never fetched it. Over the same period, those crawlers requested robots.txt thousands of times.
To be fair, 97% of llms.txt files getting zero visits in a month makes it the gym membership of technical SEO: almost everyone has one, and the logs show almost nobody going. I say that as someone who owns both.
The pattern is clear. AI search crawlers read your HTML, not your llms.txt. The readers that do show up are AI agents and developer tools, and that distinction is where llms.txt still earns its place.
So why does an llms.txt file still matter for GEO?
This is where most “llms.txt is dead” posts stop. I think they stop too early.
AI agents are becoming real visitors
Chrome’s Lighthouse 13.3 added an Agentic Browsing category, and its llms.txt audit documentation says that without the file, agents may spend more time crawling your site to work out its structure and main content. As Search Engine Journal pointed out, Google’s advice depends on the product. Search says you don’t need the file. Chrome treats it as an optional readiness check for browser-based agents.
That’s the right way to frame it. GEO (generative engine optimization) is about getting cited in AI answers. AIO is broader: getting your business understood and used by AI systems, including agents that research vendors, compare services, and fill in contact forms for their users. llms.txt does nothing for the first job. For the second, it’s a small but real advantage.
Writing it forces you to define your entity
Here’s a benefit that won’t show up in any server log study: writing a good llms.txt file is a clarity exercise. You have to describe what your company does in two sentences, list the ten pages that matter, and say what you don’t do. Most businesses I’ve seen can’t do that cleanly on the first try.
Those same sentences are what make you citable everywhere else: your About page, your Organization schema, your Google Business Profile, your LinkedIn page. AI systems reward consistent entity descriptions across sources. The file itself barely gets read, but the thinking behind it improves your whole AI search visibility.
It’s cheap insurance, and adoption keeps growing
Originality.ai’s tracking study, reported by PPC Land, counted 8.8x growth in llms.txt instances in 12 months. It concluded the file looks more like agent-readiness infrastructure than a search visibility tool. Rankability’s September 2026 adoption report found 9.3% of the world’s top 1,000 domains now serve one. Building the file takes an afternoon. Maintaining it takes minutes per quarter. If agent traffic grows the way Chrome expects, you’ll already be ready.
Why does this matter for businesses in Japan?
Japan is still early in AI adoption, which makes now a good time to get ahead. A Teikoku Databank survey of 10,312 companies found 34.5% of Japanese firms use generative AI, ranging from 63.6% of companies with more than 1,000 employees to 29.6% of companies with five or fewer.
For a Japanese SME, the practical issue is often language and format. If your service details live in Japanese-only pages, image banners, or PDF brochures, an AI agent working in English has a hard time understanding you. My recommendation is a short llms.txt with clear English descriptions (and Japanese page titles where it helps), linking to your strongest text-based pages. It costs nothing and removes friction for any agent trying to understand who you are.
How do you create an llms.txt file that actually works?
- Write your summary first. Two sentences: what you do, for whom, and where. If you can’t write them, fix your positioning before you fix your files.
- Pick 5 to 20 key pages. Services, pricing, About, contact, and your best guides. This is a curated list, not a sitemap.
- Use real Markdown links. Format each line as
- [Page name](https://url): short description. Lighthouse’s audit checks the file’s format, and links written as plain text can fail it. - Add an Optional section for blog archives, case studies, and secondary pages.
- Upload it to your root so it returns a 200 status at
/llms.txt. - Validate it with the free llms.txt Validator and Lighthouse’s Agentic Browsing audit in Chrome DevTools.
- Check your server logs after 30 days. See who actually fetches it. That’s the only honest measure.
Here’s a generic example structure for a marketing agency:
# Example Agency
> Example Agency helps SMEs in Japan get discovered in Google, ChatGPT, Gemini, and Perplexity through GEO, SEO, and AI agent strategy.
We work with B2B and hospitality businesses in Tokyo and across Japan, in English and Japanese. We do not offer paid ads management.
## Services
- [GEO and AI search visibility](https://example.com/geo/): Audits and content strategy to earn citations in AI answers
- [AI agents for business](https://example.com/ai-agents/): Designing and deploying AI agents for marketing and sales
## Company
- [About](https://example.com/about/): Team, approach, and markets served
- [Contact](https://example.com/contact/): Book a consultation
## Optional
- [Blog](https://example.com/blog/): Guides on GEO, AEO, and AI searchWhat are the most common llms.txt mistakes?
- Selling it as a ranking lever. If an agency charges you for llms.txt as a way into AI Overviews, show them Google’s AI optimization guide. Then find another agency.
- Publishing llms.txt while blocking AI crawlers in robots.txt. It happens more often than you’d think. robots.txt is the file AI bots actually read, so check it first.
- Dumping every URL into it. A 400-link llms.txt is just a worse sitemap. The whole point is curation.
- Letting it go stale. An outdated llms.txt sends agents to pages that no longer exist. Review it whenever you launch or remove a service.
- Plain-text links. Lighthouse expects Markdown link syntax. Auto-generated files from plugins sometimes get this wrong, so validate them.
- Treating it as a substitute for content. An llms.txt pointing to thin pages just helps AI find thin pages faster.
What moves AI search visibility more than llms.txt?
If you have one week to improve how AI systems see your business, spend about an hour on llms.txt and the rest here:
- Crawlable, fast HTML. Key information belongs in text, not images or PDFs. Self-referencing canonicals, a clean sitemap, and fast server response times.
- Open robots.txt for the AI crawlers you want. GPTBot, ClaudeBot, PerplexityBot, and Google’s crawlers.
- Structured data for your organization, services, and FAQs. It doesn’t guarantee AI citations, but it keeps you eligible for rich results and makes your entity explicit.
- Non-commodity content. First-hand data, real opinions, and specific numbers. That’s what Google’s guide rewards, and it’s what LLMs quote.
- Consistent mentions across the web. The same company description on directories, LinkedIn, press coverage, and partner sites.
- Agent-ready pages. As AI agents for business take on research and booking tasks, clear forms, stable layouts, and accessible markup start to matter as much as keywords.
Frequent Asked Questions about llms.txt file
Question: Is an llms.txt file required for SEO?
Answer: No. Google’s May 2026 AI optimization guide states that llms.txt is not needed for Google Search, AI Overviews, or AI Mode. Google later added that the file neither helps nor hurts rankings. Standard SEO is still the foundation for both classic and AI-powered Google results.
Question: Does ChatGPT read llms.txt files?
Answer: Very rarely. In EZY.ai’s 12-week study of 83 sites, OpenAI’s crawlers fetched llms.txt only 7 times while requesting robots.txt thousands of times. Ahrefs found AI retrieval bots made just 1.1% of all llms.txt requests across 137,210 domains in May 2026.
Question: Who actually uses the llms.txt file?
Answer: Mostly AI agents and developer tools, plus SEO audit software. Ahrefs found AI agents were the largest AI category reading the file, with Claude Code fetching it more than nearly every AI crawler. Chrome’s Lighthouse also checks for it as part of its Agentic Browsing audit.
Question: Should a small business in Japan add an llms.txt file?
Answer: Yes, as a low-cost step toward agent readiness, not as a ranking tactic. It takes an afternoon, it helps AI agents understand your services, and it’s especially useful if your key information is in Japanese-only pages, images, or PDFs. Put most of your effort into content and technical SEO.
Question: What is the difference between llms.txt and llms-full.txt?
Answer: llms.txt is a short index of your most important pages with links and descriptions. llms-full.txt puts the full text of those pages into one Markdown file so an AI can read everything in a single request. Most business websites only need llms.txt.
Conclusion: the llms.txt file is a door sign, not a billboard
The data is clear. An llms.txt file won’t get you into AI Overviews, won’t make ChatGPT cite you, and in 97% of cases won’t even get opened this month. But the web is shifting from search engines that rank pages to agents that act on them. Chrome is already measuring agent readiness, and the businesses that are easy for machines to understand will have an edge when those agents arrive. Treat your llms.txt file as a well-written door sign. It costs almost nothing, it helps the right visitors find their way in, and nobody should confuse it with your marketing.
Want AI systems to understand, cite, and choose your business? Talk to Ongito. We’re an AI marketing agency in Japan working on GEO, AI search visibility, and agent readiness.