Do You Need an llms.txt File? A Straight Answer

Short answer: for most small business websites in 2026, you do not need an llms.txt file yet, and building one should not be near the top of your list.

Across the 8 business sites we track in server logs, AI crawlers and agents hit us 39,695 times in a single week (week of Mon Jul 06 2026), and not one of those requests failed because an llms.txt file was missing. The bots read the actual pages, which is the opposite of what most ranking articles imply.

This guide gives you a straight answer, the file structure if you decide to build one, and a way to check your own server logs to see whether any AI crawler has ever asked for the file. Every number here is from our own logs and Google Search Console, with the sample stated.

What is an llms.txt file and what goes in it?

An llms.txt file is a plain markdown file placed at the root of your domain, at yoursite.com/llms.txt, that lists your most important pages with a short description of each. It was proposed in 2024 by developer Jeremy Howard through the llmstxt.org spec.

The format is deliberately simple. An llms.txt file uses one H1 for your name, a blockquote summary, and H2 sections holding markdown links with a one-line description after each link. That is the entire specification.

Example: a plumbing company names the business in the H1, summarizes the service in a blockquote, then lists services and service-area pages as links, each with a plain description so a model can tell what sits behind it without fetching the page.

What is llms-full.txt, the companion file?

There is also an optional companion file, llms-full.txt, which holds the full text content of those pages inline rather than just links. In practice, llms-full.txt files get large fast and go stale the moment you edit a page, so treat it as optional and high-maintenance.

• An llms.txt file does not block a crawler.
• It does not allow or disallow any path.
• It does not redirect a URL or change what loads.

An llms.txt file is a suggestion, not an instruction. It is a courtesy index that a model is free to read, ignore, or never fetch. Nothing breaks on your site whether it exists or not.

What is the difference between llms.txt and robots.txt?

The difference between llms.txt and robots.txt is that robots.txt controls access and llms.txt only suggests content. Robots.txt is an enforced instruction file that crawlers are expected to obey; llms.txt is an optional reading list with no enforcement behind it.

Robots.txt has been a web standard since 1994 and every major crawler honors it. Sitemap.xml, the third file people confuse here, is a machine-readable list of every URL you want indexed. Knowing which file does what stops you from expecting llms.txt to do a job it was never built for.

Does ChatGPT actually read the llms.txt file?

No major AI provider, including OpenAI, Google, Anthropic, and Perplexity, has confirmed that its models read or use llms.txt. That is the honest state of things as of July 2026. Crawler behavior is measurable even when provider intent is not.

What ChatGPT’s crawlers actually did on our servers

Across the 8 business sites we track, AI crawlers and agents generated 39,695 hits in the week of Mon Jul 06 2026. The single biggest source was chatgpt-user, the agent ChatGPT uses when it fetches a page live to answer a real person’s question, at 8,672 page fetches that week.

Here is the split that matters. In that same week, training crawlers hit the tracked sites 11,745 times (GPTBot 5,138 and ClaudeBot 6,607), versus 711 hits from PerplexityBot. Every one of those fetches was aimed at real content pages, not at an llms.txt file, because none of these sites was serving one at the time.

A crawler fetching your llms.txt file is not the same event as a model using its contents at inference time. Even a site that serves the file and sees it fetched has no visibility into whether the words inside it changed a single answer. Fetch is observable; use is not.

Is llms.txt required or just recommended?

An llms.txt file is not required by any search engine, AI provider, or web standard; it is a voluntary proposal you can adopt or ignore. Nothing penalizes you for skipping it, and no crawler treats its absence as an error.

How widely adopted is llms.txt right now?

Adoption is still small. As of 2026 the count of domains publishing an llms.txt file is roughly under a thousand, and there is no published proof of a ranking or citation benefit. A skeptic analysis by llmtxt.info highlighted an OtterlyAI study finding AI crawlers requested the file on only about 0.1% of measured visits.

Weigh the small unproven upside against a real maintenance and accuracy risk. No AI system requires the file, so its absence costs you nothing measurable today. That asymmetry is why we do not put it first for most clients.

What are the higher-ROI alternatives before you build one?

For a business deciding where to spend limited time, three fixes beat an llms.txt file on evidence every time: crawlable content, a clean robots.txt, and structured data. These are the inputs we can watch move AI crawler traffic and Google rankings in our own dashboards.

Start with content a bot can actually read

Our biggest AI-read gains came from publishing genuinely useful pages, not from config files. One researched article we wrote for a 5-location dental group earned 52,171 Google impressions in a single month while ranking on page one, measured July 2026.

1. Make your content crawlable. Confirm your main pages render as real HTML, load fast, and are not hidden behind JavaScript that bots skip or a login wall.
2. Clean up robots.txt. An accidental disallow rule can block GPTBot or ClaudeBot from your whole site. This is the one file that can truly cost you AI visibility today.
3. Add structured data. Schema markup gives every AI system explicit, machine-readable facts about your pages, and it maps to visible content, so it does not go stale silently.
4. Publish real answers. Pages that directly answer customer questions get fetched and cited far more than a curated link list ever will.

How do I create an llms.txt file for my website?

You create an llms.txt file by writing a short markdown file and uploading it to your domain root so it loads at yoursite.com/llms.txt. You can write it by hand in ten minutes or use a free generator.

Create an llms.txt file by hand

To build an llms.txt file manually, follow these steps in order.

1. Open a plain text editor. Use any editor that saves plain text. Do not use a word processor that adds formatting.
2. Write one H1 line with a single hash and your business or site name, for example # Acme Plumbing.
3. Add a blockquote summary on the next line starting with a greater-than sign: one sentence describing what you do.
4. Create H2 sections with a double hash, such as ## Services and ## About, grouping your key pages.
5. List each page as a markdown link followed by a colon and a short description, for example a services link with the words same-day drain cleaning after it.
6. Save the file as llms.txt in lowercase, with no extension changes, exactly that name.
7. Upload it to your web root using your host file manager, FTP, or a plugin, in the same folder your robots.txt sits in.

Or use a free llms.txt generator

Several free tools crawl your sitemap and output a draft llms.txt file you paste in. A generator saves time on large sites, but you still have to review every line, because a tool will happily list pages you no longer want an AI describing.

Verify the file is live

After uploading, open a browser and go to yoursite.com/llms.txt directly. If the raw markdown loads, the file is served. If you get a 404, the file is in the wrong folder or misnamed, and no bot will ever find it.

How do I check my own server logs for llms.txt requests?

You can check whether any AI crawler has ever requested your llms.txt file by searching your raw server access logs for the path /llms.txt. This is the single most useful check in this guide, and almost no other article tells business owners to run it. It turns the debate into a measurement instead of a guess.

How to search your access logs for /llms.txt

Most hosts expose access logs through the control panel. Run the check in four steps.

1. Open your access logs. In cPanel look for Raw Access or Metrics; on a managed host, ask support for the nginx or Apache access log.
2. Search the log for the text /llms.txt to find every request for that exact path.
3. Look at the user-agent column on any matching line to see whether it was GPTBot, ClaudeBot, PerplexityBot, chatgpt-user, or a human browser.
4. Count the AI requests over 30 days. If the number is zero, no AI system has asked for the file, which is the common result today.

Run the same search for GPTBot and chatgpt-user against your whole log, not just /llms.txt. Seeing real AI crawler hits on your content pages is proof the bots already read your site without any special file.

If a vendor tells you AI cannot find your site without an llms.txt file, your own server logs settle it in about five minutes. Measured behavior beats a sales claim every time.

That evidence should calm any pressure you feel to rush the file. For most business sites today, the count of AI requests for /llms.txt over 30 days is zero.

Does llms.txt help me get cited by Perplexity?

There is no published evidence that an llms.txt file increases citations from Perplexity, ChatGPT, or Google AI Overviews, and no provider has confirmed a link between the file and citation. What we can measure is that PerplexityBot hit our tracked sites only 711 times in the week of Mon Jul 06 2026, a fraction of ChatGPT-related traffic.

Citations follow useful content, not config files

A trading-education platform we run search for has 82 of its top 100 keywords on page one of Google, and 63% of its clicks are non-branded as of July 2026, meaning most visitors did not search the brand name.

That kind of broad, topic-based visibility, earned by answering the questions people actually ask, is what feeds AI answer engines. No curated link file substitutes for it.

We see the same pattern on the dental account. After our takeover, 53% of that 5-location dental group’s Google clicks came from non-branded searches, from people who did not know the practice name, in a July 2026 top-100 query sample.

The house line: anyone promising a guaranteed AI citation or ranking, from an llms.txt file or anything else, is overselling. Nobody outside OpenAI, Anthropic, Google, or Perplexity knows the exact selection rules. We describe what we measure.

Should you build one? A yes or no by business type

The honest answer to whether you need an llms.txt file depends on your business type, and most articles refuse to give a direct call per case. A local service business does not need one yet; a documentation-heavy or developer product plausibly benefits; an ecommerce store gains little.

If you land in the plausibly-yes rows, build it as a low-cost experiment and keep expectations flat. If you land in the no or low-priority rows, spend the time on the crawlable-content and schema work that already moves the numbers we track. You can always add the file later in ten minutes.

Where these numbers come from

The crawler figures come from raw nginx access logs across the 8 business sites No Time For SEO tracks, matched by user-agent (GPTBot, ClaudeBot, PerplexityBot, chatgpt-user, and others) for the week of Mon Jul 06 2026. The ranking and click figures come from Google Search Console reports on a 28-day window, verified July 2026, with client names anonymous. We report what we measured, not any provider’s internal rules.

Frequently asked questions

Do I need an llms.txt file for my small business website?

For most small business websites, you do not need an llms.txt file in 2026. No AI provider requires it, adoption sits under roughly a thousand domains, and there is no proof of a ranking or citation benefit. Across the 8 sites we track, AI crawlers made 39,695 hits in one week reading real pages, with no llms.txt file present.

Does ChatGPT read the llms.txt file?

OpenAI has not confirmed that ChatGPT reads or uses llms.txt. In our logs for the week of Mon Jul 06 2026, chatgpt-user fetched pages 8,672 times, and every fetch targeted real content, not an llms.txt file. A crawler fetching the file is different from a model using its contents at answer time.

What is the difference between llms.txt and robots.txt?

Robots.txt is an enforced standard from 1994 that controls which paths crawlers may fetch. An llms.txt file, proposed in 2024, is an optional, unenforced reading list that suggests content to language models. A bad robots.txt rule can block your whole site from AI crawlers; a missing llms.txt file costs you nothing measurable.

Is llms.txt the same as llms-full.txt?

No. An llms.txt file lists your key pages as links with short descriptions. An llms-full.txt file inlines the entire text of those pages into one large file. The full version is optional and high-maintenance, since it goes stale the moment you edit any page it copied.

Will an llms.txt file get me cited by Perplexity or AI Overviews?

There is no published evidence that an llms.txt file increases citations from Perplexity or Google AI Overviews, and no provider confirms a link. Citations in our experience follow useful content and strong rankings. A trading-education platform we run has 82 of its top 100 keywords on page one with 63% non-branded clicks as of July 2026, driven by content, not config files.

How do I check if AI crawlers request my llms.txt file?

Open your raw server access logs through your host control panel, search for the path /llms.txt, and read the user-agent on any matching line to see if it was GPTBot, ClaudeBot, PerplexityBot, or chatgpt-user. Count matches over 30 days. For most business sites today, the count is zero.

Do this now: your llms.txt checklist for this week

1. Search your server logs for GPTBot and chatgpt-user to confirm AI crawlers already read your site without any special file.
2. Search your logs for /llms.txt to see if the file has ever been requested. If the count is zero, you have your answer on urgency.
3. Open your robots.txt and confirm you are not accidentally blocking GPTBot, ClaudeBot, or PerplexityBot. This is the fix that actually protects AI visibility.
4. Check that your key pages load as real, fast HTML and are not hidden behind JavaScript or a login. Crawlable content is what earns the fetches.
5. Add or review your schema markup so every AI system gets explicit facts that stay tied to visible content.
6. Decide by business type. If you run docs, an API, or a developer tool, build an llms.txt file this week as a flat-expectation experiment. If you run a local service or ecommerce site, skip it for now and revisit later.
7. If you do build one, put a reminder in your calendar to review it every quarter so it never lists a dead page or discontinued service.

Want a hand deciding? We run this exact log check and technical AEO setup for the sites we manage, and we will tell you plainly when an llms.txt file is not worth your time. See our introduction to how we work for the approach.