How one small text file helps large language models read the right parts of your site in real time
Table of contents
- Why you should care about llms.txt
- How llms.txt differs from robots.txt and sitemap.xml
- Where the file lives and what it looks like
- How large the file should be
- What pages belong in the file
- Step by step setup on any host
- Real world example of a generative AI answer
- How often to update the file
- Frequently asked questions
- Download our prompt and build your own llms.txt
- Let PX Media handle llms.txt for you
- Key takeaways
1 Why you should care about llms.txt
Large language models llms such as ChatGPT, Gemini, Claude, and Perplexity now browse the open web in real time. These ai agents grab only a short list of urls before composing answers. If they guess wrong they may quote content that is out of date, off topic, or buried deep in an archive.
llms.txt is a new text file that lives at the root of your site and tells every ai powered crawler which pages to fetch first. A clear file helps the generative AI system find your best pages. It improves user experience and reduces errors.
2 How llms.txt differs from robots.txt and sitemap.xml
File nameMain audience What it doesTypical entryrobots.txtSearch crawlersBlocks or allows crawling Disallow: /private/sitemap.xmlSearch crawlersLists every url for indexing https://example.com/pagellms.txt AI models during a queryHighlights best pages with short summaries https://example.com/faq
robots.txt blocks. sitemap.xml lists everything. llms.txt curates the best for artificial intelligence crawlers.
3 Where the file lives and what it looks like
Place the file in the root directory of your site where it’s visible at:
https://your‑site.com/llms.txt
Lines follow this pattern in plain text:
https://your‑site.com/about/ | Brief description of the page
Keep each summary under 120 characters. Add optional footer lines so deep learning crawlers know how to credit your content.
4 How large the file should be
Stay under 100 lines when possible. A file smaller than fifty kilobytes, or about 500 lines of plain text, loads quickly. It covers many pillar pages without exceeding model context limits.
5 What pages belong in the file
(PX Media examples appear below. Swap them for your own.)
Primary services such as SEO, web design, managed hosting, customer support retainers, and white label work.
High‑traffic location hubs like Pasadena, Los Angeles, San Diego, and Las Vegas.
High‑value resources such as case studies, pricing guides, blog post highlights, or a knowledge base reference.
A contact page so visitors can reach you quickly.
At least one pillar page for every major topic you cover.
Avoid adding landing pages that require login or change minute by minute.
6 Step by step setup on any host
StepAction
- Connect with SFTP, a file manager, or SSH.
- Navigate to the root directory (often named public_html, www, or simply /).
- Create a new text file named llms.txt.
- Paste your list of urls followed by a vertical bar and a concise description, one per line.
- Save using UTF‑8 encoding.
- Open an incognito window and visit https://your‑site.com/llms.txt to confirm direct file access.
- Run a site audit tool such as SEMrush or Screaming Frog to verify user agents can reach the file.
- These instructions work on shared hosting, VPS plans, and cloud platforms alike.
7 Real world example of a generative AI answer
A user types:
“Which services does (Sample: PX Media provide in Pasadena) and how can I contact them?”
ChatGPT activates its browser and requests https://www.pxmediainc.com/llms.txt.
The file loads. It lists four (Pasadena) service URLs and the contact page.
Only those five pages are fetched. Content creation on unrelated pages stays untouched.
Headings and numbers are extracted. The reply lists each (Pasadena) service, phone number, and email. The target page for every link is exactly what (PX Media) prefers.
8 How often to update the file
Review llms.txt every quarter or whenever you publish a new pillar page. Replace any link that now redirects and add new location pages. A nightly script can provide automatic fine tuning if you prefer a hands off workflow.
9 Frequently asked questions
What is the first thing an AI tool looks for on my domain?
It checks llms.txt before any other resource, then scans the first few kilobytes to choose pages that match the query.
How many lines can I include before the file slows models down?
Plain text files up to fifty kilobytes (about five hundred lines) still load quickly, but staying under one hundred lines leaves headroom.
Does Google Search use llms.txt today?
Google has not confirmed that its ranking crawler reads the file. Generative ai tools that browse on demand are the primary consumers.
Can llms.txt replace my XML sitemap?
No. Your XML sitemap lists every public url. llms.txt is a curated shortcut for ai agents.
What happens if a listed URL later redirects?
Replace redirecting links in your quarterly review so every user agent lands on a clean 200‑status target page.
Can I include PDF files or other non HTML resources?
Yes, but limit them and make sure they load quickly. Artificial intelligence systems prefer HTML for speed.
How can I tell if the file is being read?
Check hosting analytics for hits on /llms.txt. Look for user agents such as GPTBot, Anthropic, or PerplexityBot.
10 Download our LLMS prompt and build your own llms.txt
You do not have to write a file from scratch. PX Media provides a ready‑made prompt template so you can create a draft in minutes.
How to use it
Copy the code block below or download the TXT file.
Paste it into your preferred AI chat tool.
Replace <> with your site’s domain.
Let the model crawl, then read every line for errors.
Ask the same model to critique the draft for duplicates, redirects, and summary length.
Edit as needed, save as llms.txt, upload, and test.
Always test before publishing. A quick review prevents surprises.
Prompt template (copy this into your AI chat)
PROMPT TEMPLATE: Build an llms.txt for my site
PROMPT TEMPLATE: Build an llms.txt for my site
────────────────────────────────────────────────────────────
# 1. DOMAIN TO CRAWL
<>
# Example: https://example.com
# 2. MAX PAGE LIMITS
primary_pages=12 # core or pillar content
location_pages=20 # city or region service pages
resource_pages=8 # blog posts, knowledge‑base articles, white papers
# 3. SUMMARY LENGTH
summary_limit=120 # characters including spaces
# 4. CRAWLING & OUTPUT RULES
– Crawl the domain in real time
– Use canonical URLs only; ignore “category”, “tag”, tracking params, or redirects
– Discard duplicate URLs across sections
– Group output: Core, Location hubs, Resources, Contact page
– Location hubs must be service pages; move informational articles to Resources
– Trim summaries to summary_limit characters
– Output plain text:
https://example.com/page | one line summary
# 5. SAMPLE FOOTER (replace or keep as is)
copyright: © 2025 PX Media, LLC. All rights reserved.
license: Free to use, adapt, and redistribute under CC‑BY‑SA‑4.0
author: PX Media, LLC
────────────────────────────────────────────────────────────
Download Prompt Here – direct file access.
Prompt file (TXT) – ready for any chat interface.
Optional sample llms.txt – shows a finished file in production.
Linking directly to our live llms.txt is safe because the file is public plain text. Add rel=”nofollow” to the link if you wish to minimise crawl weight.
11 Let PX Media handle llms.txt for you
PX Media provides an AI visibility tune-up. This includes checking current pages and creating a high-quality llms.txt file. file, quarterly reviews, and optional automation so you never edit the file manually. Move this offer to its own service page if you prefer an entirely educational post.
12 What we’ve learned
llms.txt guides ai models toward your best content in real time.
Keep the list short and up to date. This will help provide a great customer experience when users look for information through generative answers.
Review quarterly or automate updates.
Professional help is available if your site spans a wide range of services or locations.
Add the file and the next artificial intelligence mention of your brand will link readers to the pages you want them to see.