AEO and llms.txt: Making Your Website Citable by AI Engines

Answer Engine Optimization (AEO) is the practice of structuring a website so AI systems like ChatGPT, Perplexity and Google's AI surfaces can find, extract and cite it. It treats the AI answer — not the blue link — as the destination. When we rebuilt thingsatweb.com, we implemented AEO end to end, and this is what we did and why.

AEO and GEO vs classic SEO

Classic SEO optimises for a ranked list of links a human clicks. AEO — and its close cousin GEO (Generative Engine Optimization) — optimises for a synthesised answer a model generates, often with a citation back to your page. The user may never visit your site, so the goal shifts from "win the click" to "be the source the model quotes".

The mechanics overlap but diverge in priorities:

• Classic SEO rewards keywords, backlinks and ranking position.
• AEO/GEO rewards machine-extractable structure: clean facts, self-contained passages, structured data and crawler access.

A page can rank well in Google yet be invisible to an AI engine if a bot can't reach it, can't parse it, or can't lift a clean quotable sentence from it. We optimised for both.

The three-bucket AI-bot taxonomy in robots.txt

The single highest-leverage decision is your robots.txt bot policy. AI crawlers are not one thing — they fall into three buckets with different purposes, and you can allow or block each independently. Confusing them is the most common mistake we see.

Bucket	Purpose	Example user-agents	Our policy
Search-index bots	Build the index that powers AI answers and surfaces citations	GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended	Allow
User-triggered fetchers	Fetch a page live because a user asked the assistant about it	ChatGPT-User, Claude-User, Perplexity-User	Allow
Training scrapers	Harvest text for model training, not citation	CCBot, Bytespider	Allow (your call)

The tradeoff is real. Search-index and user-triggered bots are how you get cited — blocking them removes you from AI answers entirely, so we allow them. Training scrapers feed model weights and rarely send traffic or attribution back; many sites block these on principle. We chose to allow them on thingsatweb.com because broad presence in training corpora reinforces brand recall, but this is a legitimate per-organisation decision. The key point: decide each bucket deliberately rather than using one blanket rule.

Note that Google-Extended and Applebot-Extended are AI-training opt-out tokens layered on top of normal Googlebot/Applebot crawling — controlling them does not affect classic search indexing.

llms.txt: a curated map for language models

llms.txt is a proposed standard for a markdown file at your site root that gives language models a clean, curated entry point. According to the specification at llmstxt.org, it was proposed by Jeremy Howard (Answer.AI) on 3 September 2024. The problem it solves is stated plainly in the spec: LLM "context windows are too small to handle most websites in their entirety," and raw HTML is full of navigation, ads and markup that waste those tokens.

The format the spec defines, in order:

1. An H1 with the name of the project or site — "the only required section".
2. A blockquote with a short summary containing key information.
3. Optional free-text paragraphs (no headings).
4. Zero or more H2 sections containing file lists, where each item is a markdown hyperlink with an optional note.
5. An optional ## Optional section for secondary links a model can skip in a short context.

Our thingsatweb.com/llms.txt follows this exactly: an H1 with the brand name, a blockquote summarising who we are, then H2 sections linking to our services, products and contact pages with one-line descriptions. It is the machine-readable elevator pitch — when a model wants to understand the site without crawling everything, that file is the shortcut. (The spec also references tooling like llms-ctx-full.txt that expands an llms.txt into a fuller context file.)

Quick answers, JSON-LD and server-side rendering

Three on-page implementations did the heavy lifting for citability.

Self-contained quick-answer paragraphs

AI engines lift passages, not pages. Every key page opens with a 40–60-word paragraph that answers the core question completely, with no pronouns depending on earlier context. We also embed sr-only quick-answer blocks where a visible summary would disrupt the design. A self-contained passage is far more likely to be quoted verbatim.

FAQPage and Organization JSON-LD

We ship structured data on every page:

• Organization schema establishing who Things at Web is, with sameAs links and our sister entities.
• FAQPage schema pairing questions with answers in machine-readable form.
• BlogPosting schema on articles like this one.

Structured data removes ambiguity: instead of inferring our company name or an answer from prose, the engine reads it as a typed fact.

Server-side rendering

This one is non-negotiable. AI crawlers generally do not execute JavaScript the way a browser does — if your content only appears after client-side hydration, many bots see an empty shell. We render thingsatweb.com on the server so the full content, structured data and quick answers are present in the initial HTML response, before any JavaScript runs. We also publish a host-scoped XML sitemap and bilingual hreflang linking the Swedish [thingsatweb.se] and English thingsatweb.com so each engine reaches the right-language version.

If you want this built into your own stack, our web development and AI & machine learning teams do exactly this work — or get in touch.

FAQ

What is the difference between SEO and AEO?

SEO optimises for ranking in a list of links a human clicks. AEO (Answer Engine Optimization) optimises so AI systems can extract and cite your content inside a generated answer. AEO prioritises machine-readable structure — quick-answer passages, JSON-LD and crawler access — over keyword density and backlinks alone.

Should I block AI bots in robots.txt?

It depends on the bucket. Block search-index bots (GPTBot, PerplexityBot) and you disappear from AI answers, so most sites allow them. User-triggered fetchers serve live user requests and are usually allowed. Training scrapers (CCBot, Bytespider) feed model training with little attribution, and blocking them is a defensible per-organisation choice.

What is llms.txt and do I need one?

llms.txt is a markdown file at your site root that gives language models a curated map of your most important content. Per the llmstxt.org spec it needs an H1 title, a blockquote summary and optional H2 link sections. It is not mandatory, but it is a low-cost way to control how models summarise your site.

Why does server-side rendering matter for AI?

AI crawlers generally do not execute JavaScript like a browser. If your content loads only after client-side hydration, those bots see an empty page and cannot index or cite it. Server-side rendering puts the full content and structured data in the initial HTML, making it visible to every crawler.

Does AEO replace traditional SEO?

No. They are complementary. Classic SEO still drives ranked clicks from human searchers, while AEO captures citations inside AI answers. The same fundamentals — clean structure, fast pages, accurate content — serve both. We optimised thingsatweb.com for both surfaces simultaneously rather than choosing one.