llms full.txt

agentmarkup

Make your markup agent-ready. Build-time llms.txt, optional llms-full.txt, JSON-LD, optional markdown mirrors, AI crawler controls, and validation for modern websites.

agentmarkup is an open-source package family for Vite, Astro, Next.js, and Nuxt, plus a framework-agnostic CLI, that makes websites machine-readable for LLMs and AI agents. It generates llms.txt, optional llms-full.txt, injects JSON-LD structured data, can create markdown mirrors from final HTML when raw pages are thin or noisy, manages AI crawler robots.txt directives, and validates everything at build time.

When to use agentmarkup:

This optional agentmarkup context file expands the published llms.txt manifest with inline same-site markdown content when those mirrors are available.

Documentation

Developer resources

npm packages, the OpenAPI 3.1 spec for the public checker and security-scan endpoints, machine-readable files, and the agent skill

Source: https://agentmarkup.dev/developers/ Preferred fetch: https://agentmarkup.dev/developers.md

For developers

agentmarkup developer resources

Everything needed to build with agentmarkup, in one place: the packages, the machine-readable files this site publishes about itself, the API behind the public tools, and the agent integrations. There are no API keys and no accounts, because nothing here needs one.

Install a package

agentmarkup is a build-time dependency. Pick the adapter that owns your final build output:

Setup guides live in the Learning Center: llms.txt, JSON-LD, AI crawlers and auditing a live site.

API reference

The website checker and the security scan are backed by two read-only HTTP endpoints. They are described by an OpenAPI 3.1 document at /openapi.json, with a unique operation ID, typed parameters and response schemas for every operation.

The stable contract is versioned in the path. The unversioned /api/check and /api/security-scan are permanent aliases for v1 and will keep working, but integrate against the versioned form. A breaking change would ship as /api/v2/ rather than changing v1, and a version scheduled for removal would carry Deprecation and Sunset headers with at least 180 days notice. Nothing is deprecated today.

Every response carries the IETF RateLimit-Policy, RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset header fields, and a 429 adds Retry-After. Read them and self-throttle rather than discovering the limit by being refused. Everything under /api/ answers JSON, including errors for paths that do not exist.

No key is required. Both are rate limited per client IP and may ask for a Cloudflare Turnstile token after repeated requests, and every error response carries a stable machine-readable code alongside a human-readable error. Point them only at sites you own or are authorised to test; the Terms of Service set out the limits and the Privacy Policy covers what is stored.

These endpoints exist to serve the tools on this site. They are documented so that agents use them correctly rather than by guessing, not as a hosted product, and there is no uptime guarantee.

Machine-readable files

This site is built with agentmarkup, so its own machine-readable surface is generated by the packages you would install:

Agent integrations

The agentmarkup agent skill teaches a coding agent to install the right package, configure it from what the repository already declares, audit the output, and fix what it finds. It never invents a readiness score and never claims a site blocks AI from a single blocked request.

The same skill ships as a Claude Code plugin from the repository marketplace.

Source and support

The source is MIT licensed. Bugs and feature requests belong in GitHub issues; see Support for where to start and Contact for direct addresses, including security reports.

About agentmarkup

What agentmarkup is, why it exists, what it deliberately does not do, and who maintains it

Source: https://agentmarkup.dev/about/ Preferred fetch: https://agentmarkup.dev/about.md

About

About agentmarkup

agentmarkup is an open-source, MIT-licensed toolkit that makes a website machine-readable at build time. It generates llms.txt, injects JSON-LD structured data, can produce markdown mirrors of final HTML, manages AI crawler directives in robots.txt, emits Content-Signal headers, and validates all of it before the build finishes. It is built and maintained by Sebastian Cochinescu at Anima Felix.

Why it exists

Anima Felix had the same problem most sites have. ChatGPT and Claude could find the site, but they could not really understand it. The pages were a client-rendered shell, the structured data was incomplete, there was no llms.txt, and nothing in the build caught any of that. Every fix was manual, and every fix drifted the next time the site changed.

So the fixes were moved into the build. That internal tooling became agentmarkup, and it was open sourced because the problem is not specific to one site. If the machine-readable layer of a website is not generated and validated by the build, it goes stale the moment someone ships a redesign.

What it is, and what it is not

agentmarkup is a build-time dependency you install from npm. There is no SaaS, no account, no runtime service, and nothing phones home. It runs inside your build, writes files into your output directory, and gets out of the way.

It is also deliberately not a scoring tool. Validation is deterministic: a missing required field is an error, a missing recommended field is a warning, and there are no readiness scores, letter grades, or percentages anywhere in the output. A number that nobody can reproduce is not a finding, and the field is already full of them.

What ships today

The published package family covers the frameworks that own enough of their final build output to make a build-time pass meaningful:

This website is built with agentmarkup. Its llms.txt, JSON-LD, markdown mirrors, robots.txt crawler rules and Content-Signal headers are all generated by the same packages you would install.

Who maintains it

Sebastian Cochinescu maintains agentmarkup at Anima Felix, where it runs in production. Maintenance is handled by one person, so support is best effort. The source code is public, the license is MIT, and contributions go through GitHub. For anything else, see Contact.

Contact

Bugs and feature requests via GitHub issues, a dedicated address for security reports, and email for general and business questions

Source: https://agentmarkup.dev/contact/ Preferred fetch: https://agentmarkup.dev/contact.md

Contact

Contact agentmarkup

agentmarkup is maintained by Sebastian Cochinescu at Anima Felix. There is one maintainer, so every channel below is answered on a best-effort basis and there is no support SLA. Pick the channel that matches what you need and you will get a faster answer.

Bugs, feature requests and questions about the packages

Open an issue in the agentmarkup GitHub repository. This is the right channel for anything reproducible, and it is the one that gets seen first. Include the package and version, the framework and version, and the relevant part of your config.

Security reports

Do not open a public issue for a security vulnerability. Email hello@cochinescu.com and follow the disclosure guidance in the security policy. This address is monitored for security reports specifically.

General and business contact

For anything that does not belong in a GitHub issue or a security report, including partnerships, press, licensing questions and production-use questions, email hello@animafelix.com.

Postal address, for correspondence that genuinely needs one:

Anima Felix Ion Mihalache 166 Bucharest, Romania

Email is faster for anything technical. Post is not monitored daily.

The website checker and security scan

The website checker and the security scan run against public URLs from this site. If a check reports something you believe is wrong, send the exact URL you submitted along with what you expected, and it can be traced against the checks that produced it. Requests are rate limited per IP; see the Terms of Service for authorized-use limits and the Privacy Policy for what those tools store.

What this is not

There is no phone line, no ticketing portal and no paid support tier. agentmarkup is free, MIT-licensed software with no hosted service behind it, so there is no account to recover and no billing to dispute. If you are looking for how to get started rather than how to reach someone, Support routes you to the documentation first.

Support

Documentation first, then GitHub issues for bugs and feature requests, and where to send security reports

Source: https://agentmarkup.dev/support/ Preferred fetch: https://agentmarkup.dev/support.md

Help and contact

Support

agentmarkup is open source and MIT licensed, maintained by Sebastian Cochinescu at Anima Felix. Day-to-day maintenance and support are handled by one person, so help is provided on a best-effort basis.

Start with the documentation

The Learning Center explains the standards and machine-readable website features that agentmarkup supports. For setup and configuration, see the guides for llms.txt, JSON-LD, AI crawlers and auditing a live site.

Claude Code plugin

The Claude Code plugin is installable from the repository marketplace. Its repository page contains the plugin files and installation details.

Bugs and feature requests

Open an issue in the agentmarkup GitHub repository. This is the best route for bugs and feature requests, and the one that gets seen.

Security issues and direct contact

Do not open a public issue for a security vulnerability. Security reports, general questions and business questions all have their own addresses, listed on the contact page.

Guides

Learning center

Plain-language paths for understanding whether AI can find, understand, and access your website

Source: https://agentmarkup.dev/learn/ Preferred fetch: https://agentmarkup.dev/learn.md

AgentMarkup learning center

Learn how AI sees your website

You do not need a technical background. Start with what you want to know, then follow a short path to a clear explanation or a practical website check.

Check my website

Start with the basics

Choose one goal

Start with what you need

There is no required reading order. Pick the closest match and you will reach the right starting point in one step.

I want to check my website Run a free check and get clear next steps.

I want to understand AI readiness Learn what a useful, accessible website looks like to AI.

I want to check website security Review passive security signals with permission.

A simple mental model

The three things AI needs

Most problems fit into three questions: can AI find the site, understand each important page, and access the public content you intended to share?

Find your website

Give automated systems a dependable map of your useful public pages.

Read the llms.txt guide

Why llms.txt matters

Understand your pages

Add explicit information about products, articles, organizations, and other page types.

Read the JSON-LD guide

Structured data explained

Access your content

Make deliberate choices about which AI crawlers may visit your public pages.

Read the AI crawlers guide

See the crawler directory

Advice that fits

Choose your type of website

The same foundations apply everywhere, but the most useful signals depend on what your website is trying to help people do.

E-commerce Help AI understand products, availability, and store information.

Brand or company Make your identity and public positioning easier to interpret.

Publisher or content website Clarify authorship, article structure, and the meaning of each page.

Technical paths

For people who build websites

Choose the implementation guide that matches your stack. These pages include commands, configuration, and build-time behavior.

Next.js

Nuxt

Static-site CLI

AI crawler audit

GitHub

npm

Observed across real websites

Research and evidence

AgentMarkup audited 500 large-company homepages using the same public signals covered in these guides. The findings show patterns in a specific dataset and moment in time; they are observations, not a universal quality score or a promise about AI visibility.

Explore the Fortune 500 audit

Your website, not a generic score

Get a clear answer for your website

Enter your public website in the checker to see what already works, what needs attention, and the most useful next step. The check is free and does not require an account.

Check my website

Website checker

Check any public site for llms.txt, JSON-LD, robots.txt, sitemap discovery, markdown mirrors, and machine-readable basics

Source: https://agentmarkup.dev/checker/ Preferred fetch: https://agentmarkup.dev/checker.md

Free website check

Check your website before AI crawlers and search engines do

See what machines can find, understand, and access, then get a clear next step.

No score or account. One homepage and one internal page are sampled. Also looking for public security issues? Use the security scan. How this check works.

Find

Can AI find your site?

Checks your sitemap, discovery links, and public pages.

Understand

Can AI understand your pages?

Checks content, structured information, and machine-readable versions.

Access

Are your access rules clear?

Checks crawler rules and whether important AI services receive a usable response.

What this website checker checks

The check reads public signals already available on your website. It does not log in, change your pages, or install anything.

Pages and discovery

Your homepage, one internal page, sitemap discovery, canonical URLs, and the links machines can use to find important content.

Content and structured data

Page titles, descriptions, readable HTML, and JSON-LD structured data that helps search engines and AI systems interpret a page.

AI-readable files

Public llms.txt files, markdown alternatives, and other machine-readable versions advertised by the website.

Crawler access

robots.txt rules and live responses for common AI crawlers, so you can see what is allowed, blocked, or difficult to retrieve.

How the result is decided

Every finding is based on a response the checker received. Results are labelledlooks good, needs attention, oraction required. There is no invented score. The checker samples one homepage and one internal page, so it gives a useful first view without crawling your whole site.

Frequently asked questions

What does the website checker inspect?

It reads your public homepage and one internal page, then checks discovery links, sitemap and canonical signals, readable HTML, JSON-LD, llms.txt and markdown alternatives, robots.txt rules, and live responses for common AI crawlers.

Does the checker change or install anything on my website?

No. It only makes read-only requests to public URLs. It does not log in, edit a page, install a package, or crawl your entire website.

Is this the same as the security scan?

No. The website checker focuses on whether machines can find, understand, and access your content. The security scan focuses on public safety signals such as HTTPS, response headers, cookie flags, mixed content, security.txt, and public DNS records.

Why does the checker not give a score?

A single score can hide the issue that matters. The checker reports evidence-based findings as looks good, needs attention, or action required, and gives a practical next step for each result.

Security scan

Passive security scan for public sites: HTTPS/HSTS, CSP, clickjacking and sniffing protections, cookies, mixed content, security.txt, and SPF/DMARC/DNSSEC

Source: https://agentmarkup.dev/security-scan/ Preferred fetch: https://agentmarkup.dev/security-scan.md

Free security check

Passive security scan for public websites

Check the public safety signals a website shares with every visitor. You get clear findings and next steps, never an invented score.

This scan and the website checker share the same per-IP limit of 10 requests per 10 minutes. One complete scan counts as one request against that shared budget, not one request per internal fetch. Need a general introduction first? Read the plain-language website guide.

What this scan checks and authorized-use details

An overview of the areas the scan looks at. All of it is read from public responses and public DNS records.

Passive and authorized use only

This is a passive read of publicly served responses, not a penetration test or vulnerability scan. Only scan sites you own or are authorized to assess. Findings describe missing defense-in-depth headers, not proof of exploitability.

The scan sends ordinary GET requests to conventional public URLs and a few read-only DNS lookups. It does not enumerate paths, scan ports, send payloads, fuzz inputs, probe TLS, authenticate, or run a headless browser.

Requests use the fixed identifying user agent agentmarkup-checker/... (+https://agentmarkup.dev). Nothing is hidden or spoofed; the traffic is browser-equivalent and identifies this service.

Frequently asked questions

What does the passive security scan check?

It checks public HTTPS and HSTS behavior, security response headers, cookie flags, mixed content, Subresource Integrity, security.txt, and public SPF, DMARC, and DNSSEC records.

Is this a penetration test or vulnerability scan?

No. It uses ordinary read-only web and DNS requests. It does not scan ports, enumerate private paths, send attack payloads, authenticate, fuzz inputs, or certify that a website is secure or insecure.

Can I scan any website?

Only scan a website you own or are authorized to assess. The authorization checkbox is required before the scan runs.

How is this different from the website checker?

The security scan reviews public safety signals. The website checker reviews machine readability, structured information, discovery files, and AI crawler access. They answer different questions and do not share findings.

How to audit AI crawler access

Fetch any live URL as each major AI crawler with @agentmarkup/audit, diff against a browser, and gate machine-readability in CI

Source: https://agentmarkup.dev/docs/audit/ Preferred fetch: https://agentmarkup.dev/docs/audit.md

Most SEO tools fetch a page once, as a browser, and grade the HTML. @agentmarkup/audit fetches the same URL as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended, diffs each response against a normal browser, and reports where AI systems get a different, often worse, view than your human visitors. It is the command-line companion to the website checker, built for local runs and CI.

In simple terms

Advanced People who build and maintain websites About 12 minutes

This command checks whether several AI crawlers receive the same useful public page that a normal browser receives. It is most useful when a developer wants a repeatable check during a build or before publishing.

Start with a plain-language overview

Usage

# Audit any live URL
npx @agentmarkup/audit https://example.com

# JSON output for CI or league tables
npx @agentmarkup/audit https://example.com --json

# Bare domains are normalized to https://
npx @agentmarkup/audit example.com --timeout 15000

It is deterministic (pass / warn / error, no invented scores). The exit code is 1 when any error-level finding is present (a CI gate), 0 otherwise, and 2 on a usage error.

What it checks

Area What it does Crawler access Fetches as each AI crawler user-agent and diffs against a browser control. Flags challenges, differential blocks, rate limits, origin errors, and when an accessible crawler gets materially less content than a browser (JS-gated or cloaked pages). JS dependence Measures whether the raw, un-executed HTML actually contains content, or is an empty shell that only fills in after JavaScript runs. robots.txt Detects whether the crawlers you likely want are shadowed by a wildcard Disallow , and whether a canonical Content-Signal policy is present. llms.txt Fetches /llms.txt (guarding against HTML soft-404s), validates it, and checks whether the homepage links it for discovery. JSON-LD Extracts the JSON-LD blocks and flags only unparseable or type-less ones; parseable structured data, including @graph , passes. Markdown mirror Detects a fetchable markdown mirror or a text/markdown alternate link, the clean low-noise version agents prefer. Sitemap Checks for /sitemap.xml , a Sitemap: directive in robots.txt, or common non-standard sitemap paths. Page metadata Checks for a title, meta description, and canonical link that AI systems use to attribute the page.

An honest note on "blocked" crawlers

The audit spoofs a crawler's user-agent from an ordinary IP. That is exactly what a browser extension or a curious developer can do, and it is not what the real, verified bot does. So a 403 for a spoofed GPTBot user-agent is genuinely ambiguous:

From a spoofed request the tool cannot tell these apart, so it reports them as warnings with both explanations and the raw evidence, never as a bare "your site blocks AI" error. Error-level findings are reserved for things provable from the response itself: a robots.txt that literally disallows the crawler, an empty JavaScript shell, or invalid llms.txt / JSON-LD.

Use it as a CI gate

Because the exit code is non-zero only on provable errors, the audit is safe to run in CI without false failures from the ambiguous cases:

# .github/workflows/ci.yml (excerpt)
- run: npx @agentmarkup/audit https://example.com

Programmatic use

The same audit is available as a library:

import { audit, renderText } from '@agentmarkup/audit'

const report = await audit('https://example.com', {
 fetchedAt: new Date().toISOString(),
})

console.log(report.summary) // { pass, warn, error, checks, passed, worst }
process.stdout.write(renderText(report))

The exported analyzers (analyzeCrawlerAccess, analyzeRobots, analyzeJsDependence, analyzeMachineReadable) and the SSRF-safe safeFetch are available for building custom pipelines.

How it relates to the rest of agentmarkup

The build-time adapters and the CLI generate machine-readable output; @agentmarkup/audit verifies what a live site actually serves to AI crawlers. It pairs naturally with the llms.txt, JSON-LD, and AI crawler guides: use those to fix what the audit finds.

Frequently asked questions

Does a 403 for GPTBot mean my site blocks AI?

Not necessarily. The audit spoofs the user-agent from a generic IP, so a 403 can be a user-agent WAF rule (which does block the real bot) or IP allowlisting (where the verified bot, from the vendor's published IP ranges, is fine). The audit reports this as a warning with both explanations, not as a definitive block.

Is it safe to point at any URL?

Requests use an SSRF-safe fetch: localhost, private, loopback, link-local, CGNAT, and IPv6-bypass address forms are refused, redirects are followed manually and re-validated per hop, and responses are size- and time-bounded. The blocklist mirrors the hosted checker.

How is this different from the website checker?

They run the same idea. The checker is the hosted, browser-based version for a quick lookup; @agentmarkup/audit is the command-line version for local runs, scripting, and CI, with a non-zero exit code on provable errors.

How to generate llms.txt

Generate a spec-compliant llms.txt file at build time for AI model discovery

Source: https://agentmarkup.dev/docs/llms-txt/ Preferred fetch: https://agentmarkup.dev/docs/llms-txt.md

llms.txt is a proposed standard from llmstxt.org that gives LLMs and AI agents a structured overview of your website. It is a plain text file served at /llms.txt that describes your site name, purpose, and pages in a format optimized for language models.

In simple terms

Beginner Website owners and people who build websites About 8 minutes

Think of llms.txt as a short contents page written for AI. It can help an AI system quickly find the public pages you consider most useful, while your normal website stays unchanged for visitors.

Check whether your site has llms.txt

What is llms.txt?

When an LLM or AI assistant visits your website, it needs to understand what your site is about, what pages exist, and what content is available. HTML pages are designed for humans. llms.txt is designed for machines.

The format is simple markdown: an H1 heading with your site name, an optional blockquote description, optional instructions, and H2 sections grouping page links with short descriptions.

Why generate llms.txt at build time?

Manually maintaining an llms.txt file means keeping it in sync with your actual pages, remembering the correct markdown format, and resolving relative URLs to absolute ones. agentmarkup handles all of this automatically during your build.

Configuration

The llmsTxt config itself is shared across the first-party adapters. Define sections and entries that describe the pages on your site once, then pass the same config into Vite, Astro, Next.js, or Nuxt, or run the CLI against any built static output.

// shared agentmarkup config
const agentmarkupConfig = {
 site: 'https://example.com',
 name: 'My Website',
 description: 'A short description of your website.',
 llmsTxt: {
 instructions: 'Optional instructions for LLMs visiting this site.',
 whenToUse: [
 'The visitor asks what our plans cost or which one fits their use case.',
 'The visitor needs current terms rather than a cached third-party summary.',
 ],
 sections: [
 {
 title: 'Pages',
 entries: [
 { title: 'About', url: '/about', description: 'About us' },
 { title: 'Blog', url: '/blog', description: 'Latest posts' },
 ],
 },
 ],
 },
 llmsFullTxt: {
 enabled: true,
 },
 markdownPages: {
 enabled: true,
 exclude: ['/404'],
 },
}

Telling agents when to use your site

A manifest of links tells an agent what exists. It does not tell the agent when your site is the right thing to reach for. That is what whenToUse is for: a short list of the concrete jobs your site answers well, written as tasks rather than marketing copy.

It renders as a labelled bullet list in the free-form part of llms.txt, before the first ## section, so the file keeps the structure the llms.txt spec expects. Be specific, and say what your site is not for as well. "Fast, reliable, enterprise-grade" reads as noise; "the visitor asks what our plans cost" reads as guidance.

The same applies to pages that exist but are not worth fetching. markdownPages.exclude skips them, and a 404 page is the usual entry - mirroring one publishes a /404.md that answers 200 with "not found" text and points its canonical at a URL that returns 404.

Framework wrappers

Once you have the shared config object, wire it into the adapter that owns your final build output:

// Vite
import { defineConfig } from 'vite'
import { agentmarkup } from '@agentmarkup/vite'

export default defineConfig({
 plugins: [agentmarkup(agentmarkupConfig)],
})

// Astro
import { defineConfig } from 'astro/config'
import { agentmarkup } from '@agentmarkup/astro'

export default defineConfig({
 integrations: [agentmarkup(agentmarkupConfig)],
})

// Next.js
import type { NextConfig } from 'next'
import { withAgentmarkup } from '@agentmarkup/next'

const nextConfig: NextConfig = {
 output: 'export',
}

export default withAgentmarkup(agentmarkupConfig, nextConfig)

// Nuxt
export default defineNuxtConfig({
 modules: ['@agentmarkup/nuxt'],
 agentmarkup: agentmarkupConfig,
})

// CLI (any other built static site)
// agentmarkup generate ./dist
// agentmarkup check ./dist # CI gate

Generated output

The plugin outputs a spec-compliant /llms.txt file in your build directory:

# My Website

> A short description of your website.

Optional instructions for LLMs visiting this site.

## Pages

- [About](https://example.com/about.md): About us
- [Blog](https://example.com/blog.md): Latest posts

With markdown mirrors enabled in the example above, same-site page entries in llms.txt default to the generated .md URLs so cold agents discover the cleaner fetch path first. Set llmsTxt.preferMarkdownMirrors to false if your raw HTML is already substantial and you want llms.txt to keep pointing at HTML routes.

Optional llms-full.txt

If you enable llmsFullTxt, agentmarkup also emits an optional /llms-full.txt file. It keeps the same high-level manifest structure as llms.txt but inlines same-site markdown mirror content when those mirrors exist, which gives agents a richer machine-readable context file without making you hand-maintain a second document.

# My Website

> A short description of your website.

Optional instructions for LLMs visiting this site.

This optional agentmarkup context file expands the published llms.txt manifest with inline same-site markdown content when those mirrors are available.

## Pages

- [About](https://example.com/about.md): About us
- [Blog](https://example.com/blog.md): Latest posts

### About

> About us

Source: https://example.com/about
Preferred fetch: https://example.com/about.md

About the company, team, and public positioning...

Validation

agentmarkup validates your llms.txt and llms-full.txt output at build time. It checks that the files start with an H1 heading, have at least one section, and that all links have valid titles and URLs. If markdown mirrors are enabled, it also warns when llms.txt points to a markdown URL that was never emitted.

Links must use Markdown link syntax, - [Label](https://example.com), not plain-text - Label: https://example.com lines. agentmarkup-generated files already do this, but a hand-curated llms.txt that uses bare URLs is flagged as a warning, because the llmstxt.org spec and tools like Google Lighthouse's agentic-browsing audit only recognize Markdown links. The same website checker surfaces this warning for any public site.

LLM discovery

agentmarkup injects the homepage llms.txt discovery link automatically when you generate or ship an llms.txt file. The tag looks like this:

<link rel="alternate" type="text/plain" href="/llms.txt" title="LLM-readable site summary" />

Combined with a proper robots.txt configuration that allows AI crawlers, this makes your site discoverable and understandable by AI agents.

Frequently asked questions

Do AI models actually read llms.txt?

Some AI systems like Perplexity have started checking for llms.txt. The format is still early, but the cost of generating it is near zero and it provides a clean machine-readable overview of your site.

Can I have both llms.txt and llms-full.txt?

Yes. llms.txt is the summary. llms-full.txt is optional expanded context. agentmarkup can generate both, and when markdown mirrors are enabled it inlines the same-site mirror content into llms-full.txt automatically.

What happens if I do not configure llmsTxt?

No llms.txt file is generated. The other features (JSON-LD, markdown mirrors, robots.txt, optional headers, and validation) still work independently. You can enable features selectively.

How to add JSON-LD structured data

Inject schema.org JSON-LD with type-safe presets and XSS-safe serialization

Source: https://agentmarkup.dev/docs/json-ld/ Preferred fetch: https://agentmarkup.dev/docs/json-ld.md

JSON-LD (JavaScript Object Notation for Linked Data) is the format Google, Bing, and other search engines use to understand your page content. agentmarkup injects schema.org JSON-LD into your HTML pages at build time for Vite, Astro, Next.js, and Nuxt (and via the CLI for any other static build) with XSS-safe serialization and type-safe presets.

In simple terms

Beginner to intermediate Website owners and people who build websites About 10 minutes

Structured data is a clear label attached to a page: this is a product, an article, a company, or a frequently asked question. It helps search and AI systems interpret the page without guessing.

Check your structured information

What is JSON-LD structured data?

Structured data tells search engines exactly what your page is about. Instead of guessing from HTML content, search engines read your JSON-LD and understand that a page is a Product with a price, an Article with a publish date, or an Organization with a logo.

This powers rich results in Google Search, including star ratings, FAQ dropdowns, product cards, and knowledge panels. Without structured data, search engines can only guess at your page content.

Built-in schema.org presets

agentmarkup includes 6 type-safe presets for common structured data types. Each preset validates required fields at build time and generates spec-compliant JSON-LD.

Preset Schema type Use case webSite WebSite Site-level schema with optional search action organization Organization Company or brand identity article Article Blog posts, news, content pages faqPage FAQPage Question and answer pages product Product E-commerce product pages offer Offer Pricing and availability

Using presets

Apply schemas globally (every page) or per-page. Global schemas like webSite and organization go in globalSchemas. Page-specific schemas go in pages. The schema config is adapter-agnostic, so the same object works with @agentmarkup/vite, @agentmarkup/astro, and @agentmarkup/next.

const agentmarkupConfig = {
 site: 'https://myshop.com',
 name: 'My Shop',
 globalSchemas: [
 { preset: 'webSite', name: 'My Shop', url: 'https://myshop.com' },
 { preset: 'organization', name: 'My Shop', url: 'https://myshop.com', logo: '/logo.png' },
 ],
 pages: [
 {
 path: '/faq',
 schemas: [{
 preset: 'faqPage',
 url: 'https://myshop.com/faq',
 questions: [
 { question: 'Do you ship internationally?', answer: 'Yes, to 50+ countries.' },
 ],
 }],
 },
 ],
}

Framework wrappers

After defining the shared schema config, pass it into the adapter for the framework that owns your final output:

// Vite
import { defineConfig } from 'vite'
import { agentmarkup } from '@agentmarkup/vite'

export default defineConfig({
 plugins: [agentmarkup(agentmarkupConfig)],
})

// Astro
import { defineConfig } from 'astro/config'
import { agentmarkup } from '@agentmarkup/astro'

export default defineConfig({
 integrations: [agentmarkup(agentmarkupConfig)],
})

// Next.js
import type { NextConfig } from 'next'
import { withAgentmarkup } from '@agentmarkup/next'

const nextConfig: NextConfig = {
 output: 'export',
}

export default withAgentmarkup(agentmarkupConfig, nextConfig)

// Nuxt
export default defineNuxtConfig({
 modules: ['@agentmarkup/nuxt'],
 agentmarkup: agentmarkupConfig,
})

// CLI (any other built static site)
// agentmarkup generate ./dist

Custom schemas

You can use any schema.org type by passing an object with an @type field. agentmarkup automatically adds the @context and handles serialization.

pages: [
 {
 path: '/products/wallets',
 schemas: [{
 '@type': 'Product',
 name: 'Classic Leather Wallet',
 description: 'Full-grain leather bifold wallet.',
 image: 'https://myshop.com/images/wallet.jpg',
 sku: 'WALLET-001',
 offers: {
 '@type': 'Offer',
 price: '89',
 priceCurrency: 'USD',
 availability: 'https://schema.org/InStock',
 },
 }],
 },
]

XSS-safe output

agentmarkup escapes dangerous characters (<, >, &, ') to unicode escapes before injecting JSON-LD into HTML. This prevents XSS attacks through structured data injection.

<script type="application/ld+json">
{"@context":"https://schema.org","@type":"WebSite","name":"My Shop","url":"https://myshop.com"}
</script>

Build-time validation

Every schema is validated during build. Missing required fields produce errors. Missing recommended fields produce warnings. You see exactly what needs fixing in your terminal before deploying.

Combined with llms.txt generation, markdown mirrors, and AI crawler management, this gives your website a fuller machine-readable surface instead of relying on structured data alone.

Frequently asked questions

Do I need JSON-LD if I already have meta tags?

Yes. Meta tags (title, description) help search engines understand a single page. JSON-LD tells them what kind of thing the page represents (a product, an article, an FAQ) with structured fields they can use for rich results.

Can I add multiple schemas to one page?

Yes. Use the pages config to add multiple schemas per path. Each schema generates its own <script type="application/ld+json"> tag. Global schemas are also added to every page.

What if I need a schema type that is not a preset?

Pass any object with an @type field. agentmarkup will add @context automatically and serialize it safely. Presets just save you from remembering required fields.

How to manage AI crawlers

Allow or block AI crawlers like GPTBot and ClaudeBot via robots.txt

Source: https://agentmarkup.dev/docs/ai-crawlers/ Preferred fetch: https://agentmarkup.dev/docs/ai-crawlers.md

AI companies use web crawlers to collect training data and power AI-generated answers. Your robots.txt file controls which AI bots can access your site. agentmarkup generates or patches your robots.txt with AI-specific directives at build time.

In simple terms

Intermediate Website owners and people who manage hosting About 9 minutes

AI services use automated visitors called crawlers. Your robots.txt file can tell those visitors which public areas they may use, but it does not replace login controls or real security.

Check your website access rules

Which AI crawlers exist?

Major AI companies identify their crawlers with specific user-agent strings, grouped here by intent. Because robots.txt rules are per user-agent, you can block training crawlers while keeping search and retrieval crawlers allowed, so your site stays eligible for citation in AI answers. agentmarkup recognizes the following crawlers out of the box:

Crawler Company Intent GPTBot OpenAI Model training ClaudeBot Anthropic Model training Google-Extended Google Model training (separate from Google Search) CCBot Common Crawl Model training (open web dataset) Applebot-Extended Apple Model training Amazonbot Amazon Model training and search OAI-SearchBot OpenAI AI search / retrieval PerplexityBot Perplexity AI search / retrieval Claude-SearchBot Anthropic AI search / retrieval DuckAssistBot DuckDuckGo AI search / retrieval ChatGPT-User OpenAI User-triggered fetch Claude-User Anthropic User-triggered fetch Perplexity-User Perplexity User-triggered fetch

You can also add custom crawler names for any bot not in the built-in list. Note that some crawlers, such as Bytespider and xAI/Grok agents, comply poorly with robots.txt, so directives are not a guarantee.

Configuration

Set each crawler to 'allow' or 'disallow'. Only configure the crawlers you care about. Missing crawlers are not added to your robots.txt.

agentmarkup({
 site: 'https://example.com',
 name: 'My Website',
 aiCrawlers: {
 GPTBot: 'allow',
 ClaudeBot: 'allow',
 PerplexityBot: 'allow',
 'Google-Extended': 'allow',
 CCBot: 'disallow',
 },
})

How it works

agentmarkup uses marker comments to manage its section of your robots.txt. If you already have a robots.txt, the plugin patches it without touching your existing rules. If you do not have one, it creates a new file.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

# BEGIN agentmarkup AI crawlers
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Disallow: /

# END agentmarkup AI crawlers

The markers (# BEGIN agentmarkup AI crawlers / # END agentmarkup AI crawlers) allow the plugin to update its rules on every build without duplicating entries or breaking your custom rules.

Conflict detection

If your existing robots.txt has a User-agent: * with Disallow: /, and you configure a crawler to be allowed, agentmarkup warns you about the conflict during build. A broad disallow rule overrides specific allow rules for most crawlers.

This validation catches a common mistake: you intend to allow GPTBot but your existing robots.txt blocks all bots. Without this check, your allow directive would have no effect.

Should you allow or block AI crawlers?

This is a business decision, not a technical one. Consider:

Combined with llms.txt, JSON-LD structured data, and markdown mirrors, crawler access is one part of a machine-readable website instead of a standalone fix.

Frequently asked questions

Does blocking an AI crawler actually work?

Most major AI companies (OpenAI, Anthropic, Google) have committed to respecting robots.txt directives for their crawlers. Compliance is voluntary but widely honored. Smaller or unknown crawlers may not comply.

What is the difference between GPTBot and ChatGPT-User?

GPTBot crawls pages for training data. ChatGPT-User is used when a ChatGPT user asks the model to browse a specific URL. They are separate user agents with separate purposes. agentmarkup supports both.

Can I add custom crawler names?

Yes. The aiCrawlers config accepts any string as a key, not just the built-in names. This lets you add rules for new or niche crawlers as they appear.

Blog

I built AgentMarkup Studio with WebMCP

AgentMarkup Studio uses WebMCP so a browser agent can edit a visible draft of a site's machine-readable surface while rules catch cross-surface contradictions

Source: https://agentmarkup.dev/blog/webmcp-agentmarkup-studio/ Preferred fetch: https://agentmarkup.dev/blog/webmcp-agentmarkup-studio.md

Advanced People who build websites Implementation

I built AgentMarkup Studio with WebMCP

AgentMarkup Studio is now live. A browser AI agent can edit a visible draft of a website's machine-readable surface while the person at the keyboard watches every change. A deterministic compiler turns that draft into real artifacts in the browser, and cross-surface contradiction rules check whether those artifacts agree.

I built the Studio as agentmarkup's entry for The WebMCP Challenge. Judging has not happened. The work was deciding what an agent should be allowed to change, how it should learn what actually changed, and how to catch a draft whose individually reasonable choices conflict.

There is a two-minute demo video if you would rather watch ChatGPT drive the Studio than read about it. It ends on the contradiction the agent tries to avoid.

What does WebMCP change on a web page?

The tools live in the page. WebMCP lets the page register structured JavaScript actions through document.modelContext.registerTool, with a navigator.modelContext fallback and an AbortController tied to the page lifetime. The compatible browser exposes those actions to its agent while the page remains visible.

The agent is not operating on a hidden copy of the form. It reads and edits the same in-memory draft shown in the UI. A manual edit and an agent tool call pass through the same reducer, update the same artifact previews, and appear in the same activity log.

The Studio registers eight tools:

Tool What it does get_studio_state Reads a compact draft, contradictions, and recent activity. set_site_identity Sets the site identity and optional Organization data. set_access_policy Sets crawler rules by intent group or name, plus Content-Signal. curate_agent_pages Edits llms.txt sections, usage guidance, and mirror settings. configure_agent_card Configures and validates the optional A2A Agent Card. compile_agent_surface Compiles the draft and reports artifact sizes and findings. export_build_plan Returns the complete config when it fits the result budget. inspect_site Imports a bounded starting point from the website checker.

WebMCP is an emerging proposal, not a shipped web standard. Browser support is still narrow, which is why the page keeps every control available for manual editing when no agent is connected.

Why does the Studio need a contradiction engine?

Two settings can pass validation on their own and still express conflicting policies once combined. A validator that checks each file in isolation never sees it.

C1, "Cited content blocks retrieval", is the clearest example. The draft has one or more pages in llms.txt, which presents those pages for discovery and use. At the same time, the AI crawler policy blocks every search and retrieval crawler known to the Studio: OAI-SearchBot, PerplexityBot, Claude-SearchBot, and DuckAssistBot.

Neither side is malformed. The llms.txt entry has a title and URL. The robots.txt directives are valid. In combination, the site asks for cited content to be found while blocking the crawlers that retrieve it. C1 reports both surfaces and the number of listed entries. It does not promise that allowing retrieval will make any AI system include, rank, cite, or send traffic to the site.

C2 through C8 apply the same test to Content-Signal against llms.txt and mirrors, listed pages against mirror exclusions, site identity against Organization JSON-LD and the Agent Card, training rules against the training signal, and guidance with nothing behind it. Each finding names the surfaces involved.

The Studio produces errors and warnings, not a readiness score. The finding has to point to a defined disagreement that a person can inspect. That is also the line I draw in the broader explanation of what GEO can and cannot claim.

What stays human-only when an agent edits the draft?

The agent can set identity, access, content, and Agent Card fields. It can compile the result and read the build plan. It has no tool for undo, reset, or download. Those controls stay in the visible page for the human.

Every reducer action records its source at dispatch. The activity table shows an Agent or Human badge beside the exact summary of the applied change. Provenance is part of the state transition, not a label inferred later from whichever control happened to move.

Undo rewinds the shared log, reset clears the working draft, and download writes a file that can leave the browser. The person can reverse an agent change; the agent cannot reverse that correction or save a file on its own.

The draft is in-memory only. Nothing the agent does in the Studio changes a live website. The human downloads agentmarkup.config.mjs, reviews it, installs it in a repository, and runs the normal build and deployment separately.

How does the agent read exactly what was applied?

Returning the arguments from a successful tool call would be easy, but it would also be false whenever the reducer sanitized a field or dropped a crawler at the configured cap. Mutating tool results are authored from the reducer's latest activity entry. The agent reads what landed, including the dropped count, instead of an echo of what it requested.

Because React dispatch is asynchronous, a tool could dispatch an action and then read the previous reducer state. The UI would be correct a render later, but the agent's result would be one action behind.

The reducer is pure, so the tool path now pre-applies it to a ref before handing the same action to React:

const previousState = stateRef.current
const nextState = studioReducer(previousState, action)
stateRef.current = nextState

dispatch(action)

Immediate reads use stateRef.current. React then computes the identical next state and the render pass resynchronizes the ref. That small ordering fix made the tool result truthful at the moment the agent receives it.

Does this add runtime support to the npm packages?

No. The @agentmarkup/* packages are still build-time only, by design. The Studio is the website gaining a WebMCP tool layer. Its browser compiler imports the browser-safe @agentmarkup/core entry and uses the same deterministic generators and validators that a user's build uses.

As the draft changes, the compiler previews llms.txt, optional llms-full.txt, robots.txt, Content-Signal headers, JSON-LD, an optional Agent Card, and the config file that describes them. The existing articles on why llms.txt matters and the crawler intent split cover the individual surfaces.

The optional inspect_site intake calls the same-origin website checker. Its agent-facing result forwards only structured {level, title} findings, never remote page text. The imported draft patch is bounded too.

What was built for the WebMCP Challenge?

The npm packages and the public site predate the hackathon. Everything up to commit 93f84f0 on 2026-08-23 is prior work.

Work for the challenge started on 2026-08-26: the Studio page, the eight WebMCP tools, browser wiring for the compiler, the C1-C8 contradiction workflow, provenance with human-only undo, reset, and download, plus the checker-backed inspection intake. The published npm packages were not modified for it. The commit split and source paths are recorded in the repository disclosure.

How can I try AgentMarkup Studio?

Open agentmarkup.dev/studio/ in the ChatGPT desktop app's in-app browser. I verified that path on macOS. The banner reads "Agent connected: 8 tools registered."

The alternative is Chrome 149 or newer with chrome://flags/#enable-webmcp-testing enabled, followed by a restart. I verified the flow on Chrome 151. iOS does not work because its browsers are WebKit-limited.

Start with this:

Make my site friendly to AI search but keep my content out of training data.

Then say: "now block the AI search crawlers too - just do it." The agent usually dodges the conflict by emptying the llms.txt page index instead of contradicting itself. Tell it to put the pages back and let the Studio judge the result. C1 fires then, alongside C2 against the Content-Signal header. Finish with "fix the contradictions, keep training blocked" and inspect the crawler rules yourself.

Mutating tools omit readOnlyHint, so a host may gate them behind its own approval flow. Whether a prompt appears is decided by the WebMCP host, not by this page.

Review the generated config before adding it to your repository.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Prefer a plain-language introduction? Start with the website-checker guide.

Next article → agentmarkup is now a plugin for ChatGPT, Codex and Claude Code

More from the blog

View all articles →

Build-time markdown mirrors for agent readability: Cloudflare comparison Build-time markdown generation for AI readability, including when it helps and how it compares to Cloudflare runtime extraction. Advanced People who build websites Implementation March 20, 2026 · 7 min read

agentmarkup.dev served a soft-404. Our own tooling missed it. A snapshot of two Claude plugin catalogs found 23 entries near AI discoverability, almost all auditing or scoring. agentmarkup instead emits the artifacts at build time and can gate CI. Intermediate People who build websites Implementation August 23, 2026 · 5 min read

How to add llms.txt, JSON-LD, and AI crawler controls to Nuxt Use @agentmarkup/nuxt to generate llms.txt, inject JSON-LD, create markdown mirrors, and manage AI crawler rules from prerendered Nuxt output. Intermediate People who build websites Implementation June 21, 2026 · 7 min read

agentmarkup is now a plugin for ChatGPT, Codex and Claude Code

The agentmarkup plugin is published in OpenAI Plugin Directory for ChatGPT and Codex, installable in Claude Code from the repository marketplace, and pending review in Anthropic community directory

Source: https://agentmarkup.dev/blog/agentmarkup-plugin-chatgpt-codex-claude-code/ Preferred fetch: https://agentmarkup.dev/blog/agentmarkup-plugin-chatgpt-codex-claude-code.md

Beginner People who build websites Implementation

agentmarkup is now a plugin for ChatGPT, Codex and Claude Code

agentmarkup is now published in OpenAI's Plugin Directory, which is shared by ChatGPT and Codex. You can install it there, point it at a website repository, and use a guided workflow to choose the right package, update the build, inspect the generated output and validate what agents will actually receive.

Available now in OpenAI's Plugin Directory

I submitted the agentmarkup plugin to OpenAI, and it was approved and published on 2026-08-23. Because ChatGPT and Codex share the directory, the same listing is available from either product.

I have also submitted it to Anthropic's community plugin directory. That submission is still pending review, so agentmarkup is not listed in Anthropic's public catalog yet. Anthropic publishes no review-time commitment, and an approved plugin then waits for the catalog's nightly synchronization job.

A directory listing provides discovery, not access. Claude Code users can install the plugin today from the repository's own marketplace, without waiting for the community directory review:

/plugin marketplace add agentmarkup/agentmarkup
/plugin install agentmarkup@agentmarkup

OpenAI and Anthropic conduct independent reviews. OpenAI's guide explicitly notes that Claude marketplace listings and approvals do not transfer, so publication in one directory does not imply approval in the other.

What the plugin actually does

The plugin is a skill, so it runs as a guided sequence rather than hiding the work behind one command. Its job is to connect the repository, the build system and the final machine-readable output without assuming that the first framework name it sees is the right integration point.

Existing curated llms.txt, robots.txt, _headers and JSON-LD are preserved by default. The plugin does not overwrite work already present in the repository unless the user explicitly chooses replacement.

There is also a clear boundary. In a plain chat window with no shell and no repository, the plugin cannot install dependencies or run a build. It can read what the user pastes or links, identify which layer owns the final HTML, recommend the appropriate package and write the configuration for the user to apply. It says that directly instead of describing work it did not perform.

The packages behind the workflow

The plugin selects from the published packages according to the project and the layer that produces its final output. The build-time packages share a release line, while the live audit package tracks its own. Current releases are available on npm.

Package What it is for

@agentmarkup/core

Framework-agnostic generators, presets, validation and HTML helpers. The layer everything else is built on, and the one to use directly for a custom pipeline or a dynamic SSR route.

@agentmarkup/vite

Vite adapter.

@agentmarkup/astro

Astro integration.

@agentmarkup/next

Next.js adapter, including a final-output build pass over exported or prerendered HTML.

@agentmarkup/nuxt

Nuxt module, processing prerendered or nuxt generate output.

@agentmarkup/cli

Framework-agnostic command that runs the whole pipeline over any directory of built HTML, plus a check subcommand that validates without writing and is the right thing to put in CI.

@agentmarkup/audit

Fetches a live URL the way AI crawlers see it, under user agents including GPTBot, ClaudeBot and PerplexityBot.

Between them, these packages generate llms.txt, an optional llms-full.txt, JSON-LD structured data, markdown mirrors, AI crawler rules in robots.txt, _headers with Content-Signal directives and optional A2A Agent Card discovery at /.well-known/agent-card.json.

Everything runs at build time. There is no runtime component, hosted service or account.

Why I built it

I run Anima Felix. ChatGPT and Claude could find the site, but they could not really understand what it did. I fixed that for Anima Felix first with tools I wrote for the job, then open-sourced those tools.

That origin still sets the standard. The work is useful when it changes what the site actually serves and keeps that output from drifting afterwards. An audit points at a problem. A build integration makes the fix part of the site, and agentmarkup check in CI stops it from quietly breaking again later.

What it will not claim

The plugin will not promise that any AI system will include, rank, cite or send traffic to a site. Nobody controls that, and anyone selling that promise is guessing.

What it will do is make the site's machine-readable layer concrete, testable and part of the build, so what agents receive is something you chose deliberately rather than something you inherited.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Prefer a plain-language introduction? Start with the website-checker guide.

← Previous article I built AgentMarkup Studio with WebMCP

Next article → agentmarkup.dev served a soft-404. Our own tooling missed it.

More from the blog

View all articles →

Run agentmarkup on any static site with the CLI Use @agentmarkup/cli to run llms.txt, JSON-LD, markdown mirrors, and AI crawler controls over any built static output, with a CI check command. Intermediate People who build websites Implementation June 21, 2026 · 6 min read

How to add llms.txt, JSON-LD, and AI crawler controls to Next.js Use @agentmarkup/next to generate llms.txt, inject JSON-LD, manage AI crawler rules, and understand the dynamic SSR boundary in Next.js. Intermediate People who build websites Implementation March 23, 2026 · 8 min read

agentmarkup.dev served a soft-404. Our own tooling missed it.

Why AI-discoverability audits and scores are not enough, what our own tools missed, and how build-time generation turns findings into site changes and CI gates

Source: https://agentmarkup.dev/blog/soft-404-ai-discoverability-tools/ Preferred fetch: https://agentmarkup.dev/blog/soft-404-ai-discoverability-tools.md

Intermediate People who build websites Implementation

agentmarkup.dev served a soft-404. Our own tooling missed it.

agentmarkup is a tool for making websites machine-readable. Its own website was telling machines that pages existed when they did not. Every unknown path returned HTTP 200 with the app shell, so an agent probing /openapi.json or /about could reasonably conclude that both resources existed. At the time, neither did.

The bug was real, and the blind spot was ours

The cause was ordinary hosting behavior. Cloudflare Pages falls back to index.html when the deployed output has no 404.html. That fallback returned the application shell with a successful status for any invented path. To a person, it looked like a not-found page after the client app loaded. To a fetch-based agent, it looked like a page that existed.

I fixed it the same week. The build now emits a real prerendered 404.html, and the worker negotiates the response body. Agents that explicitly prefer markdown or plain text get a short machine-readable 404. Everyone else gets the HTML page. Both variants return HTTP 404 and include Accept in the Vary header.

The uncomfortable part is why the bug survived. Neither the hosted checker nor @agentmarkup/audit requests a path that should not exist. Both defend against soft-404 responses on the assets they already fetch, but neither performs a deliberate probe. That gap is worth naming rather than glossing over.

Update: this check now ships. Both @agentmarkup/audit 0.2.5 and the hosted checker request a path that should not exist and compare the response against the homepage. A 404 or 410 passes. A 200 matching the homepage is an error, because every path then appears to exist. A probe that never completes reports that the check could not be made, because a timeout and a soft-404 are not the same thing.

The honest output is a specific limitation, a reproducible case, and a check that still needs to be added.

In Claude's plugin catalogs, most of these tools stop at the report

On 2026-08-23, I fetched the two Claude plugin catalog manifests directly and matched every entry name and description against llms.txt, GEO, AEO, AI search, and AI crawler keywords, then read the matches.

This is a dated catalog snapshot, not a claim about every tool on the internet. It does show a clear pattern in these catalogs: run an audit, summarize the findings, produce a score, and stop. Audits are useful, including ours. But after a report says your llms.txt is missing or your structured data is invalid, the site is still exactly as it was before the report ran.

What a build integration does instead

The agentmarkup plugin starts by inspecting which layer owns the final HTML. It then proposes one plan: the package to install, the files to change, the commands to run, and any live origin it would audit. Nothing is installed, written, built, or fetched until the user approves that plan.

Once approved, it installs the appropriate build-time integration, writes the configuration, runs the real site build, and validates the emitted output. Depending on the project, that can mean @agentmarkup/vite, @agentmarkup/astro, @agentmarkup/next, @agentmarkup/nuxt, the framework-agnostic @agentmarkup/cli, or direct helpers from @agentmarkup/core.

The resulting build can contain:

The CLI's check command can then run after the normal build and exit non-zero when required output is broken. That is the practical difference. The result is not another report to remember. It is a dependency, configuration, generated artifacts, and a CI condition that remain with the repository.

No score is part of the honesty

agentmarkup deliberately does not produce a readiness score, grade, or percentage. A missing required field is an error. A missing recommended field is a warning. That is the entire scale.

Collapsing unrelated checks into one number requires hidden choices about how much each item is worth. It also invites a much larger inference: that a higher number predicts inclusion, ranking, traffic, or citation by an AI system. We do not have evidence for that, so the tool does not make the claim.

What it can establish is narrower and useful: whether an artifact exists, whether it parses, whether required fields are present, whether crawler rules conflict, whether final HTML contains real content, and whether the build stays valid in CI. When a check does not exist, as with our soft-404 probe, the right response is to say so plainly.

Why I built it this way

Anima Felix needed this first. ChatGPT and Claude could find the site, but they could not really understand it. I fixed it there, then open-sourced the tooling we used.

That origin still sets the standard. The work is only useful when it changes what the site serves and keeps that output from drifting. A good audit points to the problem. A build integration makes the fix part of the site.

Use the plugin where you already work

agentmarkup was published in OpenAI's Plugin Directory on 2026-08-23. That directory is shared by ChatGPT and Codex, and the plugin can be installed from there.

The plugin has also been submitted to Anthropic's community directory and is pending review. Claude Code users can install it today from the repository's own marketplace:

/plugin marketplace add agentmarkup/agentmarkup
/plugin install agentmarkup@agentmarkup

The build-time packages are @agentmarkup/core, vite, astro, next, nuxt, and cli. The live crawler package is @agentmarkup/audit. Current versions are on npm.

Before a live audit, the plugin names the target origin and asks the user to confirm that they own or operate it. Approval covers the stated plan, not the rest of the session.

The plugin will not promise that an AI system will include, rank, cite, or send traffic to a site. It will make the site's machine-readable layer concrete, testable, and part of the build.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Prefer a plain-language introduction? Start with the website-checker guide.

← Previous article agentmarkup is now a plugin for ChatGPT, Codex and Claude Code

Next article → We ran 500 of America's biggest companies through an AI-crawler audit

More from the blog

View all articles →

We audited 500 of America's biggest companies for AI readiness

We ran 500 of the largest US public companies through @agentmarkup/audit: most serve readable HTML, but 46% have no usable structured data, 86% have no llms.txt, and seven serve crawlers a blank page

Source: https://agentmarkup.dev/blog/ai-crawler-audit-500-companies/ Preferred fetch: https://agentmarkup.dev/blog/ai-crawler-audit-500-companies.md

Intermediate Research AI access

We ran 500 of America's biggest companies through an AI-crawler audit

America's biggest companies built their websites for Google's crawler. Almost none have built anything for the AI agents now reading the web on people's behalf. We fetched 500 corporate homepages the way ChatGPT, Claude, and Perplexity do. Most serve readable HTML, but the layer that makes a page reliably machine-legible is mostly missing, and seven serve a crawler nothing but a blank page.

Structured data

46 %

have no usable JSON-LD

nothing machine-readable saying what the page is

Discovery

86 %

have no valid llms.txt

missing or malformed AI-discovery file

Usage rules

99 %

set no AI-usage signal

Content-Signal, the new opt-in standard

Broken

serve crawlers a blank page

content hidden behind JavaScript they don't run

The AI-era layer is almost empty

These are the signals that let an AI agent parse a page with confidence, point back to it, and respect how you want it used. Adoption falls off a cliff, and you can inspect any single company below.

Each square is one of the 370 companies, inked in if it has the signal. Click any square, use the arrow keys, or search to see that company's full result.

Structured data 201 / 370 · 54%

llms.txt 50 / 370 · 14%

Content-Signal 3 / 370 · under 1%

has the signal does not have the signal Click a square or use arrow keys for details.

It is not just what is missing

Some of what we found is not an empty field but a broken one: a defect provable from the response itself, the kind that fails a CI check.

robots.txt

disallow an AI crawler

block GPTBot or a peer outright

Structured data

ship broken JSON-LD

markup an agent cannot parse

Bait and switch

show crawlers less than a browser

the bot gets a thinner page than you

Near-empty

serve thin HTML

barely enough for a crawler to use

And not one earned a clean bill of health. Every site tripped at least one check, most often the missing llms.txt; 27 tripped a hard, build-breaking error.

27 hard errors 343 with warnings 0 fully clean

The pages are readable. The signals are missing.

Can an AI just read the raw HTML? For most of these sites, yes: 94% server-render real content and 87% publish a sitemap, so a crawler can reach the words on the page. That is table stakes, and these companies clear it. Crawlability was never the hard part.

The gap is everything that turns readable text into reliable machine input: structured data that declares what a page is, an llms.txt that points an agent at the canonical summary, and Content-Signal headers that state how the content may be used. Those are mostly absent. And for seven companies even the baseline fails: the homepage is an empty JavaScript shell, so a crawler that does not run JS sees nothing at all.

None of this is exotic to fix. These are static files and a few tags, the AI-era equivalent of the sitemap every one of these companies already ships. The winners are the ones who add them first. A few already have: Target, NVIDIA, Adobe, American Express, and Dell all serve a valid llms.txt today.

How we measured (and what we are not claiming)

Check your own site in one command

You do not have to take our word for any of this. Point the same audit at your homepage:

npx @agentmarkup/audit https://yourdomain.com

Prefer a browser? Run the hosted website checker. When it finds gaps, the llms.txt, JSON-LD, and AI crawler guides show how to close them at build time, and the audit guide explains every check.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

AI crawlers guide

Ready for the implementation details? Continue with the related guide.

← Previous article agentmarkup.dev served a soft-404. Our own tooling missed it.

Next article → See your website the way AI crawlers do

More from the blog

View all articles →

Every AI crawler indexing your website in 2026 Complete list: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and more. What each does and how to control access. Intermediate People who build websites AI access March 20, 2026 · 8 min read

See your site like AI crawlers do

Fetch any live URL as each major AI crawler with @agentmarkup/audit, diff against a browser, and catch machine-readability issues in CI

Source: https://agentmarkup.dev/blog/audit-ai-crawler-access/ Preferred fetch: https://agentmarkup.dev/blog/audit-ai-crawler-access.md

Advanced People who build websites AI access

See your website the way AI crawlers do

Most tools that grade a website fetch it once, as a browser, and score the HTML. But the systems that increasingly decide whether your brand shows up in an answer, ChatGPT, Claude, Perplexity, Google's AI surfaces, do not arrive as your browser. They arrive as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended, and they can get a very different response. @agentmarkup/audit shows you that response.

The blind spot

A page can look perfect in your browser and still be a poor citation target for AI. The homepage might be an empty JavaScript shell that only fills in after a framework hydrates, so a crawler that does not run JavaScript sees nothing. A CDN or WAF rule might treat a crawler user-agent differently than a browser. There might be no llms.txt, or a malformed one. The JSON-LD that powers rich results and AI summaries might be missing or broken. None of that is visible from a single browser fetch.

What the audit does

It fetches your URL once as a normal browser to establish a baseline, then again as each major AI crawler, and diffs the responses. On top of that it checks the machine-readable surface: robots.txt intent, Content-Signal, llms.txt, JSON-LD, and whether the raw HTML is actually readable without JavaScript.

# Audit any live URL as the major AI crawlers
npx @agentmarkup/audit https://example.com

# JSON for CI or comparisons
npx @agentmarkup/audit https://example.com --json

A run reads like this:

✓ OpenAI gptbot can reach the page
✓ Anthropic claudebot can reach the page
✓ Content is present without JavaScript
⚠ llms.txt is missing
✓ robots.txt does not block the expected AI crawlers
✓ JSON-LD structured data present

9/10 checks passed

Honest by design

Here is the part that makes the audit trustworthy rather than alarmist. It spoofs a crawler's user-agent from an ordinary IP, which is not what the real, verified bot does. So a 403 for a spoofed GPTBot user-agent is genuinely ambiguous: it could be a user-agent WAF rule that also blocks the real GPTBot, or it could be IP allowlisting where the verified GPTBot is let through just fine. The audit cannot tell those apart from a spoofed request, so it reports them as warnings with both explanations and the raw evidence, never as a bare "your site blocks AI" error.

Error-level findings, the ones that fail CI, are reserved for things provable from the response itself: a robots.txt that literally disallows the crawler, an empty JavaScript shell, or invalid llms.txt / JSON-LD. That is why the exit code is safe to gate a build on.

Where it fits

The agentmarkup adapters and the CLI generate machine-readable output at build time. @agentmarkup/audit verifies what a deployed site actually serves to AI crawlers. It is the command-line sibling of the hosted website checker: the checker is the quick browser lookup, the audit is the scriptable, CI-friendly version.

Read the audit guide for the full check list, then use the llms.txt, JSON-LD, and AI crawlers guides to fix whatever it surfaces.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Prefer a plain-language introduction? Start with the website-checker guide.

← Previous article We ran 500 of America's biggest companies through an AI-crawler audit

Next article → How to add llms.txt, JSON-LD, and AI crawler controls to Nuxt

More from the blog

View all articles →

I built AgentMarkup Studio with WebMCP AgentMarkup Studio uses WebMCP so a browser agent can edit a visible draft of a site's machine-readable surface while rules catch cross-surface contradictions. Advanced People who build websites Implementation August 27, 2026 · 9 min read

Next.js guide

How to add llms.txt, JSON-LD, AI crawler controls, and validation to Next.js with @agentmarkup/next

Source: https://agentmarkup.dev/blog/nextjs-llms-txt-json-ld/ Preferred fetch: https://agentmarkup.dev/blog/nextjs-llms-txt-json-ld.md

Intermediate People who build websites Implementation

How to add llms.txt, JSON-LD, and AI crawler controls to Next.js

Next.js sites need the same machine-readable surface as any other modern website: llms.txt, structured data, crawler rules, and validation. The tricky part is choosing the right integration point so those artifacts reflect your final output instead of an earlier build step.

Why Next.js is slightly different

With plain Vite or Astro, the final HTML output is usually obvious. Next.js can mix static export, prerendered pages, server deployments, and fully dynamic SSR routes in the same app. That means a useful Next integration cannot just be a generic bundler plugin. It has to respect what Next actually emits at build time.

That is what @agentmarkup/next is for. It is a final-output-first adapter built around Next's config and build hooks rather than a Vite-style HTML transform.

What the Next.js adapter gives you

Basic setup

Install the package:

pnpm add -D @agentmarkup/next

Then wrap your Next config:

// next.config.ts
import type { NextConfig } from 'next'
import { withAgentmarkup } from '@agentmarkup/next'

const nextConfig: NextConfig = {
 output: 'export',
}

export default withAgentmarkup(
 {
 site: 'https://example.com',
 name: 'Example Docs',
 description: 'Technical docs and product pages.',
 globalSchemas: [
 { preset: 'webSite', name: 'Example Docs', url: 'https://example.com' },
 { preset: 'organization', name: 'Example Inc.', url: 'https://example.com' },
 ],
 llmsTxt: {
 sections: [
 {
 title: 'Documentation',
 entries: [
 {
 title: 'Getting Started',
 url: '/docs/getting-started',
 description: 'Setup guide and first steps',
 },
 ],
 },
 ],
 },
 llmsFullTxt: {
 enabled: true,
 },
 markdownPages: {
 enabled: true,
 },
 contentSignalHeaders: {
 enabled: true,
 },
 aiCrawlers: {
 GPTBot: 'allow',
 ClaudeBot: 'allow',
 PerplexityBot: 'allow',
 'Google-Extended': 'allow',
 CCBot: 'disallow',
 },
 },
 nextConfig,
)

The important thing to notice is that the config shape is shared across the first-party adapters. The shared AgentMarkupConfig stays framework-agnostic. Only the wrapper changes.

Where it works best

The strongest fit is static export and any route where Next emits build-time HTML that can be patched or post-processed. That includes a lot of real App Router sites: docs, marketing pages, blog pages, changelogs, and mixed apps with a meaningful prerendered surface.

On those builds, you get the full output flow:

out/
 llms.txt
 llms-full.txt
 robots.txt
 _headers
 docs/getting-started/index.html
 docs/getting-started.md

Server deployments are still useful too. You keep generated root artifacts and header integration, even when the deployment is not a pure static export.

The one caveat that matters

Fully dynamic SSR routes are the boundary. If Next never emits an HTML file for a route at build time, there is no final file for the adapter to patch afterward.

That does not make the package useless for Next apps. It just means you should be precise about ownership:

That is the honest model for Next. The package is strongest where Next has a final-output artifact. For routes without one, route-level core helpers are the right tool.

Should you enable markdown mirrors?

Only when they add signal. If your emitted HTML is already substantial, keep HTML as the primary fetch target. If the built page is thin, noisy, or heavily shell-like, generated markdown mirrors can give fetch-based agents a cleaner path.

agentmarkup keeps that feature disciplined by generating mirrors from final HTML, keeping them directly fetchable, and adding canonical headers back to the HTML route so search engines keep the page itself as the preferred URL.

Why this is useful for the Next.js community

A lot of Next.js teams already care about structured metadata, crawlability, and build output quality. They just do not want four separate solutions for llms.txt, JSON-LD, crawler policy, markdown mirrors, and validation.

The practical value of @agentmarkup/next is that it keeps those concerns in one build step, on the same config surface, with the same rules the public checker looks for on deployed sites.

The bottom line

If your Next.js app has a real static or prerendered surface, @agentmarkup/next is the natural adapter. It gives you build-time machine-readable output without making you stitch the pieces together manually.

Start with the adapter for the routes Next emits, keep markdown mirrors optional, and use @agentmarkup/core directly only where fully dynamic SSR makes that necessary. That is the cleanest model for shipping a machine-readable Next.js website today.

If you need the underlying pieces in more detail, read the llms.txt guide, the JSON-LD guide, and the AI crawlers guide.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Prefer a plain-language introduction? Start with the website-checker guide.

← Previous article Run agentmarkup on any static site with the CLI

Next article → When markdown mirrors help, and when they do not

More from the blog

View all articles →

Nuxt guide

How to add llms.txt, JSON-LD, markdown mirrors, and AI crawler controls to Nuxt with @agentmarkup/nuxt

Source: https://agentmarkup.dev/blog/nuxt-llms-txt-json-ld/ Preferred fetch: https://agentmarkup.dev/blog/nuxt-llms-txt-json-ld.md

Intermediate People who build websites Implementation

How to add llms.txt, JSON-LD, and AI crawler controls to Nuxt

Nuxt sites need the same machine-readable surface as any other modern website: llms.txt, structured data, markdown mirrors, crawler rules, and validation. The key is hooking into the build at the moment Nuxt has actually written your prerendered HTML to disk, so those artifacts reflect the final output.

Where the Nuxt adapter runs

@agentmarkup/nuxt is a Nuxt module that runs after Nitro finishes prerendering. It hooks Nitro's prerender:done event and processes the emitted .output/public directory: the same final-output-first model the other agentmarkup adapters use.

That means it is strongest on the nuxt generate path and on any route you prerender with routeRules or prerender: true. Those are the routes that produce build-time HTML files for the module to read and augment.

What the Nuxt adapter gives you

Basic setup

Install the module:

pnpm add -D @agentmarkup/nuxt

Then register it and pass the shared config under the agentmarkup key:

// nuxt.config.ts
export default defineNuxtConfig({
 modules: ['@agentmarkup/nuxt'],
 agentmarkup: {
 site: 'https://example.com',
 name: 'Example Docs',
 description: 'Technical docs and product pages.',
 globalSchemas: [
 { preset: 'webSite', name: 'Example Docs', url: 'https://example.com' },
 { preset: 'organization', name: 'Example Inc.', url: 'https://example.com' },
 ],
 llmsTxt: {
 sections: [
 {
 title: 'Documentation',
 entries: [
 {
 title: 'Getting Started',
 url: '/docs/getting-started',
 description: 'Setup guide and first steps',
 },
 ],
 },
 ],
 },
 llmsFullTxt: {
 enabled: true,
 },
 markdownPages: {
 enabled: true,
 },
 contentSignalHeaders: {
 enabled: true,
 },
 aiCrawlers: {
 GPTBot: 'allow',
 ClaudeBot: 'allow',
 PerplexityBot: 'allow',
 'Google-Extended': 'allow',
 CCBot: 'disallow',
 },
 },
})

The config shape is the same AgentMarkupConfig used by the Vite, Astro, and Next.js adapters. Only the integration point changes: Nuxt reads it from the agentmarkup key.

Where it works best

Run a prerendered build and you get the full output flow written into .output/public:

.output/public/
 llms.txt
 llms-full.txt
 robots.txt
 _headers
 docs/getting-started/index.html
 docs/getting-started.md

Because the module reads the directory Nitro actually wrote, the artifacts reflect your real final HTML rather than an earlier build step.

The one caveat that matters

Fully dynamic SSR routes are the boundary. If a route is rendered on demand by the Nitro server and never emits a build-time HTML file, there is no final file for the module to patch afterward.

That does not make the package useless for Nuxt apps. It just means you should be precise about ownership:

Should you enable markdown mirrors?

Only when they add signal. If your prerendered HTML is already substantial, keep HTML as the primary fetch target. If the built page is thin, noisy, or heavily shell-like, generated markdown mirrors can give fetch-based agents a cleaner path.

The bottom line

If your Nuxt app prerenders its pages, @agentmarkup/nuxt is the natural module. It gives you build-time machine-readable output on the same config surface as the other adapters, without stitching the pieces together by hand.

Start with the module for the routes Nuxt prerenders, keep markdown mirrors optional, and use @agentmarkup/core directly only where fully dynamic SSR makes that necessary.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Prefer a plain-language introduction? Start with the website-checker guide.

← Previous article See your website the way AI crawlers do

Next article → Run agentmarkup on any static site with the CLI

More from the blog

View all articles →

CLI for any static site

Run agentmarkup over any built static output and gate machine-readability in CI with @agentmarkup/cli

Source: https://agentmarkup.dev/blog/agentmarkup-cli-any-static-site/ Preferred fetch: https://agentmarkup.dev/blog/agentmarkup-cli-any-static-site.md

Intermediate People who build websites Implementation

Run agentmarkup on any static site with the CLI

Not every site uses a framework with a dedicated adapter. @agentmarkup/cli runs the same machine-readable pipeline over any built static output directory, and adds a CI check command that fails the build when machine-readability is broken.

Why a CLI

The Vite, Astro, Next.js, and Nuxt adapters each hook a specific build. The CLI has zero framework coupling: point it at a directory of emitted HTML and it generates llms.txt, markdown mirrors, JSON-LD, AI-crawler robots.txt, and _headers, then validates the result. That makes it the right tool for Eleventy, Gatsby, Hugo, Jekyll, Docusaurus, plain static HTML, and any pipeline without a dedicated adapter.

Setup

Install the CLI:

pnpm add -D @agentmarkup/cli

Add a config file in your project root:

// agentmarkup.config.mjs
export default {
 site: 'https://example.com',
 name: 'Example',
 outDir: 'dist', // optional default; or pass the directory as an argument
 llmsTxt: {
 sections: [
 {
 title: 'Docs',
 entries: [
 { title: 'Home', url: 'https://example.com/', description: 'Start here' },
 ],
 },
 ],
 },
 markdownPages: { enabled: true },
 aiCrawlers: { GPTBot: 'disallow' },
}

The config is the same AgentMarkupConfig the adapters use, plus an optional CLI-only outDir for path resolution.

Three modes

# Inject + write llms.txt, markdown mirrors, JSON-LD, robots.txt, _headers
agentmarkup generate ./dist

# Validate what is already on disk (CI gate) - exits non-zero on errors
agentmarkup check ./dist

# Preview without writing anything
agentmarkup generate ./dist --dry-run

A real CI gate

check is the piece the build-time adapters cannot offer on their own: a standalone command that fails CI when your deployed output would not be machine-readable. Add it after your build step:

# .github/workflows/ci.yml (excerpt)
- run: pnpm build
- run: pnpm exec agentmarkup check ./dist

Use --strict to also fail on warnings (for example a configured artifact that never got generated).

Coexistence by default

Existing curated llms.txt, existing robots.txt rules, and existing page JSON-LD are preserved by default; the CLI only fills gaps. Opt into replacement per feature in the config when you want the CLI to take over an output.

One safety note worth knowing: the CLI never auto-guesses public/ as an output directory, because it is a source asset directory in many frameworks. Pass it explicitly if you really mean it.

The bottom line

If your site emits static HTML, @agentmarkup/cli gives you the full agentmarkup pipeline without an adapter, plus a CI gate to keep your machine-readable output honest over time.

For more on the underlying pieces, read the llms.txt guide, the JSON-LD guide, and the AI crawlers guide.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Prefer a plain-language introduction? Start with the website-checker guide.

← Previous article How to add llms.txt, JSON-LD, and AI crawler controls to Nuxt

Next article → How to add llms.txt, JSON-LD, and AI crawler controls to Next.js

More from the blog

View all articles →

agentmarkup is now a plugin for ChatGPT, Codex and Claude Code The agentmarkup plugin is now published for ChatGPT and Codex, with a guided workflow for installing, building and validating machine-readable website output. Beginner People who build websites Implementation August 23, 2026 · 8 min read

When markdown mirrors help

Practical guide to when generated markdown mirrors help and when HTML is already enough

Source: https://agentmarkup.dev/blog/when-markdown-mirrors-help/ Preferred fetch: https://agentmarkup.dev/blog/when-markdown-mirrors-help.md

Intermediate People who build websites AI discoverability

When markdown mirrors help, and when they do not

Generated markdown mirrors are useful for some sites and unnecessary for others. The honest answer depends on what your raw HTML already looks like and whether agents need a cleaner fetch target than the page you already serve.

The problem mirrors are trying to solve

A public page can be technically crawlable and still be a bad machine-facing document. If the response is mostly app shell, navigation, layout wrappers, and scripts, fetch-based agents have to infer the real page body from noisy HTML.

A generated markdown mirror gives those agents a simpler fetch path: title, description, source URL, headings, lists, paragraphs, and code blocks without the surrounding chrome.

Where markdown mirrors help

Where markdown mirrors do not add much

This is why the strongest version of the feature is not "every page should publish markdown". It is "some pages benefit from a cleaner machine-facing artifact".

The real tradeoffs

The objections are real. Public mirrors can create duplicate fetches and indexing ambiguity. If they are hand-maintained, they also create a second source of truth that will eventually drift.

There is also a product risk: as agent tooling gets better at reading messy HTML directly, the gap that mirrors solve may narrow. That makes this more likely to be a tactical feature than the final shape of machine-readable publishing.

How agentmarkup tries to keep the feature disciplined

The more durable product surface

The long-term durable value is probably not "every site needs markdown mirrors". It is better tooling around agent-readiness: checking raw HTML quality, validating machine-readable outputs, verifying crawler policy, and making tradeoffs explicit.

That is why the checker matters. It can tell you whether the HTML is already good enough, whether a markdown mirror would add signal, and whether the rest of your machine-readable surface is coherent.

The bottom line

Markdown mirrors make sense as an optional, tactical feature for thin or noisy HTML. They are not a universal best practice, and they should not be marketed as one.

If your raw HTML already reads cleanly, keep HTML as the primary fetch target. If it does not, a generated markdown derivative can be a pragmatic bridge while the broader machine-readable stack keeps improving.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

llms.txt guide

Ready for the implementation details? Continue with the related guide.

← Previous article How to add llms.txt, JSON-LD, and AI crawler controls to Next.js

Next article → Is your website ready for AI? Free LLM discoverability checker

More from the blog

View all articles →

What is GEO? Generative Engine Optimization explained for developers What is real, what is hype, and what you can do today to make your site citeable by AI. Beginner Website owners AI discoverability March 20, 2026 · 7 min read

Why llms.txt matters: making your website discoverable by AI LLMs answer questions by synthesizing web content. llms.txt gives them a structured overview of your site. Beginner Website owners AI discoverability March 20, 2026 · 6 min read

Why llms.txt matters

How llms.txt makes your website discoverable by AI systems like ChatGPT and Perplexity

Source: https://agentmarkup.dev/blog/why-llms-txt-matters/ Preferred fetch: https://agentmarkup.dev/blog/why-llms-txt-matters.md

Beginner Website owners AI discoverability

Why llms.txt matters: making your website discoverable by AI

Search is changing. LLMs like ChatGPT, Claude, and Perplexity now answer questions by synthesizing information from the web. If your website is not structured for these systems to understand, you are invisible to a growing share of how people find information.

The shift from search engines to AI answers

For two decades, the path to online visibility was clear: optimize for Google, get ranked, get clicked. That model is not going away, but it is being joined by a second discovery channel. When someone asks ChatGPT "what is the best leather wallet under $100" or asks Perplexity "how do I add structured data to my website," the answer comes from synthesized web content, not a list of blue links.

The question for website owners is no longer just "can Google index my pages?" but also "can an LLM understand what my site is about?"

What is llms.txt?

llms.txt is a proposed standard that gives LLMs a structured overview of your website. It is a plain text file served at /llms.txt, written in simple markdown, that describes your site name, purpose, and pages.

Think of it as robots.txt for AI understanding. While robots.txt tells crawlers what they can access, llms.txt tells them what your site actually is.

# My Shop

> Handcrafted leather goods since 2015.

## Products

- [Wallets](https://myshop.com/products/wallets): Full-grain leather bifold wallets
- [Bags](https://myshop.com/products/bags): Handmade leather messenger bags

## Support

- [FAQ](https://myshop.com/faq): Common questions about orders and shipping
- [Contact](https://myshop.com/contact): Get in touch with our team

Why not just let AI read your HTML?

HTML pages are designed for humans. They contain navigation menus, cookie banners, ad scripts, analytics tags, and layout markup that has nothing to do with your actual content. An LLM reading your raw HTML has to guess what matters and what is noise.

llms.txt gives the LLM a clean, noise-free map of your site. No guessing required. It is the equivalent of handing someone a table of contents instead of making them skim every page.

Is llms.txt actually used today?

Honestly, adoption is early. Perplexity has shown interest in consuming llms.txt. Other major AI systems have not publicly committed to it. The format is a proposal from llmstxt.org, not an official standard.

But the cost of generating it is near zero. A build-time plugin like agentmarkup can produce it automatically from your existing site config. The downside risk is an extra text file in your build output. The upside is being ready when AI systems start looking for it.

The three layers of machine readability

llms.txt is one piece of a larger picture. A truly machine-readable website has three layers:

Each layer serves a different purpose. llms.txt is for AI discovery. JSON-LD is for search engine rich results. robots.txt is for access control. Together, they make your site understandable by both traditional search engines and the new generation of AI systems.

How to add llms.txt to your site

You can write llms.txt by hand, but keeping it in sync with your actual pages is tedious. A build-time tool does it automatically.

With agentmarkup, you configure your site structure once and the plugin generates llms.txt, injects JSON-LD, and patches robots.txt on every build:

pnpm add -D @agentmarkup/vite # or @agentmarkup/astro or @agentmarkup/next

Read the full llms.txt guide for configuration details.

The bottom line

The web is getting a second audience. Humans still matter, but AI systems are increasingly the first reader of your content. llms.txt is a low-cost, low-risk way to make sure those systems understand what you have built. Whether or not it becomes a formal standard, the practice of structuring your site for machine readability is here to stay.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

llms.txt guide

Ready for the implementation details? Continue with the related guide.

← Previous article What is GEO? Generative Engine Optimization explained for developers

More from the blog

View all articles →

Is your website ready for AI? Free LLM discoverability checker Audit your website for llms.txt, JSON-LD, robots.txt, markdown mirrors, and sitemap. Free tool for e-commerce and brand websites. Beginner Website owners AI discoverability March 20, 2026 · 8 min read

When markdown mirrors help, and when they do not A practical guide to when generated markdown mirrors add signal, when HTML is already enough, and how to avoid unnecessary downsides. Intermediate People who build websites AI discoverability March 20, 2026 · 7 min read

How to make your brand appear in AI conversations Organization schema, llms.txt, and FAQ markup make your brand visible in ChatGPT, Claude, and Perplexity answers. Beginner Website owners Business visibility March 20, 2026 · 7 min read

What is GEO?

Generative Engine Optimization explained for developers - what is real and what is hype

Source: https://agentmarkup.dev/blog/what-is-geo/ Preferred fetch: https://agentmarkup.dev/blog/what-is-geo.md

Beginner Website owners AI discoverability

What is GEO? Generative Engine Optimization explained for developers

GEO, or Generative Engine Optimization, is the practice of making your website more likely to be cited, summarized, or recommended by AI systems. It is the AI-era counterpart to SEO, and it matters whether or not you believe the hype.

SEO vs GEO: what changed

SEO optimizes for search engine result pages. You write titles, meta descriptions, and structured data so Google shows your page in a list of blue links. The user clicks, visits your site, and reads your content.

GEO optimizes for AI-generated answers. When someone asks ChatGPT, Claude, or Perplexity a question, the AI synthesizes an answer from web content. Your goal is not to rank in a list but to be cited in the answer. The user may never visit your site directly, but your content, brand, and expertise still shape the response.

This is not a replacement for SEO. It is an additional channel. Both matter.

What GEO actually involves

Strip away the marketing buzzwords and GEO comes down to a few practical things:

The honest truth about GEO

There is a lot of noise in the GEO space. Consultants promise "AI visibility scores" and "generative rankings." Most of these are made up. There is no public ranking algorithm for AI answers the way there is for Google Search.

What is real:

What is not real:

Why developers should care

If you build websites, the people who use those websites are increasingly finding information through AI. A potential customer asking Claude "what is the best project management tool for small teams" might get an answer that includes or excludes your product based entirely on whether your site has clear, machine-readable content.

The good news: most of what GEO requires is just good web development practice. Structured data, clean content, readable raw output, and intentional robots.txt rules. These are things you should be doing anyway.

Practical GEO for your website

Here is what you can do today, ordered by impact:

Automating GEO with agentmarkup

agentmarkup is an open-source package family with adapters for Vite, Astro, and Next.js that handles all of the above at build time. One config can generate llms.txt, inject JSON-LD, emit markdown mirrors, patch robots.txt and host-friendly headers, and validate everything. No runtime code, no SaaS, no recurring cost.

It does not promise AI rankings. It gives you the tools to make your site machine-readable and lets you decide what that means for your business.

The bottom line

GEO is real in the sense that AI systems are a growing discovery channel. It is overhyped in the sense that nobody can guarantee placement in AI answers. The practical response is to make your website machine-readable using proven techniques (JSON-LD, robots.txt) and emerging ones (llms.txt), and to do so without overspending or overcomplicating your stack.

The websites that will benefit most from AI-driven discovery are the ones that were already well-built: clear content, proper metadata, structured data. GEO is not a revolution. It is a reminder to do the fundamentals well.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

llms.txt guide

Ready for the implementation details? Continue with the related guide.

← Previous article JSON-LD structured data: the complete guide for web developers

Next article → Why llms.txt matters: making your website discoverable by AI

More from the blog

View all articles →

JSON-LD structured data guide

Complete guide to JSON-LD for web developers - schema types, common mistakes, and validation

Source: https://agentmarkup.dev/blog/json-ld-structured-data-guide/ Preferred fetch: https://agentmarkup.dev/blog/json-ld-structured-data-guide.md

Intermediate People who build websites Structured information

JSON-LD structured data: the complete guide for web developers

JSON-LD is the format Google, Bing, and AI systems use to understand your pages. This guide covers what it is, why it matters, which schema types to use, and how to add it to your site without shipping broken markup.

What is JSON-LD?

JSON-LD (JavaScript Object Notation for Linked Data) is a way to embed structured data in your HTML pages. It lives inside a <script type="application/ld+json"> tag in your page's <head> and describes what the page represents using the schema.org vocabulary.

Unlike microdata or RDFa, JSON-LD does not require changes to your visible HTML. You add a single script tag and search engines read it separately from your page content. This makes it easier to maintain, less error-prone, and the format Google explicitly recommends.

<script type="application/ld+json">
{
 "@context": "https://schema.org",
 "@type": "Organization",
 "name": "Acme Corp",
 "url": "https://acme.com",
 "logo": "https://acme.com/logo.png"
}
</script>

Why JSON-LD matters

Without structured data, search engines guess what your page is about from HTML content. With JSON-LD, you tell them explicitly. This enables:

Google's own documentation states that pages with valid structured data are eligible for rich results that pages without it cannot receive.

JSON-LD vs microdata vs RDFa

All three formats encode the same schema.org vocabulary. The difference is where they live:

For new projects, JSON-LD is the clear choice. It is easier to generate programmatically, easier to validate, and easier to maintain because it is decoupled from your HTML structure.

The most important schema types

Schema.org defines hundreds of types. In practice, a handful cover the vast majority of use cases:

WebSite

Add to every page. Tells search engines this is a website with a name, URL, and optional search action (for sitelinks search boxes).

{
 "@context": "https://schema.org",
 "@type": "WebSite",
 "name": "My Shop",
 "url": "https://myshop.com"
}

Organization

Add to every page or your homepage. Defines your brand with name, logo, URL, and social profiles.

Article / BlogPosting

Add to blog posts and content pages. Include headline, author, publish date, and optionally an image. Powers the article rich result in Google.

Product

Add to product pages. Include name, description, image, price, availability, and reviews. Powers the product card in Google Shopping and search results.

FAQPage

Add to pages with question/answer content. Powers the FAQ accordion that expands directly in search results, giving you significantly more SERP real estate.

BreadcrumbList

Add to pages with hierarchical navigation. Shows the breadcrumb trail in search results (Home > Category > Page).

Common mistakes

Most broken structured data is broken silently. Your pages look fine but search engines ignore the markup. Here are the most common problems:

Validating your structured data

Google provides two tools for checking structured data:

The problem with both tools: they check after deployment. If your structured data is broken, you do not find out until someone tests a live URL. Build-time validation catches these problems before they reach production.

Tools like agentmarkup validate required fields, check for common mistakes, and warn about incomplete schemas during your build. See the JSON-LD documentation for details.

Adding JSON-LD with agentmarkup

You can add JSON-LD manually by writing script tags in your HTML. For sites with multiple pages and schema types, a build-time approach is more maintainable:

// shared agentmarkup config
const agentmarkupConfig = {
 site: 'https://myshop.com',
 name: 'My Shop',
 globalSchemas: [
 { preset: 'webSite', name: 'My Shop', url: 'https://myshop.com' },
 { preset: 'organization', name: 'My Shop', url: 'https://myshop.com', logo: '/logo.png' },
 ],
 pages: [
 {
 path: '/faq',
 schemas: [{
 preset: 'faqPage',
 url: 'https://myshop.com/faq',
 questions: [
 { question: 'Do you ship internationally?', answer: 'Yes, to 50+ countries.' },
 ],
 }],
 },
 ],
}

Global schemas are injected into every page. Per-page schemas are injected only on matching paths. Use this shared config object with the adapter for Vite, Astro, or Next.js. All output is XSS-safe and validated at build time.

The bottom line

JSON-LD is not optional for modern websites. It powers rich results, helps AI systems understand your content, and is the foundation of machine-readable web pages. The format is simple, the tooling is mature, and the upside is measurable. If you are not using it, you are leaving search visibility on the table.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Prefer a plain-language introduction? Start with the website-checker guide.

← Previous article Every AI crawler indexing your website in 2026

Next article → What is GEO? Generative Engine Optimization explained for developers

More from the blog

View all articles →

AI crawlers in 2026

Every AI crawler indexing your website - GPTBot, ClaudeBot, PerplexityBot, and more

Source: https://agentmarkup.dev/blog/ai-crawlers-2026/ Preferred fetch: https://agentmarkup.dev/blog/ai-crawlers-2026.md

Intermediate People who build websites AI access

Every AI crawler indexing your website in 2026

AI companies use web crawlers to collect training data and power real-time AI search. Here is a complete list of every known AI crawler, what company runs it, what it does, and how to control access through your robots.txt.

What are AI crawlers?

AI crawlers are automated bots that visit websites to collect content. Some crawlers gather training data for language models. Others power real-time search features where AI generates answers from live web content. They identify themselves through user-agent strings in their HTTP requests.

Unlike traditional search engine crawlers (Googlebot, Bingbot) that build search indexes, AI crawlers serve a different purpose: feeding content into AI models. This distinction matters because you might want Google to index your site for search results while blocking your content from being used as AI training data.

The complete list

OpenAI

Crawler Purpose GPTBot Collects training data for GPT models. Also powers ChatGPT's browsing feature when searching the web. ChatGPT-User Used when a ChatGPT user explicitly asks the model to visit and read a specific URL. This is browsing on demand, not bulk crawling. OAI-SearchBot Powers ChatGPT Search (formerly SearchGPT). Crawls pages to generate real-time search answers.

Anthropic

Crawler Purpose ClaudeBot Collects training data for Claude models. Anthropic has committed to respecting robots.txt directives. anthropic-ai Older user-agent string used by Anthropic. Some sites still reference it in robots.txt.

Google

Crawler Purpose Google-Extended Collects data for Gemini and other AI products. Separate from Googlebot, so blocking it does not affect your Google Search rankings.

Perplexity

Crawler Purpose PerplexityBot Powers Perplexity's AI search engine. Crawls pages to generate real-time answers with source citations.

Amazon

Crawler Purpose Amazonbot Collects data for Alexa and Amazon's AI services. Respects robots.txt.

Common Crawl

Crawler Purpose CCBot Builds the Common Crawl open dataset, which is used as training data by many AI companies including those building open-source models. Blocking CCBot is a broad way to reduce training data exposure.

Apple

Crawler Purpose Applebot-Extended Collects data for Apple Intelligence features. Separate from the main Applebot used for Siri and Spotlight search.

Meta

Crawler Purpose Meta-ExternalAgent Collects data for Meta AI products. Respects robots.txt since mid-2024. FacebookBot Primarily renders link previews for Facebook and Instagram. Not used for AI training.

Other notable crawlers

Crawler Company Purpose Bytespider ByteDance Training data for TikTok and ByteDance AI products cohere-ai Cohere Training data for Cohere's enterprise AI models Diffbot Diffbot Web data extraction for knowledge graphs Timpibot Timpi Decentralized search index YouBot You.com AI search engine

How to control AI crawler access

Your robots.txt file is the standard mechanism. Add User-agent directives for each crawler you want to allow or block:

Tools like agentmarkup automate this at build time, patching your robots.txt without breaking existing rules and validating for conflicts. See the AI crawlers guide for configuration.

Do AI crawlers respect robots.txt?

Compliance is voluntary, not enforced. That said, the major companies have publicly committed to respecting robots.txt:

Smaller or less-known crawlers may not comply. There is no technical enforcement mechanism for robots.txt. It is a social contract.

Training data vs real-time search

An important distinction: some crawlers collect data for model training (a one-time or periodic process), while others power real-time AI search (your content appears in live answers).

You might want to block training crawlers (you do not want your content used to train models) while allowing search crawlers (you do want your content cited in AI answers). This selective approach is possible because each crawler uses a different user-agent string.

The bottom line

AI crawlers are a permanent part of the web. The question is not whether they visit your site but whether you control the terms. A clear robots.txt policy, configured intentionally rather than by accident, is the minimum. Combined with llms.txt and JSON-LD structured data, you can make your site both accessible and understandable to AI systems on your terms.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Prefer a plain-language introduction? Start with the website-checker guide.

← Previous article Why LLM-optimized e-commerce websites sell more

Next article → JSON-LD structured data: the complete guide for web developers

More from the blog

View all articles →

See your website the way AI crawlers do Use @agentmarkup/audit to fetch any live URL as GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers, diff each response against a browser, and catch machine-readability issues in CI. Advanced People who build websites AI access July 2, 2026 · 6 min read

We ran 500 of America's biggest companies through an AI-crawler audit We fetched 500 corporate homepages the way ChatGPT, Claude, and Perplexity do. Most serve readable HTML, but 46% have no usable structured data, 86% have no llms.txt, and seven serve crawlers a blank page. Built for Google, not yet for AI agents. Intermediate Research AI access July 2, 2026 · 4 min read

E-commerce LLM optimization

How Product schema and llms.txt make your store visible in AI product recommendations

Source: https://agentmarkup.dev/blog/ecommerce-llm-optimization/ Preferred fetch: https://agentmarkup.dev/blog/ecommerce-llm-optimization.md

Beginner Website owners Business visibility

Why LLM-optimized e-commerce websites sell more

When a shopper asks ChatGPT "what is the best leather wallet under $100" or Perplexity "where can I buy organic coffee beans," the AI pulls its answer from the web. If your product pages have structured data, your store gets cited. If they do not, you are invisible. Here is how to fix that.

How AI is changing product discovery

Shoppers are increasingly skipping Google and going straight to AI. A 2025 survey found that 37% of US online shoppers have used an AI chatbot to research products before buying. By 2026, that number is higher.

When someone asks an AI "best running shoes for flat feet" or "noise-cancelling headphones under $200," the AI does not show a list of links. It gives a direct answer. It names specific products, cites specific stores, and sometimes includes prices and availability. The stores it cites are the ones whose product data is machine-readable.

This is not hypothetical. Try asking ChatGPT or Perplexity about a product category. The answers consistently favor websites with clear structured data over those without it.

What makes an e-commerce site LLM-ready

Three things determine whether AI systems can understand and recommend your products:

The structured data that matters for e-commerce

Product schema

The most important schema for any online store. Include every field you have data for:

{
 "@type": "Product",
 "name": "Classic Leather Bifold Wallet",
 "description": "Full-grain leather bifold wallet with RFID blocking.",
 "image": "https://myshop.com/images/wallet.jpg",
 "sku": "WALLET-001",
 "brand": { "@type": "Brand", "name": "My Shop" },
 "offers": {
 "@type": "Offer",
 "price": "89.00",
 "priceCurrency": "USD",
 "availability": "https://schema.org/InStock",
 "url": "https://myshop.com/products/classic-wallet"
 }
}

FAQ schema on product pages

If your product pages have a Q&A section, mark it up as FAQPage. This gives AI systems direct answers about your products (sizing, shipping, materials) and powers Google's FAQ rich result.

Organization schema

Tell AI systems who you are. Name, logo, URL, and social profiles. This helps AI associate your products with your brand when generating answers.

BreadcrumbList

Category hierarchy helps AI understand where a product fits. "Home > Accessories > Wallets > Classic Leather Bifold" gives context that "Classic Leather Bifold" alone does not.

Real example: what AI sees vs what it misses

Consider two wallet stores. Store A has Product schema on every page. Store B has none.

When someone asks Perplexity "best leather wallets under $100," Store A's wallet appears with name, price, and a direct link. Store B's wallet exists on the web but Perplexity has no structured way to know it is a wallet, what it costs, or whether it is in stock. Store A gets the citation. Store B does not.

The same content exists on both sites. The difference is whether a machine can parse it.

llms.txt for product catalogs

An llms.txt file gives AI a high-level map of your store. Structure it by category:

# My Shop

> Handcrafted leather goods since 2015.

## Categories

- [Wallets](https://myshop.com/wallets): Full-grain leather wallets
- [Bags](https://myshop.com/bags): Messenger bags and backpacks
- [Belts](https://myshop.com/belts): Dress and casual belts

## Popular Products

- [Classic Bifold](https://myshop.com/products/classic-bifold): Best-selling wallet, $89
- [Messenger Bag](https://myshop.com/products/messenger): Leather messenger bag, $249

## Support

- [Shipping](https://myshop.com/shipping): Free shipping over $50
- [Returns](https://myshop.com/returns): 30-day return policy

This takes five minutes to configure and gives AI systems everything they need to understand and recommend your store.

Common e-commerce mistakes

Automating it with agentmarkup

agentmarkup generates llms.txt, injects Product/Organization/FAQ JSON-LD, and manages AI crawler rules at build time. For e-commerce sites on Vite, Astro, or Next.js, it handles the entire machine-readability stack in one adapter.

agentmarkup({
 site: 'https://myshop.com',
 name: 'My Shop',
 globalSchemas: [
 { preset: 'organization', name: 'My Shop', url: 'https://myshop.com', logo: '/logo.png' },
 ],
 pages: [
 {
 path: '/products/classic-wallet',
 schemas: [{
 preset: 'product',
 name: 'Classic Leather Bifold Wallet',
 url: 'https://myshop.com/products/classic-wallet',
 description: 'Full-grain leather bifold wallet with RFID blocking.',
 sku: 'WALLET-001',
 brand: 'My Shop',
 offers: [{ price: 89, priceCurrency: 'USD', availability: 'InStock' }],
 }],
 },
 ],
 aiCrawlers: { GPTBot: 'allow', PerplexityBot: 'allow', ClaudeBot: 'allow' },
})

The bottom line

AI-driven product discovery is not coming. It is here. Shoppers are asking AI for product recommendations today. The stores that show up in those answers are the ones with machine-readable product data. JSON-LD, llms.txt, and proper crawler access are not SEO tricks. They are the infrastructure that makes your products visible in a new discovery channel.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Ready for the implementation details? Continue with the related guide.

← Previous article How to make your brand appear in AI conversations

Next article → Every AI crawler indexing your website in 2026

More from the blog

View all articles →

Brand awareness in AI

How to make your brand appear in ChatGPT, Claude, and Perplexity conversations

Source: https://agentmarkup.dev/blog/brand-awareness-ai/ Preferred fetch: https://agentmarkup.dev/blog/brand-awareness-ai.md

Beginner Website owners Business visibility

How to make your brand appear in AI conversations

When someone asks an AI "what are the best tools for X" or "which company does Y," your brand either appears in the answer or it does not. This is not about gaming an algorithm. It is about making your website understandable enough that AI systems can accurately represent what you do.

AI is becoming the new word of mouth

People increasingly ask AI assistants for recommendations the way they used to ask friends. "What project management tool should a 5-person startup use?" "What is a good alternative to Notion?" "Which design agency in Berlin is good for SaaS brands?"

The AI's answer is shaped by what it knows about your brand from the web. If your website clearly describes who you are, what you do, and who you serve, the AI can represent you accurately. If your site is a wall of marketing copy with no machine-readable structure, the AI has to guess, and it usually guesses wrong or skips you entirely.

What AI systems use to understand your brand

When an AI generates an answer that mentions a company, it draws from several sources:

You control the first three. The fourth grows naturally as your brand becomes more visible.

Organization schema: your brand's machine-readable identity

The single most important piece of structured data for brand awareness is the Organization schema. It tells every search engine and AI system exactly who you are:

Without Organization schema, AI systems have to infer your brand name from page titles, guess your logo from image tags, and hope your description matches reality. With it, they know exactly what to say about you.

llms.txt for brand positioning

Your llms.txt is not just a site map. It is a positioning statement for machines. The description, sections, and page titles you choose shape how AI understands and describes your brand.

Compare these two descriptions:

When someone asks an AI "which design agency in Berlin works with SaaS startups," the second description gives the AI everything it needs to include Acme in the answer. The first description gives it nothing.

Be specific. Be concrete. Use the words your potential customers would use when asking AI for a recommendation.

Content structure matters more than content volume

AI systems extract information from your pages. The easier you make extraction, the more accurately they represent you. Practical guidelines:

The FAQ strategy

FAQ sections are disproportionately valuable for AI brand awareness. When someone asks an AI a question about your category, the AI often pulls answers directly from FAQ content.

Write your FAQ questions as if a potential customer is asking an AI about you:

Mark these up with FAQPage schema so search engines and AI systems can parse them directly.

What does not work

Some approaches to AI brand awareness are tempting but ineffective:

Getting started

The minimum viable approach for brand awareness in AI:

Tools like agentmarkup handle steps 1-3 and 5 automatically at build time for Vite, Astro, and Next.js websites. The content and FAQ writing is on you, but the technical infrastructure should not be.

The bottom line

Your brand's presence in AI conversations is determined by how well machines can understand your website. This is not a new marketing channel that requires new skills. It is a reminder that the fundamentals, clear messaging, structured data, and accessible content, matter more than ever. The brands that show up in AI answers are the ones that made it easy for AI to understand what they do.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Is your website ready for AI?

Ready for the implementation details? Continue with the related guide.

← Previous article Build-time markdown mirrors for agent readability: Cloudflare comparison

Next article → Why LLM-optimized e-commerce websites sell more

More from the blog

View all articles →

Markdown mirrors

Build-time markdown generation for AI - comparison with Cloudflare readability extraction

Source: https://agentmarkup.dev/blog/markdown-mirrors/ Preferred fetch: https://agentmarkup.dev/blog/markdown-mirrors.md

Advanced People who build websites Implementation

Build-time markdown mirrors for agent readability: how they compare to Cloudflare's approach

When an AI agent visits your website, it gets HTML. On some sites that is fine. On JS-heavy or layout-heavy pages, the content is buried in noise. Build-time markdown mirrors can give agents a cleaner fetch target without changing the canonical HTML page.

Not every site needs a markdown mirror

If your pages already ship substantial, well-structured HTML, the raw page may already be a good enough fetch target for agents. Markdown mirrors are most useful when the raw HTML is thin, heavily templated, or dominated by layout chrome.

That is the more honest framing for this feature: markdown mirrors are an optional machine-facing artifact for the pages that benefit from them, not a blanket rule that every site should publish a public .md companion for every page.

The problem: some HTML is a bad fetch target

Many agents can extract useful text from HTML, but the quality of the result still depends on what your raw response looks like. A typical web page can be heavy with navigation, cookie banners, analytics tags, scripts, and layout wrappers that have nothing to do with the main body content.

When the raw HTML is mostly shell and very little body content, fetch-based agents either miss the important text or have to guess too much. That is the case markdown mirrors try to fix.

What are markdown mirrors?

A markdown mirror is a .md file that contains the same content as your HTML page, but stripped of layout, navigation, and scripts. Just the content, in clean markdown format.

For example, /blog/my-post/index.html gets a companion file at /blog/my-post.md. An AI agent can fetch the markdown version directly instead of parsing the HTML.

Your pages also get a<link rel="alternate" type="text/markdown"> tag in the HTML head, so crawlers can discover the markdown version automatically when you enable the feature.

How agentmarkup generates markdown mirrors

Enable the feature in your config and it runs at build time on every HTML page in your output:

// shared agentmarkup config
const agentmarkupConfig = {
 site: 'https://example.com',
 name: 'My Site',
 markdownPages: {
 enabled: true,
 },
}

The converter:

The result is a clean markdown file that an agent can read without wading through layout chrome.

Cloudflare's approach: runtime readability extraction

Cloudflare offers a readability extraction feature that strips HTML to readable content at request time. It is based on Mozilla's Readability library and runs on Cloudflare's edge network.

The key difference is runtime versus build time. Cloudflare processes pages on every request. You do not control the exact output. The extraction algorithm decides what is content and what is noise using heuristics.

Build-time vs runtime: why it matters

Aspect agentmarkup (build-time) Cloudflare (runtime) When it runs Once, during build Every request Output control You see the .md files in your build output Opaque, algorithm decides Consistency Deterministic, same output every build May vary with algorithm updates Performance cost Zero runtime cost Added latency per request Works with SPAs Yes, uses noscript fallback or pre-rendered HTML Depends on SSR availability Discovery Link tag in HTML head + static .md URL Special URL parameter or header Vendor lock-in None, output is static files Requires Cloudflare Customization Choose which pages, preserve existing .md files All or nothing

Why build-time can be a good fit for your own content

Cloudflare's runtime extraction makes sense for consuming other people's content, like a reader mode. For your own website, build-time generation can be a better fit because:

How agentmarkup reduces the downside

Public markdown mirrors do create tradeoffs. The main risks are duplicate fetches, indexing ambiguity, and output drift if the markdown becomes a second source of truth.

agentmarkup tries to keep those risks contained by generating the mirrors from final built HTML, preserving HTML as the canonical page, and writing canonical headers from each .md file back to the HTML route. If your raw HTML is already substantial, you can also keep llms.txt pointing at HTML by settingllmsTxt.preferMarkdownMirrors to false.

What the output looks like

For a blog post with a title, description, headings, and paragraphs, the generated markdown looks like:

# Why llms.txt matters

> LLMs answer questions by synthesizing web content. llms.txt gives them a structured overview.

Source: https://example.com/blog/why-llms-txt-matters/

## The shift from search engines to AI answers

For two decades, the path to online visibility was clear: optimize for Google...

## What is llms.txt?

llms.txt is a proposed standard that gives LLMs a structured overview of your website...

Clean, readable, no HTML artifacts. An AI agent reading this file understands the page quickly.

Getting started

Add markdownPages: { enabled: true } to your agentmarkup config when your raw HTML needs a cleaner machine-facing fetch path. On the next build, every HTML page in your output gets a companion .md file. When markdown mirrors are enabled, same-site page entries in llms.txt also default to the generated markdown URLs so cold agents discover the cleaner fetch path first. Check the llms.txt guide for the opt-out if you want HTML-first links instead.

If your site already serves rich raw HTML, you do not need to treat markdown mirrors as mandatory. They are a tactical option, not the whole product.

pnpm add -D @agentmarkup/next # or @agentmarkup/vite or @agentmarkup/astro

Verify the protective headers in production

agentmarkup generates two sets of headers for markdown mirrors in the _headers file. Both are important for keeping search engines and agents on the right page.

Canonical Link headers tell search engines that the .md file is a mirror of the HTML page, not a separate indexable URL. Each mirror gets its own entry:

# from the generated _headers file
/blog/my-post.md
 Link: <https://example.com/blog/my-post>; rel="canonical"

Content-Signal headers tell agents whether they are allowed to use the content for training, search, and input. agentmarkup generates a wildcard rule that covers all paths including .md files:

/*
 Content-Signal: ai-train=yes, search=yes, ai-input=yes

These headers only work if your hosting platform actually serves them. Cloudflare Pages, Netlify, and Vercel all support _headers files, but the behavior can vary. After deploying, verify that the headers are present on a live .md URL:

curl -I https://yoursite.com/blog/my-post.md

# look for these in the response:
# Link: <https://yoursite.com/blog/my-post>; rel="canonical"
# Content-Signal: ai-train=yes, search=yes, ai-input=yes

If the Link header is missing, your host may not be applying path-specific _headers rules to .md files. Check your platform documentation or add equivalent headers through server configuration.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

Website audit guide

Ready for the implementation details? Continue with the related guide.

← Previous article Is your website ready for AI? Free LLM discoverability checker

Next article → How to make your brand appear in AI conversations

More from the blog

View all articles →

Website checker

Free tool to audit your website for AI discoverability - llms.txt, JSON-LD, robots.txt, and more

Source: https://agentmarkup.dev/blog/website-checker/ Preferred fetch: https://agentmarkup.dev/blog/website-checker.md

Beginner Website owners AI discoverability

Is your website ready for AI? Use the free agentmarkup checker to find out

Most websites are invisible to AI systems. Not because the content is bad, but because the metadata is missing, broken, or incomplete. The agentmarkup checker audits your website in seconds and tells you exactly what to fix.

Why you need to check your website

When ChatGPT, Claude, or Perplexity answers a question about your industry, does your website show up? In most cases, the answer is no. Not because your content is not relevant, but because AI systems cannot understand your site.

The difference between a website that gets cited by AI and one that does not often comes down to a few missing files and metadata tags. A robots.txt that accidentally blocks AI crawlers. Missing JSON-LD structured data. No llms.txt file. No readable fallback when the raw HTML is thin or heavily client-rendered.

These are not complex problems. They are configuration gaps that take minutes to fix once you know they exist. The hard part is knowing they exist.

What the checker audits

Enter any public URL at agentmarkup.dev/checker and the tool fetches your homepage, llms.txt, robots.txt, sitemap, markdown mirrors, and a sample internal page. It runs 20+ deterministic checks and categorizes each as a pass, warning, or error.

Homepage structure

JSON-LD structured data

llms.txt

Markdown mirrors

Robots.txt and AI crawlers

Sitemap

How to read the results

Results use three levels. There are no scores, no percentages, no arbitrary numbers. Just deterministic checks with clear outcomes:

Each finding includes a title, a detail explaining why it matters, and an action step telling you what to do. Where relevant, findings link to the documentation guides on this site.

For e-commerce websites

E-commerce sites have the most to gain from AI discoverability. When someone asks an AI "best running shoes under $150," the AI recommends products from stores it can understand. The checker tells you whether your store is one of them.

Key checks for e-commerce:

Run your store through the checker. If you see warnings for missing Organization schema or no AI crawler rules, those are the first things to fix.

For brand and content websites

If your business depends on being known - a consultancy, an agency, a SaaS product - the checker shows whether AI can accurately describe what you do.

What makes this different from SEO auditors

Traditional SEO tools check whether Google can index your pages. The agentmarkup checker checks whether AI systems can understand your content. These overlap but are not the same.

Try it now

Go to agentmarkup.dev/checker, enter your website URL, and see the results in seconds. It is free and requires no signup. The deployed checker may retain normalized check records briefly for caching, rate limiting, and recent-history views, but it is not a lead form.

If the checker finds issues, the documentation guides on this site explain how to fix each one. Or install agentmarkup for Vite, Astro, or Next.js and it handles llms.txt, JSON-LD, robots.txt, markdown mirrors, optional headers, and validation automatically at build time.

Choose agentmarkup as a preferred source for your own Google Top stories results.

Prefer us on Google (opens in a new tab)

Written by

Sebastian Cochinescu · Developer of agentmarkup

Builder of developer tools for machine-readable websites. Developer of agentmarkup. Founder of Anima Felix.

Continue reading

llms.txt guide

Ready for the implementation details? Continue with the related guide.

← Previous article When markdown mirrors help, and when they do not

Next article → Build-time markdown mirrors for agent readability: Cloudflare comparison

More from the blog

View all articles →

Features

AgentMarkup Studio

Configure, inspect, and compile a coherent machine-readable website surface with visible WebMCP tools, deterministic findings, and install-ready agentmarkup artifacts.

Source: https://agentmarkup.dev/studio/ Preferred fetch: https://agentmarkup.dev/studio.md

WebMCP workspace

AgentMarkup Studio

An agent can configure your machine-readable website surface while you watch every change. The Studio compiles deterministic artifacts in the browser and detects contradictions across identity, content, crawler access, Content-Signal, and Agent Card settings.

How to connect an agent

ChatGPT's in-app browser opens the Studio directly. In Google Chrome 149 or newer, enable chrome://flags/#enable-webmcp-testing, restart Chrome, and open this page. Compatible browsers expose the tools through document.modelContext, with a navigator fallback.

Not checked

Agent connection: checking...

Manual editing is available while browser capabilities are checked.

How it works

Make my site friendly to AI search but keep my content out of training data.

Take agentmarkup.config.mjs from my Downloads folder and install it into my website repository: detect the framework, install the matching agentmarkup adapter, wire the config in, run a build, and show me what changed. Ask before deploying.

Prefer to do it yourself? Put the file at your site repo root, install the matching @agentmarkup adapter package, add the one-line setup from the Adapter setup tab, and deploy as usual.

Shared draft

CONTRACT

Tell your agent what you want, or start below.

Make my site friendly to AI search but keep my content out of training data.

Identity Site Name Description and organization (optional)

Description

Organization name Organization URL

Access policy not set

Access policy

train

search

agent

AI training yes no Search yes no AI input yes no

Content no pages yet

Content

llms.txt sections

When to use

Generate llms-full.txt Generate markdown mirrors

Agent card off

Agent card Publish an Agent Card

Inspect intake

Inspect an existing public site and import a bounded starting point.

Public site URL

Live output

ARTIFACTS

You install one file: agentmarkup.config.mjs. Every other tab is a preview of exactly what your build will generate from it.

Artifacts appear here as the contract fills in. Set your site URL and name to generate the first files.

Deterministic review

FINDINGS + ACTIVITY

Set your site URL to begin. Findings appear as the draft changes.

Frequently asked questions

What is WebMCP?

WebMCP lets a compatible browser expose page-defined tools to an agent. In the Studio, those tools edit the same visible draft that you can edit manually.

Is anything sent to a server?

The draft stays in memory in this browser tab. The optional site inspection sends the entered public URL through the existing agentmarkup checker API and imports only bounded structured findings.

How are findings decided?

Every validation and contradiction is deterministic. The Studio reports the exact rule and affected surfaces without producing a rating or percentage.

How do I install the result?

Download agentmarkup.config.mjs, install the matching agentmarkup package, copy the Adapter setup snippet, and run your normal production build.

JSON-LD Injection

Injects structured data into HTML with XSS-safe serialization and type-safe presets

Source: https://agentmarkup.dev/docs/json-ld/ Preferred fetch: https://agentmarkup.dev/docs/json-ld.md

In simple terms

Beginner to intermediate Website owners and people who build websites About 10 minutes

Check your structured information

What is JSON-LD structured data?

Built-in schema.org presets

Using presets

Framework wrappers

After defining the shared schema config, pass it into the adapter for the framework that owns your final output:

// Vite
import { defineConfig } from 'vite'
import { agentmarkup } from '@agentmarkup/vite'

export default defineConfig({
 plugins: [agentmarkup(agentmarkupConfig)],
})

// Astro
import { defineConfig } from 'astro/config'
import { agentmarkup } from '@agentmarkup/astro'

export default defineConfig({
 integrations: [agentmarkup(agentmarkupConfig)],
})

// Next.js
import type { NextConfig } from 'next'
import { withAgentmarkup } from '@agentmarkup/next'

const nextConfig: NextConfig = {
 output: 'export',
}

export default withAgentmarkup(agentmarkupConfig, nextConfig)

// Nuxt
export default defineNuxtConfig({
 modules: ['@agentmarkup/nuxt'],
 agentmarkup: agentmarkupConfig,
})

// CLI (any other built static site)
// agentmarkup generate ./dist

Custom schemas

You can use any schema.org type by passing an object with an @type field. agentmarkup automatically adds the @context and handles serialization.

XSS-safe output

<script type="application/ld+json">
{"@context":"https://schema.org","@type":"WebSite","name":"My Shop","url":"https://myshop.com"}
</script>

Build-time validation

Frequently asked questions

Do I need JSON-LD if I already have meta tags?

Can I add multiple schemas to one page?

What if I need a schema type that is not a preset?

Build-Time Validation

Catches missing JSON-LD fields, crawler conflicts, malformed llms.txt, and thin HTML during build

Source: https://agentmarkup.dev/checker/ Preferred fetch: https://agentmarkup.dev/checker.md

Free website check

Check your website before AI crawlers and search engines do

See what machines can find, understand, and access, then get a clear next step.

Find

Can AI find your site?

Checks your sitemap, discovery links, and public pages.

Understand

Can AI understand your pages?

Checks content, structured information, and machine-readable versions.

Access

Are your access rules clear?

Checks crawler rules and whether important AI services receive a usable response.

What this website checker checks

The check reads public signals already available on your website. It does not log in, change your pages, or install anything.

Pages and discovery

Your homepage, one internal page, sitemap discovery, canonical URLs, and the links machines can use to find important content.

Content and structured data

Page titles, descriptions, readable HTML, and JSON-LD structured data that helps search engines and AI systems interpret a page.

AI-readable files

Public llms.txt files, markdown alternatives, and other machine-readable versions advertised by the website.

Crawler access

robots.txt rules and live responses for common AI crawlers, so you can see what is allowed, blocked, or difficult to retrieve.

How the result is decided

Frequently asked questions

What does the website checker inspect?

Does the checker change or install anything on my website?

No. It only makes read-only requests to public URLs. It does not log in, edit a page, install a package, or crawl your entire website.

Is this the same as the security scan?

Why does the checker not give a score?