How to make a Next.js site visible to AI search engines

How do you make a Next.js site visible to AI search engines?

To make a Next.js site visible to AI search engines, make sure crawlers can fetch the content as plain HTML, allow the AI crawlers you want in robots.txt, and describe the site with a sitemap, an /llms.txt file and schema.org JSON-LD. Then write content that answers questions directly and submit the sitemap to Google Search Console and Bing Webmaster Tools. This website, built with Next.js 16 and the App Router, is the worked example below.

AI search engines such as ChatGPT search, Perplexity, Claude and Google AI Overviews still depend on crawling. If a crawler cannot fetch and parse a page, the page cannot be quoted. Everything in this guide exists to make fetching and parsing easy.

Why does server-rendered HTML matter?

Server-rendered HTML matters because many AI crawlers fetch the raw HTML and do not run JavaScript. A single-page app that builds its content in the browser can look empty to them. In the Next.js App Router, pages are React Server Components by default, so the text is already in the first HTML response.

On this site, service and project pages use generateStaticParams() with dynamicParams = false. Next.js prerenders every page at build time, and an unknown slug returns a 404 instead of an empty shell. Keep important text out of client-only components, tabs that load on click, and content fetched after hydration.

How should robots.ts treat AI crawlers?

AI crawlers fall into two groups, and robots.txt can treat each group differently.

• Training crawlers collect pages to train models. Examples are GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and Meta-ExternalAgent. Google-Extended and Applebot-Extended are not separate crawlers; they are robots.txt tokens that control whether Google and Apple may use content for their AI models.
• Search and user agents fetch pages to answer questions and cite sources. Examples are OAI-SearchBot (ChatGPT search index), ChatGPT-User (fetches a page when a user asks), Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.

Blocking a training crawler does not remove a site from AI search, and blocking a search crawler does. Note that Google AI Overviews use the normal Googlebot crawl, so blocking Google-Extended does not remove a site from them. This site allows both groups, and lists the AI bots by name so the intent is explicit. Here is a shortened version of app/robots.ts:

import type { MetadataRoute } from "next";
import { site } from "@/data/site";

const aiBots = [
  "GPTBot", "OAI-SearchBot", "ChatGPT-User",
  "ClaudeBot", "Claude-User", "Claude-SearchBot",
  "PerplexityBot", "Perplexity-User",
  "Google-Extended", "Applebot-Extended", "Bingbot", "CCBot",
];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [{ userAgent: "*", allow: "/" }, { userAgent: aiBots, allow: "/" }],
    sitemap: `${site.url}/sitemap.xml`,
  };
}

Next.js turns this file into /robots.txt at build time. To opt out of training but stay in AI search, you would move the training bots into a rule with disallow: "/" and keep the search agents allowed.

How do you add a sitemap in Next.js?

A sitemap lists every URL you want crawled. In the App Router, an app/sitemap.ts file that returns a MetadataRoute.Sitemap array is served as /sitemap.xml. This site builds the list from the same data files that render the pages: the home page, each service page, each project page, the articles index, every article (with its own updated date as lastModified), /llms.txt and /llms-full.txt.

Building the sitemap from page data means a new page cannot be forgotten: publishing an article adds it to the sitemap automatically. Use a real lastModified date where you have one, so crawlers can tell which pages changed.

What are llms.txt and llms-full.txt?

llms.txt is a proposed convention: a Markdown file at the site root that summarizes the site for large language models (LLMs). It has a title, a short description, and lists of links with one-line summaries. A companion file, llms-full.txt, carries the full text of the key pages in one document, so a model can read everything in a single fetch.

This site serves /llms.txt from a route handler that builds Markdown from the same data as the pages: articles, services, case studies, skills, experience, FAQ and contact. A second route handler serves /llms-full.txt with the full text of every article, and an RSS feed at /feed.xml announces new articles. The key line is export const dynamic = "force-static", which renders the file once at build time. A shortened version:

// app/llms.txt/route.ts
import { services } from "@/data/services";
import { site } from "@/data/site";
import { serviceUrl } from "@/lib/schema";

export const dynamic = "force-static";

const body = `# ${site.fullName}

> ${site.description}

## Services
${services.map((s) => `- [${s.title}](${serviceUrl(s.slug)}): ${s.body}`).join("\n")}
`;

export function GET() {
  return new Response(body, { headers: { "Content-Type": "text/markdown; charset=utf-8" } });
}

The root layout also points to the file with alternates.types, so it appears as a <link rel="alternate"> tag in every page head.

How should you structure JSON-LD?

JSON-LD is structured data, written in the schema.org vocabulary, placed in a <script type="application/ld+json"> tag. It tells engines what a page is about in a form they do not need to guess. The most useful habit is to define the main entity once, with an @id, and reference that @id everywhere else.

On this site, lib/schema.ts defines a Person with the @id https://cyberjon.com/#person. The root layout renders it once, together with a WebSite. Every other graph, such as ProfilePage, Service, CreativeWork and FAQPage, links back with { "@id": personId } as its mainEntity, provider or creator. Service and project pages also add a BreadcrumbList.

A small component renders each graph. It escapes < so that text inside the data cannot close the script tag early:

// components/JsonLd.tsx
export default function JsonLd({ data }: { data: object }) {
  return (
    <script
      type="application/ld+json"
      dangerouslySetInnerHTML={{ __html: JSON.stringify(data).replace(/</g, "\\u003c") }}
    />
  );
}

How do metadata, canonical URLs and content help?

The Next.js Metadata API sets titles, descriptions, canonical URLs and Open Graph tags. This site's root layout sets metadataBase, a title template, and alternates.canonical. Each service page sets its own canonical path in generateMetadata(). That matters, because any page without its own canonical inherits the root layout's value and tells engines it is a copy of the home page.

Content structure matters as much as markup. Put the direct answer in the first two or three sentences, use question-shaped headings, and keep each paragraph about one idea so it can be quoted alone. A visible FAQ section, marked up as FAQPage, gives engines ready-made question-and-answer pairs.

Where should you submit the sitemap?

Submit /sitemap.xml in Google Search Console and in Bing Webmaster Tools. Google feeds Google Search and AI Overviews. Bing matters because ChatGPT search draws on Bing's index. Both tools show crawl errors and which pages are indexed.

Summary

AI search visibility in Next.js comes from a few small files: server-rendered pages, app/robots.ts, app/sitemap.ts, an /llms.txt route handler and JSON-LD built around one entity @id. Add per-page canonicals, answer-first writing and a sitemap submitted to Google and Bing. For a related build, see streaming Claude responses in a Next.js chat, or the Next.js web app development service.