headroom/docs/app/layout.tsx
chopratejas c1d2eec588 docs: improve discoverability for AI agents and search crawlers
Several signals AI agents and search engines use to discover and
install a project were misaligned or missing:

* ``docs/app/layout.tsx`` set ``metadataBase`` to
  ``https://chopratejas.github.io/headroom/`` while the live docs run
  on Vercel — every page's ``og:url`` and ``twitter:url`` resolved to
  a URL that returns 404 for ``/llms.txt``. Now points at the live
  Vercel host (overridable via ``NEXT_PUBLIC_SITE_URL`` for a future
  custom domain). Adds explicit ``openGraph`` and ``twitter`` metadata
  so social shares render a card with the project's pitch.
* No ``llms.txt`` at the GitHub repo root. AI agents crawling
  ``github.com/chopratejas/headroom/`` saw only the README. The new
  ``llms.txt`` follows the llmstxt.org convention: 1-line pitch,
  canonical docs links, copy-paste install commands (pip / npm /
  Docker / proxy / ``headroom wrap``), and entry points for the
  library, proxy, MCP server, and SDK integrations. Points at the
  Fumadocs-generated ``/llms.txt`` and ``/llms-full.txt`` for the
  full picture.
* ``pyproject.toml`` ``Documentation`` URL pointed at the GitHub
  README anchor. Updated to point at the docs site so PyPI visitors
  land on searchable docs, and adds an ``AI / LLM Index`` URL
  pointing at the Fumadocs ``/llms.txt``.
* No explicit AI-bot allow list. Added ``docs/app/robots.ts`` (Next
  13+ App Router convention) with explicit allows for GPTBot,
  ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot,
  ChatGPT-User, Cohere-AI, CCBot, and Applebot-Extended. Wildcard
  allow as the catch-all. Advertises the sitemap.
* No ``sitemap.xml`` route. Added ``docs/app/sitemap.ts`` that pulls
  every Fumadocs page out of ``source`` (same source backing
  ``/llms.txt``, search, and OG images) so search and AI crawlers
  can enumerate doc pages without scraping HTML.
* README didn't tell AI agents where to look. Added a 2-line
  pointer near the top nav row: read ``/llms.txt`` here, or fetch
  the live index / full docs blob.

Also tightened the GitHub repo description and added five topics
(``claude-code``, ``cursor``, ``tokens``, ``prompt-engineering``,
``typescript``) via ``gh repo edit`` — that's already live on the
repo, not part of this commit.

No Python or Rust code changes; ``make ci-precheck`` was run to
confirm the test slice still passes.
2026-05-13 17:36:06 -07:00

53 lines
1.9 KiB
TypeScript
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import { RootProvider } from 'fumadocs-ui/provider/next';
import './global.css';
import { Inter } from 'next/font/google';
import type { Metadata } from 'next';
const inter = Inter({
subsets: ['latin'],
});
// Canonical URL for the live docs. ``metadataBase`` resolves the og:url
// and twitter:url for every page; pointing it at the actual live site
// is what lets crawlers (search + LLM) follow the right canonical and
// pick up ``/llms.txt`` / ``/sitemap.xml`` / og images. Override at
// build time via ``NEXT_PUBLIC_SITE_URL`` (e.g. when promoting to a
// custom domain).
const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL ?? 'https://headroom-docs.vercel.app';
export const metadata: Metadata = {
title: {
default: 'Headroom — Context Optimization Layer for AI Agents',
template: '%s | Headroom',
},
description:
'Compress everything your AI agent reads — tool outputs, logs, files, RAG chunks. Same answers, fraction of the tokens. Library, proxy, MCP server. Local-first. Apache 2.0.',
metadataBase: new URL(SITE_URL),
alternates: {
canonical: '/',
},
openGraph: {
type: 'website',
siteName: 'Headroom',
title: 'Headroom — Context Optimization Layer for AI Agents',
description:
'Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 6095% fewer tokens, same answers.',
url: '/',
},
twitter: {
card: 'summary_large_image',
title: 'Headroom — Context Optimization Layer for AI Agents',
description:
'Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 6095% fewer tokens, same answers.',
},
};
export default function Layout({ children }: LayoutProps<'/'>) {
return (
<html lang="en" className={inter.className} suppressHydrationWarning>
<body className="flex flex-col min-h-screen">
<RootProvider>{children}</RootProvider>
</body>
</html>
);
}