headroom/docs
chopratejas c1d2eec588 docs: improve discoverability for AI agents and search crawlers
Several signals AI agents and search engines use to discover and
install a project were misaligned or missing:

* ``docs/app/layout.tsx`` set ``metadataBase`` to
  ``https://chopratejas.github.io/headroom/`` while the live docs run
  on Vercel — every page's ``og:url`` and ``twitter:url`` resolved to
  a URL that returns 404 for ``/llms.txt``. Now points at the live
  Vercel host (overridable via ``NEXT_PUBLIC_SITE_URL`` for a future
  custom domain). Adds explicit ``openGraph`` and ``twitter`` metadata
  so social shares render a card with the project's pitch.
* No ``llms.txt`` at the GitHub repo root. AI agents crawling
  ``github.com/chopratejas/headroom/`` saw only the README. The new
  ``llms.txt`` follows the llmstxt.org convention: 1-line pitch,
  canonical docs links, copy-paste install commands (pip / npm /
  Docker / proxy / ``headroom wrap``), and entry points for the
  library, proxy, MCP server, and SDK integrations. Points at the
  Fumadocs-generated ``/llms.txt`` and ``/llms-full.txt`` for the
  full picture.
* ``pyproject.toml`` ``Documentation`` URL pointed at the GitHub
  README anchor. Updated to point at the docs site so PyPI visitors
  land on searchable docs, and adds an ``AI / LLM Index`` URL
  pointing at the Fumadocs ``/llms.txt``.
* No explicit AI-bot allow list. Added ``docs/app/robots.ts`` (Next
  13+ App Router convention) with explicit allows for GPTBot,
  ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot,
  ChatGPT-User, Cohere-AI, CCBot, and Applebot-Extended. Wildcard
  allow as the catch-all. Advertises the sitemap.
* No ``sitemap.xml`` route. Added ``docs/app/sitemap.ts`` that pulls
  every Fumadocs page out of ``source`` (same source backing
  ``/llms.txt``, search, and OG images) so search and AI crawlers
  can enumerate doc pages without scraping HTML.
* README didn't tell AI agents where to look. Added a 2-line
  pointer near the top nav row: read ``/llms.txt`` here, or fetch
  the live index / full docs blob.

Also tightened the GitHub repo description and added five topics
(``claude-code``, ``cursor``, ``tokens``, ``prompt-engineering``,
``typescript``) via ``gh repo edit`` — that's already live on the
repo, not part of this commit.

No Python or Rust code changes; ``make ci-precheck`` was run to
confirm the test slice still passes.
2026-05-13 17:36:06 -07:00
..
app docs: improve discoverability for AI agents and search crawlers 2026-05-13 17:36:06 -07:00
components tokens saved grid 2026-04-12 13:42:20 +06:00
content/docs feat: add lean-ctx context tool support 2026-05-11 17:54:17 -04:00
overrides fix: repair release and docs pipelines 2026-04-16 12:53:51 -05:00
screenshots Merge pull request #147 from JerrettDavis/feat/anthropic-usage-insights 2026-04-12 10:54:22 -07:00
spec feat: add lean-ctx context tool support 2026-05-11 17:54:17 -04:00
.gitignore new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
auth-modes.md fix: PR-F1 classify_auth_mode helper (Phase F kickoff) 2026-05-03 17:20:14 -07:00
bedrock.md fix(proxy): PR-D3 Bedrock observability + auth-mode integration 2026-05-04 11:07:47 -07:00
bun.lock new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
next.config.mjs new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
package-lock.json chore(deps): bump the npm_and_yarn group across 3 directories with 4 updates 2026-04-24 22:15:53 +00:00
package.json chore(deps): bump the npm_and_yarn group across 3 directories with 4 updates 2026-04-24 22:15:53 +00:00
postcss.config.mjs new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
proxy.ts new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
README.md new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
source.config.ts new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
tsconfig.json new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00

docs

This is a Next.js application generated with Create Fumadocs.

Run development server:

npm run dev
# or
pnpm dev
# or
yarn dev

Open http://localhost:3000 with your browser to see the result.

Explore

In the project, you can see:

  • lib/source.ts: Code for content source adapter, loader() provides the interface to access your content.
  • lib/layout.shared.tsx: Shared options for layouts, optional but preferred to keep.
Route Description
app/(home) The route group for your landing page and other pages.
app/docs The documentation layout and pages.
app/api/search/route.ts The Route Handler for search.

Fumadocs MDX

A source.config.ts config file has been included, you can customise different options like frontmatter schema.

Read the Introduction for further details.

Learn More

To learn more about Next.js and Fumadocs, take a look at the following resources: