Several signals AI agents and search engines use to discover and
install a project were misaligned or missing:
* ``docs/app/layout.tsx`` set ``metadataBase`` to
``https://chopratejas.github.io/headroom/`` while the live docs run
on Vercel — every page's ``og:url`` and ``twitter:url`` resolved to
a URL that returns 404 for ``/llms.txt``. Now points at the live
Vercel host (overridable via ``NEXT_PUBLIC_SITE_URL`` for a future
custom domain). Adds explicit ``openGraph`` and ``twitter`` metadata
so social shares render a card with the project's pitch.
* No ``llms.txt`` at the GitHub repo root. AI agents crawling
``github.com/chopratejas/headroom/`` saw only the README. The new
``llms.txt`` follows the llmstxt.org convention: 1-line pitch,
canonical docs links, copy-paste install commands (pip / npm /
Docker / proxy / ``headroom wrap``), and entry points for the
library, proxy, MCP server, and SDK integrations. Points at the
Fumadocs-generated ``/llms.txt`` and ``/llms-full.txt`` for the
full picture.
* ``pyproject.toml`` ``Documentation`` URL pointed at the GitHub
README anchor. Updated to point at the docs site so PyPI visitors
land on searchable docs, and adds an ``AI / LLM Index`` URL
pointing at the Fumadocs ``/llms.txt``.
* No explicit AI-bot allow list. Added ``docs/app/robots.ts`` (Next
13+ App Router convention) with explicit allows for GPTBot,
ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot,
ChatGPT-User, Cohere-AI, CCBot, and Applebot-Extended. Wildcard
allow as the catch-all. Advertises the sitemap.
* No ``sitemap.xml`` route. Added ``docs/app/sitemap.ts`` that pulls
every Fumadocs page out of ``source`` (same source backing
``/llms.txt``, search, and OG images) so search and AI crawlers
can enumerate doc pages without scraping HTML.
* README didn't tell AI agents where to look. Added a 2-line
pointer near the top nav row: read ``/llms.txt`` here, or fetch
the live index / full docs blob.
Also tightened the GitHub repo description and added five topics
(``claude-code``, ``cursor``, ``tokens``, ``prompt-engineering``,
``typescript``) via ``gh repo edit`` — that's already live on the
repo, not part of this commit.
No Python or Rust code changes; ``make ci-precheck`` was run to
confirm the test slice still passes.
