deps: bump tiktoken-rs from 0.11.0 to 0.12.0 (#3147)

Bumps [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) from 0.11.0
to 0.12.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/zurawiki/tiktoken-rs/releases">tiktoken-rs's
releases</a>.</em></p>
<blockquote>
<h2>v0.12.0</h2>
<h2>Summary</h2>
<p>This release backports OpenAI <code>tiktoken</code> 0.13.0 into
<code>tiktoken-rs</code>. The main reason to upgrade is better alignment
with upstream tokenization behavior, especially the upstream Rust core
changes for large BPE pieces and error-aware encoding.</p>
<p>For most users who call the high-level model/token counting helpers,
this should behave the same aside from the new Rust compiler
requirement. Users who call lower-level <code>CoreBPE</code> encoding
methods directly should review the breaking changes below.</p>
<h2>What Changed</h2>
<ul>
<li>Backported the vendored OpenAI <code>tiktoken</code> Rust core from
0.9.0 to 0.13.0.</li>
<li>Added the upstream large-piece BPE merge path. Functionally, this
improves behavior for very large or repetitive inputs that previously
stressed the merge algorithm.</li>
<li>Changed <code>CoreBPE::encode</code> to return
<code>Result&lt;(Vec&lt;Rank&gt;, usize), EncodeError&gt;</code>,
matching upstream. Regex/tokenization failures can now be reported
instead of being hidden behind infallible APIs.</li>
<li>Updated <code>encode_as</code> and <code>count</code> to return
<code>Result</code> because they call <code>encode</code>.</li>
<li>Re-exported <code>EncodeError</code> so callers can handle encode
failures directly.</li>
<li>Aligned the vendored core with Rust 2024 and raised the crate MSRV
to Rust 1.85.</li>
<li>Synced model-to-tokenizer mappings with upstream
<code>tiktoken</code> 0.13.0 while keeping local extra prefixes
isolated.</li>
<li>Hardened asset downloads with SHA-256 checks and a repo-root-aware
asset path.</li>
</ul>
<h2>Breaking Changes</h2>
<p>If your code calls <code>CoreBPE::encode</code>, unwrap or propagate
the result before using the tokens:</p>
<pre lang="rust"><code>let allowed = bpe.special_tokens();
let (tokens, last_piece_token_len) = bpe.encode(&quot;hello
&lt;|endoftext|&gt;&quot;, &amp;allowed)?;
</code></pre>
<p>The generic helpers changed similarly:</p>
<pre lang="rust"><code>let (tokens, last_piece_token_len) =
bpe.encode_as::&lt;usize&gt;(text, &amp;allowed)?;
let token_count = bpe.count(text, &amp;allowed)?;
</code></pre>
<p><code>encode_ordinary</code>, <code>encode_ordinary_as</code>,
<code>encode_with_special_tokens</code>, and <code>count_ordinary</code>
remain infallible.</p>
<p>Projects must now build with Rust 1.85 or newer.</p>
<h2>Practical Impact</h2>
<ul>
<li>Applications processing long repeated text should see more robust
tokenization behavior.</li>
<li>Code that only uses helpers like
<code>get_chat_completion_max_tokens</code>,
<code>get_text_completion_max_tokens</code>, <code>bpe_for_model</code>,
or singleton tokenizer constructors should not need call-site
changes.</li>
<li>Code using low-level <code>CoreBPE::encode</code>,
<code>encode_as</code>, or <code>count</code> needs a small migration to
handle <code>Result</code>.</li>
</ul>
<h2>Links</h2>
<ul>
<li>PR: <a
href="https://redirect.github.com/zurawiki/tiktoken-rs/pull/164">zurawiki/tiktoken-rs#164</a></li>
<li>Upstream <code>tiktoken</code> 0.13.0: <a
href="https://github.com/openai/tiktoken/releases/tag/0.13.0">https://github.com/openai/tiktoken/releases/tag/0.13.0</a></li>
<li>Full changelog: <a
href="https://github.com/zurawiki/tiktoken-rs/compare/v0.11.0...v0.12.0">https://github.com/zurawiki/tiktoken-rs/compare/v0.11.0...v0.12.0</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="32de8dc052"><code>32de8dc</code></a>
Bump version minor to &quot;0.12.0&quot;</li>
<li><a
href="ac7e2e8b96"><code>ac7e2e8</code></a>
Backport tiktoken 0.13.0 (<a
href="https://redirect.github.com/zurawiki/tiktoken-rs/issues/164">#164</a>)</li>
<li>See full diff in <a
href="https://github.com/zurawiki/tiktoken-rs/compare/v0.11.0...v0.12.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=tiktoken-rs&package-manager=cargo&previous-version=0.11.0&new-version=0.12.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Cargo.lock

@@ -3331,7 +3331,7 @@ dependencies = [
  "pin-project-lite",
  "quinn-proto",
  "quinn-udp",
- "rustc-hash 2.1.2",
+ "rustc-hash",
  "rustls",
  "socket2",
  "thiserror 2.0.20",
@@ -3351,7 +3351,7 @@ dependencies = [
  "lru-slab",
  "rand 0.9.4",
  "ring",
- "rustc-hash 2.1.2",
+ "rustc-hash",
  "rustls",
  "rustls-pki-types",
  "slab",
@@ -3696,12 +3696,6 @@ dependencies = [
  "sqlite-wasm-rs",
 ]
 
-[[package]]
-name = "rustc-hash"
-version = "1.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
-
 [[package]]
 name = "rustc-hash"
 version = "2.1.2"
@@ -4232,9 +4226,9 @@ dependencies = [
 
 [[package]]
 name = "tiktoken-rs"
-version = "0.11.0"
+version = "0.12.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fac4a168cfc1d8ed65bf17a6ee0843ad9a68f863c63c0fb2fa7eab67838782ee"
+checksum = "027853bbf8c7763b77c5c595f1c271c7d536ced7d6f83452911b944621e57fc2"
 dependencies = [
  "anyhow",
  "base64 0.22.1",
@@ -4242,7 +4236,7 @@ dependencies = [
  "fancy-regex",
  "lazy_static",
  "regex",
- "rustc-hash 1.1.0",
+ "rustc-hash",
 ]
 
 [[package]]

crates/headroom-core/Cargo.toml

@@ -13,7 +13,7 @@ serde_json = { workspace = true }
 bytes = { workspace = true }
 thiserror = { workspace = true }
 tracing = { workspace = true }
-tiktoken-rs = "0.11"
+tiktoken-rs = "0.12"
 # `tokenizers` is the HuggingFace pure-Rust tokenizer crate. Default features
 # pull in `onig` for the BPE pre-tokenizer regex; that vendors oniguruma so it
 # builds without a system dep on macOS/Linux.