codex-as-api 0.6.1__tar.gz → 0.6.2__tar.gz

{codex_as_api-0.6.1 → codex_as_api-0.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: codex-as-api
-Version: 0.6.1
+Version: 0.6.2
 Summary: Use ChatGPT/Codex OAuth as a local OpenAI-compatible API server.
 Project-URL: Homepage, https://github.com/Eunho-J/codex-as-api
 Project-URL: Repository, https://github.com/Eunho-J/codex-as-api
@@ -18,12 +18,14 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Requires-Python: >=3.10
+Requires-Dist: regex
 Requires-Dist: tomli>=2; python_version < '3.11'
 Provides-Extra: dev
 Requires-Dist: httpx; extra == 'dev'
 Requires-Dist: mypy; extra == 'dev'
 Requires-Dist: pytest; extra == 'dev'
 Requires-Dist: ruff; extra == 'dev'
+Requires-Dist: types-regex; extra == 'dev'
 Provides-Extra: server
 Requires-Dist: fastapi>=0.100; extra == 'server'
 Requires-Dist: pydantic>=2.0; extra == 'server'
@@ -310,7 +312,11 @@ curl -N http://localhost:18080/v1/messages \
 
 ### `POST /v1/messages/count_tokens`
 
-Anthropic-compatible token counting helper. Codex OAuth does not expose a count-only endpoint equivalent to Anthropic's native API, so this route returns a conservative local estimate plus context-window metadata for the effective backend model. The estimate uses UTF-8 byte length as an upper bound for GPT/Codex BPE text tokens, then adds protocol overhead for roles, message boundaries, tools, raw request metadata, and images.
+Anthropic-compatible token counting helper. Codex OAuth does not expose a count-only endpoint equivalent to Anthropic's native API, so this route counts text locally and returns context-window metadata for the effective backend model. Normalized model-visible messages, tool calls, tool-result metadata, reasoning, and tool schemas are counted once. Request-envelope and generation-control fields are excluded, and image inputs use a separate fixed estimate so inline base64 data is not counted as text.
+
+GPT-5-family text uses a bundled port of official `tiktoken` `o200k_base` `encode_ordinary`: the same Unicode pre-tokenization regex, byte-pair merge algorithm, and merge-rank data are implemented in Python, TypeScript, and Rust. Official `tiktoken` maps `gpt-5` and its `gpt-5-`-prefixed variants to [`o200k_base`](https://github.com/openai/tiktoken/blob/08a5f3b2c987ada4fc5aa1f16c643c203fa8acaa/tiktoken/model.py#L7-L20); this project applies that GPT-5-family encoding to the bundled `gpt-5.*` Codex model IDs. The project does not depend on a `tiktoken` package or download encoding data at runtime. The last upstream synchronization check was **2026-07-14**, against [`tiktoken` 0.13.0 at `08a5f3b`](https://github.com/openai/tiktoken/tree/08a5f3b2c987ada4fc5aa1f16c643c203fa8acaa); the bundled rank file SHA-256 is `446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2d`.
+
+Official Codex also has a [`ceil(UTF-8 bytes / 4)` truncation helper](https://github.com/openai/codex/blob/393f64565ab46f09d99ca4d9bd973537e72a114b/codex-rs/utils/string/src/truncate.rs#L4-L84), but Codex documents the history estimate using that helper as [a coarse lower bound rather than a tokenizer-accurate count](https://github.com/openai/codex/blob/393f64565ab46f09d99ca4d9bd973537e72a114b/codex-rs/core/src/context_manager/history.rs#L160-L185). This endpoint therefore uses exact `o200k_base` ordinary text tokenization instead. The complete request count remains an estimate because protocol-wrapper overhead and image cost are local constants, but the former byte-as-token and raw-payload double count that could overstate ordinary Claude Code requests by about 8x is removed.
 
 ```bash
 curl http://localhost:18080/v1/messages/count_tokens \
@@ -636,7 +642,7 @@ curl -N http://localhost:18080/v1/chat/completions \
 
 ## Using with Claude Code
 
-The `/v1/messages` endpoint implements the Anthropic Messages gateway shape used by Claude Code. Compatibility was validated with Claude Code `2.1.208`, including streaming, a real `Read` tool loop, request-level GPT model routing, adaptive thinking, `--effort max`, and Fast Mode.
+The `/v1/messages` endpoint implements the Anthropic Messages gateway shape used by Claude Code. Compatibility was validated with Claude Code `2.1.209`, including streaming, real Codex OAuth chat, request-level GPT model routing, adaptive thinking, `--effort max`, and Fast Mode.
 
 Start the proxy first. `CODEX_AS_API_MODEL` is the fallback used when Claude Code sends a built-in Anthropic model name:
 
@@ -656,7 +662,7 @@ CLAUDE_CODE_ATTRIBUTION_HEADER=0 \
 claude
 ```
 
-Run `/model` and select **GPT-5.6 Sol**. The custom entry is appended to the built-in rows rather than replacing them. Claude Code `2.1.208` recognizes this GPT model ID without a custom capability declaration: `/effort low|medium|high|xhigh|max`, the picker effort control, and `claude --effort ...` are translated from Claude Code's `output_config.effort` to the Codex reasoning effort. Claude Code Fast Mode's `speed: "fast"` is translated to the Codex `priority` service tier.
+Run `/model` and select **GPT-5.6 Sol**. The custom entry is appended to the built-in rows rather than replacing them. Claude Code `2.1.209` recognizes this GPT model ID without a custom capability declaration: `/effort low|medium|high|xhigh|max`, the picker effort control, and `claude --effort ...` are translated from Claude Code's `output_config.effort` to the Codex reasoning effort. Claude Code Fast Mode's `speed: "fast"` is translated to the Codex `priority` service tier.
 
 If managed settings define an `availableModels` allowlist, that list must include the exact `ANTHROPIC_CUSTOM_MODEL_OPTION` value, such as `gpt-5.6-sol`; otherwise Claude Code hides or rejects the custom row.
 
@@ -714,7 +720,7 @@ The provider handles:
 ## Release & package publishing
 
 - Bump versions in `pyproject.toml`, `src/codex_as_api/__init__.py`, `src/codex_as_api/server.py`, `ts/package.json`, `ts/package-lock.json`, `rust/Cargo.toml`, and `rust/Cargo.lock`.
-- Publish a GitHub Release such as `v0.6.1` from the matching commit.
+- Publish a GitHub Release such as `v0.6.2` from the matching commit.
 - The manually-dispatched `Publish npm packages` workflow builds/tests the TypeScript package, runs `npm pack --dry-run`, publishes `codex-as-api` to npmjs when `NPM_TOKEN` is configured, and publishes `@eunho-j/codex-as-api` to GitHub Packages with `GITHUB_TOKEN`.
 
 Publishing to npmjs requires an authenticated npm session (`npm login` beforehand; `npm whoami` should succeed). From the repository root, the publish itself is one command:
@@ -757,6 +763,16 @@ npm test
 
 ## Release Notes
 
+### v0.6.2
+
+- Fix `/v1/messages/count_tokens` overcounting that could trigger Claude Code autocompaction on every turn.
+- Count normalized model-visible messages, tool calls, reasoning, tool-result metadata, and tool schemas once; remove the second full raw-request-body addition.
+- Port official `tiktoken` `o200k_base` ordinary encoding into Python, TypeScript, and Rust without adding a `tiktoken` package dependency or runtime rank download.
+- Match the upstream Unicode text split and BPE merge ranks last checked on 2026-07-14 against `tiktoken` 0.13.0 at `08a5f3b`.
+- Keep image input cost separate and count inline base64 images once rather than adding both a fixed image estimate and the base64 request bytes.
+- Add matching Python, TypeScript, and Rust endpoint regressions: a 4,000-byte ASCII message now returns `1,012` instead of roughly `8,000`, and non-model control fields do not change the count.
+- Validate the release with Claude Code `2.1.209` and real Codex OAuth chat.
+
 ### v0.6.1
 
 - Re-audit official Codex `0.144.4` and `main` at `393f64565ab46f09d99ca4d9bd973537e72a114b`, plus Claude Code `2.1.208`, before publishing.

{codex_as_api-0.6.1 → codex_as_api-0.6.2}/README.md

@@ -278,7 +278,11 @@ curl -N http://localhost:18080/v1/messages \
 
 ### `POST /v1/messages/count_tokens`
 
-Anthropic-compatible token counting helper. Codex OAuth does not expose a count-only endpoint equivalent to Anthropic's native API, so this route returns a conservative local estimate plus context-window metadata for the effective backend model. The estimate uses UTF-8 byte length as an upper bound for GPT/Codex BPE text tokens, then adds protocol overhead for roles, message boundaries, tools, raw request metadata, and images.
+Anthropic-compatible token counting helper. Codex OAuth does not expose a count-only endpoint equivalent to Anthropic's native API, so this route counts text locally and returns context-window metadata for the effective backend model. Normalized model-visible messages, tool calls, tool-result metadata, reasoning, and tool schemas are counted once. Request-envelope and generation-control fields are excluded, and image inputs use a separate fixed estimate so inline base64 data is not counted as text.
+
+GPT-5-family text uses a bundled port of official `tiktoken` `o200k_base` `encode_ordinary`: the same Unicode pre-tokenization regex, byte-pair merge algorithm, and merge-rank data are implemented in Python, TypeScript, and Rust. Official `tiktoken` maps `gpt-5` and its `gpt-5-`-prefixed variants to [`o200k_base`](https://github.com/openai/tiktoken/blob/08a5f3b2c987ada4fc5aa1f16c643c203fa8acaa/tiktoken/model.py#L7-L20); this project applies that GPT-5-family encoding to the bundled `gpt-5.*` Codex model IDs. The project does not depend on a `tiktoken` package or download encoding data at runtime. The last upstream synchronization check was **2026-07-14**, against [`tiktoken` 0.13.0 at `08a5f3b`](https://github.com/openai/tiktoken/tree/08a5f3b2c987ada4fc5aa1f16c643c203fa8acaa); the bundled rank file SHA-256 is `446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2d`.
+
+Official Codex also has a [`ceil(UTF-8 bytes / 4)` truncation helper](https://github.com/openai/codex/blob/393f64565ab46f09d99ca4d9bd973537e72a114b/codex-rs/utils/string/src/truncate.rs#L4-L84), but Codex documents the history estimate using that helper as [a coarse lower bound rather than a tokenizer-accurate count](https://github.com/openai/codex/blob/393f64565ab46f09d99ca4d9bd973537e72a114b/codex-rs/core/src/context_manager/history.rs#L160-L185). This endpoint therefore uses exact `o200k_base` ordinary text tokenization instead. The complete request count remains an estimate because protocol-wrapper overhead and image cost are local constants, but the former byte-as-token and raw-payload double count that could overstate ordinary Claude Code requests by about 8x is removed.
 
 ```bash
 curl http://localhost:18080/v1/messages/count_tokens \
@@ -604,7 +608,7 @@ curl -N http://localhost:18080/v1/chat/completions \
 
 ## Using with Claude Code
 
-The `/v1/messages` endpoint implements the Anthropic Messages gateway shape used by Claude Code. Compatibility was validated with Claude Code `2.1.208`, including streaming, a real `Read` tool loop, request-level GPT model routing, adaptive thinking, `--effort max`, and Fast Mode.
+The `/v1/messages` endpoint implements the Anthropic Messages gateway shape used by Claude Code. Compatibility was validated with Claude Code `2.1.209`, including streaming, real Codex OAuth chat, request-level GPT model routing, adaptive thinking, `--effort max`, and Fast Mode.
 
 Start the proxy first. `CODEX_AS_API_MODEL` is the fallback used when Claude Code sends a built-in Anthropic model name:
 
@@ -624,7 +628,7 @@ CLAUDE_CODE_ATTRIBUTION_HEADER=0 \
 claude
 ```
 
-Run `/model` and select **GPT-5.6 Sol**. The custom entry is appended to the built-in rows rather than replacing them. Claude Code `2.1.208` recognizes this GPT model ID without a custom capability declaration: `/effort low|medium|high|xhigh|max`, the picker effort control, and `claude --effort ...` are translated from Claude Code's `output_config.effort` to the Codex reasoning effort. Claude Code Fast Mode's `speed: "fast"` is translated to the Codex `priority` service tier.
+Run `/model` and select **GPT-5.6 Sol**. The custom entry is appended to the built-in rows rather than replacing them. Claude Code `2.1.209` recognizes this GPT model ID without a custom capability declaration: `/effort low|medium|high|xhigh|max`, the picker effort control, and `claude --effort ...` are translated from Claude Code's `output_config.effort` to the Codex reasoning effort. Claude Code Fast Mode's `speed: "fast"` is translated to the Codex `priority` service tier.
 
 If managed settings define an `availableModels` allowlist, that list must include the exact `ANTHROPIC_CUSTOM_MODEL_OPTION` value, such as `gpt-5.6-sol`; otherwise Claude Code hides or rejects the custom row.
 
@@ -682,7 +686,7 @@ The provider handles:
 ## Release & package publishing
 
 - Bump versions in `pyproject.toml`, `src/codex_as_api/__init__.py`, `src/codex_as_api/server.py`, `ts/package.json`, `ts/package-lock.json`, `rust/Cargo.toml`, and `rust/Cargo.lock`.
-- Publish a GitHub Release such as `v0.6.1` from the matching commit.
+- Publish a GitHub Release such as `v0.6.2` from the matching commit.
 - The manually-dispatched `Publish npm packages` workflow builds/tests the TypeScript package, runs `npm pack --dry-run`, publishes `codex-as-api` to npmjs when `NPM_TOKEN` is configured, and publishes `@eunho-j/codex-as-api` to GitHub Packages with `GITHUB_TOKEN`.
 
 Publishing to npmjs requires an authenticated npm session (`npm login` beforehand; `npm whoami` should succeed). From the repository root, the publish itself is one command:
@@ -725,6 +729,16 @@ npm test
 
 ## Release Notes
 
+### v0.6.2
+
+- Fix `/v1/messages/count_tokens` overcounting that could trigger Claude Code autocompaction on every turn.
+- Count normalized model-visible messages, tool calls, reasoning, tool-result metadata, and tool schemas once; remove the second full raw-request-body addition.
+- Port official `tiktoken` `o200k_base` ordinary encoding into Python, TypeScript, and Rust without adding a `tiktoken` package dependency or runtime rank download.
+- Match the upstream Unicode text split and BPE merge ranks last checked on 2026-07-14 against `tiktoken` 0.13.0 at `08a5f3b`.
+- Keep image input cost separate and count inline base64 images once rather than adding both a fixed image estimate and the base64 request bytes.
+- Add matching Python, TypeScript, and Rust endpoint regressions: a 4,000-byte ASCII message now returns `1,012` instead of roughly `8,000`, and non-model control fields do not change the count.
+- Validate the release with Claude Code `2.1.209` and real Codex OAuth chat.
+
 ### v0.6.1
 
 - Re-audit official Codex `0.144.4` and `main` at `393f64565ab46f09d99ca4d9bd973537e72a114b`, plus Claude Code `2.1.208`, before publishing.

{codex_as_api-0.6.1 → codex_as_api-0.6.2}/RELEASES.md

@@ -1,5 +1,23 @@
 # Release Notes
 
+## v0.6.2
+
+### Claude Code count_tokens autocompact fix
+
+- Fix the Anthropic `/v1/messages/count_tokens` estimator treating every UTF-8 byte as a token and then adding the full raw request body a second time.
+- Port official `tiktoken` `o200k_base` ordinary encoding into Python, TypeScript, and Rust: the same Unicode pre-tokenization regex, byte-pair merge algorithm, and merge ranks now count GPT-5-family text.
+- Count tool calls, tool-result identifiers, reasoning content, and normalized tool schemas once while excluding model IDs, output limits, stream flags, metadata, and other non-model request controls.
+- Keep the existing image estimate separate so URL and inline base64 images are counted once without tokenizing the base64 payload as text.
+- Keep token counting local and OAuth-free. No `tiktoken` package dependency or first-run encoding download is introduced.
+- Record the upstream synchronization point as `tiktoken` 0.13.0 commit `08a5f3b2c987ada4fc5aa1f16c643c203fa8acaa`, checked on 2026-07-14, with the official rank-file SHA-256 verified in packaged builds.
+
+### Validation
+
+- A 4,000-byte ASCII message returns `1,012` input tokens in all three implementations instead of roughly `8,000`.
+- Endpoint tests cover UTF-8 text, normalized tools, tool/reasoning history, URL and base64 images, ignored request controls, effective model limits, and zero upstream provider calls.
+- Official `tiktoken` 0.13.0 token IDs match the bundled ports across reference vectors, randomized multilingual text, code, whitespace, contractions, special-token-shaped literals, and long single-piece input.
+- Python, TypeScript, and Rust return identical counts across 204 cross-runtime endpoint cases; Claude Code 2.1.209 real Codex OAuth chat remains compatible.
+
 ## v0.6.1
 
 ### Claude Code 2.1.208 compatibility

codex_as_api-0.6.2/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,34 @@
+# Third-Party Notices
+
+## OpenAI tiktoken
+
+This project includes a port of the `o200k_base` ordinary-encoding logic and
+the corresponding merge-rank data from
+[openai/tiktoken](https://github.com/openai/tiktoken).
+
+Synchronized with tiktoken 0.13.0 at commit
+`08a5f3b2c987ada4fc5aa1f16c643c203fa8acaa` on 2026-07-14.
+The bundled rank data has SHA-256
+`446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2d`.
+
+MIT License
+
+Copyright (c) 2022 OpenAI, Shantanu Jain
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.