python-codex 0.1.8__tar.gz → 0.1.10__tar.gz

{python_codex-0.1.8 → python_codex-0.1.10}/AGENTS.md

@@ -15,6 +15,7 @@
 - `responses_server` 的 provider-specific chat payload 定制统一放在 `responses_server/payload_processors.py`：使用 `CompatServerConfig.model_provider` 选择 `provider_name -> proc_fn(outcomming_request)` 映射，并且只在真正发出 downstream `/v1/chat/completions` 前 post-process；`StreamRouter` 内部继续保留 canonical payload，避免 tool hydration / mock web_search follow-up 被 provider 改写污染。
 - `responses_server` 如果要兼容下游 `/v1/messages`，也优先保持这条边界：内部继续用 canonical chat request / chat-like chunk 流，只有真正发请求和读取 SSE 时才做 messages 适配，这样 tool hydration、mock `web_search` follow-up、provider payload post-process 都能复用。
 - 真实 vLLM `0.19.0` 的 `/v1/messages` 会对缺失 `max_tokens` 直接返回 `400`；messages 适配层必须总是补这个字段。当前约定是优先透传请求里的 `max_output_tokens`/`max_tokens`，否则回退到默认 `32000`。
+- 对 vLLM chat-completions 打开 `return_token_ids=true` 时，streaming `prompt_token_ids` 只出现在首个 chunk，后续每个 chunk 的 `choices[*].token_ids` 都是 decode delta；要在 `responses_server` 侧导出 trajectory 时，按“首个 `prompt_token_ids` + 按序拼接所有 chunk 的 `token_ids`”重建即可。
 - `pycodex` 默认是最小交互 CLI；无 prompt 时进入 REPL，并通过 `AgentRuntime` 跑外层提交循环。当前会显示最小事件流、assistant 流式输出、简单 title/history（`/title`, `/history`），并默认注册一组与原版一一对应的本地工具子集。
 - 交互 CLI 的事件流展示优先表达用户可感知的阶段（例如工具开始/完成、模型回看工具结果），不要直接把内部 `iteration` 计数暴露成主要状态文案；`iterations` 应继续保留在 `TurnResult` 等程序化结果里。
 - prompt/context 相关逻辑统一放在 `pycodex/context.py`：`AgentLoop` 只维护真实会话历史；每轮请求前由 `ContextManager` 注入 base instructions、developer message、`AGENTS.md` 指令和 `<environment_context>`，且这些注入项不写回 history。
@@ -49,5 +50,6 @@
 - `--put` 的 CLI UX 现在约定为“先打印打包文件清单和上传目标，再打印结果”，并且最后一行保留为可直接执行的 `pycodex --call SECRET-CALLID@<host:port>` 一键启动命令；后续如果再改输出，尽量保留这个末行语义。
 - `--put` 现在不是“只上传就结束”：上传成功后还会立刻跑一次真实的 `--call` 下载/解包 round-trip 测试，只有这个测试也成功才算整个 `--put` 成功。排障时如果上传成功但 CLI 仍退出非零，要继续看 call 路径而不只看 put 端。
 - `--put` 现在会先做 `/healthz` preflight，再开始扫描/压缩目录；如果用户报“卡死”，先看目标 `host:port` 是否真有 storage server 在监听。默认打包策略也已经切到白名单：只带 `config.toml`、`.env`、`AGENTS.md`、`AGENTS.override.md`、`skills/**`，以及 config 里相对引用的 `model_instructions_file`，所以像 `sessions/` 这类运行态目录不会再靠黑名单排除。
+- `--call` / portable storage paths must not rely on the process default text encoding. Always pass `encoding="utf-8"` when reading config, prompts, AGENTS files, skills, dotenv, and session history; for user-authored instructions/history, prefer `errors="replace"` so a Windows GBK locale cannot crash on UTF-8 punctuation such as U+2264 or em dash.
 - 对接真实 `~/.codex/sessions/.../rollout-*.jsonl` 时，不要假设它一定是严格的一行一个 JSON object：本机样本可能包含 pretty-printed 多行对象，且文件尾部偶尔带未完成记录。恢复历史时用 concatenated-JSON 方式读取，并容忍尾部残缺。
 - `pycodex` 本地 session 保存现在也按上游思路走：新 session 一开始就分配稳定的 uuidv7 thread/session id，并把历史增量追加到 `CODEX_HOME/sessions/.../rollout-*.jsonl`；`/resume` 列表应只展示至少有真实 user message 的 rollout，避免空白新 session 污染恢复列表。

{python_codex-0.1.8 → python_codex-0.1.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: python-codex
-Version: 0.1.8
+Version: 0.1.10
 Summary: A minimal Python extraction of Codex's main agent loop
 License-File: LICENSE
 Requires-Python: >=3.6.2

{python_codex-0.1.8 → python_codex-0.1.10}/docs/responses_server/README.md

@@ -26,6 +26,7 @@
 - vLLM chat-completions `reasoning` / `reasoning_content` -> Responses `reasoning` item 适配
 - vLLM 历史 `reasoning` item -> assistant message `reasoning` 字段回放
 - vLLM streaming `usage` -> final `response.completed.response.usage`
+- 当环境变量 `PYCODEX_DUMP` 存在时，为每条 outcomming 请求附加 `return_token_ids = true`，并把抓到的 `prompt_token_ids` / `token_ids` 以 JSONL 追加到 `{PYCODEX_DUMP}/dump.jsonl`
 - 下游 chat stream 如果半路断开，会转成上游可解析的 `response.failed` 事件，而不是直接截断 HTTP body
 - 普通 function tools
 - custom tools 的 function-wrapper 兼容适配
@@ -76,6 +77,19 @@ uv run python -m responses_server \
 默认会在本地启动一个 incomming Responses 服务；真正监听地址由 `--host` 和 `--port`
 控制。
 
+如果要导出下游 trajectory token id，可以在启动前设置：
+
+```bash
+export PYCODEX_DUMP=/tmp/pycodex-dump
+```
+
+server 会为每条实际转发到下游的请求附上 `return_token_ids = true`，并把
+trajectory 追加到 `${PYCODEX_DUMP}/dump.jsonl`，当前记录格式是：
+
+```json
+{"tokens":{"prefill":[1,2,3],"decode":[4,5,6]},"send_timestamp":2222.0}
+```
+
 如果下游 provider 需要对 chat payload 做定制化改写，可以在
 `responses_server/payload_processors.py` 里注册对应 `model_provider -> proc_fn`
 映射；server 会在真正发出每一条 outcomming `/v1/chat/completions` 请求前，

{python_codex-0.1.8 → python_codex-0.1.10}/pycodex/context.py

@@ -89,7 +89,7 @@ class ContextConfig:
         profile: 'typing.Union[str, None]' = None,
     ) -> 'ContextConfig':
         path = Path(config_path)
-        data = tomllib.loads(path.read_text())
+        data = tomllib.loads(path.read_text(encoding="utf-8"))
         selected = dict(data)
         if profile is not None:
             overrides = data.get("profiles", {}).get(profile)
@@ -162,7 +162,9 @@ class ContextManager:
         self._include_permissions_instructions = include_permissions_instructions
         self._include_skills_instructions = include_skills_instructions
         self._network_access = network_access
-        self._default_base_instructions = DEFAULT_BASE_INSTRUCTIONS_PATH.read_text()
+        self._default_base_instructions = DEFAULT_BASE_INSTRUCTIONS_PATH.read_text(
+            encoding="utf-8"
+        )
         self._workspace_metadata_turn_id: 'typing.Union[str, None]' = None
         self._workspace_metadata_cache: 'typing.Union[JSONDict, None]' = None
 
@@ -237,7 +239,10 @@ class ContextManager:
         if self._config.base_instructions is not None:
             return self._config.base_instructions
         if self._config.model_instructions_file is not None:
-            return self._config.model_instructions_file.read_text().strip()
+            return self._config.model_instructions_file.read_text(
+                encoding="utf-8",
+                errors="replace",
+            ).strip()
         resolved = self._resolve_model_instructions()
         if resolved is not None:
             return resolved
@@ -327,11 +332,11 @@ class ContextManager:
             return None
 
         sandbox_text = (
-            sandbox_prompt_path.read_text().strip().replace(
+            sandbox_prompt_path.read_text(encoding="utf-8").strip().replace(
                 "{network_access}", self._network_access
             )
         )
-        approval_text = approval_prompt_path.read_text().strip()
+        approval_text = approval_prompt_path.read_text(encoding="utf-8").strip()
         return "\n".join(
             [
                 PERMISSIONS_OPEN_TAG,
@@ -429,7 +434,7 @@ class ContextManager:
         docs: 'typing.List[str]' = []
         remaining = self._config.project_doc_max_bytes
         for path in self._discover_project_doc_paths():
-            text = path.read_text()
+            text = path.read_text(encoding="utf-8", errors="replace")
             if not text.strip():
                 continue
             if remaining is None:
@@ -437,7 +442,7 @@ class ContextManager:
                 continue
             if remaining <= 0:
                 break
-            encoded = text.encode()
+            encoded = text.encode("utf-8")
             docs.append(encoded[:remaining].decode(errors="ignore"))
             remaining -= min(len(encoded), remaining)
         if not docs:
@@ -507,15 +512,15 @@ def _normalize_int(value) -> 'typing.Union[int, None]':
 
 def _default_collaboration_instructions(mode: 'CollaborationMode') -> 'str':
     if mode == "plan":
-        return PLAN_COLLABORATION_INSTRUCTIONS_PATH.read_text()
-    return DEFAULT_COLLABORATION_INSTRUCTIONS_PATH.read_text()
+        return PLAN_COLLABORATION_INSTRUCTIONS_PATH.read_text(encoding="utf-8")
+    return DEFAULT_COLLABORATION_INSTRUCTIONS_PATH.read_text(encoding="utf-8")
 
 
 def _read_first_instruction_file(base: 'Path') -> 'typing.Union[str, None]':
     for candidate_name in (LOCAL_PROJECT_DOC_FILENAME, DEFAULT_PROJECT_DOC_FILENAME):
         candidate = base / candidate_name
         try:
-            contents = candidate.read_text()
+            contents = candidate.read_text(encoding="utf-8", errors="replace")
         except OSError:
             continue
         trimmed = contents.strip()
@@ -526,7 +531,7 @@ def _read_first_instruction_file(base: 'Path') -> 'typing.Union[str, None]':
 
 @lru_cache(maxsize=1)
 def _load_models_by_slug() -> 'typing.Dict[str, JSONDict]':
-    payload = json.loads(DEFAULT_MODELS_PATH.read_text())
+    payload = json.loads(DEFAULT_MODELS_PATH.read_text(encoding="utf-8"))
     models = payload.get("models", [])
     by_slug: 'typing.Dict[str, JSONDict]' = {}
     for model in models:
@@ -571,7 +576,7 @@ def _discover_skill_files(
 
 
 def _parse_skill_descriptor(path: 'Path', scope_rank: 'int') -> 'typing.Union[SkillDescriptor, None]':
-    text = path.read_text()
+    text = path.read_text(encoding="utf-8", errors="replace")
     if not text.startswith("---\n"):
         return None
     end_marker = "\n---\n"

{python_codex-0.1.8 → python_codex-0.1.10}/pycodex/model.py

@@ -71,7 +71,7 @@ class ResponsesProviderConfig:
         config_path: 'typing.Union[str, Path]' = DEFAULT_CODEX_CONFIG_PATH,
         profile: 'typing.Union[str, None]' = None,
     ) -> 'ResponsesProviderConfig':
-        data = tomllib.loads(Path(config_path).read_text())
+        data = tomllib.loads(Path(config_path).read_text(encoding="utf-8"))
         selected = dict(data)
         if profile is not None:
             overrides = data.get("profiles", {}).get(profile)

{python_codex-0.1.8 → python_codex-0.1.10}/pycodex/portable.py

@@ -123,7 +123,8 @@ def bootstrap_called_home(
             },
             ensure_ascii=False,
             indent=2,
-        )
+        ),
+        encoding="utf-8",
     )
     return home_dir / DEFAULT_ENTRY_CONFIG
 
@@ -199,7 +200,7 @@ def _collect_config_referenced_files(root: 'Path') -> 'typing.Set[str]':
     config_path = root / DEFAULT_ENTRY_CONFIG
     if not config_path.is_file():
         return set()
-    data = tomllib.loads(config_path.read_text())
+    data = tomllib.loads(config_path.read_text(encoding="utf-8"))
     referenced: 'typing.Set[str]' = set()
     candidates = [data]
     profiles = data.get("profiles")
@@ -352,7 +353,7 @@ def _load_cached_metadata(metadata_path: 'Path') -> 'typing.Dict[str, object]':
     if not metadata_path.is_file():
         return {}
     try:
-        payload = json.loads(metadata_path.read_text())
+        payload = json.loads(metadata_path.read_text(encoding="utf-8"))
     except (ValueError, OSError):
         return {}
     return payload if isinstance(payload, dict) else {}

{python_codex-0.1.8 → python_codex-0.1.10}/pycodex/tools/base_tool.py

@@ -30,7 +30,8 @@ EXEC_TOOLS_SNAPSHOT_PATH = (
 @lru_cache(maxsize=1)
 def _load_exec_tool_payloads() -> 'typing.Dict[str, JSONDict]':
     payloads: 'typing.Dict[str, JSONDict]' = {}
-    for payload in json.loads(EXEC_TOOLS_SNAPSHOT_PATH.read_text()):
+    raw_payloads = EXEC_TOOLS_SNAPSHOT_PATH.read_text(encoding="utf-8")
+    for payload in json.loads(raw_payloads):
         if not isinstance(payload, dict):
             continue
         name = payload.get("name")

{python_codex-0.1.8 → python_codex-0.1.10}/pycodex/utils/dotenv.py

@@ -18,7 +18,9 @@ def load_codex_dotenv(config_path: 'typing.Union[str, Path]') -> 'None':
         _LOADED_CODEX_DOTENV_HOMES.add(codex_home)
         return
 
-    for key, value in parse_dotenv(dotenv_path.read_text()).items():
+    for key, value in parse_dotenv(
+        dotenv_path.read_text(encoding="utf-8", errors="replace")
+    ).items():
         if key.upper().startswith(ILLEGAL_ENV_VAR_PREFIX):
             continue
         os.environ[key] = value

{python_codex-0.1.8 → python_codex-0.1.10}/pycodex/utils/get_env.py

@@ -98,7 +98,11 @@ def get_os_info() -> 'typing.Tuple[str, str]':
     os_release = Path("/etc/os-release")
     if os_release.is_file():
         values: 'typing.Dict[str, str]' = {}
-        for line in os_release.read_text().splitlines():
+        os_release_text = os_release.read_text(
+            encoding="utf-8",
+            errors="replace",
+        )
+        for line in os_release_text.splitlines():
             if "=" not in line:
                 continue
             key, value = line.split("=", 1)

{python_codex-0.1.8 → python_codex-0.1.10}/pycodex/utils/session_persist.py

@@ -282,7 +282,9 @@ def _latest_thread_names_by_id(codex_home: 'Path') -> 'typing.Dict[str, str]':
         return {}
 
     names_by_id: 'typing.Dict[str, str]' = {}
-    for raw_line in reversed(index_path.read_text().splitlines()):
+    for raw_line in reversed(
+        index_path.read_text(encoding="utf-8", errors="replace").splitlines()
+    ):
         line = raw_line.strip()
         if not line:
             continue
@@ -321,7 +323,7 @@ def _extract_first_user_message_preview(rollout_path: 'Path') -> 'typing.Union[s
 
 
 def _iter_rollout_entries(rollout_path: 'Path') -> 'typing.Iterable[typing.Dict[str, object]]':
-    text = rollout_path.read_text()
+    text = rollout_path.read_text(encoding="utf-8", errors="replace")
     decoder = json.JSONDecoder()
     index = 0
     parsed_entries = 0

{python_codex-0.1.8 → python_codex-0.1.10}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "python-codex"
-version = "0.1.8"
+version = "0.1.10"
 description = "A minimal Python extraction of Codex's main agent loop"
 readme = "README.md"
 requires-python = ">=3.6.2"

{python_codex-0.1.8 → python_codex-0.1.10}/responses_server/app.py

@@ -177,6 +177,7 @@ class ManagedResponseServer:
             self._app,
             host=self._config.host,
             port=self._config.port,
+            loop="asyncio",
             log_level="error",
             access_log=False,
         )

{python_codex-0.1.8 → python_codex-0.1.10}/responses_server/messages_api.py

@@ -66,6 +66,8 @@ def build_messages_request(
         "max_tokens": _resolve_max_tokens(outcomming_request),
         "stream": bool(outcomming_request.get("stream", True)),
     }
+    if isinstance(outcomming_request.get("return_token_ids"), bool):
+        payload["return_token_ids"] = bool(outcomming_request.get("return_token_ids"))
     if system_blocks:
         payload["system"] = system_blocks
 

{python_codex-0.1.8 → python_codex-0.1.10}/responses_server/payload_processors.py

@@ -32,6 +32,7 @@ class OutgoingRequest(TypedDict):
     tools: 'Optional[typing.List[typing.Dict[str, object]]]'
     tool_choice: 'Optional[object]'
     parallel_tool_calls: 'Optional[bool]'
+    return_token_ids: 'Optional[bool]'
 
 
 PayloadPostProcessor = Callable[[OutgoingRequest], OutgoingRequest]

{python_codex-0.1.8 → python_codex-0.1.10}/responses_server/server.py

@@ -3,6 +3,7 @@ from .config import CompatServerConfig
 from .payload_processors import post_process_outcomming_request
 from .session_store import SessionStore
 from .stream_router import StreamRouter
+from .trajectory_dump import TrajectoryDumpWriter
 import typing
 
 
@@ -16,6 +17,7 @@ class ResponseServer:
         self._config = config
         self._session_store = session_store or SessionStore()
         self._stream_router = stream_router or StreamRouter(config)
+        self._trajectory_dump = TrajectoryDumpWriter.from_env()
 
     @property
     def config(self) -> 'CompatServerConfig':
@@ -38,6 +40,9 @@ class ResponseServer:
         request_headers: 'typing.Dict[str, str]',
     ):
         outcomming_request = self._stream_router.build_outcomming_request(request_body)
+        if self._trajectory_dump is not None:
+            # vLLM surfaces prompt/decode token IDs only when this flag is set.
+            outcomming_request["return_token_ids"] = True
         outcomming_request = post_process_outcomming_request(
             outcomming_request,
             self._config.model_provider,
@@ -52,12 +57,9 @@ class ResponseServer:
             session_id=session_id,
             model=str(outcomming_request["model"]),
         )
-        incomming_stream = self._stream_router.open_outcomming_stream(
-            outcomming_request
-        )
         return self._stream_router.route_stream(
-            incomming_stream,
             stored_response,
             outcomming_request,
             custom_tool_names,
+            self._trajectory_dump,
         )

{python_codex-0.1.8 → python_codex-0.1.10}/responses_server/stream_router.py

@@ -27,6 +27,7 @@ from .tools.web_search import (
     hydrate_tool_call_names,
     partition_tool_calls,
 )
+from .trajectory_dump import TrajectoryDumpWriter
 import typing
 
 
@@ -285,10 +286,10 @@ class StreamRouter:
 
     def route_stream(
         self,
-        incomming_stream,
         stored_response: 'StoredResponse',
         outcomming_request: 'typing.Dict[str, object]',
         custom_tool_names: 'typing.Union[typing.Set[str], None]' = None,
+        trajectory_dump: 'typing.Union[TrajectoryDumpWriter, None]' = None,
     ):
         yield (
             "response.created",
@@ -307,7 +308,10 @@ class StreamRouter:
         reasoning_parts: 'typing.List[str]' = []
         latest_usage: 'typing.Dict[str, object]' = {}
         current_request = json.loads(json.dumps(outcomming_request))
-        current_stream = incomming_stream
+        current_stream = self._open_tracked_outcomming_stream(
+            current_request,
+            trajectory_dump,
+        )
 
         while True:
             tool_calls: 'typing.Dict[int, typing.Dict[str, object]]' = {}
@@ -352,7 +356,10 @@ class StreamRouter:
                     )
                 except ValueError as exc:
                     raise OutcommingChatError(str(exc)) from exc
-                current_stream = self.open_outcomming_stream(current_request)
+                current_stream = self._open_tracked_outcomming_stream(
+                    current_request,
+                    trajectory_dump,
+                )
                 continue
 
             for item in self._build_output_items(
@@ -394,6 +401,16 @@ class StreamRouter:
             },
         )
 
+    def _open_tracked_outcomming_stream(
+        self,
+        outcomming_request: 'typing.Dict[str, object]',
+        trajectory_dump: 'typing.Union[TrajectoryDumpWriter, None]' = None,
+    ):
+        outcomming_stream = self.open_outcomming_stream(outcomming_request)
+        if trajectory_dump is None:
+            return outcomming_stream
+        return trajectory_dump.wrap_stream(outcomming_stream)
+
     def _responses_input_to_chat_messages(
         self,
         instructions: 'str',

python_codex-0.1.10/responses_server/trajectory_dump.py

@@ -0,0 +1,105 @@
+import json
+import os
+import sys
+import threading
+import time
+import typing
+
+
+class TrajectoryDumpWriter:
+    ENV_VAR = "PYCODEX_DUMP"
+
+    def __init__(self, root_dir: 'str') -> 'None':
+        self._root_dir = os.path.abspath(root_dir)
+        self._dump_path = os.path.join(self._root_dir, "dump.jsonl")
+        self._lock = threading.Lock()
+        os.makedirs(self._root_dir, exist_ok=True)
+
+    @classmethod
+    def from_env(cls) -> 'typing.Union[TrajectoryDumpWriter, None]':
+        root_dir = str(os.environ.get(cls.ENV_VAR, "") or "").strip()
+        if not root_dir:
+            return None
+        return cls(root_dir)
+
+    def wrap_stream(self, outcomming_stream):
+        def iter_stream():
+            capture = _TrajectoryCapture(self, time.time())
+            try:
+                for chunk in outcomming_stream:
+                    capture.observe_chunk(chunk)
+                    yield chunk
+            finally:
+                capture.flush()
+
+        return iter_stream()
+
+    def _append_record(self, record: 'typing.Dict[str, object]') -> 'None':
+        serialized = json.dumps(record, ensure_ascii=False)
+        with self._lock:
+            os.makedirs(self._root_dir, exist_ok=True)
+            with open(self._dump_path, "a", encoding="utf-8") as handle:
+                handle.write(serialized)
+                handle.write("\n")
+
+
+class _TrajectoryCapture:
+    def __init__(
+        self,
+        writer: 'TrajectoryDumpWriter',
+        send_timestamp: 'float',
+    ) -> 'None':
+        self._writer = writer
+        self._send_timestamp = float(send_timestamp)
+        self._prefill_token_ids = None
+        self._decode_token_ids = []
+        self._closed = False
+
+    def observe_chunk(self, payload: 'object') -> 'None':
+        if not isinstance(payload, dict):
+            return
+        if self._prefill_token_ids is None and "prompt_token_ids" in payload:
+            normalized_prefill = _normalize_token_ids(payload.get("prompt_token_ids"))
+            if normalized_prefill is not None:
+                self._prefill_token_ids = normalized_prefill
+
+        choices = payload.get("choices") or []
+        if not isinstance(choices, list):
+            return
+        for raw_choice in choices:
+            if not isinstance(raw_choice, dict):
+                continue
+            normalized_decode = _normalize_token_ids(raw_choice.get("token_ids"))
+            if normalized_decode:
+                self._decode_token_ids.extend(normalized_decode)
+
+    def flush(self) -> 'None':
+        if self._closed:
+            return
+        self._closed = True
+        record = {
+            "tokens": {
+                "prefill": list(self._prefill_token_ids or []),
+                "decode": list(self._decode_token_ids),
+            },
+            "send_timestamp": self._send_timestamp,
+        }
+        try:
+            self._writer._append_record(record)
+        except Exception as exc:
+            print(
+                "responses_server: failed to append PYCODEX_DUMP trajectory: %s"
+                % exc,
+                file=sys.stderr,
+            )
+
+
+def _normalize_token_ids(raw_value: 'object') -> 'typing.Union[typing.List[int], None]':
+    if not isinstance(raw_value, list):
+        return None
+    token_ids = []
+    for value in raw_value:
+        if isinstance(value, bool) or not isinstance(value, int):
+            continue
+        token_ids.append(value)
+    return token_ids

{python_codex-0.1.8 → python_codex-0.1.10}/tests/responses_server/fake_chat_completions_server.py

@@ -102,20 +102,31 @@ class RunningFastAPITestServer:
             raise RuntimeError("timed out waiting for fake FastAPI server to stop")
 
 
-def build_text_chunks(text: 'str', model_id: 'str' = DEFAULT_MODEL_ID) -> 'typing.List[typing.Dict[str, object]]':
+def build_text_chunks(
+    text: 'str',
+    model_id: 'str' = DEFAULT_MODEL_ID,
+    prompt_token_ids: 'typing.Union[typing.List[int], None]' = None,
+    decode_token_ids: 'typing.Union[typing.List[int], None]' = None,
+) -> 'typing.List[typing.Dict[str, object]]':
+    first_chunk: 'typing.Dict[str, object]' = {
+        "id": "chatcmpl_mock",
+        "object": "chat.completion.chunk",
+        "model": model_id,
+        "choices": [
+            {
+                "index": 0,
+                "delta": {"role": "assistant", "content": text},
+                "finish_reason": None,
+            }
+        ],
+    }
+    if prompt_token_ids is not None:
+        first_chunk["prompt_token_ids"] = list(prompt_token_ids)
+    if decode_token_ids is not None:
+        first_chunk["choices"][0]["token_ids"] = list(decode_token_ids)
+
     return [
-        {
-            "id": "chatcmpl_mock",
-            "object": "chat.completion.chunk",
-            "model": model_id,
-            "choices": [
-                {
-                    "index": 0,
-                    "delta": {"role": "assistant", "content": text},
-                    "finish_reason": None,
-                }
-            ],
-        },
+        first_chunk,
         {
             "id": "chatcmpl_mock",
             "object": "chat.completion.chunk",

{python_codex-0.1.8 → python_codex-0.1.10}/tests/responses_server/test_server.py

@@ -82,6 +82,80 @@ def test_responses_server_streams_text_from_chat_backend(tmp_path) -> 'None':
     ]
 
 
+def test_responses_server_dumps_forwarded_chat_token_trajectory(
+    tmp_path,
+    monkeypatch,
+) -> 'None':
+    dump_root = tmp_path / "dump"
+    monkeypatch.setenv("PYCODEX_DUMP", str(dump_root))
+    capture_store = CaptureStore(tmp_path / "chat_capture")
+    fake_chat_server = build_fake_chat_server(
+        capture_store,
+        build_text_chunks(
+            "Hello",
+            prompt_token_ids=[101, 102, 103],
+            decode_token_ids=[201, 202],
+        ),
+    )
+    fake_chat_server.start()
+
+    app = ManagedResponseServer.build_app(
+        CompatServerConfig(
+            outcomming_base_url=f"http://127.0.0.1:{fake_chat_server.server_port}/v1",
+        )
+    )
+
+    try:
+        with TestClient(app) as client:
+            response = client.post(
+                "/v1/responses",
+                json={
+                    "model": "gpt-5.4",
+                    "instructions": "Be concise.",
+                    "input": [
+                        {
+                            "type": "message",
+                            "role": "user",
+                            "content": [{"type": "input_text", "text": "hi"}],
+                        }
+                    ],
+                    "tools": [],
+                    "tool_choice": "auto",
+                    "parallel_tool_calls": True,
+                    "stream": True,
+                },
+                headers={"Accept": "text/event-stream"},
+            )
+            status = response.status_code
+    finally:
+        fake_chat_server.stop()
+
+    assert status == 200
+
+    request_files = sorted((tmp_path / "chat_capture").glob("*_POST_*.json"))
+    assert len(request_files) == 1
+    request = json.loads(request_files[0].read_text())
+    assert request["body"]["return_token_ids"] is True
+
+    dump_file = dump_root / "dump.jsonl"
+    assert dump_file.exists()
+    dump_records = [
+        json.loads(line)
+        for line in dump_file.read_text().splitlines()
+        if line.strip()
+    ]
+    assert dump_records == [
+        {
+            "tokens": {
+                "prefill": [101, 102, 103],
+                "decode": [201, 202],
+            },
+            "send_timestamp": dump_records[0]["send_timestamp"],
+        }
+    ]
+    assert isinstance(dump_records[0]["send_timestamp"], float)
+
+
 def test_responses_server_streams_text_from_messages_backend(tmp_path) -> 'None':
     capture_store = CaptureStore(tmp_path / "messages_capture")
     fake_messages_server = build_fake_messages_server(
@@ -1547,6 +1621,130 @@ def test_responses_server_mocks_web_search_and_continues_chat(tmp_path) -> 'None
     }
 
 
+def test_responses_server_dumps_all_forwarded_requests_for_mock_web_search(
+    tmp_path,
+    monkeypatch,
+) -> 'None':
+    dump_root = tmp_path / "dump"
+    monkeypatch.setenv("PYCODEX_DUMP", str(dump_root))
+    capture_store = CaptureStore(tmp_path / "chat_capture")
+    fake_chat_server = build_fake_chat_server(
+        capture_store,
+        [
+            [
+                {
+                    "id": "chatcmpl_mock",
+                    "object": "chat.completion.chunk",
+                    "model": "gpt-5.4",
+                    "prompt_token_ids": [11, 12],
+                    "choices": [
+                        {
+                            "index": 0,
+                            "delta": {
+                                "tool_calls": [
+                                    {
+                                        "index": 0,
+                                        "id": "ws_1",
+                                        "type": "function",
+                                        "function": {
+                                            "arguments": '{"query":"github codex"}'
+                                        },
+                                    }
+                                ]
+                            },
+                            "token_ids": [21, 22],
+                            "finish_reason": None,
+                        }
+                    ],
+                },
+                {
+                    "id": "chatcmpl_mock",
+                    "object": "chat.completion.chunk",
+                    "model": "gpt-5.4",
+                    "choices": [
+                        {
+                            "index": 0,
+                            "delta": {},
+                            "finish_reason": "tool_calls",
+                        }
+                    ],
+                },
+            ],
+            build_text_chunks(
+                "done",
+                prompt_token_ids=[31, 32, 33],
+                decode_token_ids=[41, 42],
+            ),
+        ],
+    )
+    fake_chat_server.start()
+
+    app = ManagedResponseServer.build_app(
+        CompatServerConfig(
+            outcomming_base_url=f"http://127.0.0.1:{fake_chat_server.server_port}/v1",
+        )
+    )
+
+    try:
+        with TestClient(app) as client:
+            response = client.post(
+                "/v1/responses",
+                json={
+                    "model": "gpt-5.4",
+                    "instructions": "Be concise.",
+                    "input": [
+                        {
+                            "type": "message",
+                            "role": "user",
+                            "content": [
+                                {
+                                    "type": "input_text",
+                                    "text": "Search the web, then answer.",
+                                }
+                            ],
+                        }
+                    ],
+                    "tools": [
+                        {
+                            "type": "web_search",
+                            "external_web_access": True,
+                        }
+                    ],
+                    "tool_choice": "auto",
+                    "parallel_tool_calls": False,
+                    "stream": True,
+                },
+                headers={"Accept": "text/event-stream"},
+            )
+            status = response.status_code
+    finally:
+        fake_chat_server.stop()
+
+    assert status == 200
+
+    request_files = sorted((tmp_path / "chat_capture").glob("*_POST_*.json"))
+    assert len(request_files) == 2
+    for request_file in request_files:
+        request = json.loads(request_file.read_text())
+        assert request["body"]["return_token_ids"] is True
+
+    dump_records = [
+        json.loads(line)
+        for line in (dump_root / "dump.jsonl").read_text().splitlines()
+        if line.strip()
+    ]
+    assert len(dump_records) == 2
+    assert dump_records[0]["tokens"] == {
+        "prefill": [11, 12],
+        "decode": [21, 22],
+    }
+    assert dump_records[1]["tokens"] == {
+        "prefill": [31, 32, 33],
+        "decode": [41, 42],
+    }
+    assert dump_records[0]["send_timestamp"] <= dump_records[1]["send_timestamp"]
+
+
 def test_responses_server_turns_mock_web_search_calls_into_messages_followup(
     tmp_path,
 ) -> 'None':
@@ -1794,3 +1992,12 @@ def test_responses_server_turns_truncated_downstream_stream_into_response_failed
         or "outcomming chat stream ended before [DONE]" in body
     )
     assert "event: response.completed" not in body
+
+
+def test_managed_response_server_forces_asyncio_loop() -> 'None':
+    server = ManagedResponseServer(
+        CompatServerConfig(
+            outcomming_base_url="http://127.0.0.1:8000/v1",
+        )
+    )
+    assert server._uvicorn_config.loop == "asyncio"

{python_codex-0.1.8 → python_codex-0.1.10}/tests/test_cli.py

@@ -773,6 +773,73 @@ async def test_run_cli_bootstraps_called_home_before_loading_config(
     assert captured["provider_config"].api_key_env == "PORTABLE_API_KEY"
 
 
+@pytest.mark.asyncio
+async def test_run_cli_call_reads_called_home_text_as_utf8(
+    tmp_path,
+    monkeypatch,
+) -> 'None':
+    codex_home = tmp_path / "codex-home"
+    codex_home.mkdir()
+    _write_stored_codex_home(codex_home)
+    (codex_home / "AGENTS.md").write_text(
+        "stored rules with unicode: \u22645 min \u2014 \u4e2d\u6587\n",
+        encoding="utf-8",
+    )
+    (codex_home / "skills" / "demo" / "SKILL.md").write_text(
+        "---\n"
+        "name: demo\n"
+        "description: unicode \u2264 \u2014 \u4e2d\u6587\n"
+        "---\n"
+        "Stored skill.\n",
+        encoding="utf-8",
+    )
+    server = CodexStorageServer(tmp_path / "storage-server", port=0)
+    server.start()
+
+    original_read_text = Path.read_text
+
+    def read_text_as_gbk_by_default(path, *args, **kwargs):
+        encoding = kwargs.get("encoding")
+        if args:
+            encoding = args[0]
+        if encoding is None:
+            return path.read_bytes().decode("gbk")
+        return original_read_text(path, *args, **kwargs)
+
+    class _FakeResponsesModelClient(_ScriptedResponsesClient):
+        def __init__(
+            self,
+            config,
+            timeout_seconds,
+            session_id=None,
+            originator=None,
+            user_agent=None,
+            openai_subagent=None,
+        ) -> 'None':
+            del timeout_seconds, user_agent, openai_subagent
+            super().__init__([ModelResponse(items=[AssistantMessage(text="OK")])])
+            self._config = config
+            self.model = config.model
+            self._session_id = session_id
+            self._originator = originator or "pycodex"
+
+    monkeypatch.setattr(Path, "read_text", read_text_as_gbk_by_default)
+    monkeypatch.setattr("pycodex.cli.ResponsesModelClient", _FakeResponsesModelClient)
+    monkeypatch.setattr("pycodex.cli.configure_loguru", lambda: None)
+    monkeypatch.setattr("sys.stdin.read", lambda: "")
+    monkeypatch.delenv("CODEX_HOME", raising=False)
+    monkeypatch.delenv("PORTABLE_API_KEY", raising=False)
+
+    try:
+        stored_call = upload_codex_home(f"{codex_home}@{server.server_address}")
+        args = build_parser().parse_args(["--call", stored_call, "say ok"])
+        exit_code = await run_cli(args)
+    finally:
+        server.stop()
+
+    assert exit_code == 0
+
+
 def test_get_tools_registers_expected_builtin_tools() -> 'None':
     registry = get_tools()
     assert registry.names() == (