python-codex 0.1.0__tar.gz → 0.1.1__tar.gz

{python_codex-0.1.0 → python_codex-0.1.1}/.gitignore

@@ -3,4 +3,5 @@
 __pycache__/
 *.pyc
 .tmp/
+.pycodex-storage/
 uv.lock

{python_codex-0.1.0 → python_codex-0.1.1}/AGENTS.md

@@ -10,6 +10,7 @@
 - 优先保持主循环可测试、可替换：模型侧通过 `ModelClient` 协议接入；测试专用的 `ScriptedModelClient` 放在 `tests/fakes.py`，不要放进运行时包。
 - `ResponsesModelClient` 直接复用 `~/.codex/config.toml` 的 provider 配置；当前已验证这里的 responses provider 需要 `stream = true`，否则会返回 `400` 和 `Stream must be set to true`。
 - `responses_server` compat 层应透传请求里的 `model`；不要再做 “取 downstream /models 第一个 id 并强制覆盖请求模型” 这种兜底兼容。
+- 对 `model_provider = "vllm"`，`responses_server` 仍然走 `/v1/chat/completions` compat 路径，但要保留 reasoning：把 chat chunk 里的 `reasoning` / `reasoning_content` 翻回 Responses `reasoning` item，并把历史里的 Responses `reasoning` item 回放成下游 assistant message 的 `reasoning` 字段。
 - `responses_server` 的 provider-specific chat payload 定制统一放在 `responses_server/payload_processors.py`：使用 `CompatServerConfig.model_provider` 选择 `provider_name -> proc_fn(outcomming_request)` 映射，并且只在真正发出 downstream `/v1/chat/completions` 前 post-process；`StreamRouter` 内部继续保留 canonical payload，避免 tool hydration / mock web_search follow-up 被 provider 改写污染。
 - `pycodex` 默认是最小交互 CLI；无 prompt 时进入 REPL，并通过 `AgentRuntime` 跑外层提交循环。当前会显示最小事件流、assistant 流式输出、简单 title/history（`/title`, `/history`），并默认注册一组与原版一一对应的本地工具子集。
 - 交互 CLI 的事件流展示优先表达用户可感知的阶段（例如工具开始/完成、模型回看工具结果），不要直接把内部 `iteration` 计数暴露成主要状态文案；`iterations` 应继续保留在 `TurnResult` 等程序化结果里。
@@ -36,3 +37,8 @@
 - 用 `tests/fake_responses_server.py` 做 steer 时序对比时，不要把 proxy capture 文件的生成时刻当成“请求已到达 upstream”的信号；`build_proxy_handler(...)` 会等整条 upstream response 读完后才 `write_capture(...)`。如果要在第一条 request 仍未完成时注入 steer，应该同步等待 fake origin 自己收到第 1 条 POST。
 - 在本机做 steer fake-server 对比时，不要把用户本地 `config.toml` 里的 `service_tier` / fast-mode 设置混进“默认 steer”结论。`tests/compare_steer_request_bodies.py` 现在会给 upstream 和 `pycodex` 都生成临时 config，并去掉顶层 `service_tier` 后再比较 request body。
 - `x-codex-turn-metadata.workspaces` 的时机不是“整个 session 只发第一条请求”。当前对齐结论是：首个 turn 的后续 steer/follow-up request 也继续带 `workspaces`；切到后续新 turn 才省略。
+- 远端 Codex home 存储模式当前仍刻意只挂在 `pycodex/cli.py` 启动前：`--put`/`--call` 只负责上传或落本地 `CODEX_HOME` 并重写 `args.config`，`model/context/runtime` 继续完全按 `config_path.parent` 读取 `.env`、`AGENTS.md`、`skills/`；后续扩展时优先保持这个隔离边界，不要把分支判断散到运行时各模块里。
+- 当前存储服务原型契约固定为 `POST /v1/storage/put` 上传客户端本地加密后的 bundle，并返回/使用 `SECRET-CALLID@<host:port>` 这种 call token；`GET /v1/storage/call/<call_id>` 只下载密文，bundle 的解密和解压都在 client 侧完成。做 smoke 或排障时，优先先验证这两个 endpoint 和 call token 形状。
+- `--put` 的 CLI UX 现在约定为“先打印打包文件清单和上传目标，再打印结果”，并且最后一行保留为可直接执行的 `pycodex --call SECRET-CALLID@<host:port>` 一键启动命令；后续如果再改输出，尽量保留这个末行语义。
+- `--put` 现在不是“只上传就结束”：上传成功后还会立刻跑一次真实的 `--call` 下载/解包 round-trip 测试，只有这个测试也成功才算整个 `--put` 成功。排障时如果上传成功但 CLI 仍退出非零，要继续看 call 路径而不只看 put 端。
+- `--put` 现在会先做 `/healthz` preflight，再开始扫描/压缩目录；如果用户报“卡死”，先看目标 `host:port` 是否真有 storage server 在监听。默认打包策略也已经切到白名单：只带 `config.toml`、`.env`、`AGENTS.md`、`AGENTS.override.md`、`skills/**`，以及 config 里相对引用的 `model_instructions_file`，所以像 `sessions/` 这类运行态目录不会再靠黑名单排除。

python_codex-0.1.1/PKG-INFO

@@ -0,0 +1,355 @@
+Metadata-Version: 2.4
+Name: python-codex
+Version: 0.1.1
+Summary: A minimal Python extraction of Codex's main agent loop
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: cryptography>=3.4
+Requires-Dist: fastapi>=0.115
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: prompt-toolkit>=3.0
+Requires-Dist: requests>=2.31
+Requires-Dist: tomli>=2.0; python_version < '3.11'
+Requires-Dist: uvicorn>=0.32
+Description-Content-Type: text/markdown
+
+# pycodex
+
+English README. Chinese version: `README_ZH.md`
+
+PyPI distribution name: `python-codex`  
+Import path and CLI command remain `pycodex`.
+
+This repository extracts the core Codex agent loop from upstream Codex
+(`https://github.com/openai/codex`) into a deliberately small Python version,
+while preserving the two most important layers:
+
+- `submission_loop`: sequentially consumes submitted operations.
+- `run_turn`: keeps executing `model sample -> tool call -> feed tool result
+  back into the model` inside a single turn until a final answer is reached.
+
+Relevant Rust reference points:
+
+- `codex-rs/core/src/codex.rs` -> `submission_loop`
+- `codex-rs/core/src/codex.rs` -> `run_turn`
+- `codex-rs/core/src/codex.rs` -> `run_sampling_request`
+- `codex-rs/core/src/tools/router.rs` -> `ToolRouter`
+- `codex-rs/core/src/stream_events_utils.rs` -> `handle_output_item_done`
+
+## Quick Start
+
+Install dependencies first:
+
+```bash
+uv sync
+```
+
+Try the real entry points:
+
+```bash
+uv run pycodex "Reply with exactly OK."
+uv run pycodex
+```
+
+## Design Tradeoffs
+
+This is not a 1:1 port of the Rust implementation. The current goal is a
+minimal reusable kernel that converges on the upstream behavior over time:
+
+1. Use a thin `ModelClient` protocol to abstract the model side.
+2. Use `ToolRegistry` to manage tool specs and executors.
+3. Use `AgentLoop` to implement the core closed loop.
+4. Use `AgentRuntime` to preserve the outer submission queue so it can keep
+   converging toward Rust's `submission_loop` later.
+
+Intentionally not included yet:
+
+- TUI / streaming incremental rendering
+- MCP / connectors / sandbox / approvals
+- memory / compact / hooks / review mode
+- a full production OpenAI adapter surface
+
+All of those can be layered on later. For now, the project is focused on
+nailing the core tool-augmented reasoning loop first.
+
+## Layout
+
+- `pycodex/protocol.py`: minimal conversation item / prompt / event protocol
+- `pycodex/model.py`: model client protocol and Responses API adapter
+- `pycodex/cli.py`: single-turn and interactive `pycodex` CLI entry points
+- `pycodex/tools/base_tool.py`: `BaseTool`, `ToolRegistry`, `ToolContext`
+- `pycodex/tools/`: concrete tool implementations
+- `pycodex/agent.py`: inner turn loop
+- `pycodex/runtime.py`: outer submission queue
+- `tests/test_agent.py`: core behavior tests
+
+## Current Alignment Status
+
+Current progress is easiest to read in layers:
+
+- prompt/context alignment:
+  - on the non-interactive `exec` path, `instructions` and `input` already
+    match upstream Codex;
+  - this layer is now mainly handled by `pycodex/context.py` plus vendored
+    prompt data.
+- turn-loop semantic alignment:
+  - `AgentLoop` no longer uses a fixed 12-iteration cap by default;
+  - like upstream, it now converges naturally based on whether there is still
+    follow-up work or tool handoff to do;
+  - the local iteration-limit parameter is gone.
+- request-level alignment:
+  - the non-interactive `exec` request body is mostly aligned;
+  - the default CLI non-exec first request now also follows the upstream
+    `codex-tui` + `<collaboration_mode>` path;
+  - the default CLI two-turn main-thread request/header behavior has also been
+    captured and aligned, including omitting `workspaces` on later turns;
+  - the remaining work is now more about outer behavior branches than this
+    already-compared request/header path.
+- tool round-trip alignment:
+  - the Default-mode unavailable path for `request_user_input` is aligned to
+    real upstream captures;
+  - the Plan-mode happy path is also aligned at the tool/protocol layer based
+    on upstream source: it forces `isOther=true`, requires non-empty `options`,
+    and returns structured answers as a JSON string plus `success=true`;
+  - there is now a deterministic round-trip comparison helper,
+    `tests/compare_request_user_input_roundtrip.py`, built on the proxy mode in
+    `tests/fake_responses_server.py`; against the locally installed
+    `codex-cli 0.115.0`, the only remaining Plan-mode live-capture schema
+    difference is that `pycodex` includes `success=true` in
+    `function_call_output`.
+
+See `docs/ALIGNMENT.md` for more detailed notes.
+
+## Live Model Integration
+
+If this machine already has a Codex CLI configuration, `pycodex` can reuse the
+`model`, `model_provider`, `base_url`, and `env_key` from
+`~/.codex/config.toml` directly:
+
+```python
+from pycodex import ResponsesModelClient
+
+client = ResponsesModelClient.from_codex_config()
+```
+
+The current implementation uses the streaming OpenAI-compatible `/responses`
+endpoint. This path has already been validated against the local
+`~/.codex/config.toml` setup.
+
+When launched through the CLI, `pycodex` also loads `.env` from the same
+configuration directory before reading config (typically `~/.codex/.env`), so
+provider keys and similar environment variables can live there. To match
+upstream Codex, variables starting with `CODEX_` are not imported from `.env`.
+
+## pycodex CLI
+
+`pycodex` now defaults to a minimal interactive entry point. Internally it uses
+`AgentRuntime` to drive the turn submission loop and reuses
+`~/.codex/config.toml` by default:
+
+```bash
+pycodex
+pycodex "Summarize this repo in one sentence."
+printf 'Reply with exactly OK.' | pycodex
+pycodex --json "Reply with exactly OK."
+pycodex --profile model_proxy "Reply with exactly OK."
+pycodex --vllm-endpoint http://127.0.0.1:18000 "Reply with exactly OK."
+pycodex --put @127.0.0.1:5577
+pycodex --put /data/.codex/@127.0.0.1:5577
+pycodex --call SECRET-CALLID@127.0.0.1:5577 "Reply with exactly OK."
+pycodex doctor
+```
+
+Current behavior:
+
+- with no argv prompt and a TTY stdin, enter interactive mode
+- with an argv prompt or piped stdin, run a single turn
+- interactive mode supports `/exit` and `/quit`
+- interactive mode shows a compact event stream for user-visible phases such as
+  tool execution and model follow-up after tool results
+- assistant text is printed from streaming deltas directly
+- interactive mode supports `/history`, `/title`, and `/model`
+- `/model <name>` switches the model used by later turns in the current
+  interactive session; `/model` shows the current model and available choices
+- steer is enabled by default in interactive mode: normal input goes into the
+  runtime steer path, the current request stops at the next safe boundary, and
+  later steer text is appended to the next model request's `input` in order;
+  for explicit queueing, use `/queue <message>`, which prints
+  `[steer] queued: ...` and later `[steer] inserted: ...`
+- the default built-in tool subset currently exposed as local tools is:
+  `shell`, `shell_command`, `exec_command`, `write_stdin`, `exec`, `wait`,
+  `web_search`, `update_plan`, `request_user_input`, `request_permissions`,
+  `spawn_agent`, `send_input`, `resume_agent`, `wait_agent`, `close_agent`,
+  `apply_patch`, `grep_files`, `read_file`, `list_dir`, `view_image`
+- `--vllm-endpoint http://host:port` automatically launches a local
+  `responses_server` compatibility layer; when the URL path is empty it is
+  normalized to `/v1`, and `/responses` requests are still forwarded to the
+  downstream `/v1/chat/completions` endpoint. For `model_provider = "vllm"`,
+  reasoning is now preserved across this path: chat chunks with `reasoning` or
+  `reasoning_content` are translated back into Responses `reasoning` items, and
+  historical `reasoning` items are replayed into downstream assistant messages
+  via the `reasoning` field. Streaming token usage is also requested from vLLM
+  and forwarded to the final `response.completed.response.usage`
+- `pycodex doctor` checks config, `.env`, API keys, DNS, TCP/TLS, and an
+  optional live Responses API request
+
+Current primary uses:
+
+- verify provider / model / auth configuration
+- debug `ResponsesModelClient`
+- run minimal single-turn and multi-turn smoke tests
+
+`doctor` examples:
+
+```bash
+pycodex doctor
+pycodex doctor --skip-live
+pycodex doctor --json
+```
+
+## Portable Mode
+
+`Portable Mode` is the quickest way to bring your usual `pycodex` setup into a
+fresh machine, container, or debug image.
+
+Use it like this:
+
+```bash
+pycodex --put @127.0.0.1:5577
+pycodex --put /data/.codex/@127.0.0.1:5577
+```
+
+- `--put` prints a reusable `SECRET-CALLID@host:port` plus a final one-line
+  `pycodex --call ...` command
+- on the new environment or image, run that printed `--call` command directly
+- quickly restoring your usual `config.toml`, `.env`, `AGENTS.md`, and
+  `skills/` into a clean debug environment
+- keeping a new image focused on the bug you are debugging instead of spending
+  time rebuilding local Codex setup by hand
+- bootstrapping `pycodex` even when the target environment does not already
+  have a populated `~/.codex`
+- bare `--put` uses the current user's `~/.codex`
+- `--put /path/.codex/@host:port` lets you publish a different Codex home
+
+## Example
+
+```python
+import asyncio
+
+from pycodex import (
+    AgentLoop,
+    BaseTool,
+    ContextManager,
+    ResponsesModelClient,
+    ToolRegistry,
+)
+
+
+class EchoTool(BaseTool):
+    name = "echo"
+    description = "Echo the provided text."
+    input_schema = {
+        "type": "object",
+        "properties": {"text": {"type": "string"}},
+        "required": ["text"],
+    }
+
+    async def run(self, context, args):
+        del context
+        return args["text"]
+
+
+async def main() -> None:
+    model = ResponsesModelClient.from_codex_config()
+    context_manager = ContextManager.from_codex_config()
+
+    tools = ToolRegistry()
+    tools.register(EchoTool())
+
+    agent = AgentLoop(model, tools, context_manager)
+    result = await agent.run_turn(
+        ["Call the echo tool with text=hello, then tell me what it returned."]
+    )
+    print(result.output_text)
+
+
+asyncio.run(main())
+```
+
+## Alignment Checklist
+
+See `docs/ALIGNMENT.md` for more detail. This section keeps a high-level
+checklist for quick status scanning.
+
+### Tool Alignment
+
+Official upstream tools:
+
+- [x] `shell` - run shell commands in argv form.
+- [x] `shell_command` - run shell scripts in string form.
+- [x] `exec_command` - start long-running commands with a session.
+- [x] `write_stdin` - write stdin to an existing execution session or poll
+  output.
+- [x] `web_search` - expose provider-native web search capability.
+- [x] `update_plan` - update the task plan and maintain step status.
+- [x] `request_user_input` - ask the user structured questions and wait for an
+  answer.
+- [x] `request_permissions` - request extra permissions before continuing.
+- [x] `spawn_agent` - create and start a sub-agent.
+- [x] `send_input` - continue feeding input to an existing sub-agent.
+- [x] `resume_agent` - reopen a closed sub-agent.
+- [x] `wait_agent` - wait for a sub-agent to reach a terminal state.
+- [x] `close_agent` - close a sub-agent that is no longer needed.
+- [x] `apply_patch` - edit files precisely with a freeform patch.
+- [x] `grep_files` - search file contents by pattern.
+- [x] `read_file` - read file slices while preserving line-number semantics.
+- [x] `list_dir` - list directory tree slices.
+- [x] `view_image` - turn a local image into model-visible input.
+
+Upstream low-frequency / special-mode tools not yet modeled separately:
+
+- [ ] `wait_infinite` - long blocking wait for external events or later input.
+- [ ] `spawn_agents_on_csv` - create sub-agent jobs in bulk from CSV.
+- [ ] `report_agent_job_result` - report batch agent job results.
+- [ ] `js_repl` - JavaScript REPL / code-mode primary entry point.
+- [ ] `js_repl_reset` - reset `js_repl` state.
+- [ ] `artifacts` - generate or manage structured artifact outputs.
+- [ ] `list_mcp_resources` - list MCP resources.
+- [ ] `list_mcp_resource_templates` - list MCP resource templates.
+- [ ] `read_mcp_resource` - read MCP resource contents.
+- [ ] `multi_tool_use.parallel` - parallel wrapper around multiple developer
+  tool calls.
+
+Repository-specific compatibility / transition tools:
+
+- [x] `exec` - current local approximation of code mode.
+- [x] `wait` - current local approximation of code-mode waiting behavior.
+
+### Behavior Alignment
+
+- [x] `AgentLoop` / `AgentRuntime` main loop skeleton - turn loop and submission
+  queue are in place.
+- [x] non-interactive `exec` `instructions` alignment - base instructions match
+  upstream.
+- [x] non-interactive `exec` `input` alignment - prompt input matches upstream.
+- [x] developer/contextual-user message shape alignment - message/content shape
+  matches upstream.
+- [x] `AGENTS.md` + `<environment_context>` injection alignment - context
+  assembly order matches upstream.
+- [x] non-interactive `exec` tool subset alignment - the model-visible tool set
+  has converged.
+- [x] `include = ["reasoning.encrypted_content"]` - reasoning include field is
+  aligned.
+- [x] `prompt_cache_key` - request-level prompt cache key is implemented.
+- [x] `x-client-request-id` - request id header is implemented.
+- [x] `x-codex-turn-metadata` - turn id / sandbox header is implemented.
+- [x] `originator` - mode-aware originator header is implemented.
+- [x] exact `user-agent` string alignment - aligned on the non-interactive
+  `exec` path.
+- [x] field-by-field exec-mode tool schema alignment - currently reuses the
+  upstream snapshot directly through the tool layer.
+- [ ] full interactive-mode and non-`exec` behavior alignment - the non-exec
+  first-turn context is now on the `codex-tui` path, but continuous REPL
+  multi-turn behavior is not fully verified yet.
+- [ ] sandbox / approvals / compact / memory and other outer behavior alignment
+  - these systems are still in later scope.