pine-of-glass 0.2.4

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Thomas Mustier
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,20 @@
+# pine-of-glass
+
+Small observability and context management tools for the [Pi coding agent](https://github.com/earendil-works/pi-coding-agent).
+1. [`contextimate`](./extensions/pi-contextimate) breaks down what is filling your context window: sysprompt, AGENTS.md, Skill frontmatter, Tool schemas, and session material. Toggle with ctrl+o on start and /reload.
+2. [`traceline`](./extensions/pi-traceline) collapses tool calls to one trace line each, so you can see the full arc of what Pi did (path taken, context read, bloated tool results). Toggle with ctrl+t.
+3. `cachemire` coming soon.
+
+See each extension's own `README.md` for details, and `docs/` for deeper reference.
+
+## Installation
+- From GitHub: `pi install git:github.com/tmustier/pine-of-glass`
+- From npm (installs both `contextimate` and `traceline`): `pi install npm:pine-of-glass`
+
+## Screenshots
+### [Contextimate](./extensions/pi-contextimate)
+<img width="1477" height="589" alt="image" src="https://github.com/user-attachments/assets/8ff81aa2-f61b-4d8d-9507-f455f37c12cc" />
+
+### [Traceline](./extensions/pi-traceline)
+<img width="709" height="409" alt="image" src="https://github.com/user-attachments/assets/4a59fbae-8270-46d3-a4fc-fdf2e5c3ba8c" />
+

package/docs/pi-contextimate-codex-context-accounting.md

@@ -0,0 +1,245 @@
+# Codex context-accounting comparison for pi-contextimate
+
+This note captures how upstream OpenAI Codex estimates active context usage, especially around interruption and compaction, so `pi-contextimate` can compare its own policy against Codex without mixing the two models.
+
+Source snapshot used for this note: `openai/codex` commit [`0c5ccd18abda96efaed9e94e26ffe22def5e28ed`](https://github.com/openai/codex/tree/0c5ccd18abda96efaed9e94e26ffe22def5e28ed).
+
+## Executive summary
+
+Codex keeps two token-usage concepts:
+
+- `total_token_usage`: accumulated session spend.
+- `last_token_usage`: Codex's active-context number, used by the CLI/TUI context-window indicator.
+
+For normal completed model turns, `last_token_usage.total_tokens` comes from provider usage on `response.completed`. When local history has grown after that last successful response, Codex adds an estimate for every item after the most recent model-generated item.
+
+For compaction, Codex does **not** wait for a later assistant response to get a fresh provider usage number. After installing compacted history it recomputes a local estimate and writes that estimate into `last_token_usage.total_tokens`.
+
+Pi currently differs: after compaction, if there is no post-compaction assistant usage yet, Pi can report context usage as `null`/unknown and let the inspector fall back to heuristics.
+
+## Latest active-context estimate
+
+Codex's active-context estimator is:
+
+```text
+active_context_tokens = latest_provider_last_token_usage.total_tokens
+                      + estimated_tokens_of_items_after_last_model_generated_item
+```
+
+There is one extra branch: if the server has not indicated that past reasoning is included, Codex also adds estimated tokens for non-last encrypted reasoning items.
+
+Important boundary: `items_after_last_model_generated_item()` starts after the newest model-generated item. Model-generated items include assistant messages, reasoning, function/tool calls, shell calls, web/search/image calls, compaction items, and context-compaction items. Tool outputs and new user messages after that boundary are estimated locally.
+
+Source:
+
+- [`ContextManager::items_after_last_model_generated_item`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L303-L310)
+- [`ContextManager::get_total_token_usage`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L314-L331)
+- [`ContextManager::get_total_token_usage_breakdown`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L334-L358)
+
+## Text and item token heuristic
+
+Plain text uses a four-bytes-per-token heuristic:
+
+```text
+approx_token_count(text) = ceil(text.length / 4)
+approx_tokens_from_byte_count(bytes) = ceil(bytes / 4)
+```
+
+For full history items:
+
+```text
+estimate_item_token_count(item) = ceil(model_visible_bytes(item) / 4)
+```
+
+Source:
+
+- [`approx_token_count`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/utils/string/src/truncate.rs#L71-L83)
+- [`estimate_item_token_count`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L516-L518)
+
+## What Codex means by “model-visible bytes”
+
+“Model-visible bytes” is Codex's local proxy for what counts toward context, not simply local object size and not necessarily human-visible text.
+
+For ordinary `ResponseItem`s, Codex starts with `serde_json::to_string(item).len()` and then adjusts special payloads:
+
+1. Inline image data URLs: subtract raw base64 payload bytes and add an image-cost estimate.
+2. Encrypted function-output content items: subtract the encrypted string length and add a smaller decoded-size estimate.
+3. Encrypted reasoning / compaction blocks: do not use serialized JSON length; estimate from encrypted payload length.
+
+Source: [`estimate_response_item_model_visible_bytes`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L543-L570).
+
+### Encrypted reasoning / compaction blocks
+
+For these items:
+
+- `ResponseItem::Reasoning { encrypted_content: Some(...) }`
+- `ResponseItem::Compaction { encrypted_content }`
+- `ResponseItem::ContextCompaction { encrypted_content: Some(...) }`
+
+Codex estimates:
+
+```text
+model_visible_bytes = max(0, floor(encrypted_content.length * 3 / 4) - 650)
+tokens              = ceil(model_visible_bytes / 4)
+```
+
+So yes: this is effectively an estimate for opaque encrypted thinking / compaction payloads that are carried client-side and replayed into later requests. It is not counting visible reasoning-summary text; it is estimating the model-visible cost of the encrypted carry-forward blob.
+
+Source:
+
+- [`estimate_reasoning_length`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L504-L509)
+- reasoning/compaction branch in [`estimate_response_item_model_visible_bytes`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L543-L555)
+
+### Encrypted function outputs
+
+Encrypted function-output content uses a different adjustment:
+
+```text
+replacement_bytes = ceil(encrypted_content.length * 9 / 16)
+```
+
+Source:
+
+- [`estimate_encrypted_function_output_length`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L512-L513)
+- [`encrypted_function_output_estimate_adjustment`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L692-L714)
+
+### Images
+
+For non-`original` image detail, Codex replaces the base64 image payload with a fixed byte estimate:
+
+```text
+RESIZED_IMAGE_BYTES_ESTIMATE = 7,373 bytes
+≈ 1,844 tokens with ceil(bytes / 4)
+```
+
+For `detail: "original"`, Codex attempts to decode dimensions and estimates a 32px patch count capped at 10,000 patches, then converts patches to bytes with the same four-bytes-per-token helper.
+
+Source:
+
+- image constants and comment: [`history.rs#L521-L533`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L521-L533)
+- original-detail patch estimate: [`history.rs#L609-L642`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L609-L642)
+
+## Post-compaction estimate
+
+After compaction, Codex replaces the in-memory history and then calls `recompute_token_usage()`.
+
+That recompute does:
+
+```text
+post_compaction_context_tokens = ceil(base_instructions.length / 4)
+                               + sum(estimate_item_token_count(item) for item in compacted_history)
+```
+
+Then it stores:
+
+```text
+last_token_usage = {
+  input_tokens: 0,
+  cached_input_tokens: 0,
+  output_tokens: 0,
+  reasoning_output_tokens: 0,
+  total_tokens: post_compaction_context_tokens
+}
+```
+
+Source:
+
+- [`ContextManager::estimate_token_count_with_base_instructions`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/context_manager/history.rs#L149-L162)
+- [`Session::recompute_token_usage`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/session/mod.rs#L2970-L3005)
+- local compaction calls recompute after replacement: [`compact.rs#L292-L294`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact.rs#L292-L294)
+- remote compaction calls recompute after replacement: [`compact_remote.rs#L261-L263`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact_remote.rs#L261-L263)
+- remote compaction v2 calls recompute after replacement: [`compact_remote_v2.rs#L275-L277`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact_remote_v2.rs#L275-L277)
+
+## What is in compacted history
+
+### Local summarization compaction
+
+Local compaction builds replacement history from:
+
+1. retained recent user messages, capped by `COMPACT_USER_MESSAGE_MAX_TOKENS = 20_000` using `chars / 4`;
+2. a user-role summary message;
+3. optionally re-injected initial context before the last real user message.
+
+Source:
+
+- token cap: [`compact.rs#L47-L49`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact.rs#L47-L49)
+- replacement history builder: [`compact.rs#L474-L538`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact.rs#L474-L538)
+
+### Remote compaction v2
+
+Remote compaction v2 constructs a compact request by appending `ResponseItem::CompactionTrigger` to the prompt input. The compacted history it installs is:
+
+1. retained `user` / `developer` / `system` messages from the prompt input;
+2. filtered through `should_keep_compacted_history_item`;
+3. truncated newest-first to `RETAINED_MESSAGE_TOKEN_BUDGET = 64_000` approximate text tokens;
+4. appended with the returned `ResponseItem::Compaction { encrypted_content }`.
+
+The returned encrypted compaction item is exactly the kind of opaque payload covered by the encrypted-block estimate above.
+
+Source:
+
+- retained cap: [`compact_remote_v2.rs#L48-L50`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact_remote_v2.rs#L48-L50)
+- request appends `CompactionTrigger`: [`compact_remote_v2.rs#L209-L217`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact_remote_v2.rs#L209-L217)
+- compacted-history shape: [`compact_remote_v2.rs#L409-L421`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact_remote_v2.rs#L409-L421)
+- newest-first retained-message truncation: [`compact_remote_v2.rs#L433-L456`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact_remote_v2.rs#L433-L456)
+
+## Remote-compaction failure diagnostics
+
+When remote compaction fails, Codex logs both:
+
+- `last_api_response_total_tokens`
+- `all_history_items_model_visible_bytes`
+- `estimated_tokens_of_items_added_since_last_successful_api_response`
+- `estimated_bytes_of_items_added_since_last_successful_api_response`
+- `failing_compaction_request_model_visible_bytes`
+
+The failing request byte estimate is:
+
+```text
+instructions.length + sum(model_visible_bytes(input_item))
+```
+
+Source: [`compact_remote.rs#L335-L373`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact_remote.rs#L335-L373).
+
+Before remote compaction, Codex also tries to trim trailing Codex-generated history while the full estimate exceeds the context window. It only removes trailing Codex-generated items, not arbitrary user history.
+
+Source: [`trim_function_call_history_to_fit_context_window`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/compact_remote.rs#L376-L402).
+
+## Interruption behavior
+
+If a turn is interrupted before `response.completed`, Codex does not have fresh provider token usage for the partial turn. It records a model-visible interrupted-turn marker and emits `TurnAborted`, but it does not synthesize a new provider-like `TokenCount` from the partial response.
+
+Any later active-context calculation therefore falls back to the previous `last_token_usage.total_tokens` plus the local post-model tail estimate described above.
+
+Source:
+
+- interrupted-turn marker: [`tasks/mod.rs#L87-L106`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/tasks/mod.rs#L87-L106)
+- abort records marker and emits `TurnAborted`: [`tasks/mod.rs#L811-L864`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/tasks/mod.rs#L811-L864)
+- provider usage is recorded on `ResponseEvent::Completed`: [`turn.rs#L2000-L2014`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/core/src/session/turn.rs#L2000-L2014)
+
+## TUI/status implication
+
+Codex's UI uses `last_token_usage`, not accumulated `total_token_usage`, for context-window percentage.
+
+Source:
+
+- TUI token model: [`tui/src/token_usage.rs#L37-L50`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/tui/src/token_usage.rs#L37-L50)
+- status card uses `last_token_usage`: [`status/card.rs#L327-L337`](https://github.com/openai/codex/blob/0c5ccd18abda96efaed9e94e26ffe22def5e28ed/codex-rs/tui/src/status/card.rs#L327-L337)
+
+## Comparison policy for pi-contextimate
+
+Recommended interpretation for `pi-contextimate`:
+
+1. Keep Pi's top-line current-context usage as source of truth when `ctx.getContextUsage()` returns tokens.
+2. Keep Pi's post-compaction `null`/unknown behavior explicit; do not silently pretend a provider usage exists.
+3. Use Codex's method only as a comparison or optional Codex-native fallback after compaction:
+   - base instructions `chars / 4`;
+   - provider-shaped Responses items via serialized JSON bytes;
+   - special estimates for encrypted reasoning / compaction blocks if Pi is actually replaying Codex-native encrypted items.
+4. Do **not** estimate hidden thinking from arbitrary local encrypted/signature string length unless that string is actually a provider-native item being replayed into the model. Otherwise keep Pi's current residual bucket:
+
+```text
+Other / reasoning = Pi current-context total - locally countable visible pieces
+```
+
+That residual is safer because it absorbs opaque reasoning replay, provider overhead, images/weird blocks, and estimation error without claiming the encrypted chars are themselves the true context payload.