codex-agent-framework 0.1.9__tar.gz → 0.1.11__tar.gz

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/CHANGELOG.md

@@ -3,6 +3,31 @@
 All notable changes to this project will be documented in this file.
 
 This project loosely follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and uses semantic versioning where practical.
+## [0.1.11] - 2026-05-05
+### Added
+- Add a separate `view` tool for folders, URLs, rich document extraction, and Python object inspection.
+- Add context status counts for persistent and temporary session messages, surfaced in the TUI status bar.
+
+### Changed
+- Make `read` a strict local UTF-8 file reader with reliable forced line numbers and regex pattern search by default.
+- Keep extracted `view` output unnumbered, including search snippets, because transformed document line structure is not source-of-truth reliable.
+- Update the system prompt to direct agents to use `read` before targeted edits and `view` for broad extraction.
+
+### Fixed
+- Preserve blank lines and real source line numbers in `read` pattern snippets.
+- Apply line-based pagination before token truncation.
+
+### Tests
+- Add regression coverage for strict `read`, unnumbered `view`, blank-line preservation, regex/literal search, and persistent/temporary message status counts.
+
+## [0.1.10] - 2026-05-05
+### Fixed
+- Persist only backend `compaction_summary` items in compaction messages, avoiding bulky compacted conversation payloads while preserving summary replay.
+- Sanitize encrypted compaction summaries in local token estimates and refresh cached context status after compaction/message updates.
+
+### Tests
+- Add regression coverage for compaction payload filtering, legacy `output_items` migration, context status cache invalidation, and runtime status updates after compaction.
+
 ## [0.1.9] - 2026-05-05
 ### Changed
 - Maintenance release for the packaged distribution after validating the updated local execution environment and deploy workflow.

{codex_agent_framework-0.1.9/codex_agent_framework.egg-info → codex_agent_framework-0.1.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: codex-agent-framework
-Version: 0.1.9
+Version: 0.1.11
 Summary: A lightweight event-driven Codex agent runtime.
 Author: Baptiste
 License-Expression: MIT
@@ -400,6 +400,8 @@ python -m build
 The distribution includes prompt text files and `codex_agent/get_text/default_gitignore` through package data and `MANIFEST.in`.
 
 ## Recent changes
+- `0.1.11`: split strict line-numbered UTF-8 `read` from unnumbered extracted `view`, preserve blank lines in read snippets, and show persistent+temporary message counts in the TUI status bar.
+- `0.1.10`: persist only backend compaction summaries, drop bulky compacted conversation payloads, and refresh context status after compaction.
 - `0.1.9`: maintenance packaging release after validating the local execution environment and deploy workflow.
 - `0.1.8`: scope TUI replay/SSE catch-up to the active session and make bash/python subprocesses inherit the project Python environment, including service-launched agents.
 - `0.1.7`: add durable RAG memory, scheduled wakeups, process-isolated server runtime, tray/service controls, robust SSE replay/reconnect, richer TUI status, and improved token estimates.

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/README.md

@@ -352,6 +352,8 @@ python -m build
 The distribution includes prompt text files and `codex_agent/get_text/default_gitignore` through package data and `MANIFEST.in`.
 
 ## Recent changes
+- `0.1.11`: split strict line-numbered UTF-8 `read` from unnumbered extracted `view`, preserve blank lines in read snippets, and show persistent+temporary message counts in the TUI status bar.
+- `0.1.10`: persist only backend compaction summaries, drop bulky compacted conversation payloads, and refresh context status after compaction.
 - `0.1.9`: maintenance packaging release after validating the local execution environment and deploy workflow.
 - `0.1.8`: scope TUI replay/SSE catch-up to the active session and make bash/python subprocesses inherit the project Python environment, including service-launched agents.
 - `0.1.7`: add durable RAG memory, scheduled wakeups, process-isolated server runtime, tray/service controls, robust SSE replay/reconnect, richer TUI status, and improved token estimates.

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/codex_agent/agent.py

@@ -192,7 +192,7 @@ class AgentSession(modict):
         result = client.compact(history, model=model, instructions=instructions)
         message = CompactionMessage(
             response_id=result.response_id,
-            output_items=result.output_items,
+            summaries=CompactionMessage.summary_items(result.output_items),
             compacted_message_count=len(compacted_messages),
         )
 
@@ -825,10 +825,11 @@ class Agent:
             compact_kwargs.client = client
         message = self.session.compact(**compact_kwargs)
         self._context_status_cache = None
-        self.refresh_estimated_status_cache()
+        status = self.refresh_estimated_status_cache()
         self.emit(
             CompactionCompletedEvent,
             message=message,
+            status=status,
             compacted_message_count=plan.compacted_message_count,
             compacted_token_count=plan.compacted_token_count,
             post_compaction_message_count=len(self.session.context_messages()),
@@ -872,6 +873,7 @@ class Agent:
             if not msg.session_id:
                 msg.session_id=self.current_session_id
             self.session.messages.append(msg)
+            self._context_status_cache = None
             self.save_session()
             self.emit(MessageAddedEvent, message=msg)
         return msg
@@ -1152,6 +1154,14 @@ class Agent:
         visible_history_count = len(truncated_others[max(0, history_start):])
         sent_session_messages = visible_history_count + len(images)
         total_session_messages = len(self.session.messages) if self.session else 0
+        persistent_session_messages = sum(
+            1 for msg in session_messages
+            if msg.get("turns_left", -1) == -1
+        )
+        temporary_session_messages = sum(
+            1 for msg in session_messages
+            if isinstance(msg.get("turns_left", -1), int) and msg.get("turns_left", -1) > 0
+        )
         percent = (current_count / input_limit * 100) if input_limit else 0
         status = modict(
             used_tokens=current_count,
@@ -1165,6 +1175,8 @@ class Agent:
             sent_session_messages=sent_session_messages,
             total_session_messages=total_session_messages,
             sent_session_message_percent=(sent_session_messages / total_session_messages * 100) if total_session_messages else 0,
+            persistent_session_messages=persistent_session_messages,
+            temporary_session_messages=temporary_session_messages,
             context_session_messages=len(session_messages),
             image_messages=len(images),
             pending_messages=len(self.pending),
@@ -1197,6 +1209,8 @@ class Agent:
             sent_session_messages=estimated.sent_session_messages,
             total_session_messages=estimated.total_session_messages,
             sent_session_message_percent=estimated.sent_session_message_percent,
+            persistent_session_messages=estimated.get("persistent_session_messages", 0),
+            temporary_session_messages=estimated.get("temporary_session_messages", 0),
             quota_5h_percent=None,
             quota_7d_percent=None,
             model=self.config.get("model") or "gpt-5.4",
@@ -1221,6 +1235,8 @@ class Agent:
                 sent_session_messages=estimated.sent_session_messages,
                 total_session_messages=estimated.total_session_messages,
                 sent_session_message_percent=estimated.sent_session_message_percent,
+                persistent_session_messages=estimated.get("persistent_session_messages", 0),
+                temporary_session_messages=estimated.get("temporary_session_messages", 0),
                 model=self.config.get("model") or "gpt-5.4",
                 busy=self.busy,
                 current_turn_id=self.current_turn_id,
@@ -1242,6 +1258,8 @@ class Agent:
             sent_session_messages=estimated.sent_session_messages,
             total_session_messages=estimated.total_session_messages,
             sent_session_message_percent=estimated.sent_session_message_percent,
+            persistent_session_messages=estimated.get("persistent_session_messages", 0),
+            temporary_session_messages=estimated.get("temporary_session_messages", 0),
             model=self.config.get("model") or "gpt-5.4",
             busy=self.busy,
             current_turn_id=self.current_turn_id,
@@ -1265,6 +1283,8 @@ class Agent:
             sent_session_messages=context_status.get("sent_session_messages", 0),
             total_session_messages=context_status.get("total_session_messages", 0),
             sent_session_message_percent=context_status.get("sent_session_message_percent", 0),
+            persistent_session_messages=context_status.get("persistent_session_messages", 0),
+            temporary_session_messages=context_status.get("temporary_session_messages", 0),
             quota_5h_percent=wham.quota_5h_percent,
             quota_7d_percent=wham.quota_7d_percent,
             model=model or self.config.get("model") or "gpt-5.4",
@@ -1381,6 +1401,14 @@ class Agent:
         formatted_context = list(msg.format(context=self.shell.namespace) for msg in context)
         total_session_messages = len(self.session.messages) if self.session else 0
         sent_session_messages = len(history) + len(images)
+        persistent_session_messages = sum(
+            1 for msg in session_messages
+            if msg.get("turns_left", -1) == -1
+        )
+        temporary_session_messages = sum(
+            1 for msg in session_messages
+            if isinstance(msg.get("turns_left", -1), int) and msg.get("turns_left", -1) > 0
+        )
         token_count_source = "local_estimate"
         try:
             current_count = self.ai.count_input_tokens(
@@ -1396,6 +1424,8 @@ class Agent:
             sent_session_messages=sent_session_messages,
             total_session_messages=total_session_messages,
             sent_session_message_percent=(sent_session_messages / total_session_messages * 100) if total_session_messages else 0,
+            persistent_session_messages=persistent_session_messages,
+            temporary_session_messages=temporary_session_messages,
             token_count_source=token_count_source,
         )
         return formatted_context
@@ -1496,6 +1526,10 @@ class Agent:
                     item = item.copy()
                     item["image_url"] = "<image>"
                     return item
+                if item.get("type") == "compaction_summary" and item.get("encrypted_content"):
+                    item = item.copy()
+                    item["encrypted_content"] = "<encrypted_compaction_summary>"
+                    return {key: sanitize(child) for key, child in item.items()}
                 return {key: sanitize(child) for key, child in item.items()}
             return value
 

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/codex_agent/agent_runtime.py

@@ -247,6 +247,12 @@ class ProcessAgentRuntime:
         elif event_type == "session_loaded_event":
             self.cache.current_session_id = event.get("session_id")
             self.cache.session = event.get("session") or self.cache.session
+        elif event_type == "compaction_completed_event" and event.get("status"):
+            status = modict(event.status)
+            self.cache.status = status
+            self.cache.busy = bool(status.get("busy", self.cache.busy))
+            self.cache.current_turn_id = status.get("current_turn_id", self.cache.current_turn_id)
+            self.cache.current_session_id = status.get("session_id", self.cache.current_session_id)
         for handler in list(self.event_handlers):
             handler(event)
 

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/codex_agent/builtin_tools.py

@@ -7,43 +7,78 @@ from collections.abc import Mapping
 from .image import persist_image_url
 from .memory import format_memory_results_xml
 from .tool import ImageGenerationTool, WebSearchTool, get_agent, tool
-from .utils import read_document_content
+from .utils import read_document_content, view_document_content
 
 web_search = WebSearchTool()
 image_generation = ImageGenerationTool()
 
 
 @tool
-def read(source, start_at_line: int = 1):
+def read(source: str, start_at_line: int = 1, pattern: str = None, regex: bool = True):
     """
     description: |
-        Read and extract text content from various sources including folders, files, urls or python variables / objects.
-        This tool reads exactly one source per call. To read multiple sources at once, issue multiple `read` tool calls in parallel.
-        For folders: returns a tree view of the folder structure.
-        For files: returns the text content. PDF, DOCX, XLSX, PPTX, ODT, HTML, TXT, and more are supported.
-        For urls: returns the text content of the web page (or of the remote file if the url points to a file).
-        For python variables / objects: introspects the object or gives a str repr.
-
-        For large outputs: The output will be automatically truncated to fit token limits. When truncated, you'll see a message at the end of content indicating the truncated line range to know from where to resume reading.
+        Strictly read a local UTF-8 text file as source-of-truth content.
+        This tool always prefixes returned content with reliable source line numbers.
+        It does not extract document formats, fetch URLs, inspect Python objects, or render directory trees.
+        Use `view` for folders, URLs, PDFs, DOCX, ODT, HTML, XLSX, and other extracted or transformed content.
+
+        For large outputs: the output is automatically truncated. When truncated, a message indicates the omitted line range.
+        When `pattern` is provided, it is treated as a Python regex by default and all matches are returned as separated numbered snippets with up to 10 lines before and after each match.
     parameters:
         source:
             description: |
-                Document source - can be either:
-                - Absolute path to a local folder or file (e.g., "/home/user/document.pdf")
-                - URL (e.g., "https://example.com/report.pdf")
-                - any other data structure, or python variable living in the shell's namespace
-                Pass only one source. Use parallel `read` calls for multiple sources.
+                Local UTF-8 text file path only. For directories, URLs, rich documents, or Python objects, use `view` instead.
+            type: string
         start_at_line:
             description: |
                 Line number to start reading from (1-indexed). Default is 1 (start from beginning).
-                When reading large documents in chunks, set this to continue from where you left off.
+                When reading large files in chunks, set this to continue from where you left off.
             type: integer
+        pattern:
+            description: |
+                Optional search pattern. Treated as a Python regular expression by default.
+                Matching results are returned as separated snippets with reliable source line numbers.
+            type: string
+        regex:
+            description: |
+                Whether to treat `pattern` as a Python regular expression. Default true. Set false for literal substring search.
+            type: boolean
+    required:
+        - source
+    """
+    agent = get_agent()
+    max_tokens = agent.config.get('max_input_tokens', 8000)
+    return read_document_content(source, start_at_line=start_at_line, max_tokens=max_tokens, pattern=pattern, regex=regex)
+
+
+@tool
+def view(source, start_at_line: int = 1, search_query: str = None):
+    """
+    description: |
+        View/extract text content from varied sources using the document extraction pipeline.
+        Use this for folders, URLs, PDFs, DOCX, ODT, HTML, XLSX, archives, and Python objects.
+        This is a convenience extraction view, not a strict source-of-truth file reader; use `read` for local UTF-8 files that must be read faithfully before editing.
+
+        `view` never prefixes line numbers, including search result snippets, because extracted line structure is not reliable.
+        For large outputs: the output is automatically truncated.
+    parameters:
+        source:
+            description: |
+                Source to view/extract: local folder or file, URL, or Python object/variable.
+        start_at_line:
+            description: |
+                Line number to start viewing from (1-indexed). Pagination only; line numbers are not rendered.
+            type: integer
+        search_query:
+            description: |
+                Optional literal text query for simple search within the extracted content. Results are returned as unnumbered snippets.
+            type: string
     required:
         - source
     """
     agent = get_agent()
     max_tokens = agent.config.get('max_input_tokens', 8000)
-    return read_document_content(source, start_at_line=start_at_line, max_tokens=max_tokens)
+    return view_document_content(source, start_at_line=start_at_line, max_tokens=max_tokens, search_query=search_query)
 
 
 @tool(tool_type="custom")

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/codex_agent/chat.py

@@ -390,14 +390,17 @@ class Chat:
         quota_5h = cls._format_percent(status.get("quota_5h_percent"))
         quota_7d = cls._format_percent(status.get("quota_7d_percent"))
         sent_messages = int(status.get("sent_session_messages") or 0)
+        persistent_messages = status.get("persistent_session_messages")
+        temporary_messages = status.get("temporary_session_messages")
         total_messages = int(status.get("total_session_messages") or 0)
         model = status.get("model") or "-"
+        message_segments = cls._message_status_segments(sent_messages, persistent_messages, temporary_messages, total_messages)
         return Text.assemble(
             (" ctx ", "bold #c9d1d9 on #161b22"),
             (f"{used}/{limit} ktok ", "#8b949e on #161b22"),
             (f"{percent:.1f}%", f"{context_style} on #161b22"),
             ("  msg ", "#6e7681 on #161b22"),
-            (f"{sent_messages}/{total_messages}", "#c9d1d9 on #161b22"),
+            *message_segments,
             ("  5h ", "#6e7681 on #161b22"),
             (quota_5h, f"{cls._percent_style(status.get('quota_5h_percent'))} on #161b22"),
             ("  7d ", "#6e7681 on #161b22"),
@@ -406,6 +409,21 @@ class Chat:
             (str(model), "#c9d1d9 on #161b22"),
         )
 
+    @staticmethod
+    def _message_status_segments(sent_messages, persistent_messages, temporary_messages, total_messages):
+        if persistent_messages is None and temporary_messages is None:
+            return ((f"{sent_messages}/{total_messages}", "#c9d1d9 on #161b22"),)
+        try:
+            persistent = int(persistent_messages or 0)
+            temporary = int(temporary_messages or 0)
+        except Exception:
+            return ((f"{sent_messages}/{total_messages}", "#c9d1d9 on #161b22"),)
+        return (
+            (str(persistent), "#c9d1d9 on #161b22"),
+            (f"+{temporary}", "italic #f0883e on #161b22"),
+            (f"/{total_messages}", "#c9d1d9 on #161b22"),
+        )
+
     @staticmethod
     def _format_ktok(value):
         try:

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/codex_agent/message.py

@@ -185,11 +185,26 @@ class CompactionMessage(DeveloperMessage):
     name = "compaction"
     content = "Previous conversation context was compacted by the backend."
     response_id = ""
-    output_items = None
+    summaries = modict.factory(list)
     compacted_message_count = 0
 
+    def __init__(self, *args, **kwargs):
+        output_items = kwargs.pop("output_items", None)
+        super().__init__(*args, **kwargs)
+        if output_items is None:
+            output_items = self.pop("output_items", None)
+        if output_items is not None and not self.get("summaries"):
+            self.summaries = self.summary_items(output_items)
+
+    @staticmethod
+    def summary_items(items):
+        return [
+            item for item in items or []
+            if item.get("type") == "compaction_summary"
+        ]
+
     def to_response_format(self):
-        return list(self.get("output_items") or [])
+        return list(self.get("summaries") or [])
 
     def as_string_attrs(self):
         attrs = self.extract('name', 'type', 'timestamp')
@@ -199,7 +214,7 @@ class CompactionMessage(DeveloperMessage):
 
     def as_string_body(self):
         summary = next(
-            (item for item in self.get("output_items") or [] if item.get("type") == "compaction_summary"),
+            iter(self.get("summaries") or []),
             None,
         )
         if not summary:

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/codex_agent/prompts/system_prompt.txt

@@ -29,16 +29,17 @@ These tools are intended as general means for you to act on the user's local env
 Code runs with user priviledges on his local system, use discernment before running code that could compromise sensitive data or system integrity.
 
 Other agentic tools are provided to ease your interaction with the user and content:
-- `read` : to read folders, various file types, or webpages
-- `edit`: to edit files
-- `write`: to write/overwrite wholes files
+- `read` : to read line-numbered local utf-8 files precisely (with optional pattern search in the content)
+- `edit`: to make targeted edits to local utf-8 files.
+- `write`: to write/overwrite wholes files (use with caution)
+- `view`: to quickly access the content of folders, various file types (PDF, ODF, XLSX, HTML, ...), webpages, or python objects - when used programmatically. Works on most content types (always yields a result, even mediocre).
 - `observe` : to use your AI vision capabilities on image files or urls
 - `show`: to show any kind of content to the user by opening it with the default system backend 
 + Any other custom tools loaded for the current session
 
 Always use these specialized tools over generic bash or python calls when relevant for a task as they are optimized for agentic feedback and efficiency
 Don't use `bash` or `python` to perform targeted edits via scripts (unless automation makes sense) : always use the edit tool.
-
+Never edit blindly/guessing, always use `read` first to know what to replace before you attempt to make changes to files.
 
 # Agentic Work Loop
 

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/codex_agent/utils.py

@@ -79,15 +79,53 @@ def text_content(file):
     else:
         return None
     
-def add_line_numbers(text: str) -> str:
+def _number_lines(lines: list[str], start_line: int = 1, total_lines: int | None = None) -> str:
+    last_line = (start_line + len(lines) - 1) if total_lines is None else max(total_lines, start_line + len(lines) - 1)
+    width = len(str(max(1, last_line)))
+    return "\n".join(
+        f"{i:0{width}d}|{line}"
+        for i, line in enumerate(lines, start=start_line)
+    )
+
+
+def add_line_numbers(text: str, start_line: int = 1, total_lines: int | None = None) -> str:
+    return _number_lines(text.splitlines(), start_line=start_line, total_lines=total_lines)
+
+
+def pattern_snippets(text: str, pattern, context_lines: int = 10, regex_pattern: bool = True) -> str:
+    pattern_text = str(pattern)
     lines = text.splitlines()
-    n = len(lines)
-    width = len(str(n))
-    
-    lines_num = [f"{i:0{width}d}|{line}" 
-                  for i, line in enumerate(lines, start=1)]
-    return "\n".join(lines_num)
-    
+    total_lines = len(lines)
+    if regex_pattern:
+        try:
+            compiled_pattern = re.compile(pattern_text)
+        except re.error as exc:
+            raise ValueError(f"Invalid regex pattern {pattern_text!r}: {exc}") from exc
+        matches = [index for index, line in enumerate(lines, start=1) if compiled_pattern.search(line)]
+    else:
+        matches = [index for index, line in enumerate(lines, start=1) if pattern_text in line]
+    if not matches:
+        return _number_lines(lines, start_line=1, total_lines=total_lines)
+
+    context_lines = max(0, int(context_lines or 0))
+    ranges = []
+    for match_line in matches:
+        start = max(1, match_line - context_lines)
+        end = min(total_lines, match_line + context_lines)
+        if ranges and start <= ranges[-1][1] + 1:
+            ranges[-1] = (ranges[-1][0], max(ranges[-1][1], end))
+        else:
+            ranges.append((start, end))
+
+    parts = []
+    for number, (start, end) in enumerate(ranges, start=1):
+        snippet_lines = lines[start - 1:end]
+        parts.append(
+            f"--- match snippet {number}/{len(ranges)}: lines {start}-{end} ---\n"
+            + _number_lines(snippet_lines, start_line=start, total_lines=total_lines)
+        )
+    return "\n\n".join(parts)
+
 def short_id(length=8):
     return "".join(random.choices(string.ascii_letters,k=length))
 
@@ -162,19 +200,18 @@ def truncate(string, max_tokens=2000, start_line=1, model=None):
     Returns:
         Truncated string with indication of removed content
     """
-    # Quick check - if already under limit, return as-is
-    tokens = tokenize(string, model=model)
-    if len(tokens) <= max_tokens:
-        return string
-
-    # line-based truncation
+    # line-based pagination
     lines = string.splitlines(keepends=True)
-
-    # Skip to start_line if needed
     if start_line > 1:
         if start_line > len(lines):
             return ""
-        lines = lines[start_line - 1:]
+        string = ''.join(lines[start_line - 1:])
+        lines = string.splitlines(keepends=True)
+
+    # Quick check - if already under limit after pagination, return as-is
+    tokens = tokenize(string, model=model)
+    if len(tokens) <= max_tokens:
+        return string
 
     # Calculate the actual last line number after skipping
     remaining_lines = len(lines)
@@ -206,7 +243,7 @@ def truncate(string, max_tokens=2000, start_line=1, model=None):
 
         return result
     else:
-        return f"[Lines {start_line}-{start_line + remaining_lines - 1} truncated - first lineexceeds token limit]\n"
+        return f"[Lines {start_line}-{start_line + remaining_lines - 1} truncated - first line exceeds token limit]\n"
 
 def pack_msgs(messages):
     text = ''
@@ -387,24 +424,80 @@ class Logger:
         with open(file,'w',encoding="utf-8") as f:
             f.write(str(content)+'\n')
 
-def read_document_content(source, start_at_line=1, max_tokens=8000):
-    """
-    Read and extract text content from various document formats.
+def _render_numbered_read_text(text: str, start_at_line=1, max_tokens=8000, pattern=None, pattern_context_lines=10, regex=True):
+    effective_start_at_line = int(start_at_line or 1)
+    total_lines = len(text.splitlines())
+    if pattern:
+        text = pattern_snippets(text, pattern, context_lines=pattern_context_lines, regex_pattern=regex)
+        if max_tokens > 0:
+            text = truncate(text, max_tokens=max_tokens, start_line=1)
+        return text
+    # Apply pagination and truncation - truncate function handles smart chunking internally
+    if max_tokens > 0:
+        text = truncate(text, max_tokens=max_tokens, start_line=effective_start_at_line)
+    elif effective_start_at_line > 1:
+        lines = text.splitlines(keepends=True)
+        text = "" if effective_start_at_line > len(lines) else ''.join(lines[effective_start_at_line - 1:])
+    return add_line_numbers(text, start_line=effective_start_at_line, total_lines=total_lines)
 
-    Args:
-        source: Document source - either a local file path or URL (http/https)
-        start_at_line: Line number to start reading from (1-indexed)
-        max_tokens: Maximum tokens for truncation (0 to disable)
 
-    Returns:
-        Extracted text content, possibly truncated
-    """
-    get_text = import_module(".get_text.get_text", __package__).get_text
+def _view_search_snippets(text: str, search_query, context_lines: int = 10) -> str:
+    query = str(search_query)
+    lines = text.splitlines()
+    matches = [index for index, line in enumerate(lines, start=1) if query in line]
+    if not matches:
+        return f"No results for search_query: {query!r}"
+
+    context_lines = max(0, int(context_lines or 0))
+    ranges = []
+    for match_line in matches:
+        start = max(1, match_line - context_lines)
+        end = min(len(lines), match_line + context_lines)
+        if ranges and start <= ranges[-1][1] + 1:
+            ranges[-1] = (ranges[-1][0], max(ranges[-1][1], end))
+        else:
+            ranges.append((start, end))
+
+    parts = []
+    for number, (start, end) in enumerate(ranges, start=1):
+        snippet = "\n".join(lines[start - 1:end])
+        parts.append(f"--- search result {number}/{len(ranges)} ---\n{snippet}")
+    return "\n\n".join(parts)
+
+
+def read_document_content(source, start_at_line=1, max_tokens=8000, pattern=None, pattern_context_lines=10, regex=True):
+    """Strictly read a local UTF-8 text file without extraction, URL fetching, or directory traversal."""
+    if not isinstance(source, (str, os.PathLike)):
+        raise TypeError("read only accepts a local file path. Use view for objects or extracted document content.")
+    path = os.fspath(source)
+    if path.startswith(("http://", "https://")):
+        raise ValueError("read only accepts local UTF-8 text files. Use view for URLs.")
+    if os.path.isdir(path):
+        raise ValueError("read only accepts local UTF-8 text files. Use view for directories.")
+    if not os.path.isfile(path):
+        raise FileNotFoundError(f"File not found: {path}")
+    with open(path, "rb") as f:
+        data = f.read()
+    try:
+        text = data.decode("utf-8")
+    except UnicodeDecodeError as exc:
+        raise UnicodeDecodeError(exc.encoding, exc.object, exc.start, exc.end, f"{exc.reason}. read requires UTF-8 text files; use view for extracted content.") from exc
+    return _render_numbered_read_text(text, start_at_line=start_at_line, max_tokens=max_tokens, pattern=pattern, pattern_context_lines=pattern_context_lines, regex=regex)
 
-    text=get_text(source)
 
-    # Apply truncation - truncate function handles smart chunking internally
+def view_document_content(source, start_at_line=1, max_tokens=8000, search_query=None, search_context_lines=10):
+    """Extract text from varied sources via get_text. This view is not line-structure reliable."""
+    get_text = import_module(".get_text.get_text", __package__).get_text
+    text = get_text(source)
+    effective_start_at_line = int(start_at_line or 1)
+    if search_query:
+        text = _view_search_snippets(text, search_query, context_lines=search_context_lines)
+        if max_tokens > 0:
+            text = truncate(text, max_tokens=max_tokens, start_line=1)
+        return text
     if max_tokens > 0:
-        text = truncate(text, max_tokens=max_tokens, start_line=start_at_line)
-
+        return truncate(text, max_tokens=max_tokens, start_line=effective_start_at_line)
+    if effective_start_at_line > 1:
+        lines = text.splitlines(keepends=True)
+        return "" if effective_start_at_line > len(lines) else ''.join(lines[effective_start_at_line - 1:])
     return text

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11/codex_agent_framework.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: codex-agent-framework
-Version: 0.1.9
+Version: 0.1.11
 Summary: A lightweight event-driven Codex agent runtime.
 Author: Baptiste
 License-Expression: MIT
@@ -400,6 +400,8 @@ python -m build
 The distribution includes prompt text files and `codex_agent/get_text/default_gitignore` through package data and `MANIFEST.in`.
 
 ## Recent changes
+- `0.1.11`: split strict line-numbered UTF-8 `read` from unnumbered extracted `view`, preserve blank lines in read snippets, and show persistent+temporary message counts in the TUI status bar.
+- `0.1.10`: persist only backend compaction summaries, drop bulky compacted conversation payloads, and refresh context status after compaction.
 - `0.1.9`: maintenance packaging release after validating the local execution environment and deploy workflow.
 - `0.1.8`: scope TUI replay/SSE catch-up to the active session and make bash/python subprocesses inherit the project Python environment, including service-launched agents.
 - `0.1.7`: add durable RAG memory, scheduled wakeups, process-isolated server runtime, tray/service controls, robust SSE replay/reconnect, richer TUI status, and improved token estimates.

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "codex-agent-framework"
-version = "0.1.9"
+version = "0.1.11"
 description = "A lightweight event-driven Codex agent runtime."
 readme = "README.md"
 requires-python = ">=3.10"

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/tests/test_agent.py

@@ -120,7 +120,10 @@ class AbortableAI:
 
 class FakeCompactionResult:
     response_id = "resp_123"
-    output_items = [{"type": "compaction_summary", "id": "cs_123", "encrypted_content": "opaque"}]
+    output_items = [
+        {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "old"}]},
+        {"type": "compaction_summary", "id": "cs_123", "encrypted_content": "opaque"},
+    ]
 
 
 class FakeCompactionClient:
@@ -271,7 +274,8 @@ def test_agent_session_compact_inserts_compaction_before_remainder(tmp_path, mon
 
     assert isinstance(message, CompactionMessage)
     assert session.messages == [user, done, message, current]
-    assert message.output_items == FakeCompactionResult.output_items
+    assert "output_items" not in message
+    assert message.summaries == [FakeCompactionResult.output_items[1]]
     assert message.compacted_message_count == 2
     assert client.history == [
         {
@@ -338,6 +342,8 @@ def test_agent_compact_session_emits_volume_events(tmp_path, monkeypatch):
     assert seen[0].total_session_messages == 3
     assert seen[0].compacted_token_count > 0
     assert seen[1].message is message
+    assert seen[1].status.source == "local_estimate"
+    assert seen[1].status.context_current_tokens != 5000
     assert seen[1].compacted_message_count == 2
     assert seen[1].post_compaction_message_count == 2
     assert seen[1].total_session_messages == 4
@@ -350,6 +356,17 @@ def test_agent_compact_session_emits_volume_events(tmp_path, monkeypatch):
     assert status.quota_7d_percent == 34
 
 
+def test_agent_add_message_invalidates_context_status_cache():
+    agent = Agent(session="new", voice_enabled=False)
+    first = agent.context_status()
+
+    agent.add_message(DeveloperMessage(content="new visible context"))
+    second = agent.context_status()
+
+    assert second.total_session_messages == first.total_session_messages + 1
+    assert second.used_tokens != first.used_tokens
+
+
 def test_agent_get_context_starts_at_latest_compaction():
     agent = Agent(session="new", voice_enabled=False)
     old = DeveloperMessage(content="old hidden context")
@@ -367,6 +384,26 @@ def test_agent_get_context_starts_at_latest_compaction():
     assert "old hidden context" not in context_text
 
 
+def test_agent_local_context_estimate_does_not_count_encrypted_compaction_payload():
+    agent = Agent(session="new", voice_enabled=False)
+    encrypted_content = "x" * 100_000
+    agent.session.messages = [
+        CompactionMessage(output_items=[{
+            "type": "compaction_summary",
+            "id": "cs_123",
+            "encrypted_content": encrypted_content,
+        }]),
+        DeveloperMessage(content="small current context"),
+    ]
+
+    status = agent.context_status()
+
+    assert status.used_tokens < 10_000
+    assert encrypted_content not in str(agent.local_token_count_payload(
+        agent.response_input_token_payload(agent.session.context_messages(), response_tools=[])
+    ))
+
+
 def test_agent_get_context_filters_expired_messages():
     agent = Agent(session="new", voice_enabled=False)
     expired = DeveloperMessage(content="expired hidden context", turns_left=0)
@@ -634,6 +671,48 @@ def test_builtin_read_tool_survives_get_text_submodule_shadowing():
     assert "def read(" in agent.pending[-1].content
 
 
+def test_builtin_read_tool_accepts_pattern_and_includes_context(tmp_path):
+    file = tmp_path / "sample.txt"
+    lines = [f"line {index}" for index in range(1, 31)]
+    lines[20] = "line 21 needle"
+    file.write_text("\n".join(lines), encoding="utf-8")
+
+    agent = Agent(session="new", voice_enabled=False)
+    message = AssistantMessage()
+    message.tool_calls = [{
+        "type": "function_call",
+        "call_id": "call_1",
+        "name": "read",
+        "arguments": f'{{"source": "{file}", "pattern": "needle"}}',
+    }]
+
+    assert agent.call_tools(message) is True
+    assert isinstance(agent.pending[-1], ToolResultWrapper)
+    assert "11|line 11" in agent.pending[-1].content
+    assert "21|line 21 needle" in agent.pending[-1].content
+    assert "10|line 10" not in agent.pending[-1].content
+
+
+def test_builtin_read_tool_accepts_regex_pattern(tmp_path):
+    file = tmp_path / "sample.txt"
+    lines = ["alpha", "ticket ABC-123", *[f"line {index}" for index in range(3, 25)], "ticket XYZ-123"]
+    file.write_text("\n".join(lines), encoding="utf-8")
+
+    agent = Agent(session="new", voice_enabled=False)
+    message = AssistantMessage()
+    message.tool_calls = [{
+        "type": "function_call",
+        "call_id": "call_1",
+        "name": "read",
+        "arguments": f'{{"source": "{file}", "pattern": "ABC-\\\\d+", "regex": true}}',
+    }]
+
+    assert agent.call_tools(message) is True
+    assert isinstance(agent.pending[-1], ToolResultWrapper)
+    assert "2|ticket ABC-123" in agent.pending[-1].content
+    assert "3|ticket XYZ-123" not in agent.pending[-1].content
+
+
 def test_agent_applies_image_generation_tool_config():
     agent = Agent(
         session="new",

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/tests/test_chat.py

@@ -40,6 +40,48 @@ def test_agent_context_status_counts_only_effective_context_messages():
     assert status.context_session_messages == 2
     assert status.total_history_messages == 2
     assert status.sent_session_messages == 2
+    assert status.persistent_session_messages == 2
+    assert status.temporary_session_messages == 0
+
+
+def test_agent_context_status_counts_persistent_and_temporary_messages():
+    agent = Agent(session="new", voice_enabled=False, input_token_limit=10_000)
+    compaction = CompactionMessage(output_items=[{"type": "compaction_summary", "id": "cs_123"}])
+    persistent = DeveloperMessage(content="persistent")
+    temporary = DeveloperMessage(content="temporary", turns_left=2)
+    expired = DeveloperMessage(content="expired", turns_left=0)
+    agent.session.messages = [compaction, persistent, temporary, expired]
+
+    status = agent.context_status()
+
+    assert status.context_session_messages == 3
+    assert status.persistent_session_messages == 2
+    assert status.temporary_session_messages == 1
+
+
+def test_agent_status_preserves_persistent_and_temporary_message_counts():
+    agent = Agent(session="new", voice_enabled=False, input_token_limit=10_000)
+    agent.session.messages.append(DeveloperMessage(content="persistent"))
+    agent.session.messages.append(DeveloperMessage(content="temporary", turns_left=2))
+
+    status = agent.get_status()
+
+    assert status.persistent_session_messages == 2
+    assert status.temporary_session_messages == 1
+
+
+def test_agent_api_status_uses_precached_persistent_and_temporary_counts(monkeypatch):
+    agent = Agent(session="new", voice_enabled=False, input_token_limit=10_000)
+    agent.session.messages.append(DeveloperMessage(content="persistent"))
+    agent.session.messages.append(DeveloperMessage(content="temporary", turns_left=2))
+
+    agent.get_context()
+    monkeypatch.setattr(agent, "context_status", lambda: (_ for _ in ()).throw(AssertionError("should use cache")))
+    agent.cache_api_status(usage=modict(input_tokens=1000), model="gpt-test")
+
+    status = agent.get_status()
+    assert status.persistent_session_messages == 2
+    assert status.temporary_session_messages == 1
 
 
 def test_agent_context_status_counts_response_tool_specs():
@@ -185,6 +227,30 @@ def test_chat_status_bar_formats_compact_status():
     assert any("#161b22" in str(span.style) for span in rendered.spans)
 
 
+def test_chat_status_bar_formats_persistent_plus_temporary_messages():
+    status = modict(
+        context_current_tokens=12_300,
+        context_limit_tokens=128_000,
+        context_percent=9.609,
+        sent_session_messages=43,
+        persistent_session_messages=33,
+        temporary_session_messages=10,
+        total_session_messages=93,
+        quota_5h_percent=42,
+        quota_7d_percent=13,
+        model="gpt-5.4",
+    )
+
+    rendered = Chat.format_status(status)
+    output = rendered.plain
+
+    assert "msg 33+10/93" in output
+    assert any(
+        "#f0883e" in str(span.style) and output[span.start:span.end] == "+10"
+        for span in rendered.spans
+    )
+
+
 def test_chat_refresh_status_keeps_rich_text_styles():
     chat = Chat(client=modict(
         get_status=lambda: modict(

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/tests/test_messages.py

@@ -189,15 +189,18 @@ def test_server_tool_response_message_as_string_includes_item_metadata():
     assert '"type": "web_search_call"' in text
 
 
-def test_compaction_message_replays_backend_output_items():
-    output_items = [{"type": "compaction_summary", "id": "cs_123", "encrypted_content": "opaque"}]
+def test_compaction_message_replays_only_backend_summary_items():
+    output_items = [
+        {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "old"}]},
+        {"type": "compaction_summary", "id": "cs_123", "encrypted_content": "opaque"},
+    ]
     message = CompactionMessage(
         response_id="resp_123",
         output_items=output_items,
         compacted_message_count=4,
     )
 
-    assert message.to_response_format() == output_items
+    assert message.to_response_format() == [output_items[1]]
     text = message.as_string()
     assert text.startswith("<CompactionMessage ")
     assert "response_id='resp_123'" in text
@@ -207,6 +210,19 @@ def test_compaction_message_replays_backend_output_items():
     assert "encrypted_content" not in text
 
 
+def test_compaction_message_migrates_legacy_output_items_without_persisting_them():
+    output_items = [
+        {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "old"}]},
+        {"type": "compaction_summary", "id": "cs_123", "encrypted_content": "opaque"},
+    ]
+
+    message = CompactionMessage({"output_items": output_items})
+
+    assert "output_items" not in message
+    assert message.summaries == [output_items[1]]
+    assert message.to_response_format() == [output_items[1]]
+
+
 def test_image_generation_response_message_persists_result_as_file(tmp_path, monkeypatch):
     monkeypatch.setattr(utils, "RUNTIME_DIR", str(tmp_path))
     result = "Zm9v"

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/tests/test_server.py

@@ -605,6 +605,28 @@ def test_process_agent_runtime_discards_timed_out_commands():
     assert runtime.pending == {}
 
 
+def test_process_agent_runtime_updates_cached_status_from_compaction_event():
+    runtime = ProcessAgentRuntime()
+
+    runtime._handle_event(modict(
+        type="compaction_completed_event",
+        status=modict(
+            context_current_tokens=123,
+            context_limit_tokens=1000,
+            context_percent=12.3,
+            sent_session_messages=1,
+            total_session_messages=52,
+            session_id="session_1",
+            busy=False,
+            source="local_estimate",
+        ),
+    ))
+
+    assert runtime.cache.status.context_current_tokens == 123
+    assert runtime.cache.current_session_id == "session_1"
+    assert runtime.cache.busy is False
+
+
 def test_agent_server_subscribe_replays_missed_events_after_sequence():
     agent = Agent(session="new", voice_enabled=False)
     service = AgentServer(agent)

{codex_agent_framework-0.1.9 → codex_agent_framework-0.1.11}/tests/test_utils.py

@@ -1,5 +1,6 @@
 from io import BytesIO
 
+import pytest
 from PIL import Image as PILImage
 
 from codex_agent.message import AssistantMessage, DeveloperMessage, ServerToolResponseMessage, ToolResultMessage, UserMessage
@@ -35,6 +36,114 @@ def test_project_python_env_prefers_running_interpreter():
     assert env["PYTHON_EXECUTABLE"] == "/opt/project-python/bin/python"
 
 
+def test_read_document_content_adds_line_numbers_by_default(tmp_path):
+    file = tmp_path / "sample.txt"
+    file.write_text("alpha\nbeta\ngamma", encoding="utf-8")
+
+    content = utils.read_document_content(str(file), max_tokens=10_000)
+
+    assert content == "1|alpha\n2|beta\n3|gamma"
+
+
+def test_read_document_content_pattern_returns_all_numbered_snippets(tmp_path):
+    file = tmp_path / "sample.txt"
+    lines = [f"line {index}" for index in range(1, 61)]
+    lines[20] = "line 21 MATCH here"
+    lines[49] = "line 50 MATCH again"
+    file.write_text("\n".join(lines), encoding="utf-8")
+
+    content = utils.read_document_content(str(file), pattern="MATCH", max_tokens=10_000)
+
+    assert "--- match snippet 1/2: lines 11-31 ---" in content
+    assert "11|line 11" in content
+    assert "21|line 21 MATCH here" in content
+    assert "31|line 31" in content
+    assert "10|line 10" not in content
+    assert "--- match snippet 2/2: lines 40-60 ---" in content
+    assert "50|line 50 MATCH again" in content
+
+
+def test_read_document_content_pattern_context_is_clamped_to_file_start(tmp_path):
+    file = tmp_path / "sample.txt"
+    lines = ["line 1", "line 2 MATCH", "line 3"]
+    file.write_text("\n".join(lines), encoding="utf-8")
+
+    content = utils.read_document_content(str(file), pattern="MATCH", max_tokens=10_000)
+
+    assert content.startswith("--- match snippet 1/1: lines 1-3 ---\n1|line 1")
+    assert "2|line 2 MATCH" in content
+
+
+def test_read_document_content_pattern_is_regex_by_default(tmp_path):
+    file = tmp_path / "sample.txt"
+    lines = ["alpha", "ticket ABC-123", "ticket ABC-999", "ticket XYZ-123"]
+    file.write_text("\n".join(lines), encoding="utf-8")
+
+    content = utils.read_document_content(str(file), pattern=r"ABC-\d+", pattern_context_lines=0, max_tokens=10_000)
+
+    assert "2|ticket ABC-123" in content
+    assert "3|ticket ABC-999" in content
+    assert "4|ticket XYZ-123" not in content
+
+
+def test_read_document_content_can_treat_pattern_as_literal(tmp_path):
+    file = tmp_path / "sample.txt"
+    lines = ["ticket ABC-123", r"literal ABC-\d+ pattern"]
+    file.write_text("\n".join(lines), encoding="utf-8")
+
+    content = utils.read_document_content(str(file), pattern=r"ABC-\d+", regex=False, pattern_context_lines=0, max_tokens=10_000)
+
+    assert "1|ticket ABC-123" not in content
+    assert r"2|literal ABC-\d+ pattern" in content
+
+
+def test_read_document_content_preserves_blank_lines_and_real_line_numbers(tmp_path):
+    file = tmp_path / "sample.py"
+    file.write_text("one\n\n\n    \nfive", encoding="utf-8")
+
+    content = utils.read_document_content(str(file), max_tokens=10_000)
+
+    assert content == "1|one\n2|\n3|\n4|    \n5|five"
+
+
+def test_read_document_content_pattern_preserves_blank_lines_in_snippet_context(tmp_path):
+    file = tmp_path / "sample.py"
+    file.write_text("one\n\n\nMATCH\nfive", encoding="utf-8")
+
+    content = utils.read_document_content(str(file), pattern="MATCH", pattern_context_lines=3, max_tokens=10_000)
+
+    assert content == "--- match snippet 1/1: lines 1-5 ---\n1|one\n2|\n3|\n4|MATCH\n5|five"
+
+
+def test_read_document_content_rejects_directories_and_urls(tmp_path):
+    with pytest.raises(ValueError, match="Use view for directories"):
+        utils.read_document_content(str(tmp_path), max_tokens=10_000)
+
+    with pytest.raises(ValueError, match="Use view for URLs"):
+        utils.read_document_content("https://example.com", max_tokens=10_000)
+
+
+def test_view_document_content_keeps_extraction_pipeline_for_directories_without_line_numbers(tmp_path):
+    file = tmp_path / "sample.txt"
+    file.write_text("hello", encoding="utf-8")
+
+    content = utils.view_document_content(str(tmp_path), max_tokens=10_000)
+
+    assert "sample.txt" in content
+    assert "1|" not in content
+
+
+def test_view_document_content_search_query_returns_unnumbered_snippets(tmp_path):
+    file = tmp_path / "sample.txt"
+    file.write_text("alpha\nbeta needle\ngamma", encoding="utf-8")
+
+    content = utils.view_document_content(str(file), search_query="needle", max_tokens=10_000)
+
+    assert content.startswith("--- search result 1/1 ---\n")
+    assert "beta needle" in content
+    assert "2|" not in content
+
+
 def test_split_history_turns_keeps_current_unfinished_turn_as_remainder():
     complete_user = UserMessage(content="first")
     complete_assistant = AssistantMessage(content="done")