cli-consumption 0.4.1__tar.gz → 0.4.3__tar.gz

{cli_consumption-0.4.1 → cli_consumption-0.4.3}/CHANGELOG.md

@@ -6,6 +6,28 @@ use [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.4.3] - 2026-09-05
+
+### Added
+
+- Added opt-in `collect --incremental` automatic bounded batching for large Codex
+  stores, with restart-safe ingestion, strict metadata-only preflight staging,
+  deterministic duplicate convergence, privacy-safe partial results, and unchanged
+  per-file and per-line safety limits; authoritative subagent graphs remain untouched.
+
+## [0.4.2] - 2026-09-05
+
+### Added
+
+- Added reviewed autonomous web exports that freeze the visible dashboard layout,
+  bounded selection, privacy profile, and theme into a CSP-locked standalone file while
+  preserving query-only API and CLI export compatibility.
+- Added the provider-neutral `DashboardLayout v1` contract, a bounded widget registry,
+  deterministic defaults and retired-widget recovery, plus SQLite/PostgreSQL
+  persistence for the single authenticated dashboard operator.
+- Added an explicit, accessible dashboard edit mode with bounded undo/redo, deterministic
+  placement, and ETag/`If-Match` conflict protection for concurrent save and reset.
+
 ## [0.4.1] - 2026-09-01
 
 ### Changed
@@ -172,7 +194,9 @@ use [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 - Refreshed the provider guide for the first minor release ([#26]).
 
-[Unreleased]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.4.1...HEAD
+[Unreleased]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.4.3...HEAD
+[0.4.3]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.4.2...v0.4.3
+[0.4.2]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.4.1...v0.4.2
 [0.4.1]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.4.0...v0.4.1
 [0.4.0]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.3.3...v0.4.0
 [0.3.3]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.3.2...v0.3.3

{cli_consumption-0.4.1 → cli_consumption-0.4.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: cli-consumption
-Version: 0.4.1
+Version: 0.4.3
 Summary: Analyze and consolidate AI coding CLI consumption across machines.
 Project-URL: Homepage, https://github.com/Guillaume-Lombardo/cli-consumption
 Project-URL: Documentation, https://github.com/Guillaume-Lombardo/cli-consumption#readme
@@ -117,6 +117,10 @@ To collect one provider or select another database:
 uv run cli-consumption collect --provider codex --database usage.sqlite
 ```
 
+For a Codex store that exceeds aggregate collection limits, add `--incremental`.
+The command automatically writes bounded, restart-safe batches to the same database;
+individual file and line safety limits remain enforced.
+
 Use `--source [LABEL=]PATH` for trusted offline copies and repeated
 `--project NAME=PATH_PREFIX` mappings for stable project labels. See the
 [usage and operations guide](https://github.com/Guillaume-Lombardo/cli-consumption/blob/main/docs/usage.md)
@@ -145,7 +149,8 @@ uv run cli-consumption providers --json
 
 - Provider files are untrusted. Collection enforces discovery, file, line, SQLite row,
   structured-field, and total normalized-record limits; direct provider-file symlinks
-  are refused.
+  are refused. Opt-in Codex incremental collection resets only aggregate per-batch
+  limits and keeps a separate hard batch-count ceiling.
 - `collect --strict` and `sync --strict` refuse a batch when any provider skipped
   malformed records. Collection distinguishes `provider_limit_exceeded`,
   `provider_format_incompatible`, `invalid_snapshot`, and the unexpected-failure

{cli_consumption-0.4.1 → cli_consumption-0.4.3}/README.md

@@ -78,6 +78,10 @@ To collect one provider or select another database:
 uv run cli-consumption collect --provider codex --database usage.sqlite
 ```
 
+For a Codex store that exceeds aggregate collection limits, add `--incremental`.
+The command automatically writes bounded, restart-safe batches to the same database;
+individual file and line safety limits remain enforced.
+
 Use `--source [LABEL=]PATH` for trusted offline copies and repeated
 `--project NAME=PATH_PREFIX` mappings for stable project labels. See the
 [usage and operations guide](https://github.com/Guillaume-Lombardo/cli-consumption/blob/main/docs/usage.md)
@@ -106,7 +110,8 @@ uv run cli-consumption providers --json
 
 - Provider files are untrusted. Collection enforces discovery, file, line, SQLite row,
   structured-field, and total normalized-record limits; direct provider-file symlinks
-  are refused.
+  are refused. Opt-in Codex incremental collection resets only aggregate per-batch
+  limits and keeps a separate hard batch-count ceiling.
 - `collect --strict` and `sync --strict` refuse a batch when any provider skipped
   malformed records. Collection distinguishes `provider_limit_exceeded`,
   `provider_format_incompatible`, `invalid_snapshot`, and the unexpected-failure

{cli_consumption-0.4.1 → cli_consumption-0.4.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "cli-consumption"
-version = "0.4.1"
+version = "0.4.3"
 description = "Analyze and consolidate AI coding CLI consumption across machines."
 readme = "README.md"
 requires-python = ">=3.11"

cli_consumption-0.4.3/src/cli_consumption/INTER_FONT_LICENSE.txt

@@ -0,0 +1,93 @@
+Copyright 2016 The Inter Project Authors (https://github.com/rsms/inter) Inter-Italic[opsz,wght].ttf: Copyright 2016 The Inter Project Authors (https://github.com/rsms/inter)
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded,
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the corresponding
+Copyright Holder. This restriction only applies to the primary font name as
+presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created
+using the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.

cli_consumption-0.4.3/src/cli_consumption/adapters/base.py

@@ -0,0 +1,43 @@
+from __future__ import annotations
+
+from collections.abc import Iterator
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Protocol, runtime_checkable
+
+from cli_consumption.models import Snapshot
+
+
+@dataclass(frozen=True, slots=True)
+class CollectionBatch:
+    """One snapshot plus an optional subagent-scope authority override."""
+
+    snapshot: Snapshot
+    authoritative_subagent_scopes: frozenset[tuple[str, str]] | None = None
+
+
+class UnsupportedProviderFormat(ValueError):
+    """Raised when a detected provider store has an incompatible schema."""
+
+
+class Adapter(Protocol):
+    """Contract implemented by every supported AI CLI."""
+
+    name: str
+
+    def collect(
+        self,
+        sources: list[tuple[str, Path]],
+        project_mappings: list[tuple[str, str]],
+    ) -> Snapshot: ...
+
+
+@runtime_checkable
+class IncrementalAdapter(Adapter, Protocol):
+    """Optional adapter contract for bounded, independently ingestible batches."""
+
+    def collect_incrementally(
+        self,
+        sources: list[tuple[str, Path]],
+        project_mappings: list[tuple[str, str]],
+    ) -> Iterator[CollectionBatch]: ...

{cli_consumption-0.4.1 → cli_consumption-0.4.3}/src/cli_consumption/adapters/codex.py

@@ -3,8 +3,10 @@ from __future__ import annotations
 import hashlib
 import json
 import math
+import os
 import re
 import sqlite3
+from collections.abc import Iterator
 from datetime import UTC, datetime
 from pathlib import Path
 from typing import Any
@@ -13,11 +15,18 @@ from cli_consumption.adapters._shared import (
     MAX_BIGINT as MAX_BIGINT,
 )
 from cli_consumption.adapters._shared import (
+    ProviderDataLimitError,
     ProviderInputBudget,
     iter_bounded_jsonl_bytes,
     open_provider_sqlite,
 )
-from cli_consumption.models import TOKEN_FIELDS, Snapshot, empty_tokens
+from cli_consumption.adapters.base import CollectionBatch
+from cli_consumption.models import (
+    TOKEN_FIELDS,
+    Snapshot,
+    SnapshotValidationError,
+    empty_tokens,
+)
 
 OUTSIDE_PROJECT = "outside-project"
 TOOL_PATTERN = re.compile(r"(?:tools|collaboration)\.([A-Za-z][A-Za-z0-9_]*)\s*\(")
@@ -85,6 +94,24 @@ AGENT_ROLE_ALIASES = {
     "worker": "worker",
 }
 SAFE_DIMENSION = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:/+-]*")
+INCREMENTAL_CANDIDATES_PER_BATCH = 1_000
+
+
+def _iter_session_files(root: Path) -> Iterator[Path]:
+    """Walk a provider tree deterministically without materializing every path."""
+    pending = [root]
+    while pending:
+        directory = pending.pop()
+        with os.scandir(directory) as entries:
+            ordered = sorted(entries, key=lambda entry: entry.name)
+        child_directories: list[Path] = []
+        for entry in ordered:
+            path = Path(entry.path)
+            if entry.is_dir(follow_symlinks=False):
+                child_directories.append(path)
+            elif entry.name.endswith(".jsonl"):
+                yield path
+        pending.extend(reversed(child_directories))
 
 
 def parse_timestamp(value: object) -> datetime | None:
@@ -166,6 +193,73 @@ class CodexAdapter:
             )
         return snapshot
 
+    def collect_incrementally(
+        self,
+        sources: list[tuple[str, Path]],
+        project_mappings: list[tuple[str, str]] | None = None,
+    ) -> Iterator[CollectionBatch]:
+        """Yield deterministic, bounded snapshots from arbitrarily many rollouts."""
+        mappings = project_mappings or []
+        for machine, codex_home in sources:
+            sessions = codex_home / "sessions"
+            if not sessions.is_dir():
+                raise ValueError("Missing Codex sessions directory")
+
+            yielded_sessions = False
+            batch: list[tuple[str, Path]] = []
+            for path in _iter_session_files(sessions):
+                batch.append((machine, path))
+                if len(batch) == INCREMENTAL_CANDIDATES_PER_BATCH:
+                    yielded_sessions = True
+                    yield from self._collect_incremental_batch(batch, mappings)
+                    batch = []
+            if batch:
+                yielded_sessions = True
+                yield from self._collect_incremental_batch(batch, mappings)
+
+            if not yielded_sessions:
+                yield CollectionBatch(Snapshot(provider=self.name), frozenset())
+
+    def _collect_incremental_batch(
+        self,
+        candidates: list[tuple[str, Path]],
+        mappings: list[tuple[str, str]],
+    ) -> Iterator[CollectionBatch]:
+        try:
+            yield CollectionBatch(
+                self._collect_candidates(candidates, mappings), frozenset()
+            )
+        except ProviderDataLimitError as error:
+            if str(error) != "provider_read_limit_exceeded" or len(candidates) == 1:
+                raise
+            midpoint = len(candidates) // 2
+            yield from self._collect_incremental_batch(candidates[:midpoint], mappings)
+            yield from self._collect_incremental_batch(candidates[midpoint:], mappings)
+        except SnapshotValidationError as error:
+            if error.code != "snapshot_too_large" or len(candidates) == 1:
+                raise
+            midpoint = len(candidates) // 2
+            yield from self._collect_incremental_batch(candidates[:midpoint], mappings)
+            yield from self._collect_incremental_batch(candidates[midpoint:], mappings)
+
+    def _collect_candidates(
+        self,
+        candidates: list[tuple[str, Path]],
+        mappings: list[tuple[str, str]],
+    ) -> Snapshot:
+        budget = ProviderInputBudget()
+        selected, duplicates, malformed = self._discover_candidates(candidates, budget)
+        snapshot = Snapshot(
+            provider=self.name,
+            duplicate_conversations=duplicates,
+            malformed_records=malformed,
+        )
+        for machine, path, event_count, digest in selected:
+            self._read_rollout(
+                snapshot, machine, path, event_count, digest, mappings, budget
+            )
+        return snapshot
+
     def _read_subagents(
         self,
         state_path: Path,
@@ -215,46 +309,62 @@ class CodexAdapter:
     def _discover(
         self, sources: list[tuple[str, Path]], budget: ProviderInputBudget
     ) -> tuple[list[tuple[str, Path, int, str]], int, int]:
-        selected: dict[str, tuple[str, Path, int, str]] = {}
-        duplicates = 0
-        malformed = 0
+        candidates: list[tuple[str, Path]] = []
         for machine, codex_home in sources:
             sessions = codex_home / "sessions"
             if not sessions.is_dir():
                 raise ValueError(f"Missing Codex sessions directory: {sessions}")
-            for path in budget.sorted_paths(sessions.rglob("*.jsonl")):
-                event_count = 0
-                conversation_id = ""
-                digest = hashlib.sha256()
-                for raw_line in iter_bounded_jsonl_bytes(path, budget):
-                    digest.update(raw_line)
-                    try:
-                        event = json.loads(raw_line)
-                    except (json.JSONDecodeError, UnicodeDecodeError):
-                        malformed += 1
-                        continue
-                    if not isinstance(event, dict):
+            candidates.extend(
+                (machine, path)
+                for path in budget.sorted_paths(sessions.rglob("*.jsonl"))
+            )
+        return self._discover_candidates(candidates, budget, charge_candidates=False)
+
+    def _discover_candidates(
+        self,
+        candidates: list[tuple[str, Path]],
+        budget: ProviderInputBudget,
+        *,
+        charge_candidates: bool = True,
+    ) -> tuple[list[tuple[str, Path, int, str]], int, int]:
+        selected: dict[str, tuple[str, Path, int, str]] = {}
+        duplicates = 0
+        malformed = 0
+        for machine, path in candidates:
+            if charge_candidates:
+                budget.item()
+            event_count = 0
+            conversation_id = ""
+            digest = hashlib.sha256()
+            for raw_line in iter_bounded_jsonl_bytes(path, budget):
+                digest.update(raw_line)
+                try:
+                    event = json.loads(raw_line)
+                except (json.JSONDecodeError, UnicodeDecodeError):
+                    malformed += 1
+                    continue
+                if not isinstance(event, dict):
+                    malformed += 1
+                    continue
+                event_count += 1
+                if event.get("type") == "session_meta":
+                    payload = event.get("payload")
+                    if not isinstance(payload, dict):
                         malformed += 1
                         continue
-                    event_count += 1
-                    if event.get("type") == "session_meta":
-                        payload = event.get("payload")
-                        if not isinstance(payload, dict):
-                            malformed += 1
-                            continue
-                        candidate_id = _safe_dimension(payload.get("id"), 512)
-                        if candidate_id and not conversation_id:
-                            conversation_id = candidate_id
-                content_hash = digest.hexdigest()
-                conversation_id = conversation_id or f"content-{content_hash}"
-                candidate = (machine, path, event_count, content_hash)
-                previous = selected.get(conversation_id)
-                if previous is None:
+                    candidate_id = _safe_dimension(payload.get("id"), 512)
+                    if candidate_id and not conversation_id:
+                        conversation_id = candidate_id
+            content_hash = digest.hexdigest()
+            conversation_id = conversation_id or f"content-{content_hash}"
+            candidate = (machine, path, event_count, content_hash)
+            previous = selected.get(conversation_id)
+            if previous is None:
+                selected[conversation_id] = candidate
+            else:
+                duplicates += 1
+                if candidate[2:] > previous[2:]:
                     selected[conversation_id] = candidate
-                else:
-                    duplicates += 1
-                    if candidate[2:] > previous[2:]:
-                        selected[conversation_id] = candidate
         return list(selected.values()), duplicates, malformed
 
     def _read_rollout(

{cli_consumption-0.4.1 → cli_consumption-0.4.3}/src/cli_consumption/api.py

@@ -19,6 +19,10 @@ from sqlalchemy.engine import Engine
 from starlette.types import ASGIApp, Message, Receive, Scope, Send
 
 from cli_consumption import __version__
+from cli_consumption.dashboard_layouts import (
+    DASHBOARD_LAYOUT_VERSION,
+    MAX_DASHBOARD_LAYOUT_BYTES,
+)
 from cli_consumption.models import (
     CURRENT_SNAPSHOT_SCHEMA,
     MAX_SNAPSHOT_RECORDS,
@@ -29,6 +33,7 @@ from cli_consumption.models import (
 )
 from cli_consumption.reporting_api import (
     CACHE_HEADERS,
+    EXPORT_REQUEST_BYTES,
     EXPORT_TIMEOUT_SECONDS,
     MAX_CONCURRENT_EXPORTS,
     MAX_CONCURRENT_REPORTS,
@@ -69,6 +74,7 @@ READINESS_TABLES = (
     "subagent_scopes",
     "ingestion_runs",
     "sync_receipts",
+    "dashboard_layouts",
 )
 READINESS_RESPONSE_TIMEOUT_SECONDS = 2.0
 READINESS_CONNECT_TIMEOUT_SECONDS = 2
@@ -94,6 +100,7 @@ SAFE_ROUTES = frozenset(
         "/api/v1/reporting/conversations",
         "/api/v1/reporting/conversation",
         "/api/v1/reporting/export",
+        "/api/v1/reporting/layout",
     }
 )
 SAFE_EXCEPTION_TYPES = frozenset(
@@ -354,9 +361,12 @@ class RequestSizeLimitMiddleware:
         if scope["type"] != "http":
             await self.app(scope, receive, send)
             return
+        path = str(scope.get("path", ""))
         maximum = (
-            REPORTING_REQUEST_BYTES
-            if str(scope.get("path", "")).startswith("/api/v1/reporting/")
+            EXPORT_REQUEST_BYTES
+            if path == "/api/v1/reporting/export"
+            else REPORTING_REQUEST_BYTES
+            if path.startswith("/api/v1/reporting/")
             else self.maximum
         )
         headers = dict(scope.get("headers", []))
@@ -405,6 +415,7 @@ def _configured_credentials(
     api_token: str | None,
     read_token: str | None,
     export_token: str | None,
+    layout_token: str | None,
 ) -> list[tuple[str, frozenset[str]]]:
     credentials: list[tuple[str, frozenset[str]]] = []
     configured_values: set[str] = set()
@@ -412,6 +423,7 @@ def _configured_credentials(
         (api_token, frozenset({"ingest"})),
         (read_token, frozenset({"read"})),
         (export_token, frozenset({"read", "export"})),
+        (layout_token, frozenset({"layout"})),
     ):
         if credential is None:
             continue
@@ -436,8 +448,11 @@ def create_app(
     *,
     read_token: str | None = None,
     export_token: str | None = None,
+    layout_token: str | None = None,
 ) -> SafeExceptionBoundary:
-    credentials = _configured_credentials(api_token, read_token, export_token)
+    credentials = _configured_credentials(
+        api_token, read_token, export_token, layout_token
+    )
     initialize_database(engine)
     if engine.dialect.name == "postgresql":
         probe_engine = create_postgresql_readiness_engine(
@@ -553,8 +568,12 @@ def create_app(
             "idempotent_snapshot_uploads": True,
             "dashboard_query_versions": [1],
             "dashboard_dataset_versions": [1],
+            "dashboard_layout_versions": [DASHBOARD_LAYOUT_VERSION],
+            "dashboard_layout_mutation_scope": "layout",
+            "max_dashboard_layout_bytes": MAX_DASHBOARD_LAYOUT_BYTES,
             "cursor_versions": [1],
             "max_reporting_request_bytes": REPORTING_REQUEST_BYTES,
+            "max_export_request_bytes": EXPORT_REQUEST_BYTES,
             "max_reporting_filter_values": MAX_FILTER_VALUES,
             "max_reporting_records": MAX_REPORTING_RECORDS,
             "max_reporting_scalar_bytes": MAX_REPORTING_SCALAR_BYTES,
@@ -599,6 +618,7 @@ def create_app(
         engine,
         authorize_read=require_scopes("read"),
         authorize_export=require_scopes("read", "export"),
+        authorize_layout=require_scopes("layout"),
     )
 
     return SafeExceptionBoundary(app)