@martian-engineering/lossless-claw 0.8.2 → 0.9.0

package/docs/architecture.md

@@ -190,7 +190,7 @@ Files embedded in user messages (typically via `<file>` blocks from tool output)
 1. Parse file blocks from message content.
 2. For each block exceeding `largeFileTokenThreshold` (default 25k tokens):
    - Generate a unique file ID (`file_` prefix)
-   - Store the content to `~/.openclaw/lcm-files/<conversation_id>/<file_id>.<ext>`
+   - Store the content to `largeFilesDir/<conversation_id>/<file_id>.<ext>` (default `~/.openclaw/lcm-files/...`)
    - Generate a ~200 token exploration summary (structural analysis, key sections, etc.)
    - Insert a `large_files` record with metadata
    - Replace the file block in the message with a compact reference

package/docs/configuration.md

@@ -51,6 +51,7 @@ Most installations only need to override a handful of keys. If you want a comple
   "circuitBreakerThreshold": 5,
   "circuitBreakerCooldownMs": 1800000,
   "fallbackProviders": [],
+  "proactiveThresholdCompactionMode": "deferred",
   "cacheAwareCompaction": {
     "enabled": true,
     "maxColdCacheCatchupPasses": 2,
@@ -103,7 +104,7 @@ openclaw plugins install --link /path/to/lossless-claw
 | `enabled` | `boolean` | `true` | `LCM_ENABLED` | Enables or disables lossless-claw without uninstalling it. |
 | `databasePath` | `string` | `${OPENCLAW_STATE_DIR}/lcm.db` | `LCM_DATABASE_PATH` | Preferred path for the SQLite database. |
 | `dbPath` | `string` | alias of `databasePath` | `LCM_DATABASE_PATH` | Legacy alias for `databasePath`. Prefer `databasePath` in new config. |
-| `largeFilesDir` | `string` | `${OPENCLAW_STATE_DIR}/lcm-files` | `LCM_LARGE_FILES_DIR` | Directory where large-file text payloads are persisted. Automatically follows the active state directory. |
+| `largeFilesDir` | `string` | `${OPENCLAW_STATE_DIR}/lcm-files` | `LCM_LARGE_FILES_DIR` | Directory where externalized large files and inline images are persisted. Automatically follows the active state directory. |
 | `ignoreSessionPatterns` | `string[]` | `[]` | `LCM_IGNORE_SESSION_PATTERNS` | Session-key glob patterns that skip LCM entirely. |
 | `statelessSessionPatterns` | `string[]` | `[]` | `LCM_STATELESS_SESSION_PATTERNS` | Session-key glob patterns that may read from LCM but never write to it. |
 | `skipStatelessSessions` | `boolean` | `true` | `LCM_SKIP_STATELESS_SESSIONS` | Enforces `statelessSessionPatterns` when enabled. |
@@ -111,6 +112,7 @@ openclaw plugins install --link /path/to/lossless-claw
 | `timezone` | `string` | `TZ` or system timezone | `TZ` | IANA timezone used for timestamp rendering in summaries. |
 | `pruneHeartbeatOk` | `boolean` | `false` | `LCM_PRUNE_HEARTBEAT_OK` | Retroactively removes `HEARTBEAT_OK` turn cycles from persisted storage. |
 | `transcriptGcEnabled` | `boolean` | `false` | `LCM_TRANSCRIPT_GC_ENABLED` | Enables transcript rewrite GC during `maintain()`; disabled by default so transcript rewrites stay opt-in. |
+| `proactiveThresholdCompactionMode` | `"deferred" \| "inline"` | `"deferred"` | `LCM_PROACTIVE_THRESHOLD_COMPACTION_MODE` | Controls whether proactive threshold compaction is deferred into maintenance debt by default or run inline for legacy behavior. |
 
 > **Multi-profile note:** `OPENCLAW_STATE_DIR` (set by the host OpenClaw gateway) controls where state is stored. When two gateways run on the same host (e.g. separate bot personas), each gateway sets its own `OPENCLAW_STATE_DIR` and lossless-claw automatically uses that directory for the database, large-file payloads, auth-profile lookups, and legacy secrets — no per-profile plugin config is needed.
 
@@ -164,9 +166,11 @@ openclaw plugins install --link /path/to/lossless-claw
 | Key | Type | Default | Env override | Purpose |
 | --- | --- | --- | --- | --- |
 | `cacheAwareCompaction.enabled` | `boolean` | `true` | `LCM_CACHE_AWARE_COMPACTION_ENABLED` | Defers incremental leaf compaction more aggressively when prompt-cache telemetry indicates a hot cache. |
+| `cacheAwareCompaction.cacheTTLSeconds` | `integer` | `300` | `LCM_CACHE_TTL_SECONDS` | Fallback cache TTL used when deferred Anthropic compaction has provider/model telemetry but no explicit runtime cache-retention window. |
 | `cacheAwareCompaction.maxColdCacheCatchupPasses` | `integer` | `2` | `LCM_MAX_COLD_CACHE_CATCHUP_PASSES` | Maximum bounded catch-up passes allowed in one maintenance cycle when cache telemetry is cold. |
 | `cacheAwareCompaction.hotCachePressureFactor` | `number` | `4` | `LCM_HOT_CACHE_PRESSURE_FACTOR` | Multiplier applied to the hot-cache leaf trigger before raw-history pressure overrides cache preservation. |
 | `cacheAwareCompaction.hotCacheBudgetHeadroomRatio` | `number` | `0.2` | `LCM_HOT_CACHE_BUDGET_HEADROOM_RATIO` | Minimum fraction of the real token budget that must remain free before hot-cache incremental compaction is skipped entirely. |
+| `cacheAwareCompaction.coldCacheObservationThreshold` | `integer` | `3` | `LCM_COLD_CACHE_OBSERVATION_THRESHOLD` | Consecutive cold observations required before non-explicit cache misses are treated as truly cold. This dampens one-off routing noise and provider failover blips. |
 
 #### `dynamicLeafChunkTokens`
 
@@ -240,6 +244,28 @@ Lossless-claw treats OpenClaw reset commands differently:
 
 This keeps long-term history available while still giving users a real clean-slate reset.
 
+### Deferred proactive compaction
+
+Lossless-claw now defaults `proactiveThresholdCompactionMode` to `deferred`.
+
+- deferred mode records a single coalesced maintenance debt row per conversation
+- deferred mode persists provider/model/cache telemetry so Anthropic-family sessions can avoid rewriting a still-hot prompt cache
+- `maintain()` can still process non-prompt-mutating work when the host explicitly opts in to deferred execution, but it leaves prompt-mutating debt pending while Anthropic cache is still hot
+- `assemble()` consumes deferred prompt-mutating debt pre-assembly once the cache is cold or the next turn is already approaching overflow
+- `/lcm status` / `/lossless status` shows the current maintenance state, including pending/running/last-failure details
+- status output also surfaces the latest API/cache telemetry so operators can see whether a deferred debt item is being preserved for cache-safety reasons
+- set `proactiveThresholdCompactionMode` to `inline` only if you need the legacy inline proactive compaction behavior for compatibility
+
+### `/lcm rotate`
+
+`/lcm rotate` exists for a different use case than `/new` or `/reset`:
+
+- `/new` keeps the same active LCM conversation row and only prunes context.
+- `/reset` changes OpenClaw session flow, which is sometimes more disruptive than users want.
+- `/lcm rotate` keeps the live OpenClaw session identity and the same active LCM conversation row, but rewrites the backing transcript into a compact preserved-tail form.
+
+Before rotating, Lossless-claw replaces one rolling `rotate-latest` SQLite backup. It then rewrites the current session transcript and checkpoints the same conversation at the new transcript frontier so bootstrap does not replay the dropped transcript history. Existing summaries, context items, and conversation identity stay in place. If you want additional timestamped snapshots, run `/lcm backup` explicitly before `/lcm rotate`.
+
 ## Environment-only knobs outside plugin config
 
 These settings are not part of `plugins.entries.lossless-claw.config`, but they still affect the system:
@@ -271,6 +297,12 @@ cp "${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/lcm.db" "${OPENCLAW_STATE_DIR:-$HOME/
 sqlite3 "${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/lcm.db" ".backup ${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/lcm.db.backup"
 ```
 
+Or from a supported OpenClaw chat/native command surface:
+
+```text
+/lcm backup
+```
+
 ## Disabling lossless-claw
 
 To disable the plugin but keep it installed:

package/openclaw.plugin.json

@@ -73,6 +73,10 @@
       "label": "Large Files Directory",
       "help": "Directory for persisting large-file text payloads (default: <stateDir>/lcm-files). Uses OPENCLAW_STATE_DIR when set."
     },
+    "largeFilesDir": {
+      "label": "Large Files Directory",
+      "help": "Directory for externalized large files and inline-image payloads (default: ~/.openclaw/lcm-files)"
+    },
     "ignoreSessionPatterns": {
       "label": "Ignored Sessions",
       "help": "Glob patterns for session keys to exclude from LCM storage"
@@ -149,6 +153,10 @@
       "label": "Cache-Aware Compaction",
       "help": "When enabled, hot prompt cache defers incremental compaction while cold cache allows bounded catch-up passes"
     },
+    "cacheAwareCompaction.cacheTTLSeconds": {
+      "label": "Short Cache TTL (seconds)",
+      "help": "Treat short-lived prompt-cache entries as hot for this many seconds before deferred compaction may rewrite the prompt"
+    },
     "cacheAwareCompaction.maxColdCacheCatchupPasses": {
       "label": "Cold Cache Catch-Up Passes",
       "help": "Maximum incremental leaf passes allowed in one maintenance cycle when prompt cache is cold"
@@ -161,6 +169,10 @@
       "label": "Hot Cache Budget Headroom",
       "help": "Fraction of the real token budget that must remain free before hot-cache incremental compaction is skipped entirely"
     },
+    "cacheAwareCompaction.coldCacheObservationThreshold": {
+      "label": "Cold Cache Observation Threshold",
+      "help": "Consecutive cold observations required before non-explicit cache misses are treated as truly cold"
+    },
     "dynamicLeafChunkTokens.enabled": {
       "label": "Dynamic Leaf Chunk Tokens",
       "help": "When enabled, incremental compaction uses a larger working leaf chunk in busy sessions and keeps the static floor in quieter sessions"
@@ -181,6 +193,10 @@
       "label": "Transcript GC",
       "help": "Enable transcript rewrite GC during maintain(); disabled by default"
     },
+    "proactiveThresholdCompactionMode": {
+      "label": "Proactive Threshold Compaction Mode",
+      "help": "Choose deferred compaction debt by default or keep legacy inline proactive compaction"
+    },
     "fallbackProviders": {
       "label": "Fallback Providers",
       "help": "Explicit fallback provider/model pairs for compaction summarization (e.g., [{\"provider\": \"anthropic\", \"model\": \"claude-haiku-4-5\"}])"
@@ -249,6 +265,9 @@
       "dbPath": {
         "type": "string"
       },
+      "largeFilesDir": {
+        "type": "string"
+      },
       "ignoreSessionPatterns": {
         "type": "array",
         "items": {
@@ -324,6 +343,10 @@
           "enabled": {
             "type": "boolean"
           },
+          "cacheTTLSeconds": {
+            "type": "integer",
+            "minimum": 1
+          },
           "maxColdCacheCatchupPasses": {
             "type": "integer",
             "minimum": 1
@@ -336,6 +359,10 @@
             "type": "number",
             "minimum": 0,
             "maximum": 0.95
+          },
+          "coldCacheObservationThreshold": {
+            "type": "integer",
+            "minimum": 1
           }
         }
       },
@@ -361,6 +388,13 @@
       "transcriptGcEnabled": {
         "type": "boolean"
       },
+      "proactiveThresholdCompactionMode": {
+        "type": "string",
+        "enum": [
+          "deferred",
+          "inline"
+        ]
+      },
       "databasePath": {
         "description": "Path to LCM SQLite database (preferred key; alias of dbPath, default: <OPENCLAW_STATE_DIR>/lcm.db)",
         "type": "string"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@martian-engineering/lossless-claw",
-  "version": "0.8.2",
+  "version": "0.9.0",
   "description": "Lossless Context Management plugin for OpenClaw — DAG-based conversation summarization with incremental compaction",
   "type": "module",
   "main": "dist/index.js",

package/skills/lossless-claw/SKILL.md

@@ -13,7 +13,7 @@ Start here:
 2. If they need a quick health check, tell them to run `/lossless` (`/lcm` is the shorter alias).
 3. If they suspect summary corruption or truncation, use `/lossless doctor`.
 4. If they want high-confidence junk/session cleanup guidance, use `/lossless doctor clean` before recommending any deletes.
-5. If they ask how `/new` or `/reset` interacts with LCM, read the session-lifecycle reference before answering.
+5. If they ask how `/new`, `/reset`, or `/lossless rotate` interacts with LCM, read the session-lifecycle reference before answering.
 6. Load the relevant reference file instead of improvising details from memory.
 
 Reference map:
@@ -22,7 +22,7 @@ Reference map:
 - Internal model and data flow: `references/architecture.md`
 - Diagnostics and summary-health workflow: `references/diagnostics.md`
 - Recall tools and when to use them: `references/recall-tools.md`
-- `/new` and `/reset` behavior with current lossless-claw session mapping: `references/session-lifecycle.md`
+- `/new`, `/reset`, and `/lossless rotate` behavior with current lossless-claw session mapping: `references/session-lifecycle.md`
 
 Working rules:
 

package/skills/lossless-claw/references/config.md

@@ -87,6 +87,7 @@ Good defaults:
 - `maxColdCacheCatchupPasses: 2`
 - `hotCachePressureFactor: 4`
 - `hotCacheBudgetHeadroomRatio: 0.2`
+- `coldCacheObservationThreshold: 3`
 
 Operationally:
 
@@ -187,7 +188,6 @@ Why it matters:
 
 - defaults to `${OPENCLAW_STATE_DIR}/lcm-files`; on multi-profile hosts each profile stores files in its own state directory automatically
 - override with `LCM_LARGE_FILES_DIR` or set `largeFilesDir` in plugin config when you want an explicit path
-
 ### `largeFileThresholdTokens`
 
 Threshold for externalizing oversized tool/file payloads out of the main transcript into large-file storage.
@@ -206,6 +206,18 @@ Why it matters:
 - keep this off unless you want transcript GC to mutate the live session file during maintenance
 - the default is `false`
 
+### `proactiveThresholdCompactionMode`
+
+Controls whether proactive threshold compaction is deferred into maintenance debt or kept inline for legacy behavior.
+
+Why it matters:
+
+- `deferred` is the default and avoids foreground turn stalls by recording one coalesced maintenance row per conversation
+- `deferred` also stores provider/model/cache telemetry so Anthropic-family sessions can avoid rewriting a still-hot prompt cache
+- `inline` preserves the legacy foreground compaction path for hosts that do not yet support deferred execution
+- `/lossless status` and `/lcm status` surface pending/running/last-failure maintenance state so operators can see when compaction is queued
+- background `maintain()` can still do non-prompt-mutating work, but prompt-mutating debt is consumed pre-assembly once cache is cold or the next turn is already approaching overflow
+
 ## Compaction timing and shape
 
 ### `contextThreshold`
@@ -330,6 +342,19 @@ Why it matters:
 
 Defers incremental leaf compaction more aggressively when prompt-cache telemetry indicates a hot cache.
 
+#### `cacheAwareCompaction.cacheTTLSeconds`
+
+Fallback cache TTL used when deferred Anthropic compaction has provider/model telemetry but no explicit runtime cache-retention window.
+
+Why it matters:
+
+- lets cache-safe deferred compaction stay conservative even when the host only knows that the turn was Anthropic-family, not the exact retention tier
+- keeps prompt-mutating debt pending until the cached prefix is likely cold
+
+Default:
+
+- `300`
+
 #### `cacheAwareCompaction.maxColdCacheCatchupPasses`
 
 Maximum bounded catch-up passes allowed in one maintenance cycle when cache telemetry is cold.
@@ -360,6 +385,19 @@ Default:
 
 - `0.2`
 
+#### `cacheAwareCompaction.coldCacheObservationThreshold`
+
+Consecutive cold observations required before non-explicit cache misses are treated as truly cold.
+
+Why it matters:
+
+- prevents a single OpenRouter routing miss or provider failover blip from immediately triggering cold-cache catch-up
+- explicit cache breaks still count as cold immediately
+
+Default:
+
+- `3`
+
 ### `dynamicLeafChunkTokens`
 
 #### `dynamicLeafChunkTokens.enabled`

package/skills/lossless-claw/references/session-lifecycle.md

@@ -1,4 +1,4 @@
-# Session lifecycle (`/new` and `/reset`)
+# Session lifecycle (`/new`, `/reset`, and `/lossless rotate`)
 
 This reference describes the current behavior on `main`.
 
@@ -7,6 +7,7 @@ This reference describes the current behavior on `main`.
 For stock `lossless-claw` on current main:
 
 - OpenClaw handles `/new` and `/reset` as session-reset operations.
+- `lossless-claw` handles `/lossless rotate` (`/lcm rotate`) as transcript maintenance on the current conversation.
 - `lossless-claw` does **not** currently register its own `before_reset` hook or a custom reset policy.
 - `lossless-claw` prefers **`sessionKey`** as the stable identity for an LCM conversation.
 - When the same `sessionKey` reappears with a new `sessionId`, `lossless-claw` updates the stored `sessionId` on the existing LCM conversation row instead of creating a brand-new LCM conversation.
@@ -21,6 +22,7 @@ So today:
 
 - `/new` and `/reset` can reset the runtime session
 - but LCM history may continue in the same conversation row if the chat/thread keeps the same `sessionKey`
+- `/lossless rotate` keeps that same conversation row, summaries, and context items in place while compacting only the live transcript backing
 
 ## Why
 
@@ -32,14 +34,27 @@ Current lossless-claw conversation resolution does this:
 
 That behavior preserves continuity across session resets for the same chat identity.
 
+## `/lossless rotate`
+
+`/lossless rotate` is distinct from `/new` and `/reset`.
+
+- it does **not** create a fresh LCM conversation row
+- it does **not** archive the current conversation
+- it **does** create or replace the rolling `rotate-latest` SQLite backup first
+- it **does** rewrite the current transcript into a compact suffix-preserving form
+- it **does** refresh bootstrap state on the same conversation so dropped transcript history is not replayed
+- it **does** preserve the current conversation id, summary DAG, and active context items
+
+This makes rotate the lightweight option when the problem is transcript bloat rather than LCM conversation structure.
+
 ## Important limitation
 
-There is currently **no plugin-specific `/new` vs `/reset` split** in stock lossless-claw docs or runtime behavior.
+There is still **no plugin-specific `/new` vs `/reset` split** in stock lossless-claw docs or runtime behavior.
 
 If someone is asking for semantics like:
 
-- `/new` keeps LCM history but rotates transcript
-- `/reset` archives old LCM conversation and starts a new one
+- `/new` gives them a fresh LCM conversation row
+- `/reset` archives old LCM conversation and starts a new one for the same stable `sessionKey`
 
 that is a **design/spec topic**, not current stock behavior.
 
@@ -48,6 +63,7 @@ that is a **design/spec topic**, not current stock behavior.
 When answering users:
 
 - do not promise that `/new` or `/reset` clears LCM history
+- explain that `/lossless rotate` compacts the current transcript without splitting the LCM conversation
 - explain that current stock behavior follows `sessionKey` continuity
 - if they need a truly separate LCM history, use a different session key context (for example a different chat/thread/binding) or explicit non-MVP migration/surgery tools