logdig 0.1.1 → 0.2.0

package/QUICKSTART.md

@@ -18,13 +18,16 @@ In the wizard:
 - **Daily-notes folder:** choose the folder holding `YYYY-MM-DD.md` notes, for example `My Vault/Daily`.
 - **Summary cache:** choose `My Vault/LogDig` to keep the full summaries visible in Obsidian.
 - **Heading and length:** `# Projects` and `small` are a good first try. Your personal `# Log` stays separate.
+- **Anchor heading:** optionally choose `# Log` to create Projects at the end of the Log section, before the next same-level or higher-level heading. The default is `none` (blank): append at the end of the note. Missing anchors also append at the end; existing Projects sections stay in place.
 - **Timezone:** check that this is the timezone you journal in.
-- **Advanced settings:** leave them alone unless you use a custom Pi history folder, executable, or model.
+- **Advanced settings:** choose a custom Pi history folder, executable, model, thinking level, or concurrency here. Thinking defaults to no LogDig override; choose an explicit level such as `max` for a cost-first reasoning model if you are willing to wait longer. Concurrency defaults to four independent sessions; use `1` for sequential processing.
 - **Automatic capture:** choose **no** for now.
 - **Pi extension:** optional. The CLI works without it.
 
 Press Enter to accept defaults. A typo only re-asks that question. Review the choices before saving. Ctrl+C or declining confirmation leaves your settings unchanged. Setup writes settings only, not summaries or daily notes.
 
+To change a setting later, run `logdig config`. Choose its number or type a name such as `thinking`, then enter the new value. Available choices are shown first. Enter keeps the current value; each valid change saves immediately and refreshes the list. Type `q` and Enter to exit. Environment overrides are labeled and still take precedence. Redirected output stays read-only.
+
 ## 2. Check and preview
 
 ```sh
@@ -33,14 +36,25 @@ logdig status 7
 logdig backfill 1 --dry-run
 ```
 
-`status` shows what's already journaled, what needs updating or is missing, and which summaries would need a model request. It is read-only. The backfill preview shows projects, dates, destination files, cache hits, and which sessions need summarizing. **Neither command makes model requests or changes files.** The preview does not display transcript excerpts.
+`status` counts **logged**, **stale**, and **new** work blocks, with reusable summaries counted separately from blocks needing summarization. It is read-only. The backfill preview shows projects, dates, destination files, cache hits, and which blocks need summarizing. **Neither command makes model requests or changes files.** The preview does not display transcript excerpts.
 
-`1` means today in your chosen timezone. If nothing happened today:
+`1` means conversation activity today in your chosen timezone. Continuous overnight work stays on its starting day, so today's backfill may update yesterday's note. Resuming on a later date after at least four hours without conversation activity starts a linked continuation entry. If its preceding snapshot is missing, preview explicitly includes the earlier block needed for that link. If nothing happened today:
 
 ```sh
 logdig backfill 7 --dry-run
 ```
 
+To backfill older work without spending model requests on ongoing sessions, add `--skip-today`:
+
+```sh
+logdig backfill 7 --skip-today --dry-run
+logdig backfill 7 --skip-today
+logdig backfill 14 --skip-today
+logdig backfill 30 --skip-today
+```
+
+This selects complete calendar days through yesterday in your timezone and excludes whole work periods that continued today. Earlier periods in the same session can still be logged. Each wider run reuses unchanged summaries. The flag also works with `backfill all` and `status`.
+
 If no history is found at all, start a saved Pi session or select the right history folder in the wizard's advanced settings.
 
 ## 3. Make the first entry
@@ -51,11 +65,15 @@ Once the preview looks right, run the same range without `--dry-run`:
 logdig backfill 1
 ```
 
+To override the model and thinking level for one run, use `logdig backfill 1 --model provider/model --thinking max`. Use `--thinking default` to ignore a saved thinking override. More thinking may improve factual reconstruction, but can increase latency and token cost; available effort depends on the model. Saved thinking preferences also apply to `/journal` and automatic capture.
+
 This may send selected, redacted history to your Pi model and incur provider charges. Pi uses its existing authentication. If authentication fails, open Pi, run `/login`, then retry. Common secrets are redacted, but redaction is not a complete secret scanner.
 
-Open the daily-note date printed by the command. You will find project subheadings under `# Projects`, with timestamped summaries grouped beneath each one. Your existing text stays in place. Click a timestamp to open that entry's saved snapshot of all three summary lengths, or browse `<cache folder>/Sessions/<session-id>.md` for the latest session summary.
+Open the daily-note date printed by the command. You will find project subheadings under `# Projects`, with timestamped summaries grouped beneath each one. Your existing text stays in place. Click a timestamp to open that entry's saved snapshot of all three summary lengths, or browse `<cache folder>/Sessions/` for the latest work-block summaries.
+
+Repeating unchanged work reuses its summary and does not insert that entry again. Session renames and other bookkeeping do not trigger regeneration. New work in the same block updates that block's row; continuation blocks get separate linked entries. Manually edited blurbs and older snapshots in `Entries/` are preserved. Run only one save at a time against your journal, and let vault sync finish first.
 
-Repeating an unchanged session reuses its summary and does not insert that entry again. If a session evolves, LogDig updates its existing daily row to the latest summary, preserving manually edited blurbs and keeping older snapshots in `Entries/`. Run only one save at a time against your journal, and let vault sync finish first.
+If you used the older whole-session journal format, the first real save regenerates summaries for selected work blocks and migrates the old rows while keeping their linked snapshots. Preview before saving to check the dates and model work.
 
 ## Optional: use `/journal` inside Pi
 

package/README.md

@@ -2,7 +2,7 @@
 
 Remember what you worked on, without writing another status report.
 
-LogDig turns saved [Pi](https://pi.dev) sessions into short entries in your Obsidian daily notes. Your handwritten content stays in place. Each session also gets a cached Markdown note with **Small**, **Medium**, and **Large** summaries, so the details are there when you want them.
+LogDig turns saved [Pi](https://pi.dev) sessions into short entries in your Obsidian daily notes. Your handwritten content stays in place. Each work block also gets a cached Markdown note with **Small**, **Medium**, and **Large** summaries, so the details are there when you want them.
 
 No build step, runtime dependencies, or separate model credentials. Try a safe preview before sending any history to a model.
 
@@ -30,10 +30,11 @@ For a comfortable first try:
 - Choose the folder **inside your vault** holding your daily notes, such as `My Vault/Daily`.
 - Put the summary cache in `My Vault/LogDig` if you want to browse it in Obsidian.
 - Keep the `# Projects` heading and **small** summary unless you prefer otherwise. Your personal `# Log` section stays separate.
+- Optionally choose `# Log` as the anchor heading to create Projects after your Log section, before the next sibling heading. Leave it unset to append at the end.
 - Check the timezone. It determines the journal date and time.
 - Leave automatic capture **off** until you have tried a manual run. Pi integration is optional.
 
-`doctor` checks folder access and Pi availability without requesting a summary. It reports broken paths and a missing Pi executable together, with a next step for each.
+`doctor` checks folder access and Pi availability without requesting a summary. It reports broken paths and a missing Pi executable together, with a next step for each. Interactive terminals use colored Nerd Font status icons; set `LOGDIG_ICONS=0` for text markers, and `NO_COLOR` to disable color.
 
 **`--dry-run` never calls Pi, writes files, or creates folders.** It shows the projects, dates, destination files, matching cached summaries, and sessions that would need model requests. The preview does not print transcript excerpts.
 
@@ -43,7 +44,7 @@ When the preview looks right:
 logdig backfill 1
 ```
 
-This journals sessions whose last user message was **today in your configured timezone**, not a rolling 24-hour window. If today is quiet, preview seven days instead:
+This journals work blocks with conversation activity **today in your configured timezone**, not a rolling 24-hour window. Overnight work can update yesterday's note without moving it to today. If today is quiet, preview seven days instead:
 
 ```sh
 logdig backfill 7 --dry-run
@@ -61,7 +62,8 @@ My Vault/
 │   └── YYYY-MM-DD.md             your writing, plus project-grouped summaries under # Projects
 └── LogDig/
     ├── Sessions/
-    │   └── <session-id>.md       latest cached summary and provenance
+    │   ├── <session-id>.md       latest summary of the first work block
+│   └── <session-id>-<block-id>.md  latest summary of each continuation
     └── Entries/
         └── <entry-id>.md         snapshot of all three layers for a journal entry
 ```
@@ -88,13 +90,46 @@ More work on the same project.
 **[[<single-entry-id>|23:13]]**: One entry stays compact.
 ```
 
-A project with one entry puts the timestamp inline at the start of its summary. When a second entry arrives, both timestamps move to separate lines. Summary wording is preserved, including any edits you made.
+Each entry keeps its timestamp inline at the start of its summary, even when a project has multiple entries. Summary wording is preserved, including any edits you made.
 
-Obsidian displays each link as just the timestamp; clicking it opens that entry's detailed summary. The detailed note has two compact frontmatter properties when available: `sessionUsage` for recorded Pi-session usage and `logUsage` for LogDig's summary-generation requests, including intermediate chunks. Each shows cost, cached input, uncached input, output, and elapsed time, for example `"$3.73 ⚡12.2M ↑747k ↓62k · 1h 55m"`. Session duration is wall-clock time, including idle periods; LogDig duration is the time spent generating that summary. The original session and LogDig calls are counted separately. Historical summaries made before usage tracking have no `logUsage`; their cost cannot be recovered without making new requests. Unknown cost or token counts are omitted, not shown as zero. Keep the summary cache inside your vault so Obsidian can resolve these links. Daily summaries do not create headings or code fences; structured detail stays in the linked note. Custom section headings are supported, with project subheadings one level deeper (or bold project labels beneath a level-six heading).
+Obsidian displays each link as just the timestamp; clicking it opens that entry's detailed summary. The detailed note has two compact frontmatter properties when available: `sessionUsage` for recorded Pi usage within that work block and `logUsage` for LogDig's summary-generation requests, including intermediate chunks. Each shows cost, cached input, uncached input, output, and elapsed time, for example `"$3.73 ⚡12.2M ↑747k ↓62k · 1h 55m"`. Work-block duration is wall-clock time, including idle periods within the block; LogDig duration is the time spent generating that summary. The original session and LogDig calls are counted separately. Historical summaries made before usage tracking have no `logUsage`; their cost cannot be recovered without making new requests. Unknown cost or token counts are omitted, not shown as zero. Keep the summary cache inside your vault so Obsidian can resolve these links. Daily summaries do not create headings or code fences; structured detail stays in the linked note. Custom section headings are supported, with project subheadings one level deeper (or bold project labels beneath a level-six heading).
 
-The **last user message** supplies the date and time, even if the assistant finishes after midnight. Alternate session branches are included as explorations, not assumed to be the final result.
+### Overnight work and continuations
 
-Unchanged sessions reuse their summaries. The identifier in each timestamp link prevents duplicate entries, without HTML comments. Older project-name links and comment-wrapped entries are still recognized. When a session evolves, LogDig replaces its daily-note row with a link to the latest summary instead of adding another row. Previous linked snapshots remain unchanged in `Entries/`, and manually edited daily summaries are preserved. Changing the default does not override existing saved heading preferences; rerun setup to change them. An inline timestamp can be expanded when its project gains another entry. Headings inside frontmatter or fenced code are not insertion targets.
+A saved Pi session can contain several **work blocks**. A new block starts only at a new user message when both conditions hold:
+
+- Its local date is later than the current block's starting date.
+- At least **four hours** have passed since the preceding conversation activity. Assistant messages and tool results count as activity; session names, model switches, labels, usage records, and extension bookkeeping do not.
+
+The block's **starting date and time** supply its journal timestamp. Continuous work from 10pm to 2am stays one entry on the starting day. Returning at 11am after a long break creates a separate entry on the new day, with a **Continues** link to the preceding block's saved snapshot. Repeated saves are checkpoints, not boundaries. Alternate session branches are included as explorations, not assumed to be the final result.
+
+Date ranges select actual conversation activity, not just assigned journal dates. Today's backfill therefore catches assistant completion or continued work after midnight and updates yesterday's entry. If a selected continuation has no preceding snapshot, LogDig includes the missing earlier block(s) as prerequisites. Status and preview explicitly show these additions, including their summary-generation cost implications.
+
+### Backfill past work without touching today
+
+Use `--skip-today` with CLI `backfill` or `status` to select the last N **complete calendar days**, ending yesterday in your configured timezone. For example, `backfill 1 --skip-today` selects yesterday, and `backfill 7 --skip-today` selects seven days through yesterday. `all --skip-today` selects all past work.
+
+Work periods with conversation activity today are excluded entirely, even if they started before midnight. Earlier periods in the same session remain eligible. This avoids generating partial summaries or updating an ongoing period's existing cache or journal entry. Metadata-only activity does not exclude a period. Missing earlier continuation snapshots are still included as prerequisites.
+
+```sh
+logdig backfill 7 --skip-today --dry-run
+logdig backfill 7 --skip-today
+logdig backfill 14 --skip-today
+logdig backfill 30 --skip-today
+logdig backfill all --skip-today
+```
+
+Widening the range reuses unchanged summaries and fills the additional history. Status and preview show the selected dates and the number of excluded work periods; status JSON includes `timeframe.skipToday` and `totals.excludedToday` when the flag is set. Without the flag, the normal range still includes today.
+
+### Summary reuse and existing journals
+
+Summary freshness is based on the selected, redacted evidence and bounded earlier context, plus summary-processing version, timezone, and configured generation policy. Earlier context is drawn from the preceding block's evidence, not its generated summary, and is marked as background rather than work to repeat. Metadata-only changes do not trigger model requests; usage totals can refresh separately. Explicit LogDig model or thinking-level changes invalidate summaries, while changes to Pi's defaults do not force regeneration under the default policy.
+
+The identifier in each timestamp link prevents duplicate entries, without HTML comments. Older project-name links and comment-wrapped entries are still recognized. When a block evolves, LogDig replaces only that block's daily-note row with a link to the latest summary. Earlier blocks stay in place. Previous linked summary snapshots remain in `Entries/`, and manually edited daily summaries are preserved; usage metadata may be refreshed without regenerating the prose.
+
+**Upgrading from whole-session journaling:** old summaries use an incompatible cache key and need one regeneration per selected work block. A real save assigns legacy entries to their work blocks using their recorded journal timestamp, updates or relocates their rows as needed, and keeps the original linked snapshots. Run `backfill N --dry-run` first to see the scope and model work. Status and preview never migrate files.
+
+Saved heading preferences are not overridden by new defaults; rerun setup to change them. New entries keep their timestamps inline even when their project already has entries. Headings inside frontmatter or fenced code are not insertion targets.
 
 Missing daily-note and cache folders are created only by a real save. Raw Pi history stays in Pi's storage.
 
@@ -105,26 +140,29 @@ After a global install or `npm link`, use `logdig` from any directory. All comma
 ```text
 logdig init                         configure paths, summaries, and optional Pi integration
 logdig doctor                       check paths and Pi, with no model request
-logdig config                       show effective settings and environment overrides
+logdig config                       edit settings by number or name; q quits
 logdig --version                    show the installed version
 logdig backfill                     journal the last 3 calendar days
 logdig backfill 7 --dry-run          preview seven days without changing anything
 logdig backfill all --dry-run        preview every discoverable saved session
 logdig backfill 7                    journal seven days
-logdig backfill 7 --model provider/model
+logdig backfill 7 --skip-today       journal seven complete days through yesterday
+logdig backfill 7 --model provider/model --thinking max
 logdig backfill 7 --model default    ignore a saved model override for this run
+logdig backfill 7 --thinking default ignore a saved thinking override for this run
 logdig status                       show coverage for the last 3 calendar days
 logdig status 7                     show coverage for seven days
+logdig status 7 --skip-today        show coverage excluding work active today
 logdig status all --json            output coverage for every saved session as JSON
 logdig pi-install                   install the /journal extension
 logdig pi-uninstall                 remove it without deleting notes or summaries
 ```
 
-`--help` works before setup, including `logdig init --help`, `logdig backfill --help`, and `logdig status --help`.
+`--help` works before setup, including `logdig init --help`, `logdig config --help`, `logdig backfill --help`, and `logdig status --help`.
 
-`status` is a read-only coverage check. It reports current journal entries, sessions that need updating, missing entries, and whether each summary is cached or needs a model request. It uses the last user message date in your configured timezone. It never calls a model or writes files; scan warnings make the command exit nonzero so incomplete coverage is clear.
+`status` is a read-only coverage check. It counts work blocks: **logged** means the expected entry is present, **stale** means a previous snapshot exists but the entry needs updating, and **new** means no previous block snapshot was found. Summaries are reported separately as reusable or needing summarization. These are block counts, not exact request counts. It selects blocks by conversation activity in your configured timezone, including overnight updates and any missing continuation prerequisites. It never calls a model or writes files; scan warnings make the command exit nonzero so incomplete coverage is clear. JSON retains `sessions` as the result array, with one row per work block and both `sessionId` and `blockId`; totals use `logged`, `stale`, `new`, and `needsSummarizing`.
 
-New summaries may incur provider charges. Large sessions are summarized in chunks and may need several model requests each. Backfill shows which project it is working on before the model completes. Failed sessions are reported, other sessions continue, and the command exits nonzero if anything needs attention. Fix the issue and rerun the same command; completed summaries are reused, even if a previous attempt failed to write a daily note.
+New summaries may incur provider charges. Large work blocks are summarized in chunks and may need several model requests each. Backfill shows which project it is working on before the model completes. Failed sessions are reported, other sessions continue, and the command exits nonzero if anything needs attention. Fix the issue and rerun the same command; completed summaries are reused, even if a previous attempt failed to write a daily note.
 
 Run one backfill at a time against a given journal. The extension prevents overlapping saves within one Pi process, but separate CLI/Pi processes and external note editors are not coordinated. Let an existing save or vault sync finish first.
 
@@ -157,6 +195,10 @@ CLI backfill uses Pi's normal startup model and existing authentication unless y
 
 `/journal` uses the current Pi model, including registered providers, unless a LogDig override is set. The preview needs no available model or authentication. `doctor` checks the executable but does not validate model authentication; open Pi and run `/login` if a real save reports an authentication problem.
 
+Choose a **thinking level** alongside the model in setup's advanced settings, or use `--thinking` for one CLI backfill. Supported values are `off`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`; `default` removes the LogDig override. Saved settings use `thinkingLevel`, and `PI_JOURNAL_THINKING` overrides it. This applies to intermediate timeline extraction and final summaries, including `/journal` and automatic capture. Support and effort mapping depend on the model and Pi provider, so `max` is not necessarily a distinct supported tier on every model.
+
+More thinking can help separate proposals, failed attempts, and verified outcomes, but can increase latency and token cost. Explicitly enabled thinking allows CLI requests up to 30 minutes each, rather than the usual five; providers may impose their own timeouts. With no LogDig override, CLI backfill keeps Pi's startup thinking policy, while `/journal` keeps its existing provider-default behavior and does **not** inherit the active session's thinking level. Select an explicit level for consistent control in both paths. Changing it causes selected cached summaries to need regeneration; preview first to see the scope.
+
 Selected user prompts, assistant conclusions, tool actions, test results, and error excerpts are sent to the chosen model after common credential redaction. System prompts, hidden reasoning, and image payloads are excluded. **Redaction is not a comprehensive secret scanner.** Review your provider's data handling before processing sensitive sessions or enabling automatic capture. Summaries can also contain private project details, so treat your cache and vault accordingly.
 
 Model summaries are aids to memory, not proof of completed work. Keep Pi history as the source of truth.
@@ -169,13 +211,36 @@ Settings contain paths and preferences, never provider credentials:
 - macOS: `~/Library/Application Support/LogDig/settings.json`
 - Windows: `%APPDATA%\LogDig\settings.json`
 
+### Change one setting
+
+Run `logdig config` in a terminal to see a numbered list of settings and current saved values. Enter a number (including `10`, `11`, or `12`) or a setting name such as `thinking` or `parallel`. Allowed values and a short explanation appear before you enter a new value. Enter keeps the current value; invalid input re-asks only that field. Each valid change saves immediately and shows the updated list. Type `q` and Enter at either prompt to exit. Ctrl+C or ending input also exits; completed saves are kept.
+
+Use `default` to clear a model or thinking override, and `none` to clear the anchor heading. Active environment overrides are labeled next to their saved preferences and are never copied into the settings file. Changing a preference does not defeat its environment override. Enabling automatic capture still requires the Pi extension (`logdig pi-install`). Configuration never requests summaries, writes notes, or creates note/cache folders. Run `init` first if those folders have not been configured.
+
+The menu uses Node's built-in line prompts with no runtime dependencies. Save confirmations follow the existing Nerd Font, `LOGDIG_ICONS=0`, and `NO_COLOR` conventions. `TERM=dumb` uses plain prompts. When input or output is redirected, `config` stays read-only and shows effective settings as formatted JSON with the settings path and override names, as before. For example: `logdig config > settings-report.txt`.
+
+### Journal preferences
+
+`dailyHeader` selects the section for project entries (default `# Projects`). The optional `dailyHeaderAnchor` selects where to create that section:
+
+```json
+{
+  "dailyHeader": "# Projects",
+  "dailyHeaderAnchor": "# Log"
+}
+```
+
+This inserts a new Projects section after the entire Log section, including its subheadings, and before the next same-level or higher-level heading. Use the same heading level for both settings to keep the sections as siblings. The anchor must be one Markdown heading, including its `#` level, and matches the first occurrence outside frontmatter and fenced code. The default is `""` (blank): append at the end of the note. A missing anchor also falls back to the end. Existing Projects sections stay where they are; this setting does not relocate them or regenerate summaries. Run `config` to edit the anchor or choose `none` to clear it. `PI_JOURNAL_DAILY_HEADER_ANCHOR` overrides the saved setting, including an empty value to clear it.
+
+Backfill and automatic capture process up to **four independent sessions concurrently** by default. Set `concurrency` in the settings file or choose “Maximum parallel sessions” in setup's advanced settings. It must be a positive integer; use `1` for sequential processing or to reduce provider rate-limit pressure. Work blocks and extraction requests within each session remain sequential to preserve continuation links. Shared daily-note updates are serialized to avoid overwriting entries. CLI result rows appear in session order even when parallel sessions finish out of order. Session numbers are zero-padded to match the total, for example `[01/65]`, followed by the date, a fixed-width status column, project, and short session ID. Each work period has one final result row; resumed sessions can have multiple dated rows with the same session number. In interactive terminals, a bounded live panel below the permanent results shows active workers (`CHECKING` or `SUMMARIZING`), the completed-session count, and sessions that are `DONE` but awaiting ordered output. The panel redraws in place and clears when the run finishes. A slow earlier session can delay display of later result rows, but does not block processing or live feedback. Redirected output and `TERM=dumb` use plain `Active` and `Finished` log lines instead of cursor controls. Dry-run output stays static. Blocks outside the selected range that are needed for continuation links are marked `prerequisite`. The final checked count distinguishes work blocks from sessions. Status reports retain chronological session order. Changing concurrency does not invalidate cached summaries. Avoid running separate LogDig commands against the same notes at the same time; the write queue is local to one run.
+
 Set `LOGDIG_CONFIG_PATH` to choose another settings file. Existing `PI_JOURNAL_*` environment variables remain supported and override saved values. Setup, `config`, and `doctor` name active overrides so you can see why a saved preference is not taking effect.
 
 - **No saved history found:** create a saved Pi session, or use setup's advanced settings to select your history folder. This is especially useful with a custom Pi session directory.
-- **Notes went to the wrong folder:** run `config`, check environment overrides, then run `init` to choose the daily-notes folder rather than the vault root.
-- **Pi cannot start:** run `doctor`, then use setup's advanced settings to set the executable path.
+- **Notes went to the wrong folder:** run `config`, check environment overrides, and edit the daily-notes folder rather than the vault root.
+- **Pi cannot start:** run `doctor`, then use `config` to set the executable path.
 - **A summary or note failed:** read the error, fix the path, permissions, authentication, or provider issue, and rerun. Successful cached work is kept.
-- **Want to stop automatic capture:** rerun `init` and choose “no”, or set `PI_JOURNAL_AUTO=0`.
+- **Want to stop automatic capture:** run `config` and set automatic capture to `no`, or set `PI_JOURNAL_AUTO=0`.
 
 ## Development