@vrtmrz/livesync-commonlib 0.1.42 → 0.1.44

package/docs/database-diagnostics.md

@@ -0,0 +1,69 @@
+---
+date: 2026-10-19
+commonlib_version: '0.1.43'
+livesync_version: '1.0.38'
+status: unreleased
+---
+
+# Database diagnostics
+
+The focused `/diagnostics` entry supplies read-only database usage analysis and integrity inspection. Hosts choose original names or temporary pseudonyms before each run, provide progress and cancellation controls, and present the resulting reports. Analysis uses the original identifiers; name projection happens afterwards.
+
+Usage retains the existing available-revision selection and incremental chunk accounting. The first processed reference counts a chunk as unique; later references count it as shared. Sizes count JavaScript string units, as in the existing report. These columns do not measure physical database size or establish whether a chunk can be deleted.
+
+Usage progress emits a `reading` phase while it discovers revisions, with `idsProcessed` and `revisionsProcessed` retained for existing consumers. `processed` counts settled available-revision reads, including failures; `total` grows as available revisions are discovered and is not final until enumeration finishes. The analyser does not pre-scan to calculate a total. The separate `accounting` phase has a fixed work-item total and yields between batches so the host can respond to cancellation. Cancellation during reading retains collected evidence; cancellation during accounting omits incomplete rows and orphan statistics. A failing progress callback does not change the result.
+
+Integrity inspection enumerates actual stored document IDs, including logically deleted normal-file Metadata and current conflict branches. It uses the same identity classifier as the existing Metadata inspector. It reads content only for groups with identity discrepancies, conflicting branches, duplicate path claims, or differences between the inspected local and remote Metadata. Content comparisons use reconstructed bytes, including the Vault file where available. Missing, unreadable, and uninspected content remain distinct outcomes. Equal content within this scope does not establish that an entry is safe to delete.
+
+Standard integrity observations include recorded document type, size, and total, unique, and embedded Chunk-reference counts. When suspect bytes have already been read, they also include measured decoded-content and Vault sizes, their differences from recorded or reconstructed sizes, and the Vault-to-Metadata modification-time difference. Case-only and NFC-equivalent path group membership is included once for each group. These facts use the normal decoded and decrypted Metadata and reads already needed for inspection; they do not add database, HTTP, or Vault reads. Facts which were not inspected remain absent or `not-checked`.
+
+Integrity inspection classifies up to four Metadata documents or processes up to four suspect content groups concurrently. Hosts can set `concurrency` between one and sixteen for comparison or resource limits. Local and remote Metadata scans remain separate, and results retain enumeration order regardless of completion order. Sources with `getMany` fetch current Metadata and required Chunks in groups of up to 64 documents. Metadata classification follows each bulk read; content groups read their Chunk batches and Vault files sequentially. The CouchDB reader separately decodes up to four documents per bulk response, so `concurrency` limits analyser work rather than total concurrent decryption across active groups. Exact conflict revisions still use individual reads. Cancellation stops dispatching new work and drains active reads before releasing the remote source; collected findings remain available.
+
+Integrity progress identifies the phase and source. Metadata progress counts completed documents, including excluded IDs, against the source's `doc_count` when available. Content progress counts completed suspect groups against their known total. Missing document totals remain unknown until enumeration finishes. A source changed during inspection can change the total and makes the report partial. Hosts should label these separate units rather than combine them into one percentage.
+
+The optional `onLog` callback receives phase start and finish events, with progress events at most once every five seconds while work completes. Events contain counts, elapsed time, the concurrency limit, active and peak reads, read operations, and summed read duration. Read operations count requested documents; a bulk read counts once towards active reads and elapsed read duration. Summed read duration includes overlapping reads. HTTP request counts belong to the host's injected fetch boundary and can differ from document reads. Grouping has its own timing event. These events contain no paths, document identifiers, connection details, or raw errors. Progress and log callback failures do not interrupt inspection.
+
+The first implementation supports the local database and CouchDB. Internal-file, customisation, and plug-in storage Metadata, older revision history, and other remote backends are explicitly outside integrity inspection. Usage continues to cover the revisions available to its local source. A remote failure retains the local findings and marks the integrity report partial.
+
+`includeRemote` defaults to true. Setting it to false does not open the remote source: local identity, conflict, and duplicate-path checks still run, with content comparisons against the Vault for suspect groups. Remote coverage and cross-source comparisons remain `not-checked`, and `scope.remote` is `excluded` in JSON and Markdown reports. A local-only run can complete within that declared scope; excluding a remote is distinct from a failed connection. Hosts should collect this choice for each run rather than change synchronisation settings.
+
+The CouchDB reader only sends GET requests. It reads existing synchronisation parameters when decryption needs a Security Seed and never creates missing parameters, initialises a database, or invokes replication admission. Local content reads do not fetch or cache remote Chunks. Owned requests are cancelled and released when the run ends; host-native request adapters may finish an already dispatched request after cancellation, but its result is discarded.
+
+Bulk reads use `_all_docs` with selected `keys`, `include_docs`, and optional `conflicts`. The reader limits each request to 64 documents and a 6,000-character encoded URL; an individual key which exceeds that URL limit uses the existing single-document GET. ID enumeration does not include document bodies, so unrelated Chunks are not downloaded. Embedded Chunks require no request, and repeated Chunk IDs within one batch share a read. Missing rows, deleted documents, unreadable results, and failed HTTP requests retain distinct evidence; a failed batch does not establish that its documents are absent. Bulk results use the same decryption and decompression as individual reads and are released after each batch.
+
+Database update sequences and the host's configuration identity are checked again before completion. Storage readers must detect changes while reading a file. These observations can detect concurrent changes; they do not provide an atomic snapshot or prevent a file from changing later.
+
+Reports project a whitelist of facts. Pseudonyms preserve exact path equality across sources and keep document, revision, and Chunk identifiers separate. Case and Unicode normalisation facts are computed before projection. Maps are local to one run and are not persisted. Original-name mode still excludes credentials, connection addresses, encryption keys, raw errors, and file contents. JSON and Markdown use the same projected integrity result; TSV uses the same name mapper for Usage.
+
+Each pseudonym report uses a random eight-character public salt. Path tokens have the form `<salt>$<hash>`; `<hash>` is the first 128 bits of a SHA-256 digest, shown as 32 lowercase hexadecimal characters. The digest is domain-separated by name kind and uses the exact stored Vault-relative path, without lower-casing or Unicode NFC normalisation. The mapper keeps no persistent salt or correspondence table. A consumer can parse a complete path token with `parseDiagnosticPathToken` and pass its salt to `createDiagnosticNameMapper('pseudonym', parsed.salt)` to reproduce path, document, revision, and Chunk pseudonyms for a targeted follow-up. Revision pseudonyms retain a valid generation prefix, such as `7-psu…`.
+
+The salt is public, so someone with the report can test guessed path values against a token; pseudonyms are not encryption. Earlier ordinal pseudonyms cannot be looked up this way. The path-token parser only extracts a salt and hash from a complete token; it does not recover the original path.
+
+### Compact Integrity Markdown report
+
+Integrity Markdown retains its status, name mode, versions, settings, scope, coverage, and count summary, with separate counts for content and Vault checks which were not performed. **State counts** count observations, not unique files; each local or remote observation contributes to its state total. The **Review index** lists references to observations which match review conditions, before **Findings**. Those conditions are identity mismatches, current conflicts, case- or NFC-equivalent paths, non-NFC paths, a missing group in the other inspected source, missing or unreadable content or Vault files, unavailable Chunks, different or unavailable comparisons, non-zero measured differences, and absent Metadata. These are review candidates, not findings of corruption. An observation outside the index is still present in **Findings**, and unchecked content or Vault state is not evidence of health.
+
+Each observation uses a compact field order while preserving every observation and its complete path, document ID, revision, and reference values:
+
+```text
+### `remote:4410` source=remote
+path: `<salt>$<hash>`
+doc: `<document-id>` rev=`<revision>` expected=self
+state: consistent-winner-logicallyLive-NFC-newnote conflicts=0
+paths: case=none nfcPeers=none | otherDB=present
+data: recorded=47584B chunks=1/1/0 read=0
+check: content=not-checked vault=not-checked@`vault:remote:4410`
+measure: content=47584B vault=47584B deltaContent=0B deltaVault=0B deltaMtime=0ms
+```
+
+The `state` value orders identity, current branch, logical deletion, exact stored-path NFC status, and Metadata type. Identity is `consistent`, `documentIdMismatch`, or `namespaceMismatch`; JSON keeps the values `document-id-mismatch` and `namespace-mismatch` for the latter two states. The branch is `winner` or `conflict`; `winner` can coexist with a non-zero conflict count. `logicallyLive` and `logicallyDeleted` describe the deletion marker. `NFC` means the stored path is normalised to Unicode NFC; `nonNFC` means it is not. The type is `plain`, `newnote`, `notes`, or `unknown`, and is `notRecorded` when Metadata is absent. `consistent` confirms the identity check only; it does not establish readable or equal content.
+
+`expected=self` means the actual document ID exactly matches the expected ID. Otherwise, the field contains the expected ID, or `?` if it was not recorded. In `paths`, `case` and `nfcPeers` report case-equivalent and NFC-equivalent path groups: `none` means no equivalent path was found, `yes` means one was found, and `yes@case:1` or `yes@nfc:1` adds the group reference when available. `otherDB` describes whether the same identity and path group is present in the other inspected source; it does not compare content. An excluded or unavailable source is `not-checked`.
+
+In `data`, `chunks` is the total/unique/embedded Chunk-reference count, and `read` is the number of Chunk reads, not HTTP requests. When Metadata is absent, its recorded size and Chunk counts are `?`. A `?` means a value was not recorded; it is distinct from zero, missing, unreadable, and `not-checked`. The `check` line distinguishes `available`, `missing`, `unreadable`, and `not-checked` for content and Vault evidence; a recorded reason follows the status in parentheses. The optional `measure` line includes only available values: decoded content size and Vault size in bytes, `deltaContent` (decoded content minus recorded Metadata size), `deltaVault` (Vault size minus decoded content size), and `deltaMtime` (Vault modification time minus Metadata modification time) in milliseconds. Positive differences have a `+` prefix, negative differences retain `-`, and zero is `0`; these measurements describe collected evidence and do not select an authoritative version.
+
+Unavailable Chunk entries, comparisons, case/NFC equivalence groups, and errors retain their names, reasons, and references. The collapsible details section and fenced text copy remain available for sharing the report. The compact Markdown representation does not change the JSON data or schema.
+
+Integrity Markdown wraps the complete report in a `details` element and a fenced text block for pasting into an issue. The fence is longer than any backtick run in the report, including original names. This keeps the full evidence together without expanding every observation in the issue body.
+
+Hosts may pass Commonlib, plug-in, and application versions to either report projector through its optional third argument. Only those three version fields are retained. Integrity reports also retain the five identity-related settings used for the inspection; credentials and the active connection configuration remain outside the report.

package/docs/development.md

@@ -8,6 +8,7 @@ The source tree is larger than the supported package surface. Consumers may impo
 
 - `src/index.ts`, `src/context.ts`, `src/settings.ts`, `src/remoteConfigurations.ts`, `src/p2p/index.ts`, `src/platform/browser/index.ts`, and `src/platform/node/index.ts` define the focused entries;
 - `src/replication/index.ts` defines the focused provider-capability and interaction contract for hosts which compose replication;
+- `src/diagnostics/index.ts` defines read-only Usage and integrity inspection, report projection, and formatting; hosts own the run dialogue and output destination;
 - `src/rpc/index.ts` defines a transitional entry for the existing LiveSync P2P composition, not a stable Commonlib 1.0 contract;
 - `_tools/build-package.mjs` compiles those entries and creates the publishable manifest under `.package`;
 - `docs/migration/downstream-imports.json` is the reviewed inventory from which explicit `compat/*` exports are generated;

package/docs/proven-in-use.md

@@ -8,6 +8,14 @@ The code paths below identify the maintained Self-hosted LiveSync composition po
 
 ## Obsidian plug-in
 
+Unreleased remote-inspection tests compare individual and bulk reads against CouchDB 3.5.0. A fixture with 65 Metadata documents and 130 Chunks produces identical integrity reports while reducing HTTP requests from 198 to 8. Encrypted bulk reads preserve current conflicts, exact revision access, missing-document evidence, and unchanged database and Security Seed state. Unit tests cover bounded GET batches, encoded URL limits, per-document failures, cancellation, and explicitly excluded remote coverage. These counts describe the fixture; they do not establish large-Vault elapsed-time improvements.
+
+Unreleased [database diagnostics](database-diagnostics.md) tests cover preserved Usage accounting, normal-file identity classification, current conflict branches, byte comparisons, embedded Chunks, missing and unreadable content, cancellation, concurrent changes, and consistent pseudonyms. Real in-memory PouchDB tests verify pagination and conflict revision reads while retaining the original database handle. A CouchDB 3.5.0 integration test verifies GET-only access, encrypted Metadata and Chunk reads with an existing Security Seed, missing-seed failure without initialisation, and unchanged database update sequences and Seed documents. Focused tests also reject ciphertext which remains undecoded after an ID mismatch. Commonlib owns the result contract and compact Integrity Markdown formatter; Self-hosted LiveSync owns the serviceFeature, run-time name choice, dialogue lifecycle, and report copy and download controls. Its real Obsidian 1.12.7 test on Linux verifies cancellation before a run, both name modes, the nine-column Usage report, copy and download payloads, local integrity findings when CouchDB is unavailable, and unchanged settings, database update sequences, and Vault file names. Mobile emulation verifies the choice dialogue's viewport, safe-area, and touch-target layout; native iOS and Android behaviour remains unverified.
+
+The Commonlib `/diagnostics` entry provides a path-token parser and a mapper which can reuse a report's public salt to reproduce path, document, revision, and Chunk pseudonyms for a targeted follow-up. Its Usage progress contract separates revision reading from cancellable accounting. Self-hosted LiveSync owns the current-Vault lookup, local-only candidate display, follow-up choice, and dialogue lifecycle; the public salt permits guessed-path checks, while older ordinal pseudonyms cannot be looked up.
+
+Self-hosted LiveSync's real Obsidian 1.12.7 test on Linux also verifies local lookup, selection between distinct document IDs for one path, and a follow-up report which reuses the selected ID and original salt. It checks that compact Markdown retains every JSON observation reference and pseudonym, and that copy and save preserve the complete Markdown output. Desktop and mobile-mode viewports verify the candidate list, report controls, and horizontal overflow. These host tests complement the package contracts; they do not establish native iOS or Android behaviour.
+
 Unreleased content-evidence tests replicate 99, 100, and 599 updates between real in-memory PouchDB databases, including a receiver with `revs_limit: 100`. They verify unchanged reflection after the recorded revision is pruned, including a recreated handler, and preserve same-size edits, intentional reverts, and records without valid content evidence. Separate tests verify legacy upgrades and changes during hashing. These are package-owned tests; they do not establish Windows, mobile, or remote CouchDB behaviour.
 
 The exact local packed candidate also passes Self-hosted LiveSync 1.0.37's checks, unit tests, application builds, and CLI E2E. Its focused real-Obsidian `stale-file-restart` scenario prunes the recorded revision before restarting, verifies unchanged reflection and legacy-record upgrades, and preserves an unknown-origin file. The Review Harness exercises the exported hash helper in desktop Obsidian. Maintainer-supplied Android and iOS reports additionally verify the helper and its benchmark on those devices; these timing measurements do not establish complete mobile synchronisation behaviour or peak memory use.
@@ -92,7 +100,9 @@ The CLI extends the same neutral context with its database path and injected `St
 
 The CLI filesystem composition receives a configured base directory and constructs the rooted Node storage adapter in `src/apps/cli/adapters/NodeFileSystemAdapter.ts`. Commonlib's shared storage result table checks the focused root contract; the consumer adds its file cache, Vault-shaped operations, case handling, and command policy.
 
-The unreleased [Unicode path identity contract](unicode-path-identity.md) adds opt-in NFC document IDs for CLI and Obsidian use with CouchDB. Commonlib tests exercise real in-memory Metadata lookup across different host normalisers. CLI temporary-filesystem tests cover resolution, updates, deletions, unchanged physical spellings, and ambiguous NFC paths. File-reflection tests preserve NFC-equivalent physical names while retaining case-only rename behaviour. With the packed candidate installed, the Linux CLI-to-real-Obsidian scenario verifies creation, remote updates, and daemon-applied deletions with and without E2EE and Path Obfuscation. A separate run migrates two legacy CLI databases with Rebuild and Fetch before the Obsidian round trip, preserving both NFD physical names and replacing the remote document ID. These checks do not establish native macOS or iOS behaviour.
+The [Unicode path identity contract](unicode-path-identity.md), introduced in Commonlib `0.1.42`, provides opt-in NFC document IDs for CLI and Obsidian use with CouchDB. Commonlib tests exercise real in-memory Metadata lookup across different host normalisers. CLI temporary-filesystem tests cover resolution, updates, deletions, unchanged physical spellings, and ambiguous NFC paths. File-reflection tests preserve NFC-equivalent physical names while retaining case-only rename behaviour. With the packed candidate installed, the Linux CLI-to-real-Obsidian scenario verifies creation, remote updates, and daemon-applied deletions with and without E2EE and Path Obfuscation. A separate run migrates two legacy CLI databases with Rebuild and Fetch before the Obsidian round trip, preserving both NFD physical names and replacing the remote document ID. These checks do not establish native macOS or iOS behaviour.
+
+The unreleased reconstruction refinement excludes unconflicted logical deletions only when replacing the local database. Rebuilder tests exercise real PouchDB enumeration before the reset boundary across Rebuild, Standard Fetch, and Fast Fetch. Real CouchDB tests cover deletion propagation, fully tombstoned documents, unresolved conflict leaves, and a stale local copy after a remote deletion. Non-deleted duplicates and unresolved conflicts still stop reconstruction, and remote-only Rebuild retains its ID check. These package-owned tests establish preflight decisions; the maintained downstream migration scenario separately owns complete CLI reconstruction and real-Obsidian Fast Fetch with the exact packed candidate.
 
 Unreleased Node timestamp tests verify that `stat` and `trystat` retain an available birth time when change time advances, fall back when birth time is zero, and return integer milliseconds. Packed-package checks exercise the shared `getNodeFileCreationTime` export. With the exact locally packed candidate installed, downstream CLI tests verify that an in-place edit retains creation time in local Metadata. The CLI-to-real-Obsidian scenario additionally verifies the edited content and creation time in received Metadata. These checks do not establish the reporter's ZFS behaviour or physical-file creation times in Obsidian.
 

package/docs/unicode-path-identity.md

@@ -0,0 +1,98 @@
+---
+date: 2026-10-19
+commonlib-version: "0.1.42"
+self-hosted-livesync-version: "1.0.38"
+status: unreleased
+---
+
+# Unicode path identity
+
+`pathNormalizationVersion` controls the logical path used to derive a normal-file
+Metadata document ID. An absent value or `0` preserves legacy behaviour. Value
+`1` applies Unicode NFC before the existing case and ID derivation rules. It
+does not alter the Metadata path or require physical files to be renamed.
+Unsupported values are rejected.
+
+NFC identity shipped in Commonlib `0.1.42` and Self-hosted LiveSync `1.0.38`.
+The deletion handling below is an unreleased refinement of reconstruction.
+
+## Package contract
+
+The Commonlib settings entry exposes `PathNormalizationVersion`,
+`PathIdentitySettings`, `getPathNormalizationVersion`, and
+`normalisePathForIdentity`. Consumers must use the shared path identity
+implementation for Metadata IDs rather than add a host-specific normalisation
+rule. Host storage adapters remain responsible for mapping a logical path back
+to its existing physical entry.
+
+## Database compatibility
+
+Version `1` uses `nfc-path-identity-v1` in the existing generation-13
+`used_features` list. This is separate from `idDerivationVersion`, which selects
+the cryptographic ID key. The NFC setting is a required Tweak even when
+encryption is disabled.
+
+A fresh database declares the feature before its first Metadata mutation. An
+existing database without the feature cannot acquire it through an ordinary
+Metadata write or replication while it already contains Metadata. A client
+configured for version `0` refuses an NFC database. These checks also apply
+when ordinary configuration-mismatch warnings are disabled. The existing
+version-document reception hook stops an active replication session when it
+detects incompatible path identity. It does not revoke server credentials or
+atomically cancel work already admitted by another device.
+
+Fast Fetch checkpoints include the path identity version, so a checkpoint
+created under version `0` cannot resume under version `1`, or vice versa.
+
+## Migration contract
+
+1. Update every participating client, stop synchronisation, and retain a
+   complete backup.
+2. Select a complete, authoritative Vault. Resolve distinct files or Metadata
+   documents which would derive the same NFC document ID. Case-insensitive
+   handling also applies its existing case rule.
+3. Set version `1` on the source and Rebuild both databases from that Vault.
+4. Set version `1` on the remaining clients and Fetch the rebuilt remote before
+   resuming synchronisation.
+
+Reconstruction checks the available storage listing and local Metadata before
+resetting either database. When the local database is not open, the storage
+check conservatively includes every listed file. A collision stops the
+operation; it never chooses a file to keep or merges separate histories.
+Ordinary synchronisation does not rename legacy document IDs in place.
+
+Full Rebuild, Standard Fetch, and Fast Fetch replace the local database. Their
+Metadata check requests conflict information and excludes a logically deleted
+document (`deleted: true`) only when it has no unresolved conflict leaves.
+A deleted winner with any unresolved conflict remains a collision candidate.
+Remote-only Rebuild retains logically deleted Metadata and still requires its
+stored document IDs to match the selected identity.
+
+A fully tombstoned document is already absent from ordinary Metadata
+enumeration. A deletion made only on the remote does not exclude a still
+non-deleted local document until that deletion has been replicated.
+
+The Setup URI shares `pathNormalizationVersion`; an absent value decodes as
+legacy mode, and unsupported versions are rejected. Importing a shared value
+does not itself perform the required reconstruction.
+
+## Supported hosts
+
+The initial support boundary is the maintained CLI and Obsidian with CouchDB.
+`PathService` owns logical ID derivation. The CLI resolves NFC-equivalent names
+against directory entries while preserving the actual entry spelling; its
+generic Node storage contract remains exact-path access. Obsidian supplies its
+normalised storage paths and preserves an existing physical spelling when it
+is NFC-equivalent to the received path.
+
+Journal and P2P refuse version `1`. Direct file access refuses a remote which
+declares the NFC feature. Other host adapters must establish physical path
+resolution before offering this setting. The regression tests cover ASCII,
+NFC, NFD, and mixed paths through in-memory PouchDB Metadata lookup, with and
+without path obfuscation, plus database admission, reconstruction ordering,
+collisions, and physical path handling. These tests do not establish native
+iOS or macOS filesystem behaviour.
+
+The full host setting, Rebuild/Fetch, CLI, and Obsidian migration contract is
+in the [Self-hosted LiveSync Unicode path identity design](https://github.com/vrtmrz/obsidian-livesync/blob/main/docs/design_docs/unicode_path_identity.md)
+and its [decision record](https://github.com/vrtmrz/obsidian-livesync/blob/main/docs/adr/2026_10_unicode_path_identity.md).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vrtmrz/livesync-commonlib",
-  "version": "0.1.42",
+  "version": "0.1.44",
   "description": "Shared data, storage, and replication primitives for Self-hosted LiveSync clients.",
   "type": "module",
   "types": "./dist/index.d.ts",
@@ -47,6 +47,11 @@
       "import": "./dist/context.js",
       "default": "./dist/context.js"
     },
+    "./diagnostics": {
+      "types": "./dist/diagnostics/index.d.ts",
+      "import": "./dist/diagnostics/index.js",
+      "default": "./dist/diagnostics/index.js"
+    },
     "./hashing": {
       "types": "./dist/hashing.d.ts",
       "import": "./dist/hashing.js",
@@ -643,6 +648,9 @@
       "context": [
         "dist/context.d.ts"
       ],
+      "diagnostics": [
+        "dist/diagnostics/index.d.ts"
+      ],
       "hashing": [
         "dist/hashing.d.ts"
       ],
@@ -997,6 +1005,7 @@
     }
   },
   "dependencies": {
+    "@aws-crypto/sha256-js": "5.2.0",
     "@aws-sdk/client-s3": "^3.808.0",
     "@smithy/fetch-http-handler": "^5.3.10",
     "@smithy/md5-js": "^4.2.9",