@vrtmrz/livesync-commonlib 0.1.28 → 0.1.30

package/docs/development.md

@@ -87,6 +87,8 @@ The unreleased watcher deletion checks and the separate database-to-storage refl
 
 File-path parsing recognises only the leading `i:`, `ix:`, and `ps:` namespaces. Colons elsewhere belong to the path, including the complete remainder after one namespace. Document-ID parsing additionally recognises `h:` and `f:`; those document IDs are not ordinary storage paths. Keep normalisation, selection patterns, and database lookup on the complete path. `pathPrefixes.unit.spec.ts`, `typeUtils.unit.spec.ts`, and `PathService.unit.spec.ts` cover these conversions, and `ServiceFileHandlerBase.colonPaths.unit.spec.ts` exercises storage and reflection with real PouchDB Metadata and Chunks. Host filename restrictions remain a separate contract: a host must not interpret an unsupported name as permission to write a shortened one.
 
+Before HKDF decryption, an obfuscated Metadata document's `path` field contains an encrypted payload beginning `/\:` rather than a logical file path or a document ID. Read that field directly for decryption; apply path and ID conversion only after obtaining the decoded path. `encryption.unit.spec.ts` covers this round trip, the legacy V1 path fallback, and decoded PouchDB replication events. `StreamingFetch.unit.spec.ts` covers decryption before the initial fetch writes and checkpoints a document.
+
 The revision-tree safety rules are documented in [Conflict resolution and file provenance](conflict-resolution.md). `ConflictManager.unit.spec.ts` owns common-ancestor and conservative merge behaviour, deterministic pair selection for multiple live leaves, and reconstruction from the committed tree between stages. `EntryManagerImpls.unit.spec.ts` owns the distinction between force-writing below an exact revision and conditionally advancing a current live leaf through PouchDB's ordinary revision check. `ServiceDatabaseFileAccessBase.unit.spec.ts` owns all-branch content discovery, short-circuiting history existence checks after an exact match, exact-revision logical deletion, and the public live-base result and event boundary. `ServiceFileHandlerBase.unit.spec.ts` owns the injected provenance policy for edit, deletion, and rename operations, together with user-selected reflection, child creation, strict matching, matching-provenance cleanup after exact branch deletion, and stale-revision refusal. `LiveSyncLocalDB.unit.spec.ts` owns chunk reachability across current winners, shared chunks, live conflicts, available merge ancestry, conflict resolution, and PouchDB replication of collection and recreation. Extend the relevant sides when changing conflict resolution, compaction handling, provenance, chunk collection, or database-to-storage overwrite protection.
 
 `LiveSyncReplicator.unit.spec.ts` owns the CouchDB remote-chunk batch contract: a mixed `_all_docs` result must return each available document even when another keyed row reports an error. `ChunkFetcher.unit.spec.ts` owns the consumer behaviour: it writes valid returned chunks and emits `missingChunkRemote` only for requested IDs which are absent from the returned set. Extend both sides when changing remote chunk acquisition or missing-chunk delivery.

package/docs/proven-in-use.md

@@ -10,6 +10,12 @@ The code paths below identify the maintained Self-hosted LiveSync composition po
 
 Unreleased colon-path regression tests establish that Commonlib preserves ordinary file paths through Metadata storage, Chunk retrieval, and storage reflection, with path obfuscation enabled or disabled. They use real PouchDB with injected storage. This result does not establish which filenames a host's file-creation API accepts; that boundary requires host runtime verification.
 
+Commonlib's encryption tests verify that an obfuscated Metadata document retains its document ID while its HKDF-encrypted path and file properties round-trip, that V1-obfuscated paths remain readable through the fallback, and that PouchDB replication presents a decoded path to both local storage and change events. The Fast Fetch test verifies decryption before storage and checkpointing. On 26th September 2026, a local packed candidate of this fix passed LiveSync 1.0.31's `npm run check` and unit tests, as well as real Obsidian two-Vault synchronisation, encrypted CLI-to-Obsidian transfer, and encrypted Metadata retrieval through Fast Fetch during manual CouchDB onboarding.
+
+The unreleased 0.1.30 packed candidate adds encryption of Hidden File Sync and Customisation Sync Metadata when CouchDB, E2EE V2, and Property Encryption are active. Focused Commonlib tests cover the `i:f:`, `ix:f:`, and legacy `ps:f:` representations, legacy and new setting defaults, feature declaration, concurrent version updates, and unknown-feature rejection. With that packed candidate in LiveSync, two real-Obsidian scenarios checked the raw CouchDB Metadata marker, zeroed times and size, empty public Chunk-reference list, and generation 13 feature declaration before restoring the files in a second Vault. The Hidden File Sync scenario also rebuilt the remote and verified the accepted writer's feature declaration and encrypted Metadata before the second Vault fetched them. The CLI-to-Obsidian encrypted note scenario also passed with matching effective settings. A separate real-Obsidian scenario kept continuous CouchDB replication active whilst the remote version document acquired an unknown feature; the local document was observed, the active Replicator retired, another replication was refused, and previously accepted Vault content remained intact. After restarting the Vault, both finite replication and the actual continuous connection attempt were refused by the remote declaration, without a KV feature history. These runs establish the tested local package and host build; they do not validate a published registry artefact or recovery under a future compatible client.
+
+Additional real-CouchDB integration tests exercise Fast Fetch admission before local reset, generation 12 acceptance, and generation 13 retrieval with plaintext and encrypted internal Metadata together. A real-Obsidian migration scenario enables the preference through the settings UI without Rebuild, verifies that unchanged Metadata retains its original revision while rewritten entries are encrypted with stable IDs, and checks a second device's mismatch rejection and restoration after alignment. After turning the preference OFF, Fast Fetch still retrieves both representations and their Chunks with matching content; the remote feature declaration remains. These checks use the local packed candidate with LiveSync 1.0.32.
+
 The plug-in extends the neutral `ServiceContext` with Obsidian-owned capabilities in `src/modules/services/ObsidianServiceContext.ts`. One context instance carries the LiveSync-owned translator and event hub through the Service Hub rather than placing those dependencies in Commonlib globals. Commonlib supplies its typed English fallback when another host omits the translator.
 
 The consumer contract in `test/contracts/serviceContext.ts` checks shared result semantics and exact context identity across the composed services. The real-Obsidian suite documented in `test/e2e-obsidian/README.md` then covers plug-in loading, representative Svelte dialogue mounts, Vault reflection, CouchDB and Object Storage synchronisation, a two-device P2P Setup URI round-trip, two-Vault behaviour, and other Obsidian-owned boundaries.

package/docs/remote-feature-compatibility.md

@@ -0,0 +1,177 @@
+---
+date: 2026-09-27
+commonlib-version: "0.1.30"
+self-hosted-livesync-version: "1.0.32"
+status: unreleased
+---
+
+# Remote feature compatibility
+
+This document defines the persisted feature contract for CouchDB databases and
+the checks required of Commonlib consumers. It describes unreleased behaviour
+being implemented in this branch.
+The first feature is encrypted Metadata for Hidden File Sync and Customisation
+Sync. Journal and P2P retain their existing transport contracts.
+
+## Ownership and compatibility dimensions
+
+Commonlib owns the wire representation, validation, feature identifiers, shared
+assessment, and admission requirements. Hosts own presentation, scheduling, and
+the application of received documents to their files.
+
+Keep these independent dimensions separate:
+
+- the remote protocol generation states how to interpret the control document;
+- the used-feature set states which representations the database may contain;
+- the supported-feature set belongs to the running client implementation;
+- Tweak values describe settings which participating writers should share; and
+- the existing Chunk version range describes Chunk format compatibility.
+
+A feature is supported only when the client can interpret its data and preserve
+its dependencies during applicable operations, including maintenance. Ignoring
+an unfamiliar file path is not sufficient: encrypted Chunk references must not
+be mistaken for an empty reference set.
+
+## Control document
+
+Extend the existing ordinary document, preserving its existing identifier:
+
+```json
+{
+  "_id": "obsydian_livesync_version",
+  "type": "versioninfo",
+  "version": 13,
+  "used_features": ["encrypted-internal-metadata-v1"]
+}
+```
+
+Generation 13 is the introduction boundary. It makes legacy clients
+which understand only generation 12 reject the enabled remote through their
+existing version check. Subsequent independent features use the feature list;
+they do not each require another protocol generation.
+
+Generation 12 remains readable without a feature list. Connecting with a newer
+client must not promote a legacy database. The existing local compatibility
+acknowledgement and settings-schema version remain separate concerns; adding
+this remote contract must not implicitly advance those markers for every user.
+
+The document remains readable before Metadata decryption and is replicated to
+local databases. The `_local/obsydian_livesync_milestone` document continues to
+hold participant and Tweak information; it is not the sole source of feature
+requirements. The ordinary version document also provides the existing route
+for observing compatibility changes during replication.
+
+### Validation and unknown identifiers
+
+Read `used_features` as an array of opaque, non-empty strings. Preserve unknown
+strings before comparing them with the client-owned supported set. A closed
+enum must not discard the evidence needed to diagnose an unknown feature.
+Order and duplicates do not change the set's meaning.
+
+Assessment returns distinct outcomes for a compatible document, an unsupported
+protocol generation, unknown feature identifiers, and malformed control data.
+Unknown identifiers are included in the result so any host can render a generic
+message such as `Unknown features are in use: example-feature-v2` without a
+feature-specific label or translation. Hosts render identifiers as text.
+
+A missing list is accepted as legacy input only in the legacy generation. A
+new-generation document with a missing or invalid list is not silently treated
+as an empty set. Deleting or corrupting the control document does not authorise
+ordinary data operations or recreate a legacy declaration over a non-empty DB.
+
+### Updating requirements
+
+Declare a feature before writing documents which depend on it. Introduce the
+new generation and its initial feature set in the same control-document update.
+Preserve existing identifiers and unrelated fields when updating the document.
+On a revision conflict, fetch and reassess the current document before merging
+or retrying; never replace another writer's requirements with a stale snapshot.
+
+The used-feature set is retained when a writer turns its corresponding setting
+off. Older documents or revisions may still depend on that feature. Removing
+a requirement needs a verified data-history transition, such as an explicit
+reconstruction which removes that representation. Manually shortening the declaration is not a supported migration.
+
+This is a client compatibility contract, not CouchDB access control. It cannot
+retrofit checks into legacy direct-access clients or revoke requests which a
+server has already accepted.
+
+## Admission and changes during replication
+
+Use the same assessment before ordinary replication, Fast Fetch, direct file
+access, and cleaned-remote recovery before counting Chunk references. Declare
+required write features after admission, including for the writer accepted by a
+Rebuild lock. A locked-out device cannot declare or upload a new representation.
+
+Fast Fetch reads the version document with its configured HTTP credentials and
+custom headers before opening or resetting the local database, including when
+resuming a checkpoint. Rejection leaves local data and the checkpoint intact.
+An empty remote and legacy generations retain their existing completion and
+migration paths; a missing version document in a populated remote is rejected.
+
+The remote document is the source of requirements. The milestone retains its
+participant, Tweak comparison, and lock responsibilities. Hosts do not need a
+second persistent feature list or rejection history in KV storage.
+
+The existing received-version handler also uses the shared assessment. An
+unknown identifier, unsupported generation, or malformed document requests
+Replicator retirement and reports the reason. Supported requirements do not
+interrupt the connection merely because the revision or feature list changed.
+Do not await retirement inside the change callback: the owner may be waiting
+for that callback to settle before it can drain and close the operation.
+
+A received notification can follow persistence to the local database. The
+handler is a best-effort response to an exceptional live change; it does not
+fence every queued file application or atomically revoke admitted requests.
+Use the established rollout workflow: update every synchronising client, enable
+the preference, and perform the strongly recommended manual Rebuild. Rebuild
+uses the existing remote lock. Merely changing the preference does not lock the
+remote, so continuing without rebuilding requires compatible devices first,
+including those with an active connection.
+
+The next connection assesses the current remote declaration, including after
+restart. Existing host snapshot and startup behaviour remain unchanged. After
+updating, use the host's normal reconciliation or Fetch facilities as needed;
+this contract does not add permanent local rejection flags or repair unrelated
+snapshot inconsistencies.
+
+## First feature: encrypted internal Metadata
+
+`encrypted-internal-metadata-v1` denotes the first specification of this feature,
+not the legacy E2EE V1 algorithm. It uses E2EE V2's existing HKDF Metadata
+representation for obfuscated `i:`, `ix:`, and supported legacy `ps:` entries.
+Document IDs and path-to-ID conversion remain unchanged.
+
+Supporting readers decode by the encrypted Metadata marker, independently of
+the current write preference. Existing plaintext and encrypted Metadata may
+coexist. Activating the preference affects subsequent writes; it does not
+rewrite unchanged documents or erase earlier plaintext revisions. Reconstruction
+is strongly recommended for existing-data privacy, and remains a manual choice.
+
+Garbage Collection V3 remains a beta, manually initiated operation. Its
+workflow first performs an ordinary bidirectional synchronisation, which checks
+the remote feature contract. It does not add another check at every collection
+step. The separate cleaned-remote recovery path checks the local contract
+before its first Chunk-reference count because that path does not begin with
+ordinary synchronisation.
+
+## Verification required before acceptance
+
+Protect the wire contract with tests for unknown names, multiple unknown names,
+malformed lists, legacy absence, new-generation absence, duplicate and reordered
+entries, concurrent updates, and preservation of existing fields. Use a made-up
+future identifier so the test does not depend on a known feature's label.
+
+Protect runtime behaviour with tests for initial admission, an admitted writer
+on a locked remote, unknown names received at the same generation, and
+retirement without a circular wait. Keep direct access, Fast Fetch, and
+maintenance admission checks covered. Host tests should preserve existing
+startup and queue semantics, including snapshot failures.
+
+Use real CouchDB and downstream Obsidian and CLI tests to validate encrypted
+Metadata, file restoration, receipt of a changed declaration, and rejection
+before synchronisation after restart. These checks do not establish atomic
+cancellation timing or automatic recovery under a future compatible client.
+
+Related contracts: [local database lifecycle](database-lifecycle.md) and
+[settings lifecycle](settings-lifecycle.md).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vrtmrz/livesync-commonlib",
-  "version": "0.1.28",
+  "version": "0.1.30",
   "description": "Shared data, storage, and replication primitives for Self-hosted LiveSync clients.",
   "type": "module",
   "types": "./dist/index.d.ts",