@vrtmrz/livesync-commonlib 0.1.43 → 0.2.0

package/README.md

@@ -14,12 +14,19 @@ This package is primarily for maintainers of those clients and for integrations
 
 ## Status and versioning
 
-The package is pre-1.0. Pin an exact reviewed version, and review changes before upgrading. An entry in the package export map means that the import is deliberate and package-tested; it does not make every semantic detail of that entry a final 1.0 guarantee.
+Commonlib is pre-1.0. The compatibility policy described here begins with 0.2.0 and does not retroactively apply to published 0.1.x versions. Within 0.2.x, Commonlib preserves published APIs, types, behaviour, supported persisted data, and exported `compat/*` paths; intentional breaking changes use 0.3.0. Consumers may declare ranges such as `^0.2.0`; retain a lockfile and use `npm ci` for reproducible installs. An entry in the package export map means that the import is deliberate and package-tested; it does not make every semantic detail of that entry a final 1.0 guarantee.
 
 Treat every `compat/*` subpath as a migration-only interface. A source-directory import, Git submodule, or consumer-defined TypeScript alias is not a supported package boundary.
 
 The package is ESM-only and declares Node 20 or later. Browser entry points are built with browser conditions, but the host remains responsible for checking the availability of browser capabilities such as the File System Access API.
 
+Fast Fetch imports all current leaf revisions listed by CouchDB, including
+deleted conflict leaves. A host may supply the optional
+`initial-fetch-fast-forward` resource to hand a fresh, complete import to
+ordinary replication before resuming local writers. Existing hosts can omit
+this optimisation. Resume cursors retain their existing meaning; this resource
+does not provide an arbitrary-time checkpoint jump or On Demand repair.
+
 ## Package entry points
 
 | Entry point                                        | Intended use                                                                      | Current contract status                                                                     |
@@ -36,7 +43,8 @@ The package is ESM-only and declares Node 20 or later. Browser entry points are
 | `@vrtmrz/livesync-commonlib/remote-configurations` | Multiple-remote profile creation, naming, and selection                           | Focused, package-tested pre-1.0 contract                                                    |
 | `@vrtmrz/livesync-commonlib/settings`              | New-Vault defaults, stored-setting fallbacks, and settings migration results      | Focused, package-tested pre-1.0 contract                                                    |
 | `@vrtmrz/livesync-commonlib/setup-uri`             | Encrypted Setup URI generation, availability, and import                          | Focused, package-tested pre-1.0 contract                                                    |
-| `@vrtmrz/livesync-commonlib/compat/*`              | Exact legacy imports still required by existing clients                           | Migration-only; paths may be removed as consumers migrate                                   |
+| `@vrtmrz/livesync-commonlib/ui-commands` | Instance-owned typed UI command declarations and handlers | Focused, package-tested contract |
+| `@vrtmrz/livesync-commonlib/compat/*`              | Exact legacy imports still required by existing clients                           | Migration-only; retained within 0.2.x, with removals treated as 0.3.0 breaking changes       |
 | `@vrtmrz/livesync-commonlib/package.json`          | Package metadata for tooling                                                      | Metadata export, not a runtime API                                                          |
 
 Import only the entry point required by the current runtime. Browser code must not import the Node entry.
@@ -75,6 +83,31 @@ After the Self-hosted LiveSync 1.0 work, the planned high-level composition is a
 
 The factory is not implemented yet. Until it is available, service implementations exposed under `compat/*` retain their owning client's lifecycle rather than becoming a general public composition API.
 
+## UI commands
+
+Import `declareUICommand` and `UICommandRegistry` from `@vrtmrz/livesync-commonlib/ui-commands`. A declaration binds a stable command ID to its request and result types. Each `UIService` owns its own `commands` registry; declarations do not register global handlers.
+
+```ts
+import { declareUICommand } from "@vrtmrz/livesync-commonlib/ui-commands";
+
+const showDevices = declareUICommand<void, 'closed'>('example:show-devices');
+const unregister = services.UI.commands.register(showDevices, async () => {
+    await openHostDeviceDialogue();
+    return 'closed';
+});
+const result = await services.UI.commands.invoke(showDevices, undefined);
+// result is { status: 'handled', value: 'closed' } or { status: 'unavailable' }.
+unregister();
+```
+
+A feature releases its handler when it unloads. The host dialogue owner remains responsible for active UI work. Duplicate handlers and conflicting declarations with the same ID are rejected; handler failures propagate. Calling an unavailable or disposed registry settles immediately. `UIService` disposes its registry on the context's plug-in-unloaded event.
+
+## Joined device records
+
+The `/replication` entry includes the `joined-devices` finite resource contract. A provider binds each resource to one settings snapshot; its owner calls `dispose()` in `finally`. `read()` returns recorded device IDs, acceptance, versions, platform, and connection times where available. An absent milestone is explicit. Joined-device operations never initialise a missing CouchDB database. These records do not establish online presence.
+
+`forget(id)` removes one other device from all participation maps and the accepted list, while retaining the lock state, preferred tweaks, and unknown fields. The CouchDB implementation rereads the existing milestone before editing and verifies removal after saving. CouchDB local documents do not reject stale revisions, so concurrent changes to other device records can be overwritten. A failed verification is reported to the caller for manual refresh and retry. Object Storage checks ETags with conditional writes and retries conflicts a bounded number of times. Missing records, unsupported conditional writes, and exhausted conflicts have distinct results. Forgetting is not credential revocation: an unlocked remote permits a returning device to register again.
+
 ## Rooted storage
 
 Both platform storage factories receive an existing root from the host. They never select a directory, request browser permission, infer a process directory, or persist a handle.
@@ -122,7 +155,9 @@ The focused Setup URI entry offers an Ephemeral mode bound to the current fixed
 
 ## Database diagnostics
 
-The focused `/diagnostics` entry provides read-only Usage and integrity reports, with original names or temporary pseudonyms selected by the host for each run. Integrity inspection covers normal-file Metadata, current conflict branches, and content comparisons for suspect groups in the local database and CouchDB. See [Database diagnostics](docs/database-diagnostics.md) for coverage, cancellation, and privacy boundaries.
+The focused `/diagnostics` entry provides read-only Usage and integrity reports, with original names or temporary pseudonyms selected by the host for each run. Integrity inspection covers normal-file Metadata, current conflict branches, and content comparisons for suspect groups in the local database and, by default, CouchDB; the host can exclude the remote source for one run. See [Database diagnostics](docs/database-diagnostics.md) for coverage, progress, cancellation, pseudonym lookup, and privacy boundaries.
+
+Integrity Markdown places a compact state summary and review index before all findings, while retaining each observation and its follow-up references. This formatting does not change the projected JSON data or schema; see [the Markdown field guide](docs/database-diagnostics.md#compact-integrity-markdown-report) for the meaning and limits of its review index.
 
 ```typescript
 import {
@@ -130,17 +165,33 @@ import {
     createDiagnosticNameMapper,
     createLocalDiagnosticSource,
     formatUsageTSV,
+    parseDiagnosticPathToken,
     projectUsageReport,
 } from '@vrtmrz/livesync-commonlib/diagnostics';
 
 const result = await analyseDatabaseUsage({ source: createLocalDiagnosticSource(localPouchDB) });
-const report = projectUsageReport(result, createDiagnosticNameMapper('pseudonym'));
+const mapper = createDiagnosticNameMapper('pseudonym');
+const report = projectUsageReport(result, mapper);
 if (report.status === 'complete') {
     const tsv = formatUsageTSV(report);
     // Present the report through the host's own UI or output stream.
 }
 ```
 
+Path pseudonyms include the report salt. A consumer can parse that token and supply its salt to a new mapper for a targeted follow-up against exact local Metadata paths:
+
+```typescript
+const pathToken = report.rows[0]?.path;
+const parsed = pathToken === undefined ? undefined : parseDiagnosticPathToken(pathToken);
+if (parsed) {
+    const followUpMapper = createDiagnosticNameMapper('pseudonym', parsed.salt);
+    const candidateToken = followUpMapper.path(storedVaultRelativePath);
+    // Compare this token with the path pseudonym on this device.
+}
+```
+
+The parser accepts only complete path tokens; it does not reverse them to recover a path. The salt is public, so a guessed path can be checked. Older ordinal pseudonyms have no corresponding lookup.
+
 ## Chunk ID calculation
 
 The `/hashing` entry exports `HashManager` and its `HashManagerCoreOptions` type. It uses the same implementation as database writes, including legacy algorithms and independent ID keys. Hosts supply their settings service and await `initialise()` before calling `computeHash(piece)`.
@@ -173,11 +224,11 @@ The accepted replacement direction is an asynchronously created file client with
 
 Package developers should read [the developer guide](docs/development.md). The focused contracts are described in [the storage guide](docs/platform-storage.md), [the standard-I/O guide](docs/platform-standard-io.md), [the settings lifecycle guide](docs/settings-lifecycle.md), [the remote configuration profile guide](docs/remote-configurations.md), [the remote feature compatibility contract](docs/remote-feature-compatibility.md), [the conflict-resolution guide](docs/conflict-resolution.md), and [the P2P transport lifecycle guide](docs/p2p-transport-lifecycle.md).
 
-## Proven in maintained hosts
+## Maintained-host examples and evidence
 
 The package boundary is exercised by Self-hosted LiveSync's Obsidian plug-in, CLI, WebApp, and WebPeer compositions. Their validation is not uniform: the Obsidian plug-in and CLI have the strongest device and long-running synchronisation evidence, while WebApp and WebPeer have focused adapter and runtime tests, browser smoke tests, a cross-application P2P scenario ending at the production CLI, and public Pages smoke validation.
 
-See [Proven in maintained hosts](docs/proven-in-use.md) for the actual composition points, downstream tests, and remaining limits. The examples deliberately separate focused public entries which a new consumer may copy from `compat/*` imports which exist to migrate the current application.
+See [Maintained-host examples and evidence](docs/proven-in-use.md) for composition points, downstream tests, and remaining limits. This optional page records examples and evidence; it does not define package support or create a release or adoption requirement. The examples deliberately separate focused public entries which a new consumer may copy from `compat/*` imports which exist to migrate the current application.
 
 ## Licence
 

package/dist/API/DirectFileManipulatorV2.d.ts

@@ -24,6 +24,8 @@ export type DirectFileManipulatorOptions = {
     maxChunksInEden?: number;
     maxTotalLengthInEden?: number;
     maxAgeInEden?: number;
+    /** Enables synchronisation of dot-prefixed internal files. */
+    syncInternalFiles?: boolean;
     /**
      * @deprecated use chunkSplitterVersion instead.
      */

package/dist/API/DirectFileManipulatorV2.js

@@ -1,3 +1,4 @@
+import { MemoryValueStore } from "@vrtmrz/value-store";
 import {
   DEFAULT_SETTINGS,
   DEFAULT_SYNC_PARAMETERS,
@@ -119,6 +120,7 @@ class DirectFileManipulator {
     const getSettings = () => this.settings;
     const context = new ServiceContext();
     this.services = new HeadlessServiceHub(context, {
+      deviceValues: new MemoryValueStore(),
       pouchDB: PouchDB,
       database: this.getBoundDatabaseService(() => this.options, this.runtimeOptions),
       getPathObfuscationPassphrase: () => this.options.obfuscatePassphrase ?? false,
@@ -253,6 +255,7 @@ class DirectFileManipulator {
         handleFilenameCaseSensitive: this.options.handleFilenameCaseSensitive ?? DEFAULT_SETTINGS.handleFilenameCaseSensitive,
         usePathObfuscation: !!this.options.obfuscatePassphrase,
         encryptInternalMetadata: this.options.encryptInternalMetadata ?? false,
+        syncInternalFiles: this.options.syncInternalFiles ?? DEFAULT_SETTINGS.syncInternalFiles,
         E2EEAlgorithm: this.options.E2EEAlgorithm ?? E2EEAlgorithms.V2
       }
     };

package/dist/common/configForDoc.d.ts

@@ -53,6 +53,7 @@ export declare const DoctorRegulationV0_25_0: DoctorRegulation;
 export declare const DoctorRegulationV0_25_27: DoctorRegulation;
 export declare const DoctorRegulationV1_0_0: DoctorRegulation;
 export declare const DoctorRegulationV1_0_33: DoctorRegulation;
+export declare const DoctorRegulationV1_0_40: DoctorRegulation;
 export declare const DoctorRegulation: DoctorRegulation;
 export declare function checkUnsuitableValues(setting: Partial<ObsidianLiveSyncSettings>, regulation?: DoctorRegulation): DoctorRegulation;
 export declare const RebuildOptions: {

package/dist/common/configForDoc.js

@@ -139,7 +139,16 @@ const DoctorRegulationV1_0_33 = {
     }
   }
 };
-const DoctorRegulation = DoctorRegulationV1_0_33;
+const DoctorRegulationV1_0_40 = {
+  // Only withdraw a recommendation. Retain the consultation version so users
+  // who dismissed 1.0.33 are not prompted again for unchanged recommendations.
+  version: DoctorRegulationV1_0_33.version,
+  rules: {
+    ...DoctorRegulationV1_0_33.rules,
+    customChunkSize: void 0
+  }
+};
+const DoctorRegulation = DoctorRegulationV1_0_40;
 function checkUnsuitableValues(setting, regulation = DoctorRegulation) {
   const result = {
     version: regulation.version,
@@ -339,6 +348,7 @@ export {
   DoctorRegulationV0_25_27,
   DoctorRegulationV1_0_0,
   DoctorRegulationV1_0_33,
+  DoctorRegulationV1_0_40,
   RebuildOptions,
   RuleLevel,
   checkUnsuitableValues,

package/dist/common/models/db.definition.d.ts

@@ -20,6 +20,7 @@ export interface DeviceInfo {
      */
     plugin_version: string;
     progress: string;
+    platform?: string;
 }
 export interface NodeData extends DeviceInfo {
     /**

package/dist/common/models/setting.const.preferred.js

@@ -1,6 +1,7 @@
 import { E2EEAlgorithms } from "./setting.const.js";
 const PREFERRED_BASE = {
   syncMaxSizeInMB: 50,
+  customChunkSize: 0,
   chunkSplitterVersion: "v3-rabin-karp",
   usePluginSyncV2: true,
   handleFilenameCaseSensitive: false,
@@ -8,21 +9,18 @@ const PREFERRED_BASE = {
 };
 const PREFERRED_SETTING_CLOUDANT = {
   ...PREFERRED_BASE,
-  customChunkSize: 0,
   sendChunksBulkMaxSize: 1,
   concurrencyOfReadChunksOnline: 100,
   minimumIntervalOfReadChunksOnline: 333
 };
 const PREFERRED_SETTING_SELF_HOSTED = {
   ...PREFERRED_BASE,
-  customChunkSize: 60,
   sendChunksBulkMaxSize: 1,
   concurrencyOfReadChunksOnline: 30,
   minimumIntervalOfReadChunksOnline: 25
 };
 const PREFERRED_JOURNAL_SYNC = {
   ...PREFERRED_BASE,
-  customChunkSize: 10,
   concurrencyOfReadChunksOnline: 30,
   minimumIntervalOfReadChunksOnline: 25
 };

package/dist/diagnostics/couchdb.js

@@ -6,6 +6,8 @@ import { decompressDoc } from "../pouchdb/compress.js";
 import { AuthorizationHeaderGenerator, generateCredentialObject } from "../replication/httplib.js";
 import { base64ToArrayBuffer } from "../string_and_binary/convert.js";
 import { DiagnosticReadError } from "./types.js";
+const MAX_BULK_DOCUMENTS = 64;
+const MAX_BULK_URL_LENGTH = 6e3;
 function createCouchDBDiagnosticSource({
   settings: originalSettings,
   fetch,
@@ -30,13 +32,16 @@ function createCouchDBDiagnosticSource({
   const checkCancelled = () => {
     if (controller.signal.aborted) throw new DOMException("Diagnostic cancelled", "AbortError");
   };
+  const requestURL = (path, parameters = {}) => {
+    const requestURL2 = new URL(`${root}${path}`);
+    for (const [key, value] of Object.entries(parameters)) requestURL2.searchParams.set(key, value);
+    return requestURL2.toString();
+  };
   const readJSON = async (path, parameters = {}) => {
     checkCancelled();
-    const requestURL = new URL(`${root}${path}`);
-    for (const [key, value] of Object.entries(parameters)) requestURL.searchParams.set(key, value);
     headers.set("authorization", await auth.getAuthorizationHeader(credentials));
     checkCancelled();
-    const response = await fetch(requestURL.toString(), {
+    const response = await fetch(requestURL(path, parameters), {
       method: "GET",
       headers,
       cache: "no-store",
@@ -73,10 +78,20 @@ function createCouchDBDiagnosticSource({
     () => {
     }
   ).outgoing : async (doc) => doc;
-  return {
+  const decode = async (id, value) => {
+    if (!value || typeof value !== "object" || !("_id" in value) || value._id !== id || !("_rev" in value) || typeof value._rev !== "string") {
+      throw new DiagnosticReadError("document-unreadable");
+    }
+    const decoded = await decompressDoc(await decrypt(value));
+    if ("e_" in decoded && decoded.e_ === true || "path" in decoded && typeof decoded.path === "string" && (isEncryptedMeta(decoded) || isObfuscatedEntry(decoded) && isPathProbablyObfuscated(decoded.path)))
+      throw new DiagnosticReadError("document-unreadable");
+    checkCancelled();
+    return decoded;
+  };
+  const source = {
     async info() {
       const info = await readJSON("");
-      return { update_seq: info.update_seq };
+      return { update_seq: info.update_seq, doc_count: info.doc_count };
     },
     async *enumerateIds() {
       let previous;
@@ -106,16 +121,73 @@ function createCouchDBDiagnosticSource({
       if (options.conflicts) parameters.conflicts = "true";
       if (options.revs) parameters.revs = "true";
       if (options.revs_info) parameters.revs_info = "true";
-      const raw = await readJSON(`/${encodeURIComponent(id)}`, parameters);
-      if (raw._id !== id || typeof raw._rev !== "string") {
-        throw new DiagnosticReadError("document-unreadable");
-      }
-      const decoded = await decompressDoc(await decrypt(raw));
-      if ("e_" in decoded && decoded.e_ === true || "path" in decoded && typeof decoded.path === "string" && (isEncryptedMeta(decoded) || isObfuscatedEntry(decoded) && isPathProbablyObfuscated(decoded.path))) {
-        throw new DiagnosticReadError("document-unreadable");
+      return await decode(id, await readJSON(`/${encodeURIComponent(id)}`, parameters));
+    },
+    async getMany(ids, options = {}) {
+      const result = [];
+      const parameters = (keys) => ({
+        keys: JSON.stringify(keys),
+        include_docs: "true",
+        ...options.conflicts ? { conflicts: "true" } : {}
+      });
+      const readBatch = async (keys) => {
+        if (!keys.length) return;
+        let rows;
+        try {
+          const page = await readJSON("/_all_docs", parameters(keys));
+          if (!Array.isArray(page.rows)) throw new DiagnosticReadError("document-unreadable");
+          rows = page.rows;
+        } catch {
+          checkCancelled();
+          result.push(...keys.map((id) => ({ id, error: new DiagnosticReadError("document-unreadable") })));
+          return;
+        }
+        for (let offset = 0; offset < keys.length; offset += 4) {
+          checkCancelled();
+          const decoded = await Promise.allSettled(
+            keys.slice(offset, offset + 4).map(async (id, index) => {
+              try {
+                const row = rows[offset + index];
+                if (!row || row.key !== id) throw new DiagnosticReadError("document-unreadable");
+                if (row.error === "not_found" || row.value?.deleted)
+                  throw new DiagnosticReadError("document-unreadable", true);
+                if (row.error || row.id !== id) throw new DiagnosticReadError("document-unreadable");
+                return { id, document: await decode(id, row.doc) };
+              } catch (error) {
+                checkCancelled();
+                return {
+                  id,
+                  error: error instanceof DiagnosticReadError ? error : new DiagnosticReadError("document-unreadable")
+                };
+              }
+            })
+          );
+          checkCancelled();
+          for (const outcome of decoded) {
+            if (outcome.status === "rejected") throw outcome.reason;
+            result.push(outcome.value);
+          }
+        }
+      };
+      let pending = [];
+      for (const id of ids) {
+        checkCancelled();
+        if (pending.length >= MAX_BULK_DOCUMENTS || requestURL("/_all_docs", parameters([...pending, id])).length > MAX_BULK_URL_LENGTH) {
+          await readBatch(pending);
+          pending = [];
+        }
+        if (requestURL("/_all_docs", parameters([id])).length > MAX_BULK_URL_LENGTH) {
+          try {
+            result.push({ id, document: await source.get(id, options) });
+          } catch (error) {
+            checkCancelled();
+            result.push({ id, error });
+          }
+        } else pending.push(id);
       }
+      await readBatch(pending);
       checkCancelled();
-      return decoded;
+      return result;
     },
     close() {
       signal?.removeEventListener("abort", abort);
@@ -123,6 +195,7 @@ function createCouchDBDiagnosticSource({
       return Promise.resolve();
     }
   };
+  return source;
 }
 export {
   createCouchDBDiagnosticSource

package/dist/diagnostics/index.d.ts

@@ -4,7 +4,7 @@ export { analyseDatabaseIntegrity } from "./integrity.js";
 export { createLocalDiagnosticSource } from "./local.js";
 export { createCouchDBDiagnosticSource } from "./couchdb.js";
 export type { CouchDBDiagnosticSourceOptions } from "./couchdb.js";
-export { createDiagnosticNameMapper, projectIntegrityReport, projectUsageReport, formatDiagnosticJSON, formatIntegrityMarkdown, formatUsageTSV, } from "./reports.js";
+export { createDiagnosticNameMapper, parseDiagnosticPathToken, projectIntegrityReport, projectUsageReport, formatDiagnosticJSON, formatIntegrityMarkdown, formatUsageTSV, } from "./reports.js";
 export type { DiagnosticNameMapper, DiagnosticReportVersions, ProjectedDiagnosticReport, ProjectedDatabaseIntegrityReport, ProjectedDatabaseUsageReport, } from "./reports.js";
 export { DiagnosticReadError } from "./types.js";
-export type { DiagnosticStatus, DiagnosticNameMode, DiagnosticSourceName, DiagnosticReason, DiagnosticReadOptions, DiagnosticDatabaseSource, DiagnosticError, DiagnosticContent, DiagnosticCoverage, DiagnosticObservation, DiagnosticIdentitySettings, DatabaseIntegrityResult, DatabaseIntegrityOptions, } from "./types.js";
+export type { DiagnosticStatus, DiagnosticNameMode, DiagnosticSourceName, DiagnosticReason, DiagnosticReadOptions, DiagnosticDocumentReadResult, DiagnosticDatabaseSource, DiagnosticError, DiagnosticContent, DiagnosticCoverage, DiagnosticObservation, DiagnosticPathEquivalenceGroup, DiagnosticIdentitySettings, DatabaseIntegrityResult, DatabaseIntegrityOptions, DatabaseIntegrityProgress, DatabaseIntegrityLog, } from "./types.js";

package/dist/diagnostics/index.js

@@ -4,6 +4,7 @@ import { createLocalDiagnosticSource } from "./local.js";
 import { createCouchDBDiagnosticSource } from "./couchdb.js";
 import {
   createDiagnosticNameMapper,
+  parseDiagnosticPathToken,
   projectIntegrityReport,
   projectUsageReport,
   formatDiagnosticJSON,
@@ -21,6 +22,7 @@ export {
   formatDiagnosticJSON,
   formatIntegrityMarkdown,
   formatUsageTSV,
+  parseDiagnosticPathToken,
   projectIntegrityReport,
   projectUsageReport
 };