@rljson/fs-agent 0.0.3 → 0.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.architecture.md +47 -0
- package/README.public.md +90 -2
- package/dist/README.architecture.md +47 -0
- package/dist/README.public.md +90 -2
- package/dist/fs-agent.d.ts +115 -5
- package/dist/fs-agent.js +259 -77
- package/dist/fs-db-adapter.d.ts +10 -3
- package/dist/index.d.ts +1 -1
- package/package.json +16 -16
package/README.architecture.md
CHANGED
|
@@ -447,3 +447,50 @@ This peer-to-peer pattern with server coordination enables:
|
|
|
447
447
|
✅ **Real-world deployment**: Works across networks, not just in-memory mocks
|
|
448
448
|
|
|
449
449
|
**This is why clients must NEVER access server Io/Bs directly** - it would bypass the entire peer-to-peer mechanism and make the system only work in single-process scenarios.
|
|
450
|
+
|
|
451
|
+
## Bounce-Back Prevention
|
|
452
|
+
|
|
453
|
+
Bidirectional sync creates a potential infinite loop:
|
|
454
|
+
|
|
455
|
+
```
|
|
456
|
+
Client A writes file → syncToDb stores tree → broadcasts ref →
|
|
457
|
+
Client B receives ref → syncFromDb restores file → fs watcher fires →
|
|
458
|
+
syncToDb stores tree → broadcasts ref → Client A receives ref → ...
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
FsAgent uses three layers of deduplication to break the loop:
|
|
462
|
+
|
|
463
|
+
1. **Ref-level dedup**: After `syncToDb` stores a tree, it compares the
|
|
464
|
+
resulting ref against `_lastSentRef`. If identical, no broadcast occurs.
|
|
465
|
+
|
|
466
|
+
2. **Content-key dedup**: Even when refs differ (e.g. different mtimes produce
|
|
467
|
+
different hashes), `syncToDb` computes a content key from file paths +
|
|
468
|
+
blobIds. If the content key matches `_lastSentContentKey`, the broadcast
|
|
469
|
+
is skipped.
|
|
470
|
+
|
|
471
|
+
3. **Content comparison before restore**: In `syncFromDb`, before restoring
|
|
472
|
+
an incoming tree, FsAgent compares its file content map against the current
|
|
473
|
+
filesystem. If they are equivalent, the restore is skipped entirely,
|
|
474
|
+
preventing the filesystem watcher from firing.
|
|
475
|
+
|
|
476
|
+
Additionally, all sync callbacks are **debounced** (default 300ms) to coalesce
|
|
477
|
+
rapid filesystem events (e.g. multi-file saves, editor autosave) into a single
|
|
478
|
+
sync operation.
|
|
479
|
+
|
|
480
|
+
## Known Constraints
|
|
481
|
+
|
|
482
|
+
### macOS Finder Paste and Rename
|
|
483
|
+
|
|
484
|
+
Bidirectional sync relies on Node.js `fs.watch` (FSEvents on macOS). This works
|
|
485
|
+
reliably for programmatic file operations (Node.js `writeFile`/`copyFile`,
|
|
486
|
+
terminal commands, editor saves) but **not** for macOS Finder's paste-and-rename
|
|
487
|
+
workflow.
|
|
488
|
+
|
|
489
|
+
Finder generates rapid, non-atomic, multi-step event sequences (create temp →
|
|
490
|
+
write → rename → delete temp) that FSEvents may coalesce, reorder, or split
|
|
491
|
+
unpredictably. This causes intermediate filesystem states to be scanned and
|
|
492
|
+
broadcast before the operation completes, leading to sync conflicts.
|
|
493
|
+
|
|
494
|
+
**This is a known limitation and is not supported.** Use programmatic operations
|
|
495
|
+
or terminal commands instead of Finder drag-and-drop or paste-and-rename for
|
|
496
|
+
files in synced folders.
|
package/README.public.md
CHANGED
|
@@ -21,9 +21,11 @@ found in the LICENSE file in the root of this package.
|
|
|
21
21
|
- 💾 **Smart Storage**: Content-addressed blob storage eliminates duplicate file content
|
|
22
22
|
- 📜 **Version History**: Every change is tracked with complete insert history
|
|
23
23
|
- 🔁 **Bidirectional Sync**: Changes in either direction are automatically propagated
|
|
24
|
-
- 🛡️ **Loop Prevention**:
|
|
24
|
+
- 🛡️ **Loop Prevention**: Content-based dedup, ref tracking, and debounce prevent infinite sync loops
|
|
25
|
+
- ⏱️ **Timeout Guards**: Every async operation is bounded to prevent silent hangs
|
|
26
|
+
- 🎯 **Debounce**: Rapid filesystem events are coalesced into a single sync cycle
|
|
25
27
|
- ✅ **Type-Safe**: Full TypeScript support with comprehensive type definitions
|
|
26
|
-
- 🧪 **Battle-Tested**: 100% test coverage with
|
|
28
|
+
- 🧪 **Battle-Tested**: 100% test coverage with 204 tests (SocketMock + Socket.IO)
|
|
27
29
|
- 🧹 **Target Cleanup**: Optional `cleanTarget` restore prunes stale files and directories
|
|
28
30
|
|
|
29
31
|
## Installation
|
|
@@ -312,6 +314,37 @@ Stops all syncing and cleans up resources.
|
|
|
312
314
|
agent.dispose();
|
|
313
315
|
```
|
|
314
316
|
|
|
317
|
+
##### `static fromClient(filePath, treeKey, client, socket, options?): Promise<FsAgent>`
|
|
318
|
+
|
|
319
|
+
Factory method that creates an FsAgent wired to a `@rljson/server` Client. Returns an enhanced agent with simplified sync methods:
|
|
320
|
+
|
|
321
|
+
```typescript
|
|
322
|
+
import { Client } from '@rljson/server';
|
|
323
|
+
|
|
324
|
+
const agent = await FsAgent.fromClient(
|
|
325
|
+
'./my-project',
|
|
326
|
+
'sharedTree',
|
|
327
|
+
client,
|
|
328
|
+
socket,
|
|
329
|
+
{ ignore: ['node_modules', '.git'] },
|
|
330
|
+
);
|
|
331
|
+
|
|
332
|
+
// Simplified sync (no need to manage Db/Connector manually)
|
|
333
|
+
const stopToDb = await agent.syncToDbSimple({ notify: true });
|
|
334
|
+
const stopFromDb = await agent.syncFromDbSimple({ cleanTarget: true });
|
|
335
|
+
|
|
336
|
+
// Stop when done
|
|
337
|
+
stopToDb();
|
|
338
|
+
stopFromDb();
|
|
339
|
+
```
|
|
340
|
+
|
|
341
|
+
**Reconnect handling:** When the Client supports `onDisconnect` / `onReconnect` (available since `@rljson/server` ≥ 0.0.10), `fromClient()` automatically:
|
|
342
|
+
|
|
343
|
+
- **Pauses** the filesystem watcher on disconnect (prevents futile sync attempts while the server is unreachable)
|
|
344
|
+
- **Resumes** the watcher on reconnect (the server's bootstrap message triggers a catch-up sync via the Connector)
|
|
345
|
+
|
|
346
|
+
No application code is needed — the wiring is built into `fromClient()`. Clients created with older server versions that lack these methods continue to work without interruption.
|
|
347
|
+
|
|
315
348
|
### FsScanner
|
|
316
349
|
|
|
317
350
|
Low-level filesystem scanner (usually accessed via `agent.scanner`).
|
|
@@ -700,6 +733,36 @@ try {
|
|
|
700
733
|
}
|
|
701
734
|
```
|
|
702
735
|
|
|
736
|
+
## Timeout & Debounce Configuration
|
|
737
|
+
|
|
738
|
+
Every async operation in FsAgent is guarded by a timeout to prevent silent hangs.
|
|
739
|
+
Rapid filesystem events are debounced to coalesce into a single sync cycle.
|
|
740
|
+
|
|
741
|
+
```typescript
|
|
742
|
+
const agent = new FsAgent('./my-project', bs, {
|
|
743
|
+
timeouts: {
|
|
744
|
+
dbQuery: 10_000, // Single db.get() query (default: 10s)
|
|
745
|
+
fetchTree: 20_000, // Fetching an entire tree from the DB (default: 20s)
|
|
746
|
+
extract: 15_000, // Filesystem extract/scan (default: 15s)
|
|
747
|
+
restore: 15_000, // Filesystem restore (default: 15s)
|
|
748
|
+
syncCallback: 25_000, // Overall syncFromDb callback (default: 25s)
|
|
749
|
+
debounceMs: 300, // Coalesce rapid FS events (default: 300ms)
|
|
750
|
+
},
|
|
751
|
+
});
|
|
752
|
+
```
|
|
753
|
+
|
|
754
|
+
## Bounce-Back Prevention
|
|
755
|
+
|
|
756
|
+
Bidirectional sync can cause infinite loops when both clients detect each
|
|
757
|
+
other's restores as changes. FsAgent prevents this with three layers:
|
|
758
|
+
|
|
759
|
+
1. **Ref tracking** (`_lastSentRef`): Skips re-broadcast if the stored tree
|
|
760
|
+
produces the same ref as the last one we sent.
|
|
761
|
+
2. **Content-key dedup** (`_lastSentContentKey`): Compares file paths + blobIds
|
|
762
|
+
(ignoring mtimes) to detect identical content with different tree hashes.
|
|
763
|
+
3. **Content comparison before restore**: `syncFromDb` compares the incoming
|
|
764
|
+
tree's content with the current filesystem and skips restore if identical.
|
|
765
|
+
|
|
703
766
|
## Performance
|
|
704
767
|
|
|
705
768
|
- **Efficient Scanning**: Only scans changed directories
|
|
@@ -707,6 +770,8 @@ try {
|
|
|
707
770
|
- **Content-Addressed**: Fast lookups via hashes
|
|
708
771
|
- **Optimized Watching**: Uses native filesystem events
|
|
709
772
|
- **Smart Sync**: Only syncs when changes detected
|
|
773
|
+
- **Debounced Events**: Rapid changes coalesced into single operations
|
|
774
|
+
- **Bounded Operations**: All async operations time out to prevent hangs
|
|
710
775
|
|
|
711
776
|
## Troubleshooting
|
|
712
777
|
|
|
@@ -763,6 +828,29 @@ ignore: ['*'];
|
|
|
763
828
|
ignore: ['node_modules', '*.log'];
|
|
764
829
|
```
|
|
765
830
|
|
|
831
|
+
## Known Constraints
|
|
832
|
+
|
|
833
|
+
### macOS Finder paste and rename
|
|
834
|
+
|
|
835
|
+
Bidirectional sync relies on Node.js `fs.watch` (FSEvents on macOS) to detect
|
|
836
|
+
filesystem changes. This works reliably for **programmatic** file operations:
|
|
837
|
+
|
|
838
|
+
- ✅ `writeFile`, `copyFile`, `rename` from Node.js or shell commands
|
|
839
|
+
- ✅ Editor saves (VS Code, Vim, etc.)
|
|
840
|
+
- ✅ Terminal commands (`echo`, `cp`, `mv`, `touch`, etc.)
|
|
841
|
+
- ✅ Application-generated file changes
|
|
842
|
+
|
|
843
|
+
However, **macOS Finder's paste-and-rename workflow** generates rapid,
|
|
844
|
+
non-atomic, multi-step event sequences (create temporary file → write → rename →
|
|
845
|
+
delete temporary) that FSEvents may coalesce, reorder, or split unpredictably.
|
|
846
|
+
This can cause intermediate states to be scanned and broadcast before the
|
|
847
|
+
operation completes, leading to sync conflicts.
|
|
848
|
+
|
|
849
|
+
**This is a known limitation of filesystem watching on macOS and is not
|
|
850
|
+
supported.** If you need to add or rename files in synced folders, use
|
|
851
|
+
programmatic operations or terminal commands instead of Finder drag-and-drop or
|
|
852
|
+
paste-and-rename.
|
|
853
|
+
|
|
766
854
|
## Related Packages
|
|
767
855
|
|
|
768
856
|
- `@rljson/db` - RLJSON database
|
|
@@ -447,3 +447,50 @@ This peer-to-peer pattern with server coordination enables:
|
|
|
447
447
|
✅ **Real-world deployment**: Works across networks, not just in-memory mocks
|
|
448
448
|
|
|
449
449
|
**This is why clients must NEVER access server Io/Bs directly** - it would bypass the entire peer-to-peer mechanism and make the system only work in single-process scenarios.
|
|
450
|
+
|
|
451
|
+
## Bounce-Back Prevention
|
|
452
|
+
|
|
453
|
+
Bidirectional sync creates a potential infinite loop:
|
|
454
|
+
|
|
455
|
+
```
|
|
456
|
+
Client A writes file → syncToDb stores tree → broadcasts ref →
|
|
457
|
+
Client B receives ref → syncFromDb restores file → fs watcher fires →
|
|
458
|
+
syncToDb stores tree → broadcasts ref → Client A receives ref → ...
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
FsAgent uses three layers of deduplication to break the loop:
|
|
462
|
+
|
|
463
|
+
1. **Ref-level dedup**: After `syncToDb` stores a tree, it compares the
|
|
464
|
+
resulting ref against `_lastSentRef`. If identical, no broadcast occurs.
|
|
465
|
+
|
|
466
|
+
2. **Content-key dedup**: Even when refs differ (e.g. different mtimes produce
|
|
467
|
+
different hashes), `syncToDb` computes a content key from file paths +
|
|
468
|
+
blobIds. If the content key matches `_lastSentContentKey`, the broadcast
|
|
469
|
+
is skipped.
|
|
470
|
+
|
|
471
|
+
3. **Content comparison before restore**: In `syncFromDb`, before restoring
|
|
472
|
+
an incoming tree, FsAgent compares its file content map against the current
|
|
473
|
+
filesystem. If they are equivalent, the restore is skipped entirely,
|
|
474
|
+
preventing the filesystem watcher from firing.
|
|
475
|
+
|
|
476
|
+
Additionally, all sync callbacks are **debounced** (default 300ms) to coalesce
|
|
477
|
+
rapid filesystem events (e.g. multi-file saves, editor autosave) into a single
|
|
478
|
+
sync operation.
|
|
479
|
+
|
|
480
|
+
## Known Constraints
|
|
481
|
+
|
|
482
|
+
### macOS Finder Paste and Rename
|
|
483
|
+
|
|
484
|
+
Bidirectional sync relies on Node.js `fs.watch` (FSEvents on macOS). This works
|
|
485
|
+
reliably for programmatic file operations (Node.js `writeFile`/`copyFile`,
|
|
486
|
+
terminal commands, editor saves) but **not** for macOS Finder's paste-and-rename
|
|
487
|
+
workflow.
|
|
488
|
+
|
|
489
|
+
Finder generates rapid, non-atomic, multi-step event sequences (create temp →
|
|
490
|
+
write → rename → delete temp) that FSEvents may coalesce, reorder, or split
|
|
491
|
+
unpredictably. This causes intermediate filesystem states to be scanned and
|
|
492
|
+
broadcast before the operation completes, leading to sync conflicts.
|
|
493
|
+
|
|
494
|
+
**This is a known limitation and is not supported.** Use programmatic operations
|
|
495
|
+
or terminal commands instead of Finder drag-and-drop or paste-and-rename for
|
|
496
|
+
files in synced folders.
|
package/dist/README.public.md
CHANGED
|
@@ -21,9 +21,11 @@ found in the LICENSE file in the root of this package.
|
|
|
21
21
|
- 💾 **Smart Storage**: Content-addressed blob storage eliminates duplicate file content
|
|
22
22
|
- 📜 **Version History**: Every change is tracked with complete insert history
|
|
23
23
|
- 🔁 **Bidirectional Sync**: Changes in either direction are automatically propagated
|
|
24
|
-
- 🛡️ **Loop Prevention**:
|
|
24
|
+
- 🛡️ **Loop Prevention**: Content-based dedup, ref tracking, and debounce prevent infinite sync loops
|
|
25
|
+
- ⏱️ **Timeout Guards**: Every async operation is bounded to prevent silent hangs
|
|
26
|
+
- 🎯 **Debounce**: Rapid filesystem events are coalesced into a single sync cycle
|
|
25
27
|
- ✅ **Type-Safe**: Full TypeScript support with comprehensive type definitions
|
|
26
|
-
- 🧪 **Battle-Tested**: 100% test coverage with
|
|
28
|
+
- 🧪 **Battle-Tested**: 100% test coverage with 204 tests (SocketMock + Socket.IO)
|
|
27
29
|
- 🧹 **Target Cleanup**: Optional `cleanTarget` restore prunes stale files and directories
|
|
28
30
|
|
|
29
31
|
## Installation
|
|
@@ -312,6 +314,37 @@ Stops all syncing and cleans up resources.
|
|
|
312
314
|
agent.dispose();
|
|
313
315
|
```
|
|
314
316
|
|
|
317
|
+
##### `static fromClient(filePath, treeKey, client, socket, options?): Promise<FsAgent>`
|
|
318
|
+
|
|
319
|
+
Factory method that creates an FsAgent wired to a `@rljson/server` Client. Returns an enhanced agent with simplified sync methods:
|
|
320
|
+
|
|
321
|
+
```typescript
|
|
322
|
+
import { Client } from '@rljson/server';
|
|
323
|
+
|
|
324
|
+
const agent = await FsAgent.fromClient(
|
|
325
|
+
'./my-project',
|
|
326
|
+
'sharedTree',
|
|
327
|
+
client,
|
|
328
|
+
socket,
|
|
329
|
+
{ ignore: ['node_modules', '.git'] },
|
|
330
|
+
);
|
|
331
|
+
|
|
332
|
+
// Simplified sync (no need to manage Db/Connector manually)
|
|
333
|
+
const stopToDb = await agent.syncToDbSimple({ notify: true });
|
|
334
|
+
const stopFromDb = await agent.syncFromDbSimple({ cleanTarget: true });
|
|
335
|
+
|
|
336
|
+
// Stop when done
|
|
337
|
+
stopToDb();
|
|
338
|
+
stopFromDb();
|
|
339
|
+
```
|
|
340
|
+
|
|
341
|
+
**Reconnect handling:** When the Client supports `onDisconnect` / `onReconnect` (available since `@rljson/server` ≥ 0.0.10), `fromClient()` automatically:
|
|
342
|
+
|
|
343
|
+
- **Pauses** the filesystem watcher on disconnect (prevents futile sync attempts while the server is unreachable)
|
|
344
|
+
- **Resumes** the watcher on reconnect (the server's bootstrap message triggers a catch-up sync via the Connector)
|
|
345
|
+
|
|
346
|
+
No application code is needed — the wiring is built into `fromClient()`. Clients created with older server versions that lack these methods continue to work without interruption.
|
|
347
|
+
|
|
315
348
|
### FsScanner
|
|
316
349
|
|
|
317
350
|
Low-level filesystem scanner (usually accessed via `agent.scanner`).
|
|
@@ -700,6 +733,36 @@ try {
|
|
|
700
733
|
}
|
|
701
734
|
```
|
|
702
735
|
|
|
736
|
+
## Timeout & Debounce Configuration
|
|
737
|
+
|
|
738
|
+
Every async operation in FsAgent is guarded by a timeout to prevent silent hangs.
|
|
739
|
+
Rapid filesystem events are debounced to coalesce into a single sync cycle.
|
|
740
|
+
|
|
741
|
+
```typescript
|
|
742
|
+
const agent = new FsAgent('./my-project', bs, {
|
|
743
|
+
timeouts: {
|
|
744
|
+
dbQuery: 10_000, // Single db.get() query (default: 10s)
|
|
745
|
+
fetchTree: 20_000, // Fetching an entire tree from the DB (default: 20s)
|
|
746
|
+
extract: 15_000, // Filesystem extract/scan (default: 15s)
|
|
747
|
+
restore: 15_000, // Filesystem restore (default: 15s)
|
|
748
|
+
syncCallback: 25_000, // Overall syncFromDb callback (default: 25s)
|
|
749
|
+
debounceMs: 300, // Coalesce rapid FS events (default: 300ms)
|
|
750
|
+
},
|
|
751
|
+
});
|
|
752
|
+
```
|
|
753
|
+
|
|
754
|
+
## Bounce-Back Prevention
|
|
755
|
+
|
|
756
|
+
Bidirectional sync can cause infinite loops when both clients detect each
|
|
757
|
+
other's restores as changes. FsAgent prevents this with three layers:
|
|
758
|
+
|
|
759
|
+
1. **Ref tracking** (`_lastSentRef`): Skips re-broadcast if the stored tree
|
|
760
|
+
produces the same ref as the last one we sent.
|
|
761
|
+
2. **Content-key dedup** (`_lastSentContentKey`): Compares file paths + blobIds
|
|
762
|
+
(ignoring mtimes) to detect identical content with different tree hashes.
|
|
763
|
+
3. **Content comparison before restore**: `syncFromDb` compares the incoming
|
|
764
|
+
tree's content with the current filesystem and skips restore if identical.
|
|
765
|
+
|
|
703
766
|
## Performance
|
|
704
767
|
|
|
705
768
|
- **Efficient Scanning**: Only scans changed directories
|
|
@@ -707,6 +770,8 @@ try {
|
|
|
707
770
|
- **Content-Addressed**: Fast lookups via hashes
|
|
708
771
|
- **Optimized Watching**: Uses native filesystem events
|
|
709
772
|
- **Smart Sync**: Only syncs when changes detected
|
|
773
|
+
- **Debounced Events**: Rapid changes coalesced into single operations
|
|
774
|
+
- **Bounded Operations**: All async operations time out to prevent hangs
|
|
710
775
|
|
|
711
776
|
## Troubleshooting
|
|
712
777
|
|
|
@@ -763,6 +828,29 @@ ignore: ['*'];
|
|
|
763
828
|
ignore: ['node_modules', '*.log'];
|
|
764
829
|
```
|
|
765
830
|
|
|
831
|
+
## Known Constraints
|
|
832
|
+
|
|
833
|
+
### macOS Finder paste and rename
|
|
834
|
+
|
|
835
|
+
Bidirectional sync relies on Node.js `fs.watch` (FSEvents on macOS) to detect
|
|
836
|
+
filesystem changes. This works reliably for **programmatic** file operations:
|
|
837
|
+
|
|
838
|
+
- ✅ `writeFile`, `copyFile`, `rename` from Node.js or shell commands
|
|
839
|
+
- ✅ Editor saves (VS Code, Vim, etc.)
|
|
840
|
+
- ✅ Terminal commands (`echo`, `cp`, `mv`, `touch`, etc.)
|
|
841
|
+
- ✅ Application-generated file changes
|
|
842
|
+
|
|
843
|
+
However, **macOS Finder's paste-and-rename workflow** generates rapid,
|
|
844
|
+
non-atomic, multi-step event sequences (create temporary file → write → rename →
|
|
845
|
+
delete temporary) that FSEvents may coalesce, reorder, or split unpredictably.
|
|
846
|
+
This can cause intermediate states to be scanned and broadcast before the
|
|
847
|
+
operation completes, leading to sync conflicts.
|
|
848
|
+
|
|
849
|
+
**This is a known limitation of filesystem watching on macOS and is not
|
|
850
|
+
supported.** If you need to add or rename files in synced folders, use
|
|
851
|
+
programmatic operations or terminal commands instead of Finder drag-and-drop or
|
|
852
|
+
paste-and-rename.
|
|
853
|
+
|
|
766
854
|
## Related Packages
|
|
767
855
|
|
|
768
856
|
- `@rljson/db` - RLJSON database
|
package/dist/fs-agent.d.ts
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { Bs } from '@rljson/bs';
|
|
2
|
+
import { ClientId, SyncConfig } from '@rljson/rljson';
|
|
2
3
|
import { FsBlobAdapter } from './fs-blob-adapter.ts';
|
|
3
4
|
import { StoreFsTreeOptions } from './fs-db-adapter.ts';
|
|
4
5
|
import { FsScanner, FsTree } from './fs-scanner.ts';
|
|
@@ -23,12 +24,56 @@ export interface FsAgentOptions {
|
|
|
23
24
|
bidirectional?: boolean;
|
|
24
25
|
/** Restore options applied when syncing from DB */
|
|
25
26
|
restoreOptions?: RestoreOptions;
|
|
27
|
+
/** Timeout configuration for async operations */
|
|
28
|
+
timeouts?: TimeoutConfig;
|
|
29
|
+
/**
|
|
30
|
+
* Centralized sync protocol configuration.
|
|
31
|
+
* When provided, this SyncConfig is forwarded to every Connector
|
|
32
|
+
* created by {@link FsAgent.fromClient}, and governs whether
|
|
33
|
+
* `sendWithAck()` (when `requireAck` is true) or `send()` is used
|
|
34
|
+
* in {@link FsAgent.syncToDb}.
|
|
35
|
+
*
|
|
36
|
+
* The same SyncConfig should also be passed to the Server and Client
|
|
37
|
+
* constructors so that every layer uses the same protocol settings.
|
|
38
|
+
*/
|
|
39
|
+
syncConfig?: SyncConfig;
|
|
40
|
+
/**
|
|
41
|
+
* Stable client identity. When provided, it is forwarded to every
|
|
42
|
+
* Connector created by {@link FsAgent.fromClient}. When omitted but
|
|
43
|
+
* `syncConfig.includeClientIdentity` is true, each Connector
|
|
44
|
+
* auto-generates its own identity.
|
|
45
|
+
*/
|
|
46
|
+
clientIdentity?: ClientId;
|
|
26
47
|
}
|
|
27
48
|
/** Restore options */
|
|
28
49
|
export interface RestoreOptions {
|
|
29
50
|
/** Remove files/dirs on target that are not present in the tree */
|
|
30
51
|
cleanTarget?: boolean;
|
|
31
52
|
}
|
|
53
|
+
/**
|
|
54
|
+
* Timeout configuration for async operations (milliseconds).
|
|
55
|
+
* Every async operation in FsAgent is guarded by a timeout to prevent
|
|
56
|
+
* silent hangs in socket communication, filesystem I/O, or database queries.
|
|
57
|
+
*/
|
|
58
|
+
export interface TimeoutConfig {
|
|
59
|
+
/** Timeout for a single db.get() query. Default: 10 000 ms */
|
|
60
|
+
dbQuery?: number;
|
|
61
|
+
/** Timeout for fetching an entire tree from the DB. Default: 20 000 ms */
|
|
62
|
+
fetchTree?: number;
|
|
63
|
+
/** Timeout for a filesystem extract / scan. Default: 15 000 ms */
|
|
64
|
+
extract?: number;
|
|
65
|
+
/** Timeout for a filesystem restore. Default: 15 000 ms */
|
|
66
|
+
restore?: number;
|
|
67
|
+
/** Timeout for the overall syncFromDb callback. Default: 25 000 ms */
|
|
68
|
+
syncCallback?: number;
|
|
69
|
+
/**
|
|
70
|
+
* Debounce delay for sync callbacks (milliseconds). Default: 300 ms.
|
|
71
|
+
* Rapid filesystem events (e.g. macOS Finder "Keep Both" copy+rename)
|
|
72
|
+
* are coalesced into a single sync operation after this quiet period.
|
|
73
|
+
* Also applies to incoming database refs in syncFromDb.
|
|
74
|
+
*/
|
|
75
|
+
debounceMs?: number;
|
|
76
|
+
}
|
|
32
77
|
/**
|
|
33
78
|
* Orchestrates filesystem operations with tree structures and blob storage
|
|
34
79
|
*/
|
|
@@ -42,6 +87,9 @@ export declare class FsAgent {
|
|
|
42
87
|
private _stopSync?;
|
|
43
88
|
private _stopSyncFromDb?;
|
|
44
89
|
private _lastSentRef?;
|
|
90
|
+
/** Content fingerprint of the last tree we broadcasted (paths+blobIds) */
|
|
91
|
+
private _lastSentContentKey?;
|
|
92
|
+
private _timeouts;
|
|
45
93
|
constructor(rootPath: string, bs?: Bs, options?: FsAgentOptions);
|
|
46
94
|
/**
|
|
47
95
|
* Gets the root path
|
|
@@ -59,6 +107,27 @@ export declare class FsAgent {
|
|
|
59
107
|
* Gets the adapter instance
|
|
60
108
|
*/
|
|
61
109
|
get adapter(): FsBlobAdapter;
|
|
110
|
+
/**
|
|
111
|
+
* Gets the current timeout configuration
|
|
112
|
+
*/
|
|
113
|
+
get timeouts(): Required<TimeoutConfig>;
|
|
114
|
+
/**
|
|
115
|
+
* Wraps a promise with a timeout.
|
|
116
|
+
* Rejects with a descriptive error if the promise does not settle
|
|
117
|
+
* within the given number of milliseconds.
|
|
118
|
+
* @param promise - The promise to guard
|
|
119
|
+
* @param ms - Maximum allowed time in milliseconds
|
|
120
|
+
* @param label - Human-readable label included in the error message
|
|
121
|
+
*/
|
|
122
|
+
private static _withTimeout;
|
|
123
|
+
/**
|
|
124
|
+
* Sends a ref through the connector.
|
|
125
|
+
* Uses `sendWithAck()` when the connector has `requireAck` enabled,
|
|
126
|
+
* otherwise falls back to fire-and-forget `send()`.
|
|
127
|
+
* @param connector - The Connector to send through
|
|
128
|
+
* @param ref - The ref to broadcast
|
|
129
|
+
*/
|
|
130
|
+
private _sendRef;
|
|
62
131
|
/**
|
|
63
132
|
* Starts automatic syncing to database
|
|
64
133
|
* Note: Auto-sync requires Connector which is not available in constructor.
|
|
@@ -130,6 +199,15 @@ export declare class FsAgent {
|
|
|
130
199
|
* @returns Array of all tree nodes in the tree
|
|
131
200
|
*/
|
|
132
201
|
private _fetchTreeRecursively;
|
|
202
|
+
/**
|
|
203
|
+
* Fetches tree from database without restoring to filesystem.
|
|
204
|
+
* Separated from loadFromDb to allow content comparison before restore.
|
|
205
|
+
* @param db - Database instance
|
|
206
|
+
* @param treeKey - Tree table key
|
|
207
|
+
* @param rootRef - Root tree reference (hash)
|
|
208
|
+
* @returns FsTree structure ready for restore
|
|
209
|
+
*/
|
|
210
|
+
private _fetchTreeFromDb;
|
|
133
211
|
/**
|
|
134
212
|
* Loads tree from database and restores to filesystem
|
|
135
213
|
* Writes to filesystem from DB trees and Bs blobs
|
|
@@ -159,10 +237,37 @@ export declare class FsAgent {
|
|
|
159
237
|
* @param db - Database instance
|
|
160
238
|
* @param connector - Connector instance for socket-based sync
|
|
161
239
|
* @param treeKey - Tree table key
|
|
162
|
-
* @param options - Storage options (e.g.,
|
|
240
|
+
* @param options - Storage options (e.g., skipNotification)
|
|
163
241
|
* @returns Function to stop watching
|
|
164
242
|
*/
|
|
165
243
|
syncToDb(db: Db, connector: Connector, treeKey: string, options?: StoreFsTreeOptions): Promise<() => void>;
|
|
244
|
+
/**
|
|
245
|
+
* Builds a map of relativePath → blobId for all files in a tree.
|
|
246
|
+
* Used to compare trees by content rather than by hash (which includes mtime).
|
|
247
|
+
* @param tree - Tree structure to extract file content map from
|
|
248
|
+
*/
|
|
249
|
+
private _getFileContentMap;
|
|
250
|
+
/**
|
|
251
|
+
* Derives a deterministic string key from a content map so that two trees
|
|
252
|
+
* with identical file paths + blobIds produce the same key regardless of
|
|
253
|
+
* mtime differences.
|
|
254
|
+
* @param map - Content map (relativePath → blobId)
|
|
255
|
+
*/
|
|
256
|
+
private _contentKeyFromMap;
|
|
257
|
+
/**
|
|
258
|
+
* Derives a deterministic content key from an FsTree.
|
|
259
|
+
* @param tree - Tree structure to derive content key from
|
|
260
|
+
*/
|
|
261
|
+
private _contentKeyFromTree;
|
|
262
|
+
/**
|
|
263
|
+
* Compares two trees by file content (relativePath + blobId).
|
|
264
|
+
* Ignores mtime differences — trees are equivalent if they have the same
|
|
265
|
+
* files with the same content. This prevents bounce-back restores from
|
|
266
|
+
* destroying locally-created files during bidirectional sync.
|
|
267
|
+
* @param a - First tree to compare
|
|
268
|
+
* @param b - Second tree to compare
|
|
269
|
+
*/
|
|
270
|
+
private _treesHaveEquivalentContent;
|
|
166
271
|
/**
|
|
167
272
|
* Watches database for tree changes and syncs to filesystem
|
|
168
273
|
* Uses Connector for socket-based notifications
|
|
@@ -181,16 +286,21 @@ export declare class FsAgent {
|
|
|
181
286
|
* @param treeKey - Tree table key (route will be `/${treeKey}`)
|
|
182
287
|
* @param client - Client instance with io and bs properties
|
|
183
288
|
* @param socket - Socket instance for connector communication
|
|
184
|
-
* @param options - Optional FsAgent options (db and treeKey are set automatically)
|
|
289
|
+
* @param options - Optional FsAgent options (db and treeKey are set automatically).
|
|
290
|
+
* `syncConfig` and `clientIdentity` from these options are forwarded to
|
|
291
|
+
* the Connector so that a single config origin governs all layers.
|
|
185
292
|
* @returns Configured FsAgent instance with simplified sync API
|
|
186
293
|
* @example
|
|
187
294
|
* ```typescript
|
|
188
|
-
* const
|
|
295
|
+
* const syncConfig: SyncConfig = { requireAck: true, maxDedupSetSize: 5000 };
|
|
296
|
+
* const agent = await FsAgent.fromClient(
|
|
297
|
+
* './my-folder', 'sharedTree', client, socket, { syncConfig },
|
|
298
|
+
* );
|
|
189
299
|
* // Simplified sync methods - no db/connector/treeKey needed
|
|
190
|
-
* await agent.syncToDbSimple(
|
|
300
|
+
* await agent.syncToDbSimple();
|
|
191
301
|
* await agent.syncFromDbSimple({ cleanTarget: true });
|
|
192
302
|
* // Original methods still work
|
|
193
|
-
* await agent.syncToDb(db, connector, treeKey
|
|
303
|
+
* await agent.syncToDb(db, connector, treeKey);
|
|
194
304
|
* ```
|
|
195
305
|
*/
|
|
196
306
|
static fromClient(filePath: string, treeKey: string, client: any, // Client type from \@rljson/server
|
package/dist/fs-agent.js
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { BsMem } from "@rljson/bs";
|
|
2
|
-
import {
|
|
2
|
+
import { Route, createTreesTableCfg } from "@rljson/rljson";
|
|
3
3
|
import { stat, readFile, mkdir, writeFile, readdir, utimes, rm } from "fs/promises";
|
|
4
4
|
import { dirname, join, sep } from "path";
|
|
5
5
|
import { hip } from "@rljson/hash";
|
|
@@ -124,7 +124,6 @@ class FsDbAdapter {
|
|
|
124
124
|
* @returns The root tree reference
|
|
125
125
|
*/
|
|
126
126
|
async storeFsTree(fsTree, options = {}) {
|
|
127
|
-
const { notify = false } = options;
|
|
128
127
|
if (!fsTree) {
|
|
129
128
|
throw new Error("fsTree cannot be null or undefined");
|
|
130
129
|
}
|
|
@@ -153,37 +152,10 @@ class FsDbAdapter {
|
|
|
153
152
|
(tree) => tree._hash !== fsTree.rootHash
|
|
154
153
|
);
|
|
155
154
|
trees.push(rootTree);
|
|
156
|
-
const
|
|
157
|
-
|
|
158
|
-
_data: trees
|
|
159
|
-
};
|
|
160
|
-
try {
|
|
161
|
-
await this.db.core.import({
|
|
162
|
-
[this.treeKey]: treeTable
|
|
163
|
-
});
|
|
164
|
-
} catch (error) {
|
|
165
|
-
throw new Error(
|
|
166
|
-
`Failed to import tree data into database: ${error instanceof Error ? error.message : String(error)}`
|
|
167
|
-
);
|
|
168
|
-
}
|
|
169
|
-
const treeRootRef = trees[trees.length - 1]._hash;
|
|
170
|
-
const historyRow = {
|
|
171
|
-
timeId: timeId(),
|
|
172
|
-
route: `/${this.treeKey}/${treeRootRef}`,
|
|
173
|
-
[`${this.treeKey}Ref`]: treeRootRef
|
|
174
|
-
};
|
|
175
|
-
const historyTable = {
|
|
176
|
-
_type: "insertHistory",
|
|
177
|
-
_data: [historyRow]
|
|
178
|
-
};
|
|
179
|
-
await this.db.core.import({
|
|
180
|
-
[`${this.treeKey}InsertHistory`]: historyTable
|
|
155
|
+
const results = await this.db.insertTrees(this.treeKey, trees, {
|
|
156
|
+
skipNotification: options.skipNotification
|
|
181
157
|
});
|
|
182
|
-
|
|
183
|
-
const treeKeyRoute = Route.fromFlat(`/${this.treeKey}`);
|
|
184
|
-
this.db.notify.notify(treeKeyRoute, historyRow);
|
|
185
|
-
}
|
|
186
|
-
return treeRootRef;
|
|
158
|
+
return results[0][`${this.treeKey}Ref`];
|
|
187
159
|
}
|
|
188
160
|
/**
|
|
189
161
|
* Get the tree table key
|
|
@@ -499,6 +471,14 @@ class FsScanner {
|
|
|
499
471
|
});
|
|
500
472
|
}
|
|
501
473
|
}
|
|
474
|
+
const DEFAULT_TIMEOUTS = {
|
|
475
|
+
dbQuery: 1e4,
|
|
476
|
+
fetchTree: 2e4,
|
|
477
|
+
extract: 15e3,
|
|
478
|
+
restore: 15e3,
|
|
479
|
+
syncCallback: 25e3,
|
|
480
|
+
debounceMs: 300
|
|
481
|
+
};
|
|
502
482
|
class FsAgent {
|
|
503
483
|
_scanner;
|
|
504
484
|
_adapter;
|
|
@@ -509,11 +489,15 @@ class FsAgent {
|
|
|
509
489
|
_stopSync;
|
|
510
490
|
_stopSyncFromDb;
|
|
511
491
|
_lastSentRef;
|
|
492
|
+
/** Content fingerprint of the last tree we broadcasted (paths+blobIds) */
|
|
493
|
+
_lastSentContentKey;
|
|
494
|
+
_timeouts;
|
|
512
495
|
constructor(rootPath, bs, options = {}) {
|
|
513
496
|
this._rootPath = rootPath;
|
|
514
497
|
this._bs = bs || new BsMem();
|
|
515
498
|
this._db = options.db;
|
|
516
499
|
this._treeKey = options.treeKey;
|
|
500
|
+
this._timeouts = { ...DEFAULT_TIMEOUTS, ...options.timeouts };
|
|
517
501
|
this._scanner = new FsScanner(rootPath, { ...options, bs: this._bs });
|
|
518
502
|
this._adapter = new FsBlobAdapter(this._bs);
|
|
519
503
|
if (this._db && this._treeKey) {
|
|
@@ -547,6 +531,51 @@ class FsAgent {
|
|
|
547
531
|
get adapter() {
|
|
548
532
|
return this._adapter;
|
|
549
533
|
}
|
|
534
|
+
/**
|
|
535
|
+
* Gets the current timeout configuration
|
|
536
|
+
*/
|
|
537
|
+
get timeouts() {
|
|
538
|
+
return this._timeouts;
|
|
539
|
+
}
|
|
540
|
+
/**
|
|
541
|
+
* Wraps a promise with a timeout.
|
|
542
|
+
* Rejects with a descriptive error if the promise does not settle
|
|
543
|
+
* within the given number of milliseconds.
|
|
544
|
+
* @param promise - The promise to guard
|
|
545
|
+
* @param ms - Maximum allowed time in milliseconds
|
|
546
|
+
* @param label - Human-readable label included in the error message
|
|
547
|
+
*/
|
|
548
|
+
static _withTimeout(promise, ms, label) {
|
|
549
|
+
return new Promise((resolve, reject) => {
|
|
550
|
+
const timer = setTimeout(() => {
|
|
551
|
+
reject(new Error(`Timeout after ${ms}ms: ${label}`));
|
|
552
|
+
}, ms);
|
|
553
|
+
promise.then(
|
|
554
|
+
(value) => {
|
|
555
|
+
clearTimeout(timer);
|
|
556
|
+
resolve(value);
|
|
557
|
+
},
|
|
558
|
+
(err) => {
|
|
559
|
+
clearTimeout(timer);
|
|
560
|
+
reject(err);
|
|
561
|
+
}
|
|
562
|
+
);
|
|
563
|
+
});
|
|
564
|
+
}
|
|
565
|
+
/**
|
|
566
|
+
* Sends a ref through the connector.
|
|
567
|
+
* Uses `sendWithAck()` when the connector has `requireAck` enabled,
|
|
568
|
+
* otherwise falls back to fire-and-forget `send()`.
|
|
569
|
+
* @param connector - The Connector to send through
|
|
570
|
+
* @param ref - The ref to broadcast
|
|
571
|
+
*/
|
|
572
|
+
async _sendRef(connector, ref) {
|
|
573
|
+
if (connector.syncConfig?.requireAck) {
|
|
574
|
+
await connector.sendWithAck(ref);
|
|
575
|
+
} else {
|
|
576
|
+
connector.send(ref);
|
|
577
|
+
}
|
|
578
|
+
}
|
|
550
579
|
/**
|
|
551
580
|
* Starts automatic syncing to database
|
|
552
581
|
* Note: Auto-sync requires Connector which is not available in constructor.
|
|
@@ -727,8 +756,15 @@ class FsAgent {
|
|
|
727
756
|
processed.add(currentHash);
|
|
728
757
|
let result;
|
|
729
758
|
try {
|
|
730
|
-
result = await
|
|
731
|
-
|
|
759
|
+
result = await FsAgent._withTimeout(
|
|
760
|
+
db.get(route, { _hash: currentHash }),
|
|
761
|
+
this._timeouts.dbQuery,
|
|
762
|
+
`db.get(${treeKey}, _hash=${currentHash.slice(0, 8)}…)`
|
|
763
|
+
);
|
|
764
|
+
} catch (error) {
|
|
765
|
+
if (error instanceof Error && error.message.startsWith("Timeout")) {
|
|
766
|
+
throw error;
|
|
767
|
+
}
|
|
732
768
|
continue;
|
|
733
769
|
}
|
|
734
770
|
const treeData = result?.rljson?.[treeKey];
|
|
@@ -751,24 +787,22 @@ class FsAgent {
|
|
|
751
787
|
return Array.from(fetchedNodes.values());
|
|
752
788
|
}
|
|
753
789
|
/**
|
|
754
|
-
*
|
|
755
|
-
*
|
|
790
|
+
* Fetches tree from database without restoring to filesystem.
|
|
791
|
+
* Separated from loadFromDb to allow content comparison before restore.
|
|
756
792
|
* @param db - Database instance
|
|
757
793
|
* @param treeKey - Tree table key
|
|
758
794
|
* @param rootRef - Root tree reference (hash)
|
|
759
|
-
* @
|
|
760
|
-
* @param options - Restore options
|
|
795
|
+
* @returns FsTree structure ready for restore
|
|
761
796
|
*/
|
|
762
|
-
async
|
|
797
|
+
async _fetchTreeFromDb(db, treeKey, rootRef) {
|
|
763
798
|
if (!rootRef || rootRef.trim() === "") {
|
|
764
799
|
throw new Error("rootRef cannot be empty");
|
|
765
800
|
}
|
|
766
801
|
const route = Route.fromFlat(treeKey);
|
|
767
|
-
const allNodes = await
|
|
768
|
-
db,
|
|
769
|
-
|
|
770
|
-
treeKey,
|
|
771
|
-
rootRef
|
|
802
|
+
const allNodes = await FsAgent._withTimeout(
|
|
803
|
+
this._fetchTreeRecursively(db, route, treeKey, rootRef),
|
|
804
|
+
this._timeouts.fetchTree,
|
|
805
|
+
`fetchTree(${treeKey}@${rootRef.slice(0, 8)}…)`
|
|
772
806
|
);
|
|
773
807
|
if (allNodes.length === 0) {
|
|
774
808
|
throw new Error(
|
|
@@ -786,10 +820,19 @@ class FsAgent {
|
|
|
786
820
|
`Root tree node "${rootRef}" not found in tree data. Available hashes: ${Array.from(trees.keys()).slice(0, 5).join(", ")}${trees.size > 5 ? "..." : ""}`
|
|
787
821
|
);
|
|
788
822
|
}
|
|
789
|
-
|
|
790
|
-
|
|
791
|
-
|
|
792
|
-
|
|
823
|
+
return { rootHash: rootRef, trees };
|
|
824
|
+
}
|
|
825
|
+
/**
|
|
826
|
+
* Loads tree from database and restores to filesystem
|
|
827
|
+
* Writes to filesystem from DB trees and Bs blobs
|
|
828
|
+
* @param db - Database instance
|
|
829
|
+
* @param treeKey - Tree table key
|
|
830
|
+
* @param rootRef - Root tree reference (hash)
|
|
831
|
+
* @param targetPath - Optional target path (defaults to rootPath)
|
|
832
|
+
* @param options - Restore options
|
|
833
|
+
*/
|
|
834
|
+
async loadFromDb(db, treeKey, rootRef, targetPath, options) {
|
|
835
|
+
const fsTree = await this._fetchTreeFromDb(db, treeKey, rootRef);
|
|
793
836
|
await this.restore(fsTree, targetPath, options);
|
|
794
837
|
}
|
|
795
838
|
/**
|
|
@@ -845,33 +888,115 @@ class FsAgent {
|
|
|
845
888
|
* @param db - Database instance
|
|
846
889
|
* @param connector - Connector instance for socket-based sync
|
|
847
890
|
* @param treeKey - Tree table key
|
|
848
|
-
* @param options - Storage options (e.g.,
|
|
891
|
+
* @param options - Storage options (e.g., skipNotification)
|
|
849
892
|
* @returns Function to stop watching
|
|
850
893
|
*/
|
|
851
894
|
async syncToDb(db, connector, treeKey, options) {
|
|
852
|
-
const initialRef = await
|
|
895
|
+
const initialRef = await FsAgent._withTimeout(
|
|
896
|
+
this.storeInDb(db, treeKey, options),
|
|
897
|
+
this._timeouts.fetchTree,
|
|
898
|
+
`syncToDb → initial storeInDb(${treeKey})`
|
|
899
|
+
);
|
|
853
900
|
if (initialRef) {
|
|
854
901
|
this._lastSentRef = initialRef;
|
|
855
|
-
|
|
902
|
+
const currentTree = this._scanner.tree;
|
|
903
|
+
if (currentTree) {
|
|
904
|
+
this._lastSentContentKey = this._contentKeyFromTree(currentTree);
|
|
905
|
+
}
|
|
906
|
+
await this._sendRef(connector, initialRef);
|
|
856
907
|
}
|
|
857
|
-
|
|
858
|
-
|
|
859
|
-
if (
|
|
860
|
-
|
|
861
|
-
|
|
862
|
-
|
|
863
|
-
if (
|
|
864
|
-
|
|
908
|
+
let debounceTimer = null;
|
|
909
|
+
const debouncedSync = () => {
|
|
910
|
+
if (debounceTimer) clearTimeout(debounceTimer);
|
|
911
|
+
debounceTimer = setTimeout(async () => {
|
|
912
|
+
debounceTimer = null;
|
|
913
|
+
const tree = this._scanner.tree;
|
|
914
|
+
if (tree) {
|
|
915
|
+
try {
|
|
916
|
+
const contentKey = this._contentKeyFromTree(tree);
|
|
917
|
+
if (contentKey === this._lastSentContentKey) {
|
|
918
|
+
return;
|
|
919
|
+
}
|
|
920
|
+
const dbAdapter = new FsDbAdapter(db, treeKey);
|
|
921
|
+
const ref = await FsAgent._withTimeout(
|
|
922
|
+
dbAdapter.storeFsTree(tree, options),
|
|
923
|
+
this._timeouts.fetchTree,
|
|
924
|
+
`syncToDb → storeFsTree(${treeKey})`
|
|
925
|
+
);
|
|
926
|
+
if (ref === this._lastSentRef) {
|
|
927
|
+
return;
|
|
928
|
+
}
|
|
929
|
+
this._lastSentRef = ref;
|
|
930
|
+
this._lastSentContentKey = contentKey;
|
|
931
|
+
if (ref) {
|
|
932
|
+
await this._sendRef(connector, ref);
|
|
933
|
+
}
|
|
934
|
+
} catch {
|
|
935
|
+
}
|
|
865
936
|
}
|
|
866
|
-
}
|
|
937
|
+
}, this._timeouts.debounceMs);
|
|
867
938
|
};
|
|
868
|
-
this._scanner.onChange(
|
|
939
|
+
this._scanner.onChange(debouncedSync);
|
|
869
940
|
await this._scanner.watch();
|
|
870
941
|
return () => {
|
|
871
|
-
|
|
942
|
+
if (debounceTimer) clearTimeout(debounceTimer);
|
|
943
|
+
this._scanner.offChange(debouncedSync);
|
|
872
944
|
this._scanner.stopWatch();
|
|
873
945
|
};
|
|
874
946
|
}
|
|
947
|
+
/**
|
|
948
|
+
* Builds a map of relativePath → blobId for all files in a tree.
|
|
949
|
+
* Used to compare trees by content rather than by hash (which includes mtime).
|
|
950
|
+
* @param tree - Tree structure to extract file content map from
|
|
951
|
+
*/
|
|
952
|
+
_getFileContentMap(tree) {
|
|
953
|
+
const map = /* @__PURE__ */ new Map();
|
|
954
|
+
for (const [, node] of tree.trees) {
|
|
955
|
+
const meta = node?.meta;
|
|
956
|
+
if (meta?.type === "file") {
|
|
957
|
+
map.set(meta.relativePath, meta.blobId ?? "");
|
|
958
|
+
} else if (meta?.type === "directory" && meta.relativePath !== ".") {
|
|
959
|
+
map.set(meta.relativePath, "<dir>");
|
|
960
|
+
}
|
|
961
|
+
}
|
|
962
|
+
return map;
|
|
963
|
+
}
|
|
964
|
+
/**
|
|
965
|
+
* Derives a deterministic string key from a content map so that two trees
|
|
966
|
+
* with identical file paths + blobIds produce the same key regardless of
|
|
967
|
+
* mtime differences.
|
|
968
|
+
* @param map - Content map (relativePath → blobId)
|
|
969
|
+
*/
|
|
970
|
+
_contentKeyFromMap(map) {
|
|
971
|
+
const sorted = Array.from(map.entries()).sort(
|
|
972
|
+
(a, b) => a[0].localeCompare(b[0])
|
|
973
|
+
);
|
|
974
|
+
return sorted.map(([p, b]) => `${p}:${b}`).join("\n");
|
|
975
|
+
}
|
|
976
|
+
/**
|
|
977
|
+
* Derives a deterministic content key from an FsTree.
|
|
978
|
+
* @param tree - Tree structure to derive content key from
|
|
979
|
+
*/
|
|
980
|
+
_contentKeyFromTree(tree) {
|
|
981
|
+
return this._contentKeyFromMap(this._getFileContentMap(tree));
|
|
982
|
+
}
|
|
983
|
+
/**
|
|
984
|
+
* Compares two trees by file content (relativePath + blobId).
|
|
985
|
+
* Ignores mtime differences — trees are equivalent if they have the same
|
|
986
|
+
* files with the same content. This prevents bounce-back restores from
|
|
987
|
+
* destroying locally-created files during bidirectional sync.
|
|
988
|
+
* @param a - First tree to compare
|
|
989
|
+
* @param b - Second tree to compare
|
|
990
|
+
*/
|
|
991
|
+
_treesHaveEquivalentContent(a, b) {
|
|
992
|
+
const aFiles = this._getFileContentMap(a);
|
|
993
|
+
const bFiles = this._getFileContentMap(b);
|
|
994
|
+
if (aFiles.size !== bFiles.size) return false;
|
|
995
|
+
for (const [path, blobId] of aFiles) {
|
|
996
|
+
if (bFiles.get(path) !== blobId) return false;
|
|
997
|
+
}
|
|
998
|
+
return true;
|
|
999
|
+
}
|
|
875
1000
|
/**
|
|
876
1001
|
* Watches database for tree changes and syncs to filesystem
|
|
877
1002
|
* Uses Connector for socket-based notifications
|
|
@@ -885,24 +1010,60 @@ class FsAgent {
|
|
|
885
1010
|
if (!this._scanner["_watcher"]) {
|
|
886
1011
|
await this._scanner.watch();
|
|
887
1012
|
}
|
|
888
|
-
|
|
889
|
-
|
|
890
|
-
|
|
891
|
-
}
|
|
892
|
-
if (treeRef === this._lastSentRef) {
|
|
893
|
-
return;
|
|
894
|
-
}
|
|
1013
|
+
let pendingRef = null;
|
|
1014
|
+
let fromDbTimer = null;
|
|
1015
|
+
const processRef = async (treeRef) => {
|
|
895
1016
|
this._scanner.pauseWatch();
|
|
896
1017
|
try {
|
|
897
|
-
await
|
|
1018
|
+
const incomingTree = await FsAgent._withTimeout(
|
|
1019
|
+
this._fetchTreeFromDb(db, treeKey, treeRef),
|
|
1020
|
+
this._timeouts.fetchTree,
|
|
1021
|
+
`syncFromDb → fetchTree(${treeKey}@${treeRef.slice(0, 8)}…)`
|
|
1022
|
+
);
|
|
1023
|
+
const currentTree = await FsAgent._withTimeout(
|
|
1024
|
+
this.extract(),
|
|
1025
|
+
this._timeouts.extract,
|
|
1026
|
+
`syncFromDb → extract(${this._rootPath})`
|
|
1027
|
+
);
|
|
1028
|
+
if (this._treesHaveEquivalentContent(currentTree, incomingTree)) {
|
|
1029
|
+
return;
|
|
1030
|
+
}
|
|
1031
|
+
await FsAgent._withTimeout(
|
|
1032
|
+
this.restore(incomingTree, void 0, restoreOptions),
|
|
1033
|
+
this._timeouts.restore,
|
|
1034
|
+
`syncFromDb → restore(${treeKey})`
|
|
1035
|
+
);
|
|
1036
|
+
const postRestoreTree = await this._scanner.scan();
|
|
1037
|
+
const dbAdapter = new FsDbAdapter(db, treeKey);
|
|
1038
|
+
const postRestoreRef = await dbAdapter.storeFsTree(postRestoreTree, {
|
|
1039
|
+
skipNotification: true
|
|
1040
|
+
});
|
|
1041
|
+
this._lastSentRef = postRestoreRef;
|
|
1042
|
+
this._lastSentContentKey = this._contentKeyFromTree(postRestoreTree);
|
|
898
1043
|
} catch {
|
|
899
1044
|
} finally {
|
|
900
1045
|
this._scanner.resumeWatch();
|
|
901
1046
|
}
|
|
902
1047
|
};
|
|
1048
|
+
const syncCallback = async (treeRef) => {
|
|
1049
|
+
if (!treeRef || typeof treeRef !== "string") {
|
|
1050
|
+
return;
|
|
1051
|
+
}
|
|
1052
|
+
pendingRef = treeRef;
|
|
1053
|
+
if (fromDbTimer) clearTimeout(fromDbTimer);
|
|
1054
|
+
fromDbTimer = setTimeout(async () => {
|
|
1055
|
+
fromDbTimer = null;
|
|
1056
|
+
const ref = pendingRef;
|
|
1057
|
+
pendingRef = null;
|
|
1058
|
+
if (ref) {
|
|
1059
|
+
await processRef(ref);
|
|
1060
|
+
}
|
|
1061
|
+
}, this._timeouts.debounceMs);
|
|
1062
|
+
};
|
|
903
1063
|
connector.listen(syncCallback);
|
|
904
1064
|
return () => {
|
|
905
|
-
|
|
1065
|
+
if (fromDbTimer) clearTimeout(fromDbTimer);
|
|
1066
|
+
connector.tearDown();
|
|
906
1067
|
};
|
|
907
1068
|
}
|
|
908
1069
|
/**
|
|
@@ -913,16 +1074,21 @@ class FsAgent {
|
|
|
913
1074
|
* @param treeKey - Tree table key (route will be `/${treeKey}`)
|
|
914
1075
|
* @param client - Client instance with io and bs properties
|
|
915
1076
|
* @param socket - Socket instance for connector communication
|
|
916
|
-
* @param options - Optional FsAgent options (db and treeKey are set automatically)
|
|
1077
|
+
* @param options - Optional FsAgent options (db and treeKey are set automatically).
|
|
1078
|
+
* `syncConfig` and `clientIdentity` from these options are forwarded to
|
|
1079
|
+
* the Connector so that a single config origin governs all layers.
|
|
917
1080
|
* @returns Configured FsAgent instance with simplified sync API
|
|
918
1081
|
* @example
|
|
919
1082
|
* ```typescript
|
|
920
|
-
* const
|
|
1083
|
+
* const syncConfig: SyncConfig = { requireAck: true, maxDedupSetSize: 5000 };
|
|
1084
|
+
* const agent = await FsAgent.fromClient(
|
|
1085
|
+
* './my-folder', 'sharedTree', client, socket, { syncConfig },
|
|
1086
|
+
* );
|
|
921
1087
|
* // Simplified sync methods - no db/connector/treeKey needed
|
|
922
|
-
* await agent.syncToDbSimple(
|
|
1088
|
+
* await agent.syncToDbSimple();
|
|
923
1089
|
* await agent.syncFromDbSimple({ cleanTarget: true });
|
|
924
1090
|
* // Original methods still work
|
|
925
|
-
* await agent.syncToDb(db, connector, treeKey
|
|
1091
|
+
* await agent.syncToDb(db, connector, treeKey);
|
|
926
1092
|
* ```
|
|
927
1093
|
*/
|
|
928
1094
|
static async fromClient(filePath, treeKey, client, socket, options) {
|
|
@@ -935,7 +1101,13 @@ class FsAgent {
|
|
|
935
1101
|
const { Db: Db2, Connector } = await import("@rljson/db");
|
|
936
1102
|
const db = new Db2(client.io);
|
|
937
1103
|
const route = Route.fromFlat(`/${treeKey}`);
|
|
938
|
-
const connector = new Connector(
|
|
1104
|
+
const connector = new Connector(
|
|
1105
|
+
db,
|
|
1106
|
+
route,
|
|
1107
|
+
socket,
|
|
1108
|
+
options?.syncConfig,
|
|
1109
|
+
options?.clientIdentity
|
|
1110
|
+
);
|
|
939
1111
|
const agent = new FsAgent(filePath, client.bs, options);
|
|
940
1112
|
const enhancedAgent = agent;
|
|
941
1113
|
enhancedAgent.syncToDbSimple = async (syncOptions) => {
|
|
@@ -944,6 +1116,16 @@ class FsAgent {
|
|
|
944
1116
|
enhancedAgent.syncFromDbSimple = async (restoreOpts) => {
|
|
945
1117
|
return agent.syncFromDb(db, connector, treeKey, restoreOpts);
|
|
946
1118
|
};
|
|
1119
|
+
if (typeof client.onDisconnect === "function") {
|
|
1120
|
+
client.onDisconnect(() => {
|
|
1121
|
+
agent.scanner.pauseWatch();
|
|
1122
|
+
});
|
|
1123
|
+
}
|
|
1124
|
+
if (typeof client.onReconnect === "function") {
|
|
1125
|
+
client.onReconnect(() => {
|
|
1126
|
+
agent.scanner.resumeWatch();
|
|
1127
|
+
});
|
|
1128
|
+
}
|
|
947
1129
|
return enhancedAgent;
|
|
948
1130
|
}
|
|
949
1131
|
/** Example instance for test purposes */
|
|
@@ -999,7 +1181,7 @@ async function runClientServerSetup(opts = {}) {
|
|
|
999
1181
|
const agentB = new FsAgent(folderB, clientB.bs);
|
|
1000
1182
|
const helloPathA = join(folderA, "hello.txt");
|
|
1001
1183
|
await writeFile(helloPathA, "Hello from Client A");
|
|
1002
|
-
const rootRef = await agentA.storeInDb(clientDbA, treeKey, {
|
|
1184
|
+
const rootRef = await agentA.storeInDb(clientDbA, treeKey, { skipNotification: true });
|
|
1003
1185
|
await agentB.loadFromDb(clientDbB, treeKey, rootRef);
|
|
1004
1186
|
const contentB = await readFile(join(folderB, "hello.txt"), "utf8");
|
|
1005
1187
|
const cleanup = async () => {
|
package/dist/fs-db-adapter.d.ts
CHANGED
|
@@ -5,12 +5,19 @@ import { FsTree } from './fs-scanner.js';
|
|
|
5
5
|
*/
|
|
6
6
|
export interface StoreFsTreeOptions {
|
|
7
7
|
/**
|
|
8
|
-
* Whether to
|
|
8
|
+
* Whether to skip notifications after storing (defaults to false).
|
|
9
|
+
* When false, observers (e.g. Connector) are notified automatically
|
|
10
|
+
* via the standard db.insertTrees() pipeline.
|
|
9
11
|
*/
|
|
10
|
-
|
|
12
|
+
skipNotification?: boolean;
|
|
11
13
|
}
|
|
12
14
|
/**
|
|
13
|
-
* Adapter for storing filesystem trees in a database
|
|
15
|
+
* Adapter for storing filesystem trees in a database.
|
|
16
|
+
*
|
|
17
|
+
* Uses `db.insertTrees()` to go through the standard insert pipeline:
|
|
18
|
+
* - TreeController writes each node
|
|
19
|
+
* - InsertHistoryRow is created automatically
|
|
20
|
+
* - `notify.notify()` fires so Connector observers broadcast the ref
|
|
14
21
|
*/
|
|
15
22
|
export declare class FsDbAdapter {
|
|
16
23
|
private db;
|
package/dist/index.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
export { FsAgent, type FsAgentOptions } from './fs-agent.ts';
|
|
1
|
+
export { FsAgent, type FsAgentOptions, type RestoreOptions, type TimeoutConfig, } from './fs-agent.ts';
|
|
2
2
|
export { FsBlobAdapter, type BlobToFileOptions, type FileBlobMeta, type FileToBlobOptions, } from './fs-blob-adapter.ts';
|
|
3
3
|
export { FsDbAdapter, type StoreFsTreeOptions } from './fs-db-adapter.ts';
|
|
4
4
|
export { FsScanner, type FsChange, type FsChangeCallback, type FsChangeType, type FsNodeMeta, type FsScanOptions, type FsTree, } from './fs-scanner.ts';
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@rljson/fs-agent",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.5",
|
|
4
4
|
"description": "Rljson fs-agent description",
|
|
5
5
|
"homepage": "https://github.com/rljson/fs-agent",
|
|
6
6
|
"bugs": "https://github.com/rljson/fs-agent/issues",
|
|
@@ -20,40 +20,40 @@
|
|
|
20
20
|
],
|
|
21
21
|
"type": "module",
|
|
22
22
|
"devDependencies": {
|
|
23
|
-
"@rljson/server": "^0.0.
|
|
24
|
-
"@types/node": "^25.
|
|
25
|
-
"@typescript-eslint/eslint-plugin": "^8.
|
|
26
|
-
"@typescript-eslint/parser": "^8.
|
|
23
|
+
"@rljson/server": "^0.0.10",
|
|
24
|
+
"@types/node": "^25.3.1",
|
|
25
|
+
"@typescript-eslint/eslint-plugin": "^8.56.1",
|
|
26
|
+
"@typescript-eslint/parser": "^8.56.1",
|
|
27
27
|
"@vitest/coverage-v8": "^4.0.18",
|
|
28
28
|
"cross-env": "^10.1.0",
|
|
29
|
-
"eslint": "~9.39.
|
|
30
|
-
"eslint-plugin-jsdoc": "^62.
|
|
31
|
-
"eslint-plugin-tsdoc": "^0.5.
|
|
29
|
+
"eslint": "~9.39.3",
|
|
30
|
+
"eslint-plugin-jsdoc": "^62.7.1",
|
|
31
|
+
"eslint-plugin-tsdoc": "^0.5.2",
|
|
32
32
|
"globals": "^17.3.0",
|
|
33
33
|
"jsdoc": "^4.0.5",
|
|
34
34
|
"read-pkg": "^10.1.0",
|
|
35
35
|
"typescript": "~5.9.3",
|
|
36
|
-
"typescript-eslint": "^8.
|
|
36
|
+
"typescript-eslint": "^8.56.1",
|
|
37
37
|
"vite": "^7.3.1",
|
|
38
38
|
"vite-node": "^5.3.0",
|
|
39
39
|
"vite-plugin-dts": "^4.5.4",
|
|
40
|
-
"vite-tsconfig-paths": "^6.1.
|
|
40
|
+
"vite-tsconfig-paths": "^6.1.1",
|
|
41
41
|
"vitest": "^4.0.18",
|
|
42
42
|
"vitest-dom": "^0.1.1"
|
|
43
43
|
},
|
|
44
44
|
"dependencies": {
|
|
45
|
-
"@rljson/bs": "^0.0.
|
|
46
|
-
"@rljson/db": "^0.0.
|
|
45
|
+
"@rljson/bs": "^0.0.21",
|
|
46
|
+
"@rljson/db": "^0.0.15",
|
|
47
47
|
"@rljson/hash": "^0.0.18",
|
|
48
|
-
"@rljson/io": "^0.0.
|
|
48
|
+
"@rljson/io": "^0.0.66",
|
|
49
49
|
"@rljson/json": "^0.0.23",
|
|
50
|
-
"@rljson/rljson": "^0.0.
|
|
50
|
+
"@rljson/rljson": "^0.0.78",
|
|
51
51
|
"socket.io": "^4.8.3",
|
|
52
52
|
"socket.io-client": "^4.8.3"
|
|
53
53
|
},
|
|
54
54
|
"scripts": {
|
|
55
|
-
"build": "
|
|
56
|
-
"test": "cross-env NODE_OPTIONS=--max-old-space-size=8192
|
|
55
|
+
"build": "pnpm exec vite build && tsc && node scripts/copy-readme-to-dist.js",
|
|
56
|
+
"test": "cross-env NODE_OPTIONS=--max-old-space-size=8192 pnpm exec vitest run --coverage && pnpm run lint",
|
|
57
57
|
"prebuild": "npm run test",
|
|
58
58
|
"lint": "pnpm exec eslint .",
|
|
59
59
|
"updateGoldens": "cross-env UPDATE_GOLDENS=true pnpm test"
|