@pixlcore/xycli 1.0.6 → 1.0.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,13 @@
1
1
  # xyCLI Changelog
2
2
 
3
+ ## Version v1.0.6
4
+
5
+ > September 22, 2026
6
+
7
+ - [`985c132`](https://github.com/pixlcore/xycli/commit/985c132607e1dcdabbadc829d6f21e599a2eb52b): Version 1.0.6
8
+ - [`f0aac32`](https://github.com/pixlcore/xycli/commit/f0aac320cbd301a5dc45862edaec9f842c2c6c79): Feature: Add "xy upgrade" command.
9
+ - [`a9b653b`](https://github.com/pixlcore/xycli/commit/a9b653b9649afd15d93cb3d29ee4f23da47bde0a): Bug Fix: Sync files with similar stems and neighbors can get mixed up. Fixes #4.
10
+
3
11
  ## Version v1.0.5
4
12
 
5
13
  > September 21, 2026
package/README.md CHANGED
@@ -499,7 +499,7 @@ cd xyops-automation
499
499
  xy sync setup events plugins categories --file_props script,params.script
500
500
  ```
501
501
 
502
- Setup creates one file per definition and can extract scripts into adjacent source files. Events and workflows are automatically grouped into subfolders named after their Categories. You can reorganize the tree afterward; sync scans subdirectories and matches definitions by resource type and exact ID.
502
+ Setup creates one file per definition and can extract scripts into adjacent source files. The same `--file_props script,params.script` selection also extracts matching scripts from workflow nodes automatically. Events and workflows are grouped into subfolders named after their Categories. You can reorganize the tree afterward; sync scans subdirectories and matches definitions by resource type and exact ID.
503
503
 
504
504
  Preview a push or pull, then apply the direction you want:
505
505
 
package/docs/help.md CHANGED
@@ -200,6 +200,8 @@ xy sync ./ --down events,plugins
200
200
  xy sync ./ --up categories,plugins --down events --dry
201
201
  xy sync ./ --up events --down events --dry
202
202
  xy sync ./ --up all --dry
203
+ xy sync ./ --up buckets --dry
204
+ xy sync ./ --down buckets
203
205
  xy sync --up events,plugins --verbose
204
206
  ```
205
207
 
@@ -212,7 +214,7 @@ Pass one or more base directories, or omit them to scan the current directory. F
212
214
  - `--dry` previews changes without applying them.
213
215
  - `--verbose` shows full diffs and API requests and responses.
214
216
 
215
- You must enable at least one direction. `TYPES` is a comma-separated list of `alerts`, `api_keys`, `categories`, `channels`, `events`, `groups`, `monitors`, `plugins`, `tags`, or `web_hooks`. Use `all` to select every supported type. Workflows are included under `events`. Buckets, Secrets, Users, and Roles are not supported.
217
+ You must enable at least one direction. `TYPES` is a comma-separated list of `alerts`, `api_keys`, `buckets`, `categories`, `channels`, `events`, `groups`, `monitors`, `plugins`, `tags`, or `web_hooks`. Use `all` to select every supported type. Workflows are included under `events`. Bucket sync requires xyOps v1.1.2 or later. Secrets, Users, and Roles are not supported.
216
218
 
217
219
  Enable both directions for the same type to use experimental two-way sync. The newest modification time wins, including external property files. Git checkouts can reset filesystem timestamps, so prefer one-way sync when reliable conflict handling matters. Dry runs never update xyOps, write files, run completion commands, or send notifications.
218
220
 
@@ -222,6 +224,8 @@ Repeated sources with the same type and ID generate warnings, even if their cont
222
224
 
223
225
  External string properties are loaded automatically from adjacent files named `BASENAME-PROPERTY.EXT`, such as `My-Plugin-script.js` beside `My-Plugin.json`, or `My-Event-params.script.sh` beside `My-Event.json`. The extension is your choice. Keep only one external file per property, and rename its basename along with the JSON file.
224
226
 
227
+ Bucket metadata uses a normal XYPDF source. For a Bucket titled `My Bucket Name`, its separate JSON data and uploaded files live beside `buckets/My-Bucket-Name.json` in `buckets/My-Bucket-Name/data.json` and `buckets/My-Bucket-Name/files/`. Both `data.json` and `files/` are required for a selected local bucket. Bucket data upsync replaces the entire remote data object. Bucket files are matched by xyOps filename normalization and transferred according to size and modification time. Two-way bucket sync uses the data record and file timestamps from xyOps; a transfer updates local file times to match. Bucket files missing locally are deleted remotely only with up-only `--delete buckets`.
228
+
225
229
  **Delete mode**
226
230
 
227
231
  Add `--delete TYPES` to delete xyOps objects that have no matching source in the scanned directories. Deletion requires up-only sync and a complete local inventory. The CLI rejects deletion if upsync is disabled or any downsync direction is enabled, including saved defaults and dry runs. Use `--down false` to disable inherited downsync. See [Delete mode](sync.md#delete-mode) before using it.
@@ -282,6 +286,7 @@ Export existing xyOps definitions into a starter sync tree in the current direct
282
286
 
283
287
  ```sh
284
288
  xy sync setup events plugins categories --dry
289
+ xy sync setup buckets
285
290
  xy sync setup events plugins categories --file_props script,params.script
286
291
  xy sync setup events --file_props params.script --default_ext ps1
287
292
  xy sync setup events plugins --new
@@ -296,6 +301,8 @@ Choose one or more resource types separated by spaces, or use `all`. Supported t
296
301
 
297
302
  Setup creates one folder per resource type and one XYPDF JSON file per object, using its title as a filename slug. Selecting `events` exports both Events and workflows, with workflows placed in a separate `workflows` folder. Events and workflows are grouped into subfolders using their Category title as a slug. Both are synced with `--up events` or `--down events`. Setup stops before writing if an Event or workflow selected for export refers to a missing Category.
298
303
 
304
+ For each Bucket, setup also writes `BUCKET-SLUG/data.json` and downloads its files into `BUCKET-SLUG/files/` beside the XYPDF file. `--new` skips Bucket definitions already present locally, including their content. `--force` permits overwriting existing Bucket content files.
305
+
299
306
  - `--file_props PATHS` extracts string properties into adjacent files.
300
307
  - `--default_ext EXT` sets the fallback neighbor extension instead of `txt`.
301
308
  - `--new` writes only remote definitions not already represented locally.
@@ -308,6 +315,8 @@ Setup creates one folder per resource type and one XYPDF JSON file per object, u
308
315
 
309
316
  For `--file_props`, use comma-separated property names or dot paths, such as `script,params.script`. Only nonempty string values are extracted. The JSON value becomes `(External)`, and sync loads the external file automatically. Setup chooses extensions from executable commands, shebangs, and JSON content when possible. Use `--default_ext ps1` (or `--default_ext .ps1`) to replace the `.txt` fallback for properties whose type cannot be detected. The override does not replace an extension that setup successfully detects.
310
317
 
318
+ The same property selection also checks every workflow node automatically. For example, `params.script` in a node's `data` becomes a neighbor such as `My-Workflow-workflow.nt3sr3y4.params.script.sh`. The node ID in the filename identifies which node owns the property. No additional setup option is needed.
319
+
311
320
  Use `--new` from the root of an existing sync tree to add definitions that exist in xyOps but have no local source. Setup scans recursively and matches existing sources by item type and exact ID, so renamed files and custom folder layouts are preserved. New definitions use the normal suggested type and Category folders. Malformed, duplicate, unsupported, or orphaned local sources stop the operation before any new files are written. `--new` cannot be combined with `--force`, and it does not create new definitions in xyOps.
312
321
 
313
322
  An explicitly supplied `--down_cmd` runs from the current setup directory after at least one definition is written. It does not run when `--new` finds nothing or during a dry run. Saved sync completion commands are not inherited by setup.
package/docs/sync.md CHANGED
@@ -54,6 +54,7 @@ Use these names with `--up`, `--down`, `--delete`, and setup:
54
54
  | --- | --- |
55
55
  | `alerts` | Alert definitions, rather than their invocations. |
56
56
  | `api_keys` | API Key definitions. |
57
+ | `buckets` | Bucket definitions, JSON data, and uploaded files. Requires xyOps 1.1.2 or later. |
57
58
  | `categories` | Event Categories. |
58
59
  | `channels` | Notification Channels. |
59
60
  | `events` | Events and workflows. |
@@ -65,7 +66,7 @@ Use these names with `--up`, `--down`, `--delete`, and setup:
65
66
 
66
67
  `all` selects every supported type. Supply multiple types as a comma-separated string without spaces, such as `events,plugins,categories`.
67
68
 
68
- Buckets, Secret Vaults, Users, and Roles are not supported by sync. Individual workers, jobs, Tickets, Alert history, and snapshots are also outside its definition-sync scope. Workflows use the `events` selection even though setup places them in a separate folder (for convenience).
69
+ Secret Vaults, Users, and Roles are not supported by sync. Individual workers, jobs, Tickets, Alert history, and snapshots are also outside its definition-sync scope. Workflows use the `events` selection even though setup places them in a separate folder (for convenience).
69
70
 
70
71
  ### Permissions
71
72
 
@@ -96,7 +97,7 @@ Setup always writes below the current working directory. Type names after `setup
96
97
  xy sync setup events plugins categories --file_props script,params.script --dry
97
98
  ```
98
99
 
99
- The `--file_props` switch will externalize specific JSON property paths into individually named files, for easy editing. This is great for storing Plugin scripts and Event shell scripts in separate files alongside the main JSON definition files.
100
+ The `--file_props` switch will externalize specific JSON property paths into individually named files, for easy editing. This is great for storing Plugin scripts, Event shell scripts, and workflow node scripts in separate files alongside the main JSON definition files.
100
101
 
101
102
  The preview lists the folders and files that would be written. `--dry` creates no directories or files.
102
103
 
@@ -122,10 +123,13 @@ xyops-automation/
122
123
  workflows/
123
124
  Deployments/
124
125
  Release-Pipeline.json
126
+ Release-Pipeline-workflow.nt3sr3y4.params.script.sh
125
127
  ```
126
128
 
127
129
  Setup creates one folder per selected resource type and one JSON file per definition. Event and workflow exports are separated into `events/` and `workflows/`, then grouped into subfolders using their Category titles. Both participate in normal sync through the `events` selection.
128
130
 
131
+ Bucket setup also creates a sibling directory for each Bucket's content. For example, a Bucket titled `My Bucket Name` has `buckets/My-Bucket-Name.json`, `buckets/My-Bucket-Name/data.json`, and `buckets/My-Bucket-Name/files/` for its definition, data, and uploaded files. Move or rename the sibling directory with the XYPDF file. A normal Bucket sync requires both the data file and files directory, even when they contain an empty object and no files. Bucket data upsync replaces the full remote data object, including removed keys. File transfers match xyOps normalized filenames and align local modification times with the server's timestamps after each transfer.
132
+
129
133
  Folder and filenames come from Category and definition titles: sequences of non-word characters become hyphens, and leading or trailing hyphens are removed. For example, the Event `Nightly Backup` in the `Operations` Category becomes `events/Operations/Nightly-Backup.json`. Different titles can produce the same path, so review the preview for collisions. Setup stops before writing anything if an Event or workflow selected for export refers to a Category that is not present in xyOps.
130
134
 
131
135
  ### 4. Review and version the tree
@@ -242,9 +246,12 @@ This extracts the selected content, if found, and moves it to a neighboring file
242
246
  | --- | --- | --- |
243
247
  | `My-Plugin.json` | `script` | `My-Plugin-script.js` |
244
248
  | `My-Event.json` | `params.script` | `My-Event-params.script.sh` |
249
+ | `Release-Pipeline.json` | `workflow.nt3sr3y4.params.script` | `Release-Pipeline-workflow.nt3sr3y4.params.script.sh` |
245
250
 
246
251
  The general naming pattern is **`BASENAME-PROPERTY.EXT`**, in the same directory as `BASENAME.json`. Dot paths such as `params.script` identify nested properties. Setup chooses an extension from the executable command or a script's shebang when possible, falling back to JSON-looking content or `.txt`. Normal sync accepts your chosen extension.
247
252
 
253
+ The same `--file_props script,params.script` selection checks every node in a workflow automatically. If a node has a nonempty string at `data.params.script`, setup writes a neighbor for that node and leaves `(External)` in the workflow JSON. In its filename, `workflow.NODE_ID.params.script` identifies the node by its ID and then reads `data.params.script`. No additional option is needed for workflow nodes.
254
+
248
255
  Use `--default_ext EXT` to replace the `.txt` fallback when setup cannot detect a type. For example, `--default_ext ps1` and `--default_ext .ps1` both produce `.ps1` neighbors. Detected interpreter and JSON extensions still take precedence. The fallback applies to every undetected property selected by that setup command, so group properties with the same expected file type or run setup separately when needed.
249
256
 
250
257
  External properties are detected automatically when sync runs; you do not need to repeat `--file_props`. The neighbor's text replaces that property in the local definition before comparison and before an upload. A pull writes the latest value into the existing neighbor file.
@@ -435,7 +442,23 @@ For each differing definition, the engine compares xyOps's `modified` timestamp
435
442
 
436
443
  Two-way sync is experimental. Keep clocks synchronized, review dry runs, and preserve a history of both configuration and scripts. Git checkout, clone, restore, or file copying can give old content a new filesystem timestamp.
437
444
 
438
- For automatic two-way operation, use a persistent tree and run every minute. Both directions are considered on each run; you do not need separate alternating up and down jobs. Newly created definitions still need a setup export or individual local source, and remotely deleted definitions leave their old source files behind until you remove them.
445
+ Bucket data and files use their own modification times, separate from the Bucket definition. When contents differ, two-way sync compares `data.json` with the data record's modification time and each local file with the matching manifest date. Equal timestamps with differing Bucket content are reported as conflicts. After a transfer, xyCLI sets the local modification time to the server's returned time.
446
+
447
+ Bucket file comparison uses size and modification time because the manifest has no content hash. A same-size change with an unchanged timestamp cannot be detected.
448
+
449
+ For automatic two-way operation, use a persistent tree and run every minute. Both directions are considered on each run; you do not need separate alternating up and down jobs. Newly created definitions still need a setup export or individual local source.
450
+
451
+ ### Remove a definition during two-way sync
452
+
453
+ Two-way sync does not propagate deletions. Definition deletions are usually infrequent, so coordinating them manually is generally practical. Keep `--delete false` in the scheduled command, and remove a definition from both sides deliberately:
454
+
455
+ 1. Delete the definition in the xyOps web interface first.
456
+ 2. Remove its local XYPDF JSON file and any neighboring external property files. If the tree is stored in Git, commit and push their removal so the dedicated sync checkout receives the deletion.
457
+ 3. Run sync again, or resume the schedule, after the local files are gone. Pause the schedule during these steps if you want to avoid a failed pass between the two removals.
458
+
459
+ If a sync runs after the xyOps deletion but before local cleanup, the leftover source produces a missing-object warning. The run exits with a failure status before updating or downloading any definitions, and a script that stops on failure will not reach a following `sync setup --new` step. The deleted definition is not recreated. Remove the stale files and retry.
460
+
461
+ Do not remove the Git files first while the definition still exists in xyOps. A scheduled `sync setup --new` pass can export the still-existing definition back into the repository. See the [Two-Way Git Sync Tutorial](two-way-git-tutorial.md#delete-a-definition-from-both-sides) for the Git workflow.
439
462
 
440
463
  ## Remote-management warnings
441
464
 
@@ -495,6 +518,8 @@ Deleting an Event does not automatically remove its dependencies. Remote API rul
495
518
 
496
519
  Deletion only begins after every update has completed and the sync-tracking state has been written without warnings or errors. If either phase fails, the entire delete pass is skipped for that run. Earlier successful updates remain in place, so correct the error and retry; definitions already updated will compare equal. Once deletion begins, successful earlier deletions are not rolled back if a later deletion fails.
497
520
 
521
+ With `--delete buckets`, a remote Bucket file missing from a selected local Bucket's `files/` directory is deleted individually. Removing the Bucket XYPDF source deletes the whole Bucket, including all its data and files. A missing or invalid `data.json`, incomplete file inventory, or failed transfer prevents Bucket file deletion and the whole-object delete pass.
522
+
498
523
  An unavailable base directory stops the scan, but an existing empty directory can look like an empty inventory. Do not treat that guard as protection against an empty mount, accidental file removal, or missing exports of your own definitions.
499
524
 
500
525
  Keep deletion disabled in general-purpose cron, Git hooks, and CI until you have a deliberate policy for maintaining and checking the complete inventory. Disable saved deletion for other runs with `--delete false`.
@@ -67,7 +67,7 @@ xy sync setup events --file_props script,params.script --default_ext ps1 --dry
67
67
 
68
68
  The `events` selection includes both Events and workflows. Setup places workflows in their own `workflows/` directory, then groups Events and workflows into subfolders named after their Categories.
69
69
 
70
- `--file_props script,params.script` extracts common Plugin and Event script properties into neighboring files. If your Plugin uses another property, add its dot path to the list. For example, use `script,params.script,params.command` when an Event stores its PowerShell source in `params.command`.
70
+ `--file_props script,params.script` extracts common script properties into neighboring files. It also checks each workflow node automatically, so a node's `data.params.script` gets its own file named with that node's ID. If a property uses another path, add its dot path to the list. For example, use `script,params.script,params.command` when an Event stores its PowerShell source in `params.command`.
71
71
 
72
72
  The optional `--default_ext ps1` uses `.ps1` when setup cannot detect a neighbor file's language from an executable command, shebang, or JSON content. Remove it or choose another extension if PowerShell is not the appropriate fallback for your repository.
73
73
 
@@ -108,7 +108,7 @@ This considers both directions for every local Event and workflow source:
108
108
  - A newer xyOps definition is downloaded to its existing local file.
109
109
  - Identical definitions are left alone.
110
110
 
111
- The `--delete false` switch is intentional. Two-way sync does not propagate deletions safely. Delete an item deliberately on both sides, and keep deletion disabled in the scheduled command.
111
+ The `--delete false` switch is intentional. Two-way sync does not propagate deletions. Keep deletion disabled in the scheduled command, and follow the [coordinated deletion steps](#delete-a-definition-from-both-sides) when retiring an item.
112
112
 
113
113
  Review the displayed diffs carefully. If the wrong side would win, check the clocks and file modification times before continuing.
114
114
 
@@ -245,6 +245,18 @@ When someone creates a new Event in xyOps, the ordinary sync pass does not know
245
245
 
246
246
  `--new` is additive. It does not overwrite existing sources, and it matches existing definitions by type and exact ID rather than filename. This preserves renamed files and custom folder layouts.
247
247
 
248
+ ### Delete a definition from both sides
249
+
250
+ Event and workflow deletions are usually infrequent, so it is practical to coordinate the xyOps and Git changes manually. Pause the scheduled script if you want to avoid a failed pass while you make both changes:
251
+
252
+ 1. Delete the definition in the xyOps web interface first.
253
+ 2. Remove its XYPDF JSON file and all neighboring property files from Git, then commit and push the deletion. Make sure the dedicated sync checkout receives that commit before its next sync pass.
254
+ 3. Resume the scheduled script and check its next run.
255
+
256
+ If the script runs after the xyOps deletion but before the Git deletion reaches its checkout, ordinary sync reports a missing-object warning and exits with a failure status. It does not recreate the definition or process other changes. The script's `set -e` also prevents the following `sync setup --new` pass. Once the stale files are removed, the next run can proceed normally.
257
+
258
+ Do not remove the Git files first. While the definition still exists in xyOps, `sync setup --new` can export it again and commit the files back to Git. This is a manual deletion procedure, not automatic two-way deletion. See the [Sync Guide](sync.md#remove-a-definition-during-two-way-sync) for the general behavior.
259
+
248
260
  ## Troubleshooting
249
261
 
250
262
  | Symptom | What to check |
@@ -255,7 +267,7 @@ When someone creates a new Event in xyOps, the ordinary sync pass does not know
255
267
  | The wrong side wins a two-way conflict | Check NTP, the xyOps conductor clock, and the modification times of the JSON and neighboring files. A checkout, restore, or file copy can refresh local timestamps. |
256
268
  | A new Event is not exported | Confirm `sync setup events --new` runs from the repository root and that the API Key can see the Event. Add the correct property path if its script is stored somewhere other than `params.script`. |
257
269
  | A new Git file is not created in xyOps | Ordinary sync updates definitions that already exist on both sides. Create or import the definition into xyOps first, then let sync manage subsequent changes. |
258
- | A deleted item reappears or causes an error | Two-way deletion is not supported. Remove the local source and the xyOps definition deliberately as one coordinated operation. |
270
+ | A deleted item reappears or causes an error | Follow the [coordinated deletion steps](#delete-a-definition-from-both-sides): delete in xyOps first, then remove and commit its local files. A leftover source stops sync until it is removed. |
259
271
  | A cron run is skipped | Another cron invocation may hold the `flock` lock. The lock file itself may remain when no lock is held, so check for a running sync process rather than treating the file as stale. |
260
272
  | A completion command times out | Increase `--cmd_timeout`, investigate Git network latency, and confirm that Git cannot wait for interactive input. |
261
273
 
package/index.js CHANGED
@@ -39,6 +39,14 @@ Uncatch.on('uncaughtException', function(err) {
39
39
  cli.progress.end();
40
40
  });
41
41
 
42
+ cli.emoji = function(emoji) {
43
+ // override cli.emoji with our own TTY-aware version (FUTURE: Remove this patch after pixl-cli v1.1.7+)
44
+ // Draw the emoji, restore its starting cursor position, then explicitly move
45
+ // forward two cells. This overrides each terminal's automatic cursor advance.
46
+ if (!cli.tty()) return emoji;
47
+ return '\u001b7' + emoji + '\u001b8\u001b[2C';
48
+ };
49
+
42
50
  const { api } = require('@pixlcore/xyops-sdk');
43
51
  var highlight = require('cli-highlight').highlight;
44
52
  const Tools = cli.Tools;
@@ -357,10 +365,7 @@ const app = {
357
365
  print( "\n " + cyan.bold("Request:") );
358
366
  this.jsonOutput(this.args);
359
367
 
360
- if (this.dry) {
361
- this.toast('⚠️', 'orange', bold("DRY RUN: ") + "Exiting without sending request.");
362
- return;
363
- }
368
+ if (this.dry) return;
364
369
 
365
370
  cli.progress.start({ amount: 1, pct: false, text: gray('→ ' + name) });
366
371
  var { err, data } = await this.api[name](this.args);
@@ -453,6 +458,7 @@ Tools.mergeHashInto( app, require('./lib/webhook.js') );
453
458
  Tools.mergeHashInto( app, require('./lib/transfer.js') );
454
459
  Tools.mergeHashInto( app, require('./lib/marketplace.js') );
455
460
  Tools.mergeHashInto( app, require('./lib/sync.js') );
461
+ Tools.mergeHashInto( app, require('./lib/sync-buckets.js') );
456
462
 
457
463
  global.app = app;
458
464
 
package/lib/buckets.js CHANGED
@@ -218,7 +218,6 @@ module.exports = {
218
218
  if (this.dry) {
219
219
  if (files.length) {
220
220
  this.jsonOutput({ files: files });
221
- this.toast('⚠️', 'orange', bold("DRY RUN: ") + "Exiting without sending request.");
222
221
  }
223
222
  return;
224
223
  }
@@ -475,10 +474,7 @@ module.exports = {
475
474
  output: dest_file
476
475
  });
477
476
  }
478
- if (this.dry) {
479
- this.toast('⚠️', 'orange', bold("DRY RUN: ") + "Exiting without sending request.");
480
- return;
481
- }
477
+ if (this.dry) return;
482
478
 
483
479
  println( "\n " + this.color('theme').bold('BUCKET FILE: ' + file.filename) );
484
480
  println( " " + gray("Downloading to file: ") + dest_file );
package/lib/jobs.js CHANGED
@@ -414,8 +414,8 @@ module.exports = {
414
414
 
415
415
  this.printNiceHTML( job.html.content );
416
416
 
417
- if (job.table.caption) {
418
- println( " " + gray(job.table.caption) );
417
+ if (job.html.caption) {
418
+ println( " " + gray(job.html.caption) );
419
419
  }
420
420
  } // job.html
421
421
 
@@ -0,0 +1,315 @@
1
+ // Filesystem sync for storage bucket data and binary files.
2
+ // Bucket definitions themselves remain in the ordinary XYPDF sync engine.
3
+
4
+ const fs = require('fs');
5
+ const Path = require('path');
6
+ const Crypto = require('crypto');
7
+ const Diff = require('diff');
8
+ const cli = require('pixl-cli');
9
+ const Tools = cli.Tools;
10
+
11
+ module.exports = {
12
+
13
+ requireBucketSyncVersion() {
14
+ // The replace flag and separate data modification time arrived in 1.1.2.
15
+ // Keep the older minimum version for every other sync resource.
16
+ if (!this.xyopsVersion || this.compareVersions(this.xyopsVersion, '1.1.2') < 0) {
17
+ this.die("Bucket sync requires xyOps v1.1.2 or later (server: " + (this.xyopsVersion || 'unknown') + ").");
18
+ }
19
+ },
20
+
21
+ getSyncBucketPaths(item) {
22
+ // The local XYPDF source owns its sibling content directory. The bucket ID
23
+ // identifies the remote object, so title edits never move existing files.
24
+ var dir = item.file.replace(/\.json$/i, '');
25
+ return { dir, data: Path.join(dir, 'data.json'), files: Path.join(dir, 'files') };
26
+ },
27
+
28
+ cleanSyncBucketFilename(filename) {
29
+ // Match xyOps cleanFilename(), which is used for manifest identity. The
30
+ // server's cleanURLFilename() is only for the storage URL component.
31
+ return filename.replace(/[^\w\-\+\.\,\s\(\)\[\]\{\}\'\"\!\&\^\%\$\#\@\*\?\~]+/g, '_');
32
+ },
33
+
34
+ validateSyncBucketFilename(filename) {
35
+ // A server manifest must never be allowed to escape the bucket files dir.
36
+ if (!filename || (typeof(filename) != 'string') || (filename == '.') || (filename == '..') ||
37
+ filename.includes('/') || filename.includes('\\') || filename.includes('\0') || (Path.basename(filename) != filename)) {
38
+ throw new Error("Unsafe bucket filename: " + filename);
39
+ }
40
+ },
41
+
42
+ async getSyncBucket(id) {
43
+ // Read both independently stored content records and their timestamps.
44
+ var { err, data } = await this.api.getBucket({ id });
45
+ if (err) throw err;
46
+ if (!data || !Tools.isaHash(data.data) || !Array.isArray(data.files) ||
47
+ !data.meta || (typeof(data.meta.mod) != 'number') || !Number.isFinite(data.meta.mod)) {
48
+ throw new Error("Incomplete bucket data or file manifest: " + id);
49
+ }
50
+
51
+ var names = new Set();
52
+ data.files.forEach( file => {
53
+ this.validateSyncBucketFilename(file.filename);
54
+ if ((typeof(file.date) != 'number') || !Number.isFinite(file.date) ||
55
+ (typeof(file.size) != 'number') || !Number.isSafeInteger(file.size) || (file.size < 0) ||
56
+ (typeof(file.path) != 'string') || !file.path.startsWith('files/bucket/' + id + '/') ||
57
+ Path.posix.normalize(file.path) != file.path) {
58
+ throw new Error("Invalid file manifest entry in bucket: " + id);
59
+ }
60
+ if (names.has(file.filename)) throw new Error("Duplicate bucket filename in manifest: " + file.filename);
61
+ names.add(file.filename);
62
+ });
63
+ return data;
64
+ },
65
+
66
+ readSyncBucketLocal(item) {
67
+ // A missing data file or files directory is an incomplete inventory. This
68
+ // is an error even when the remote bucket currently happens to be empty.
69
+ var paths = this.getSyncBucketPaths(item);
70
+ var stat = fs.lstatSync(paths.data);
71
+ if (!stat.isFile() || stat.isSymbolicLink()) throw new Error("Invalid bucket data file: " + paths.data);
72
+ var data = JSON.parse(fs.readFileSync(paths.data, 'utf8'));
73
+ if (!Tools.isaHash(data)) throw new Error("Bucket data must be a JSON object: " + paths.data);
74
+
75
+ var dir_stat = fs.lstatSync(paths.files);
76
+ if (!dir_stat.isDirectory() || dir_stat.isSymbolicLink()) throw new Error("Invalid bucket files directory: " + paths.files);
77
+
78
+ var files = new Map();
79
+ fs.readdirSync(paths.files, { withFileTypes: true }).forEach( entry => {
80
+ var path = Path.join(paths.files, entry.name);
81
+ if (!entry.isFile()) throw new Error("Bucket files directory must contain only regular files: " + path);
82
+ var name = this.cleanSyncBucketFilename(entry.name);
83
+ this.validateSyncBucketFilename(name);
84
+ if (files.has(name)) throw new Error("Bucket filenames collide after normalization: " + path);
85
+ files.set(name, { path, stat: fs.statSync(path) });
86
+ });
87
+ return { paths, data, data_stat: stat, files };
88
+ },
89
+
90
+ stampSyncBucketFile(path, epoch) {
91
+ // Keep the access time intact. Server times are Unix seconds; Node accepts
92
+ // the same unit for utimes(), including fractional seconds if provided.
93
+ var stat = fs.statSync(path);
94
+ fs.utimesSync(path, stat.atime, Number(epoch));
95
+ },
96
+
97
+ printSyncBucketHeading(icon, label, title, id, deleting) {
98
+ // Match the standard sync heading: direction, action, display title and
99
+ // a quiet internal ID. Give destructive operations the existing red style.
100
+ var action = deleting ? cli.chalk.red.bold : cli.chalk.green.bold;
101
+ var name = deleting ? cli.chalk.red.bold : cli.chalk.cyan.bold;
102
+ println( "\n " + cli.emoji(icon) + " " + action(label) + ": " + name(title) + cli.chalk.gray(" (" + id + ")") );
103
+ },
104
+
105
+ async downloadSyncBucketFile(id, file, destination) {
106
+ // Download to a private sibling first. A failed or partial response must
107
+ // never replace the previous local copy of a bucket file.
108
+ var temp = destination + '.xy-sync-' + Crypto.randomBytes(8).toString('hex') + '.tmp';
109
+ try {
110
+ var { err } = await this.api.file({
111
+ path: file.path.replace(/^files\//, ''),
112
+ download: file.filename
113
+ }, { download: temp });
114
+ if (err) throw err;
115
+ var stat = fs.statSync(temp);
116
+ if (stat.size !== Number(file.size)) throw new Error("Downloaded bucket file size differs from manifest: " + file.filename);
117
+ this.stampSyncBucketFile(temp, file.date);
118
+ fs.renameSync(temp, destination);
119
+ }
120
+ finally {
121
+ if (fs.existsSync(temp)) fs.unlinkSync(temp);
122
+ }
123
+ },
124
+
125
+ async setupBucketContents(plans, opts) {
126
+ // Fetch every selected bucket before writing any bucket extras. Honor the
127
+ // ordinary setup --force, --new and --dry rules for all content files.
128
+ var prepared = [];
129
+ var paths_seen = new Set();
130
+ for (const item of plans) {
131
+ var paths = this.getSyncBucketPaths(item);
132
+ if (paths_seen.has(paths.dir)) this.die("Bucket setup paths collide: " + paths.dir);
133
+ paths_seen.add(paths.dir);
134
+ for (const dir of [ paths.dir, paths.files ]) {
135
+ if (fs.existsSync(dir) && (!fs.lstatSync(dir).isDirectory() || fs.lstatSync(dir).isSymbolicLink())) {
136
+ this.die("Invalid bucket setup directory: " + dir);
137
+ }
138
+ }
139
+ var snapshot = await this.getSyncBucket(item.id);
140
+ var files = snapshot.files.map( file => ({ file, path: Path.join(paths.files, file.filename) }) );
141
+ if (!opts.force && !opts.dry) {
142
+ if (fs.existsSync(paths.data)) this.die("File exists, add '--force' to overwrite all: " + paths.data);
143
+ files.forEach( entry => {
144
+ if (fs.existsSync(entry.path)) this.die("File exists, add '--force' to overwrite all: " + entry.path);
145
+ });
146
+ }
147
+ prepared.push({ item, paths, snapshot, files });
148
+ }
149
+
150
+ for (const entry of prepared) {
151
+ // Mirror the setup tree above: a colored folder heading, then short
152
+ // names for the data file and nested uploaded-file directory.
153
+ println( "\n " + cli.emoji('🪣') + " " + cli.chalk.bold.cyan(Path.basename(entry.paths.dir) + '/') );
154
+ println( " " + cli.emoji('📄') + " " + cli.chalk.bold.green('data.json') );
155
+ println( " " + cli.emoji('📂') + " " + cli.chalk.bold.cyan('files/') );
156
+ entry.files.forEach( file => {
157
+ println( " " + cli.emoji('📎') + " " + cli.chalk.bold.yellow(file.file.filename) );
158
+ });
159
+ if (opts.dry) continue;
160
+ Tools.mkdirp.sync(entry.paths.files);
161
+ Tools.writeFileAtomicSync(entry.paths.data, JSON.stringify(entry.snapshot.data, null, '\t') + '\n');
162
+ this.stampSyncBucketFile(entry.paths.data, entry.snapshot.meta.mod);
163
+ for (const file of entry.files) {
164
+ await this.downloadSyncBucketFile(entry.item.id, file.file, file.path);
165
+ }
166
+ }
167
+ },
168
+
169
+ async syncBucketContents(items, sconfig) {
170
+ if (!items.length || this.errors.length || this.warnings.length) return;
171
+ var prepared = [];
172
+
173
+ // Preflight every selected local inventory and every remote manifest before
174
+ // applying changes. A missing data.json or unreadable files dir must never
175
+ // turn into a remote deletion candidate.
176
+ try {
177
+ for (const item of items) {
178
+ prepared.push({ item, local: this.readSyncBucketLocal(item), remote: await this.getSyncBucket(item.data.id) });
179
+ }
180
+ }
181
+ catch (err) {
182
+ this.logSyncError("Bucket sync preflight failed: " + err);
183
+ return;
184
+ }
185
+
186
+ var up = sconfig.up && sconfig.up.includes('buckets');
187
+ var down = sconfig.down && sconfig.down.includes('buckets');
188
+ var delete_files = sconfig.delete && sconfig.delete.includes('buckets');
189
+ var deletes = [];
190
+
191
+ for (const entry of prepared) {
192
+ var id = entry.item.data.id;
193
+ var local = entry.local;
194
+ var remote = entry.remote;
195
+ var title = (remote.bucket && remote.bucket.title) || entry.item.data.title || id;
196
+ var same_data = Tools.stableStringify(local.data) === Tools.stableStringify(remote.data);
197
+
198
+ try {
199
+ if (!same_data) {
200
+ var local_time = local.data_stat.mtimeMs / 1000;
201
+ var remote_time = Number(remote.meta.mod);
202
+ if (up && down && local_time == remote_time) throw new Error("Bucket data conflict with equal timestamps: " + id);
203
+ var data_direction = up && down ? (local_time > remote_time ? 'up' : 'down') : (up ? 'up' : 'down');
204
+ var data_changes = Diff.diffLines(
205
+ Tools.stablePrettyStringify(remote.data),
206
+ Tools.stablePrettyStringify(local.data)
207
+ );
208
+ this.printSyncBucketHeading(data_direction == 'up' ? '⬆️' : '⬇️', 'Updating bucket data', title, id);
209
+ println( "\n" + this.markdown(this.formatSyncDiff(data_changes)) );
210
+ if (!this.dry && data_direction == 'up') {
211
+ await this.callStandardAPI('writeBucketData', { id, data: local.data, replace: true }, { text: 'Writing bucket data...', throw: true });
212
+ var updated = await this.getSyncBucket(id);
213
+ if (Tools.stableStringify(updated.data) !== Tools.stableStringify(local.data)) {
214
+ throw new Error("Bucket data changed during upload: " + id);
215
+ }
216
+ if (Tools.stableStringify(JSON.parse(fs.readFileSync(local.paths.data, 'utf8'))) !== Tools.stableStringify(local.data)) {
217
+ throw new Error("Local bucket data changed during upload: " + local.paths.data);
218
+ }
219
+ this.stampSyncBucketFile(local.paths.data, updated.meta.mod);
220
+ this.upSynced = true;
221
+ }
222
+ else if (!this.dry) {
223
+ Tools.writeFileAtomicSync(local.paths.data, JSON.stringify(remote.data, null, '\t') + '\n');
224
+ this.stampSyncBucketFile(local.paths.data, remote.meta.mod);
225
+ this.downSynced = true;
226
+ }
227
+ }
228
+
229
+ var manifest = new Map(remote.files.map( file => [file.filename, file] ));
230
+ var uploads = [];
231
+ var downloads = [];
232
+ for (const [name, source] of local.files) {
233
+ var target = manifest.get(name);
234
+ if (!target) {
235
+ if (up) uploads.push({ name, source });
236
+ continue;
237
+ }
238
+ var local_time = source.stat.mtimeMs / 1000;
239
+ var remote_time = Number(target.date);
240
+ var size_differs = source.stat.size !== Number(target.size);
241
+ if (up && down) {
242
+ if (local_time > remote_time) uploads.push({ name, source });
243
+ else if (remote_time > local_time) downloads.push({ name, target, path: source.path });
244
+ else if (size_differs) throw new Error("Bucket file size conflict with equal timestamps: " + source.path);
245
+ }
246
+ else if (up && (size_differs || local_time > remote_time)) uploads.push({ name, source });
247
+ else if (down && (size_differs || remote_time > local_time)) downloads.push({ name, target, path: source.path });
248
+ }
249
+ for (const file of remote.files) {
250
+ if (!local.files.has(file.filename)) {
251
+ if (down) downloads.push({ name: file.filename, target: file, path: Path.join(local.paths.files, file.filename) });
252
+ else if (delete_files) deletes.push({ id, title, file });
253
+ }
254
+ }
255
+
256
+ if (uploads.length) {
257
+ var upload_label = "Uploading " + uploads.length + " bucket " + (uploads.length == 1 ? 'file' : 'files');
258
+ this.printSyncBucketHeading('⬆️', upload_label, title, id);
259
+ uploads.forEach( entry => {
260
+ println( " " + cli.emoji('📎') + " " + cli.chalk.bold.yellow(Path.basename(entry.source.path)) );
261
+ });
262
+ if (!this.dry) {
263
+ var result = await this.callStandardAPI('uploadBucketFiles', { id }, {
264
+ text: 'Uploading bucket files...', files: uploads.map( entry => entry.source.path ), throw: true
265
+ });
266
+ var updated = new Map((result.files || []).map( file => [file.filename, file] ));
267
+ for (const entry of uploads) {
268
+ var file = updated.get(entry.name);
269
+ if (!file || (typeof(file.date) != 'number') || !Number.isFinite(file.date)) {
270
+ throw new Error("Uploaded bucket file missing or invalid in response: " + entry.name);
271
+ }
272
+ var current = fs.statSync(entry.source.path);
273
+ if ((current.size !== entry.source.stat.size) || (current.mtimeMs !== entry.source.stat.mtimeMs)) {
274
+ throw new Error("Local bucket file changed during upload: " + entry.source.path);
275
+ }
276
+ this.stampSyncBucketFile(entry.source.path, file.date);
277
+ }
278
+ this.upSynced = true;
279
+ }
280
+ }
281
+ for (const entry of downloads) {
282
+ this.printSyncBucketHeading('⬇️', 'Downloading bucket file', title, id);
283
+ println( " " + cli.emoji('📎') + " " + cli.chalk.bold.yellow(entry.name) );
284
+ if (!this.dry) {
285
+ await this.downloadSyncBucketFile(id, entry.target, entry.path);
286
+ this.downSynced = true;
287
+ }
288
+ }
289
+ }
290
+ catch (err) {
291
+ this.logSyncError("Bucket sync failed for " + id + ": " + err);
292
+ }
293
+ }
294
+
295
+ // Deletions are the final content phase. Any earlier warning or error
296
+ // suppresses all bucket file deletes, just like whole-object deletion.
297
+ if (deletes.length && (this.errors.length || this.warnings.length)) {
298
+ this.logSyncWarning("Bucket file delete pass skipped because sync encountered warnings or errors.");
299
+ return;
300
+ }
301
+ for (const entry of deletes) {
302
+ this.printSyncBucketHeading('🗑️', 'Deleting bucket file', entry.title, entry.id, true);
303
+ println( " " + cli.emoji('📎') + " " + cli.chalk.bold.yellow(entry.file.filename) );
304
+ if (this.dry) continue;
305
+ try {
306
+ await this.callStandardAPI('deleteBucketFile', { id: entry.id, path: entry.file.path }, { text: 'Deleting bucket file...', throw: true });
307
+ this.upSynced = true;
308
+ }
309
+ catch (err) {
310
+ this.logSyncError("Bucket file deletion failed: " + err);
311
+ return;
312
+ }
313
+ }
314
+ }
315
+ };