@pixlcore/xycli 1.0.8 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -3
- package/docs/help.md +21 -7
- package/docs/sync.md +85 -21
- package/docs/two-way-delete-tutorial.md +259 -0
- package/docs/two-way-git-tutorial.md +13 -7
- package/index.js +22 -1
- package/lib/apikey.js +1 -1
- package/lib/marketplace.js +1 -1
- package/lib/sync-buckets.js +35 -1
- package/lib/sync-syshook.js +207 -0
- package/lib/sync.js +161 -26
- package/lib/transfer.js +3 -3
- package/lib/utils.js +6 -2
- package/package.json +2 -2
- package/test/README.md +3 -3
- package/test/help.test.js +56 -0
- package/test/sync-buckets-unit.test.js +31 -0
- package/test/sync-buckets.test.js +36 -0
- package/test/sync-syshook-unit.test.js +231 -0
- package/test/sync-unit.test.js +215 -7
- package/test/sync.test.js +42 -0
- package/test/tickets.test.js +4 -1
- package/test/toast.test.js +3 -3
|
@@ -0,0 +1,259 @@
|
|
|
1
|
+
# Two-Way Git Sync with Event-Driven Deletes
|
|
2
|
+
|
|
3
|
+
This tutorial keeps Events, workflows, and Plugins in both xyOps and Git. Existing definitions can be edited on either side, and deletions can start on either side. New definitions are created in the xyOps UI and exported to Git. Every sync runs because Git accepted a push or xyOps recorded an activity. There are no cron jobs or recurring sync schedules.
|
|
4
|
+
|
|
5
|
+
Use xyCLI version **v1.1.0+** (with down-sync delete support) and xyOps version **v1.1.3+** (with System Activity triggers). Both features must be available before you enable this workflow.
|
|
6
|
+
|
|
7
|
+
> [!IMPORTANT]
|
|
8
|
+
> **Two-way deletion requires authoritative triggers in both directions.** Run `--up --delete` only in response to an accepted Git change, and run `--down --delete` only in response to xyOps activity. Each invocation has exactly one authoritative side. A single command with both `--up` and `--down` cannot use `--delete`.
|
|
9
|
+
|
|
10
|
+
| Authoritative action | Trigger | Command | Result |
|
|
11
|
+
| --- | --- | --- | --- |
|
|
12
|
+
| Git accepts a contributor push | Git `post-receive` hook or GitHub `push` webhook | `xy sync ... --up events,plugins --down false --delete events,plugins` | Git updates existing definitions and removes missing ones from xyOps. |
|
|
13
|
+
| xyOps updates or deletes an Event or Plugin | **System Activity** Event | `xy sync ... --up false --down events,plugins --delete events,plugins` | xyOps updates existing files and removes missing XYPDF files and their owned neighbors. |
|
|
14
|
+
| xyOps creates an Event or Plugin | Separate **System Activity** Event | `xy sync setup events plugins --new ...` | New definitions are exported, committed, and pushed to Git. |
|
|
15
|
+
|
|
16
|
+
> [!CAUTION]
|
|
17
|
+
> There is a potential race condition between creating an item in xyOps and exporting it to Git. If a Git-triggered up-delete runs before the new XYPDF file reaches the checkout, it can delete that newly created item from xyOps. The shared job queue prevents runs from overlapping, but it cannot guarantee which trigger arrives first. See [Ordering and recovery](#ordering-and-recovery).
|
|
18
|
+
|
|
19
|
+
The examples in this tutorial include **Events** (which includes Workflows) and **Plugins**. To manage other resource types, add them to the commands and choose their corresponding activity types.
|
|
20
|
+
|
|
21
|
+
## Before you begin
|
|
22
|
+
|
|
23
|
+
Use one persistent Git checkout on a machine running xySat. Target all three xyOps Events below to that machine, where the checkout, xyCLI configuration, and Git credentials are available. The xyOps conductor does not need a copy of the repository. Use one operating-system account for all three jobs.
|
|
24
|
+
|
|
25
|
+
You will need:
|
|
26
|
+
|
|
27
|
+
- A xyOps API Key that can see the **complete** Event and Plugin inventories, update and delete those types, and update global sync state. A restricted or partial inventory is unsafe for up-only deletion.
|
|
28
|
+
- A dedicated repository and a persistent checkout, such as `/srv/xyops-automation`, with noninteractive Git pull and push access.
|
|
29
|
+
- A Git author identity for commits generated by xyOps.
|
|
30
|
+
- Shared xyOps job capacity and queue limits on all three Events.
|
|
31
|
+
- A way to monitor failed xyOps jobs and Git webhook deliveries, then retry them after checking both sides.
|
|
32
|
+
|
|
33
|
+
This is a full-inventory workflow. Do not point it at a partial export, empty mount, fresh clone missing files, or API Key with an incomplete view. Stock and Marketplace definitions are naturally excluded.
|
|
34
|
+
|
|
35
|
+
## 1. Prepare the checkout
|
|
36
|
+
|
|
37
|
+
Clone the repository on the sync worker, then configure the account that will run the jobs:
|
|
38
|
+
|
|
39
|
+
```sh
|
|
40
|
+
git clone git@github.com:YOUR_ORG/xyops-automation.git /srv/xyops-automation
|
|
41
|
+
cd /srv/xyops-automation
|
|
42
|
+
git config user.name "xyOps Sync"
|
|
43
|
+
git config user.email "xyops-sync@example.com"
|
|
44
|
+
xy config --base_url https://xyops.example.com --api_key YOUR_API_KEY
|
|
45
|
+
xy dashboard
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Keep the API Key and Git credentials outside the repository. Use `main` as the dedicated branch in the examples below, or replace it consistently with your branch name.
|
|
49
|
+
|
|
50
|
+
Export a complete starter inventory from the repository root. `--file_props` keeps common Plugin and Event scripts in neighboring files:
|
|
51
|
+
|
|
52
|
+
```sh
|
|
53
|
+
xy sync setup events plugins --file_props script,params.script --dry
|
|
54
|
+
xy sync setup events plugins --file_props script,params.script
|
|
55
|
+
git status --short
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Review the generated files, including scripts and potentially sensitive configuration. Confirm that every non-stock Event, workflow, and Plugin visible to the sync API Key is present. Then commit and push the initial tree:
|
|
59
|
+
|
|
60
|
+
```sh
|
|
61
|
+
git add --all
|
|
62
|
+
git commit -m "Export xyOps automation"
|
|
63
|
+
git push -u origin main
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Create the automation Events described below only after this inventory exists. Export them before enabling the Git trigger, so an early up-only delete pass cannot mistake an automation Event for a Git deletion.
|
|
67
|
+
|
|
68
|
+
## 2. Add a commit helper
|
|
69
|
+
|
|
70
|
+
Save this script as `/usr/local/sbin/commit-xyops-sync` on the sync worker. Both xyOps-originated jobs will use it as their explicit `--down_cmd`, to run after local changes have been made:
|
|
71
|
+
|
|
72
|
+
```sh
|
|
73
|
+
#!/bin/sh
|
|
74
|
+
set -eu
|
|
75
|
+
|
|
76
|
+
cd /srv/xyops-automation
|
|
77
|
+
|
|
78
|
+
# This checkout is dedicated to xyOps automation. Stage removals as well as
|
|
79
|
+
# new and updated files, then avoid an empty commit if nothing changed in Git.
|
|
80
|
+
git add --all
|
|
81
|
+
if git diff --cached --quiet; then
|
|
82
|
+
exit 0
|
|
83
|
+
fi
|
|
84
|
+
|
|
85
|
+
# The Git trigger should ignore commits carrying this prefix. They came from
|
|
86
|
+
# xyOps and must not cause a second authoritative up-delete run.
|
|
87
|
+
git commit -m "[xyops-sync] Export xyOps changes"
|
|
88
|
+
git push origin HEAD:main
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Make it executable with `chmod 755 /usr/local/sbin/commit-xyops-sync`. This helper assumes the repository contains only the automation tree. If it contains unrelated work, narrow the staged paths and review the resulting commit before using it unattended.
|
|
92
|
+
|
|
93
|
+
## 3. Add one sync script for the three jobs
|
|
94
|
+
|
|
95
|
+
Save this as `/usr/local/sbin/xyops-git-sync` on the sync worker:
|
|
96
|
+
|
|
97
|
+
```sh
|
|
98
|
+
#!/bin/sh
|
|
99
|
+
set -eu
|
|
100
|
+
|
|
101
|
+
# Include the directories containing Node.js, xyCLI, and Git on this worker.
|
|
102
|
+
PATH="/usr/local/bin:/usr/bin:/bin"
|
|
103
|
+
export PATH
|
|
104
|
+
|
|
105
|
+
REPO_DIR="/srv/xyops-automation"
|
|
106
|
+
XY_BIN="/usr/local/bin/xy"
|
|
107
|
+
COMMIT_CMD="/usr/local/sbin/commit-xyops-sync"
|
|
108
|
+
SYNC_TYPES="events,plugins"
|
|
109
|
+
MODE="${1:-}"
|
|
110
|
+
|
|
111
|
+
cd "$REPO_DIR"
|
|
112
|
+
|
|
113
|
+
# Work only from the dedicated branch and a clean, existing checkout. A
|
|
114
|
+
# missing tree or leftover change must never be treated as a deletion.
|
|
115
|
+
if [ "$(git symbolic-ref --quiet --short HEAD)" != main ]; then
|
|
116
|
+
echo "Sync stopped: checkout is not on main." >&2
|
|
117
|
+
exit 1
|
|
118
|
+
fi
|
|
119
|
+
if [ -n "$(git status --porcelain)" ]; then
|
|
120
|
+
echo "Sync stopped: checkout has uncommitted changes." >&2
|
|
121
|
+
exit 1
|
|
122
|
+
fi
|
|
123
|
+
if [ ! -d events ] || [ ! -d plugins ]; then
|
|
124
|
+
echo "Sync stopped: Event or Plugin inventory directory is missing." >&2
|
|
125
|
+
exit 1
|
|
126
|
+
fi
|
|
127
|
+
|
|
128
|
+
# Pull accepted Git changes before scanning. A failed fast-forward requires
|
|
129
|
+
# review rather than a reset that could discard a xyOps-generated commit.
|
|
130
|
+
git pull --quiet --ff-only origin main
|
|
131
|
+
|
|
132
|
+
case "$MODE" in
|
|
133
|
+
up)
|
|
134
|
+
"$XY_BIN" sync "$REPO_DIR" \
|
|
135
|
+
--up "$SYNC_TYPES" --down false --delete "$SYNC_TYPES" \
|
|
136
|
+
--up_cmd false --down_cmd false
|
|
137
|
+
;;
|
|
138
|
+
|
|
139
|
+
down)
|
|
140
|
+
"$XY_BIN" sync "$REPO_DIR" \
|
|
141
|
+
--up false --down "$SYNC_TYPES" --delete "$SYNC_TYPES" \
|
|
142
|
+
--down_cmd "$COMMIT_CMD" --cmd_timeout 120
|
|
143
|
+
;;
|
|
144
|
+
|
|
145
|
+
create)
|
|
146
|
+
# Setup scans the current directory and adds only remote definitions
|
|
147
|
+
# whose type and ID are not already represented in the checkout.
|
|
148
|
+
"$XY_BIN" sync --setup "$SYNC_TYPES" --new \
|
|
149
|
+
--file_props script,params.script \
|
|
150
|
+
--down_cmd "$COMMIT_CMD" --cmd_timeout 120
|
|
151
|
+
;;
|
|
152
|
+
|
|
153
|
+
*)
|
|
154
|
+
echo "Usage: $0 up|down|create" >&2
|
|
155
|
+
exit 2
|
|
156
|
+
;;
|
|
157
|
+
esac
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
Make it executable with `chmod 755 /usr/local/sbin/xyops-git-sync`. Change the paths and `PATH` for your worker. The directory checks are a basic guard against a missing checkout. They do not prove that the inventory is complete, so inspect it before enabling deletion and after any Git or API failure.
|
|
161
|
+
|
|
162
|
+
Set a suitable job timeout for Git network operations. All three Events will share a xyOps capacity pool so only one can use this checkout at a time.
|
|
163
|
+
|
|
164
|
+
## 4. Configure the xyOps Activity Events
|
|
165
|
+
|
|
166
|
+
Create three Shell Plugin Events in xyOps, initially disabled. Target only the sync worker and give each Event the indicated trigger and script. The [System Activity trigger](https://docs.xyops.io/triggers#activity) lets you select multiple activity types in the UI; it launches a job when any of them occur. Note that the xyOps scheduler must be enabled for Activity triggers to run.
|
|
167
|
+
|
|
168
|
+
| Event | Trigger | Select these activities | Shell script body |
|
|
169
|
+
| --- | --- | --- | --- |
|
|
170
|
+
| **Sync deletions and updates from xyOps** | System Activity | `event_update`, `plugin_update`, `event_delete`, `plugin_delete` | `/usr/local/sbin/xyops-git-sync down` |
|
|
171
|
+
| **Export creations from xyOps** | System Activity | `event_create`, `plugin_create` | `/usr/local/sbin/xyops-git-sync create` |
|
|
172
|
+
| **Apply accepted Git pushes** | Manual | No activity selection | `/usr/local/sbin/xyops-git-sync up` |
|
|
173
|
+
|
|
174
|
+
The update activities make xyOps UI edits flow back to Git too. The separate creation Event runs `sync setup --new`, because ordinary downsync updates existing local sources but does not create files for new xyOps definitions. The setup command's `--down_cmd` runs only when it writes files; the down-delete command's `--down_cmd` runs only after local changes. Neither creates an empty Git commit on a no-op run.
|
|
175
|
+
|
|
176
|
+
The Git Event has an enabled **Manual** trigger so a Git hook or verified webhook receiver can launch it with `xy run` or the [`run_event` API](https://docs.xyops.io/api#run_event). It has no schedule. The Git push is still the authoritative reason for that run.
|
|
177
|
+
|
|
178
|
+
On **each** of these three Events, set a **[Max Jobs Limit](https://docs.xyops.io/#Docs/limits/max-jobs-limit)** of `1` with the same **[Shared Capacity Key](https://docs.xyops.io/#Docs/limits/shared-capacity-key)**, such as `xyops_git_sync`. Also set a **[Max Queue Limit](https://docs.xyops.io/#Docs/limits/max-queue-limit)** with the same positive amount on each Event, sized for the bursts you expect. This creates one shared, FIFO job queue for all sync operations. Without queue capacity, a trigger arriving while another sync job runs may be aborted instead of waiting. See xyOps's [shared capacity and queue limits](https://docs.xyops.io/limits#shared-capacity-key). Use the same priority on all three Events unless you deliberately want to change queue order, and monitor jobs that fail or exhaust the queue.
|
|
179
|
+
|
|
180
|
+
Before enabling any of the three Events, export their new definitions from the checkout root:
|
|
181
|
+
|
|
182
|
+
```sh
|
|
183
|
+
cd /srv/xyops-automation
|
|
184
|
+
/usr/local/sbin/xyops-git-sync create
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
The `create` mode uses `sync setup --new` and its `--down_cmd` to commit and push those files. Review that commit and confirm the tree is complete. Then enable the three Events and test with disposable definitions before allowing normal Git pushes to trigger up-delete.
|
|
188
|
+
|
|
189
|
+
## 5. Trigger upsync from accepted Git pushes
|
|
190
|
+
|
|
191
|
+
Choose the Git trigger that matches your repository hosting. Both options launch the **same** Git Event after a push has been accepted. Do not enable both for one repository.
|
|
192
|
+
|
|
193
|
+
### Self-hosted Git: a `post-receive` hook
|
|
194
|
+
|
|
195
|
+
On a self-hosted **bare remote repository**, save the following as `hooks/post-receive` and make it executable. This hook runs on the Git server after refs have been updated, not in each contributor's clone or the sync worker's checkout. The Git server needs xyCLI credentials that permit launching the Git Event, but it does not need the sync checkout.
|
|
196
|
+
|
|
197
|
+
```sh
|
|
198
|
+
#!/bin/sh
|
|
199
|
+
set -eu
|
|
200
|
+
|
|
201
|
+
deploy=false
|
|
202
|
+
while read -r old_oid new_oid ref; do
|
|
203
|
+
[ "$ref" = refs/heads/main ] || continue
|
|
204
|
+
git cat-file -e "$new_oid^{commit}" 2>/dev/null || continue
|
|
205
|
+
|
|
206
|
+
# A commit generated by the down/create helper is already from xyOps.
|
|
207
|
+
# Do not turn its push into another authoritative up-delete run.
|
|
208
|
+
case "$(git log -1 --format=%s "$new_oid")" in
|
|
209
|
+
'[xyops-sync]'*) continue ;;
|
|
210
|
+
esac
|
|
211
|
+
deploy=true
|
|
212
|
+
done
|
|
213
|
+
|
|
214
|
+
[ "$deploy" = true ] || exit 0
|
|
215
|
+
/usr/local/bin/xy run SYNC_UP_EVENT_ID
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
Replace `SYNC_UP_EVENT_ID` with the xyOps Git Event's exact ID. Git's [`post-receive` hook](https://git-scm.com/docs/githooks#post-receive) runs after a push succeeds; a later hook or job failure cannot undo that push. Monitor the launched job and retry it after fixing any problem.
|
|
219
|
+
|
|
220
|
+
### GitHub: a `push` webhook
|
|
221
|
+
|
|
222
|
+
If GitHub hosts the repository, use its **push** webhook to notify your existing webhook receiver. In GitHub, open **Settings > Webhooks > Add webhook**, set its payload URL to that receiver, choose `application/json`, set a secret, and select only the **Pushes** event. The receiver should:
|
|
223
|
+
|
|
224
|
+
1. Verify `X-Hub-Signature-256` against the raw request body, and accept only the intended repository and `refs/heads/main`.
|
|
225
|
+
2. Ignore GitHub's initial `ping`, deleted branches, and commits whose head message starts with `[xyops-sync]`.
|
|
226
|
+
3. Deduplicate by `X-GitHub-Delivery`, then launch `SYNC_UP_EVENT_ID` through xyOps's `run_event` API using an API Key with `run_jobs` access.
|
|
227
|
+
4. Acknowledge only after recording the delivery for processing, and monitor the resulting xyOps job. Retry a failed job or redeliver a missed webhook after reviewing the Git and xyOps state.
|
|
228
|
+
|
|
229
|
+
The API request made by the verified receiver is a JSON `POST` to `/api/app/run_event/v1` with an `X-API-Key` header and body `{"id":"SYNC_UP_EVENT_ID"}`. The Event runs on the sync worker, so the receiver does not need the checkout. Do not put an xyOps API Key in the GitHub payload URL. A direct xyOps Magic Link can start an Event, but it does not verify GitHub's webhook signature; use a receiver that does that verification for this workflow. See GitHub's [webhook setup](https://docs.github.com/en/webhooks/using-webhooks/creating-webhooks), [signature validation](https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries), and [delivery guidance](https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks).
|
|
230
|
+
|
|
231
|
+
## 6. Test each direction before enabling automatic deletion
|
|
232
|
+
|
|
233
|
+
Run these tests with disposable Events and Plugins, not production definitions:
|
|
234
|
+
|
|
235
|
+
1. Create an Event or Plugin in xyOps. Confirm that the **create** Activity Event exports its XYPDF, commits it, and pushes it to Git.
|
|
236
|
+
2. Edit that definition in the xyOps UI. Confirm that the **down** Activity Event downloads the change and commits it.
|
|
237
|
+
3. Delete it in xyOps. Confirm that the **down** Event removes the XYPDF and owned neighbor files and pushes the deletion.
|
|
238
|
+
4. Edit an existing definition in Git and push. Confirm that the accepted-push trigger runs **up** and updates xyOps.
|
|
239
|
+
5. Delete a different existing definition in Git and push. Confirm that **up** deletes only that definition in xyOps.
|
|
240
|
+
|
|
241
|
+
Before the first live delete, preview each direction against the complete checkout and inspect every proposed deletion:
|
|
242
|
+
|
|
243
|
+
```sh
|
|
244
|
+
cd /srv/xyops-automation
|
|
245
|
+
xy sync . --up events,plugins --down false --delete events,plugins --dry
|
|
246
|
+
xy sync . --up false --down events,plugins --delete events,plugins --dry
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
Remember that a dry run previews the inventory **at that moment**. It cannot establish what a later Git push or xyOps activity means. Keep recoverable Git history and xyOps backups.
|
|
250
|
+
|
|
251
|
+
## Ordering and recovery
|
|
252
|
+
|
|
253
|
+
Authoritative triggers tell each pass which side should win. They do not make a full-inventory sweep transactional or remove all races. For example, an Event newly created in xyOps but not yet exported looks missing to an up-delete run; a new Git XYPDF that has not been created in xyOps looks missing to a down-delete run. This tutorial permits **creation only in xyOps**, so contributors should edit or delete existing XYPDF sources, not add new IDs in Git.
|
|
254
|
+
|
|
255
|
+
Keep all three jobs on the same worker and checkout, serialize the entire Git plus xyCLI sequence with their shared capacity key, and process pending xyOps creations before allowing a Git up-delete pass to treat the inventory as complete. The queue prevents overlap, but it cannot guarantee that independently delivered Git and xyOps triggers arrive in causal order. If changes on both sides occur together, pause automatic deletion, inspect the Git diff and both inventories, resolve the intended order, and rerun the appropriate job. Do not assume a successful webhook acknowledgment means the xyOps job succeeded.
|
|
256
|
+
|
|
257
|
+
If a job or push fails, leave the checkout and xyOps as they are until you have inspected them. A failed Git push can leave a local commit that needs to be pushed or reconciled before another `git pull --ff-only`. A failed Activity launch or webhook delivery needs a deliberate retry; there is no cron pass to pick it up later. GitHub [does not automatically redeliver failed webhook deliveries](https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries).
|
|
258
|
+
|
|
259
|
+
For all sync options and deletion safeguards, see the [Filesystem Sync Guide](sync.md#delete-mode). For a scheduled, modification-time-based workflow without automatic two-way deletion, see the [Two-Way Git Sync Tutorial](two-way-git-tutorial.md).
|
|
@@ -108,7 +108,9 @@ This considers both directions for every local Event and workflow source:
|
|
|
108
108
|
- A newer xyOps definition is downloaded to its existing local file.
|
|
109
109
|
- Identical definitions are left alone.
|
|
110
110
|
|
|
111
|
-
The `--delete false` switch is intentional.
|
|
111
|
+
The `--delete false` switch is intentional. A single command with both directions does not propagate deletions. Keep deletion disabled in this scheduled command, and follow the [coordinated deletion steps](#delete-a-definition-from-both-sides) when retiring an item. Separate one-way [up and down delete modes](sync.md#delete-mode) are available for workflows that can coordinate their triggers and inventories.
|
|
112
|
+
|
|
113
|
+
If you want deletion in both directions without a recurring schedule, use the separate [event-driven two-way delete tutorial](two-way-delete-tutorial.md).
|
|
112
114
|
|
|
113
115
|
Review the displayed diffs carefully. If the wrong side would win, check the clocks and file modification times before continuing.
|
|
114
116
|
|
|
@@ -151,6 +153,7 @@ GIT_DOWN_CMD="git add --all && git commit -m 'Sync from xyOps' && git push"
|
|
|
151
153
|
--up events \
|
|
152
154
|
--down events \
|
|
153
155
|
--delete false \
|
|
156
|
+
--ignore_missing \
|
|
154
157
|
--down_cmd "$GIT_DOWN_CMD" \
|
|
155
158
|
--cmd_timeout 120
|
|
156
159
|
|
|
@@ -159,6 +162,7 @@ GIT_DOWN_CMD="git add --all && git commit -m 'Sync from xyOps' && git push"
|
|
|
159
162
|
# The setup completion command runs only when new files were actually written.
|
|
160
163
|
"$XY_BIN" sync setup events \
|
|
161
164
|
--new \
|
|
165
|
+
--ignore_missing \
|
|
162
166
|
--file_props script,params.script \
|
|
163
167
|
--default_ext ps1 \
|
|
164
168
|
--down_cmd "$GIT_DOWN_CMD" \
|
|
@@ -185,6 +189,8 @@ The two uses of `--down_cmd` are deliberately identical:
|
|
|
185
189
|
- `sync setup --new` runs it only after at least one new local definition is written.
|
|
186
190
|
- A no-op or dry run does not create a commit or push.
|
|
187
191
|
|
|
192
|
+
The two `--ignore_missing` switches let the script continue while a definition deleted in xyOps still has a local XYPDF source. Other definitions can sync and new xyOps definitions can be exported during that interval. The stale source and its neighboring files remain in Git until you remove and push them. Other scan problems still stop the run.
|
|
193
|
+
|
|
188
194
|
The command stages the entire dedicated repository with `git add --all`. Do not use this script in a checkout containing unrelated work. If your repository contains other material, narrow the Git paths in `GIT_DOWN_CMD`.
|
|
189
195
|
|
|
190
196
|
## 5. Schedule the script with cron
|
|
@@ -207,7 +213,7 @@ Run the script every minute, with `flock` outside the script:
|
|
|
207
213
|
* * * * * /usr/bin/flock --nonblock --conflict-exit-code 0 /tmp/xyops-git-sync.lock /usr/local/sbin/xyops-git-sync >/dev/null
|
|
208
214
|
```
|
|
209
215
|
|
|
210
|
-
Cron normally emails job output when local mail delivery is configured. This entry discards routine standard output so successful runs remain silent, but leaves standard error alone so failures can still trigger mail. The xyCLI commands
|
|
216
|
+
Cron normally emails job output when local mail delivery is configured. This entry discards routine standard output so successful runs remain silent, but leaves standard error alone so failures can still trigger mail. The xyCLI commands show progress when run manually; you can add `--quiet` to suppress that progress while keeping warnings and errors on standard error.
|
|
211
217
|
|
|
212
218
|
To keep successful output in a log instead, replace `>/dev/null` with an append redirect to a file that the sync account can write. Do not place that log inside the Git repository because the completion command stages the whole tree.
|
|
213
219
|
|
|
@@ -247,13 +253,13 @@ When someone creates a new Event in xyOps, the ordinary sync pass does not know
|
|
|
247
253
|
|
|
248
254
|
### Delete a definition from both sides
|
|
249
255
|
|
|
250
|
-
|
|
256
|
+
You can automate deletions made in xyOps with [System Hook deletion](sync.md#system-hook-deletion). For manual coordination, the `--ignore_missing` switches let the scheduled script keep running while you complete these steps:
|
|
251
257
|
|
|
252
258
|
1. Delete the definition in the xyOps web interface first.
|
|
253
|
-
2. Remove its XYPDF JSON file and all neighboring property files from Git, then commit and push the deletion.
|
|
254
|
-
3.
|
|
259
|
+
2. Remove its XYPDF JSON file and all neighboring property files from Git, then commit and push the deletion.
|
|
260
|
+
3. Check that the dedicated sync checkout receives the commit and completes a later sync run.
|
|
255
261
|
|
|
256
|
-
If the script runs after the xyOps deletion but before the Git deletion reaches its checkout,
|
|
262
|
+
If the script runs after the xyOps deletion but before the Git deletion reaches its checkout, both sync commands skip that local source without treating it as a warning. They leave its files in place and continue processing other definitions. Remove and push the stale files to complete the deletion. Without `--ignore_missing` on both commands, the missing source stops the script.
|
|
257
263
|
|
|
258
264
|
Do not remove the Git files first. While the definition still exists in xyOps, `sync setup --new` can export it again and commit the files back to Git. This is a manual deletion procedure, not automatic two-way deletion. See the [Sync Guide](sync.md#remove-a-definition-during-two-way-sync) for the general behavior.
|
|
259
265
|
|
|
@@ -267,7 +273,7 @@ Do not remove the Git files first. While the definition still exists in xyOps, `
|
|
|
267
273
|
| The wrong side wins a two-way conflict | Check NTP, the xyOps conductor clock, and the modification times of the JSON and neighboring files. A checkout, restore, or file copy can refresh local timestamps. |
|
|
268
274
|
| A new Event is not exported | Confirm `sync setup events --new` runs from the repository root and that the API Key can see the Event. Add the correct property path if its script is stored somewhere other than `params.script`. |
|
|
269
275
|
| A new Git file is not created in xyOps | Ordinary sync updates definitions that already exist on both sides. Create or import the definition into xyOps first, then let sync manage subsequent changes. |
|
|
270
|
-
| A deleted item reappears or causes an error | Follow the [coordinated deletion steps](#delete-a-definition-from-both-sides): delete in xyOps first, then remove and commit its local files.
|
|
276
|
+
| A deleted item reappears or causes an error | Follow the [coordinated deletion steps](#delete-a-definition-from-both-sides): delete in xyOps first, then remove and commit its local files. Use `--ignore_missing` on both sync commands to keep the script running during the gap. |
|
|
271
277
|
| A cron run is skipped | Another cron invocation may hold the `flock` lock. The lock file itself may remain when no lock is held, so check for a running sync process rather than treating the file as stale. |
|
|
272
278
|
| A completion command times out | Increase `--cmd_timeout`, investigate Git network latency, and confirm that Git cannot wait for interactive input. |
|
|
273
279
|
|
package/index.js
CHANGED
|
@@ -18,6 +18,15 @@ if (!process.env.XYOPS_USER_AGENT) {
|
|
|
18
18
|
process.env.XYOPS_USER_AGENT = `xyOps CLI v${pkg.version} (Node.js); ${os.hostname()} (${os.platform()}/${os.arch()})`;
|
|
19
19
|
}
|
|
20
20
|
|
|
21
|
+
// Warnings and errors must reach stderr even in quiet mode. Install this
|
|
22
|
+
// before cli.global() so its global warn() helper is bound to cli as usual.
|
|
23
|
+
cli.warn = function(msg) {
|
|
24
|
+
if (this.progress.running) this.progress.erase();
|
|
25
|
+
process.stderr.write(msg);
|
|
26
|
+
if (this.progress.running) this.progress.draw();
|
|
27
|
+
this.log(msg);
|
|
28
|
+
};
|
|
29
|
+
|
|
21
30
|
cli.global();
|
|
22
31
|
|
|
23
32
|
cli.mapArgs({
|
|
@@ -26,7 +35,8 @@ cli.mapArgs({
|
|
|
26
35
|
'd': 'debug',
|
|
27
36
|
'f': 'format',
|
|
28
37
|
'r': 'raw',
|
|
29
|
-
'h': 'help'
|
|
38
|
+
'h': 'help',
|
|
39
|
+
'ignore-missing': 'ignore_missing'
|
|
30
40
|
});
|
|
31
41
|
|
|
32
42
|
process.on('SIGINT', function() { cli.progress.end(); process.exit(130); } );
|
|
@@ -240,6 +250,14 @@ const app = {
|
|
|
240
250
|
delete args.quiet;
|
|
241
251
|
delete args.verbose;
|
|
242
252
|
|
|
253
|
+
// A System Hook deletion uses only its STDIN payload and local XYPDF
|
|
254
|
+
// files. Run it before the normal xyOps API credential and config fetch.
|
|
255
|
+
if ((cmd == 'sync') && ('syshook' in args)) {
|
|
256
|
+
await this.cmd_sync();
|
|
257
|
+
print("\n");
|
|
258
|
+
return;
|
|
259
|
+
}
|
|
260
|
+
|
|
243
261
|
// create temp dir if needed
|
|
244
262
|
this.tempDir = this.config.temp_dir || Path.join( os.tmpdir(), 'xyops', 'cli' );
|
|
245
263
|
if (!fs.existsSync(this.tempDir)) Tools.mkdirpSync( this.tempDir );
|
|
@@ -413,6 +431,8 @@ const app = {
|
|
|
413
431
|
|
|
414
432
|
dieUsage(heading) {
|
|
415
433
|
// print help and exit
|
|
434
|
+
if (cli.args.quiet) this.die("Invalid usage. Run 'xy help " + heading + "' for details.");
|
|
435
|
+
|
|
416
436
|
this.printHelp(heading);
|
|
417
437
|
println("");
|
|
418
438
|
process.exit(1);
|
|
@@ -459,6 +479,7 @@ Tools.mergeHashInto( app, require('./lib/transfer.js') );
|
|
|
459
479
|
Tools.mergeHashInto( app, require('./lib/marketplace.js') );
|
|
460
480
|
Tools.mergeHashInto( app, require('./lib/sync.js') );
|
|
461
481
|
Tools.mergeHashInto( app, require('./lib/sync-buckets.js') );
|
|
482
|
+
Tools.mergeHashInto( app, require('./lib/sync-syshook.js') );
|
|
462
483
|
|
|
463
484
|
global.app = app;
|
|
464
485
|
|
package/lib/apikey.js
CHANGED
|
@@ -241,7 +241,7 @@ module.exports = {
|
|
|
241
241
|
println( "\n " + cyan.bold("API Key Secret: ") + bold(data.plain_key) );
|
|
242
242
|
println( " " + yellow.bold("Save this secret now. It will never be displayed again.") );
|
|
243
243
|
if (state_error) {
|
|
244
|
-
|
|
244
|
+
warnln( "\n " + red.bold("Disable Error: ") + (state_error.message || state_error) );
|
|
245
245
|
process.exitCode = 1;
|
|
246
246
|
return;
|
|
247
247
|
}
|
package/lib/marketplace.js
CHANGED
|
@@ -235,7 +235,7 @@ module.exports = {
|
|
|
235
235
|
]
|
|
236
236
|
});
|
|
237
237
|
|
|
238
|
-
if (warning)
|
|
238
|
+
if (warning) warnln("\n " + yellow(warning));
|
|
239
239
|
println("\n This is the Plugin definition that will be " + (operation == 'create' ? 'created' : 'used to update the installed Plugin') + " when you confirm:");
|
|
240
240
|
this.jsonOutput(display.plugin);
|
|
241
241
|
if (display.script) this.printPluginScript(plugin);
|
package/lib/sync-buckets.js
CHANGED
|
@@ -87,6 +87,25 @@ module.exports = {
|
|
|
87
87
|
return { paths, data, data_stat: stat, files };
|
|
88
88
|
},
|
|
89
89
|
|
|
90
|
+
getSyncBucketLocalDeletePlan(item) {
|
|
91
|
+
// A deleted Bucket owns its sibling data.json and uploaded files. Only
|
|
92
|
+
// remove that directory when it has the exact structure produced by
|
|
93
|
+
// setup; an unexpected entry could belong to someone else.
|
|
94
|
+
var local = this.readSyncBucketLocal(item);
|
|
95
|
+
var dir_stat = fs.lstatSync(local.paths.dir);
|
|
96
|
+
if (!dir_stat.isDirectory() || dir_stat.isSymbolicLink()) {
|
|
97
|
+
throw new Error("Invalid bucket content directory: " + local.paths.dir);
|
|
98
|
+
}
|
|
99
|
+
var entries = fs.readdirSync(local.paths.dir).sort();
|
|
100
|
+
if (entries.length !== 2 || entries[0] !== 'data.json' || entries[1] !== 'files') {
|
|
101
|
+
throw new Error("Unexpected file in bucket content directory: " + local.paths.dir);
|
|
102
|
+
}
|
|
103
|
+
return {
|
|
104
|
+
files: [...local.files.values()].map( entry => entry.path ).concat(local.paths.data),
|
|
105
|
+
dirs: [local.paths.files, local.paths.dir]
|
|
106
|
+
};
|
|
107
|
+
},
|
|
108
|
+
|
|
90
109
|
stampSyncBucketFile(path, epoch) {
|
|
91
110
|
// Keep the access time intact. Server times are Unix seconds; Node accepts
|
|
92
111
|
// the same unit for utimes(), including fractional seconds if provided.
|
|
@@ -187,6 +206,7 @@ module.exports = {
|
|
|
187
206
|
var down = sconfig.down && sconfig.down.includes('buckets');
|
|
188
207
|
var delete_files = sconfig.delete && sconfig.delete.includes('buckets');
|
|
189
208
|
var deletes = [];
|
|
209
|
+
var local_deletes = [];
|
|
190
210
|
|
|
191
211
|
for (const entry of prepared) {
|
|
192
212
|
var id = entry.item.data.id;
|
|
@@ -233,6 +253,7 @@ module.exports = {
|
|
|
233
253
|
var target = manifest.get(name);
|
|
234
254
|
if (!target) {
|
|
235
255
|
if (up) uploads.push({ name, source });
|
|
256
|
+
else if (delete_files) local_deletes.push({ id, title, name, path: source.path });
|
|
236
257
|
continue;
|
|
237
258
|
}
|
|
238
259
|
var local_time = source.stat.mtimeMs / 1000;
|
|
@@ -294,7 +315,7 @@ module.exports = {
|
|
|
294
315
|
|
|
295
316
|
// Deletions are the final content phase. Any earlier warning or error
|
|
296
317
|
// suppresses all bucket file deletes, just like whole-object deletion.
|
|
297
|
-
if (deletes.length && (this.errors.length || this.warnings.length)) {
|
|
318
|
+
if ((deletes.length || local_deletes.length) && (this.errors.length || this.warnings.length)) {
|
|
298
319
|
this.logSyncWarning("Bucket file delete pass skipped because sync encountered warnings or errors.");
|
|
299
320
|
return;
|
|
300
321
|
}
|
|
@@ -311,5 +332,18 @@ module.exports = {
|
|
|
311
332
|
return;
|
|
312
333
|
}
|
|
313
334
|
}
|
|
335
|
+
for (const entry of local_deletes) {
|
|
336
|
+
this.printSyncBucketHeading('🗑️', this.dry ? 'Would delete local bucket file' : 'Deleting local bucket file', entry.title, entry.id, true);
|
|
337
|
+
println( " " + cli.emoji('📎') + " " + cli.chalk.bold.yellow(entry.name) );
|
|
338
|
+
if (this.dry) continue;
|
|
339
|
+
try {
|
|
340
|
+
fs.unlinkSync(entry.path);
|
|
341
|
+
this.downSynced = true;
|
|
342
|
+
}
|
|
343
|
+
catch (err) {
|
|
344
|
+
this.logSyncError("Local bucket file deletion failed: " + err);
|
|
345
|
+
return;
|
|
346
|
+
}
|
|
347
|
+
}
|
|
314
348
|
}
|
|
315
349
|
};
|