@pipefy/pipefy-process-coder 0.1.2 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -83,6 +83,23 @@ webhooks", and conflating them licenses deleting a client's live integrations.
83
83
  Applies to iPaaS flows identically: a pipe whose iPaaS is off gets `unknown`, and
84
84
  its flows are not compared at all.
85
85
 
86
+ **Rows and columns are covered separately.** `coverage` says whether every *row*
87
+ of a group was read; `columns` (in `_meta.json`) whether every *column* was. A
88
+ GraphQL reconstruction reads every phase, field and label and still lacks
89
+ `color_uuid` or a label's `uuid`, so its rows are `complete` and its columns
90
+ `partial`. The diff reads them for different questions: a missing column on a
91
+ partial side is "unseen", never "set to null"; a missing *row* is only unseen
92
+ when the rows are partial, and only then is a create or delete held back. These
93
+ were once one flag, and a reconstruction marked everything `partial` — which,
94
+ while the snapshot download was failing on every pipe, left every pull able to
95
+ update and unable to create or delete anything.
96
+
97
+ **And a held-back create or delete stops the plan.** Between a workspace and its
98
+ own baseline each one is an edit someone made, so `plan` and `apply` refuse,
99
+ naming them, rather than apply the rest (`suppressedEdits`, `--force` to apply
100
+ the rest anyway). Applying 20 updates without the 2 phase creates they pointed
101
+ at would have left a pipe with no final phase.
102
+
86
103
  ### Refuse rather than guess, per column
87
104
 
88
105
  `src/apply/registry.ts` records, for every entity and operation, whether a write
@@ -252,6 +269,9 @@ live running. Do not re-derive them. The shapes that bite most often:
252
269
  | `updateAutomation` | works, across every field map dimension. The recipes' delete-and-recreate was inherited, not measured |
253
270
  | internal `AutomationActionParams.aiBehaviorParams` | an **object** (`AiBehaviorParams`), the same type the public `aiAgents` read selects. Selected bare, the whole `loadAutomation` document fails validation — so *every* automation in the pipe loads without `action_params`, not only the AI ones. One selection, `AI_BEHAVIOR_PARAMS_SELECTION`, serves both reads |
254
271
  | a snapshot's `signedUrl` | `app.pipefy.com/storage/v1/signed/…`, and it can answer **HTTP 500** with Pipefy's own error page while `createRepoSnapshot` and the `UPLOADED` status both succeed — measured on two unrelated pipes, old snapshots and new. A pull falls back to reconstruction rather than write a workspace with no pipe in it; `--no-fallback` refuses instead |
272
+ | internal `automations(organizationId:, repoId:)` | the organization must be **the pipe's own**. Asked under another organization the credential also belongs to, it answers `Couldn't find Repo with 'id'=…` about a pipe that exists — so a saved default org silently left every automation unknown on pipes from other orgs. The repo's `organization_id` wins; the default is only the fallback |
273
+ | `Pipe.pipe_relations` | **does not exist** publicly. The relations are `parentsRelations` and `childrenRelations`, and their `parent` / `child` are a **union** (`RepoTypes`: Pipe \| Table) — `{ id }` is refused until spread with `... on Pipe { id } ... on Table { id }`. The reconstruction asked for the missing field for its whole life, so pipe_relations was always `unknown` without a snapshot |
274
+ | reconstruction without a snapshot | every group the compiler can write now has a public read: `Phase.teamMembers`, `Pipe.preferences`, `Pipe.publicForm` (its `submitterEmailCollectionMethod` is the payload's `submitter_email_required`). With rows `complete`, a pull whose snapshot download fails supports the whole loop. Only read-only groups stay `unknown` |
255
275
  | iPaaS `IMPORT_FLOW` | migrates the tree it is given; never use it |
256
276
  | iPaaS connections | the host marks any credential `ACTIVE` **without validating it** |
257
277
  | `POST /app-connections/{id}` | accepts `scope` and `projectIds` and returns 200 ignoring both |
@@ -126,3 +126,55 @@ export const renderAdopted = (adopted) => {
126
126
  lines.push(` …and ${adopted.length - 8} more`);
127
127
  return lines;
128
128
  };
129
+ /**
130
+ * The baseline after an apply that verified, when the read-back was not a
131
+ * snapshot: the workspace for every group the baseline could see, the old
132
+ * baseline for every group it could not.
133
+ *
134
+ * Without this only created ids reached the baseline (mergeAdoptedIntoBaseline),
135
+ * so every update and delete an apply had just made stayed "pending" in the
136
+ * next diff. An update resent is harmless; a delete resent is refused by the
137
+ * API, about a field the apply itself removed. Found on the first real apply
138
+ * after snapshot downloads started failing, when this became the only path.
139
+ *
140
+ * Why the workspace is safe to adopt here, when a reconstruction is not (see
141
+ * the post-snapshot branch in apply.ts): verify has just confirmed that the
142
+ * live pipe satisfies everything the workspace states, and the workspace
143
+ * carries every row the baseline did unless the author removed it — which the
144
+ * apply has also done. A group `unknown` on the baseline was never diffed, so
145
+ * an edit there was never applied, and adopting it would mark unapplied work
146
+ * done. Those groups keep the baseline's copy.
147
+ */
148
+ /**
149
+ * Phases carry two folded groups that may be unknown on their own. The phase
150
+ * and field rows come from the workspace; a folded list the baseline could not
151
+ * see keeps the baseline's copy, matched by phase id.
152
+ */
153
+ const advancePhases = (baseline, applied, jumpsSeen, teamSeen) => {
154
+ if (jumpsSeen && teamSeen)
155
+ return applied.phases;
156
+ const before = new Map(baseline.phases.map((ph) => [String(ph['id']), ph]));
157
+ return applied.phases.map((ph) => {
158
+ const old = before.get(String(ph['id']));
159
+ return {
160
+ ...ph,
161
+ jump_target_ids: jumpsSeen ? ph.jump_target_ids : (old?.jump_target_ids ?? []),
162
+ team_member_ids: teamSeen ? ph.team_member_ids : (old?.team_member_ids ?? []),
163
+ };
164
+ });
165
+ };
166
+ export const advanceBaseline = (baseline, applied) => {
167
+ const seen = (...groups) => groups.every((g) => (baseline.meta.coverage[g] ?? 'unknown') !== 'unknown');
168
+ return {
169
+ ...baseline,
170
+ repo: applied.repo,
171
+ phases: seen('phases', 'fields') ? advancePhases(baseline, applied, seen('phase_jumps'), seen('phase_team_memberships')) : baseline.phases,
172
+ automations: seen('automations') ? applied.automations : baseline.automations,
173
+ field_conditions: seen('field_conditions') ? applied.field_conditions : baseline.field_conditions,
174
+ labels: seen('labels') ? applied.labels : baseline.labels,
175
+ webhooks: seen('webhooks') ? applied.webhooks : baseline.webhooks,
176
+ pipe_relations: seen('pipe_relations') ? applied.pipe_relations : baseline.pipe_relations,
177
+ public_form: seen('public_forms') ? applied.public_form : baseline.public_form,
178
+ preferences: seen('repo_preferences') ? applied.preferences : baseline.preferences,
179
+ };
180
+ };
@@ -83,6 +83,8 @@ export const packEnvelope = (tree) => ({
83
83
  readAt: tree.meta.readAt,
84
84
  repoKind: tree.meta.repoKind,
85
85
  coverage: tree.meta.coverage,
86
+ ...(tree.meta.columns ? { columns: tree.meta.columns } : {}),
87
+ repo_columns: tree.meta.repo_columns,
86
88
  notes: tree.meta.notes,
87
89
  payload: pack(tree),
88
90
  });
@@ -155,6 +155,7 @@ export const unpack = (env) => {
155
155
  readAt: env.readAt,
156
156
  repoKind: env.repoKind,
157
157
  coverage: env.coverage,
158
+ ...(env.columns ? { columns: env.columns } : {}),
158
159
  repo_columns: env.repo_columns ?? 'complete',
159
160
  notes: env.notes,
160
161
  },
@@ -10,7 +10,7 @@ import { unpack } from "../codec/unpack.js";
10
10
  import { packEnvelope } from "../codec/pack.js";
11
11
  import { upsertRepo, writeLock } from "../workspace/lock.js";
12
12
  import { renderStamp, stampCreated } from "../workspace/stamp.js";
13
- import { adoptCreatedIds, mergeAdoptedIntoBaseline, renderAdopted } from "../apply/adopt.js";
13
+ import { adoptCreatedIds, advanceBaseline, mergeAdoptedIntoBaseline, renderAdopted } from "../apply/adopt.js";
14
14
  import { baselineFile, planFile, planFileFor, findPlanFile } from "../workspace/layout.js";
15
15
  import { diffTrees } from "../diff/diff.js";
16
16
  import { compile } from "../apply/compile.js";
@@ -19,7 +19,7 @@ import { execute } from "../apply/execute.js";
19
19
  import { validateTree, blockingErrors } from "../validate/index.js";
20
20
  import { connect, openWorkspace, resolveWorkspace } from "./context.js";
21
21
  import { checkGit } from "../util/git.js";
22
- import { loadBaseline, loadBaselineEnvelope, refFor } from "./diff.js";
22
+ import { loadBaseline, loadBaselineEnvelope, refFor, refuseSuppressedEdits } from "./diff.js";
23
23
  import { loadPlan, renderPlan, warnUnverifiedSteps } from "./plan.js";
24
24
  import { verifyRepo } from "./verify.js";
25
25
  import { restoreToVersion } from "./misc.js";
@@ -438,6 +438,7 @@ const applyOneRepo = async (p) => {
438
438
  }
439
439
  rec?.before(await loadBaselineEnvelope(root, repo.id));
440
440
  const changeSet = diffTrees(baseline, edited);
441
+ refuseSuppressedEdits(changeSet, bool(p, 'force'));
441
442
  if (!changeSet.changes.length) {
442
443
  /**
443
444
  * The pipe half is clean. That is not the same as nothing to apply: flows
@@ -702,6 +703,7 @@ const applyOneRepo = async (p) => {
702
703
  */
703
704
  const takeSnapshot = caps.snapshots && (bool(p, 'post-snapshot') || (await confirmPostSnapshot(p)));
704
705
  const verified = await verifyRepo({
706
+ parsed: p,
705
707
  net,
706
708
  root,
707
709
  lock,
@@ -710,8 +712,15 @@ const applyOneRepo = async (p) => {
710
712
  idMap: result.idMap,
711
713
  preferReconstruct: !takeSnapshot,
712
714
  });
715
+ /**
716
+ * What was actually read, not what was asked for. A requested snapshot can
717
+ * come back as a reconstruction when the download fails (readLive), and
718
+ * everything below that adopts the read-back wholesale is only safe on a
719
+ * real snapshot — so it keys on this, never on takeSnapshot.
720
+ */
721
+ const realRead = verified.envelope.source === 'snapshot';
713
722
  if (!verified.ok) {
714
- warn(takeSnapshot
723
+ warn(realRead
715
724
  ? 'verification found differences between what was applied and what the workspace asks for:'
716
725
  : 'a free, GraphQL-reconstructed comparison found differences — coverage gaps may hide more (see below):');
717
726
  out(verified.render);
@@ -730,7 +739,7 @@ const applyOneRepo = async (p) => {
730
739
  * GraphQL reconstruction's partial coverage risks the same "adopt an
731
740
  * absence as a deletion" mistake the coverage doctrine exists to avoid.
732
741
  */
733
- if (takeSnapshot) {
742
+ if (realRead) {
734
743
  const authored = await readTree(dir);
735
744
  const previous = await loadBaselineEnvelope(root, repo.id);
736
745
  const stamp = stampCreated(authored, unpack(verified.envelope), previous ? unpack(previous) : null);
@@ -754,10 +763,20 @@ const applyOneRepo = async (p) => {
754
763
  await reportRun(rec, 'partial', `all ${result.done} steps applied, but verification found differences`);
755
764
  return 1;
756
765
  }
757
- ok(takeSnapshot ? 'verified — the pipe now matches the workspace' : 'verified against a free GraphQL reconstruction — the pipe matches the workspace');
766
+ ok(realRead ? 'verified — the pipe now matches the workspace' : 'verified against a free GraphQL reconstruction — the pipe matches the workspace');
758
767
  rec?.after(verified.envelope);
759
768
  rec?.verified(true);
760
- if (takeSnapshot) {
769
+ if (!realRead) {
770
+ // The read-back is not adopted (see advanceBaseline); the verified
771
+ // workspace is, for every group the baseline could see.
772
+ const previousEnvelope = await loadBaselineEnvelope(root, repo.id);
773
+ if (previousEnvelope) {
774
+ const next = advanceBaseline(unpack(previousEnvelope), await readTree(dir));
775
+ await writeJson(baselineFile(root, repo.id), { ...packEnvelope(next), readAt: new Date().toISOString() });
776
+ info(dim(' baseline advanced to the applied workspace — the next diff starts from what is now live'));
777
+ }
778
+ }
779
+ if (realRead) {
761
780
  /**
762
781
  * Adopt the live state as both the workspace and the baseline —
763
782
  * strictly more complete than the id-only adoption above, since it also
@@ -1,7 +1,7 @@
1
1
  import { dirname, isAbsolute, join, resolve } from 'node:path';
2
2
  import { LIMITS, credentialProvider, readOrgId } from "../config.js";
3
3
  import { exists } from "../util/fsx.js";
4
- import { UserError, debug } from "../util/log.js";
4
+ import { UserError, debug, warn } from "../util/log.js";
5
5
  import { PipefyClient } from "../pipefy/client.js";
6
6
  import { InternalApi } from "../pipefy/internal.js";
7
7
  import { SnapshotClient } from "../pipefy/snapshot.js";
@@ -10,7 +10,8 @@ import { IpaasClient } from "../pipefy/ipaas.js";
10
10
  import { detect } from "../pipefy/capability.js";
11
11
  import { paths } from "../workspace/layout.js";
12
12
  import { readLock } from "../workspace/lock.js";
13
- import { str } from "../cli/args.js";
13
+ import { bool, str } from "../cli/args.js";
14
+ import { fullCoverage } from "../model/payload.js";
14
15
  export const connect = async (p) => {
15
16
  /**
16
17
  * One credential, shared by the public client and the internal adapter, and
@@ -77,3 +78,40 @@ export const openWorkspace = async (p, positionalIndex = 0) => {
77
78
  return { root, lock, repo, repoDir: join(root, repo.dir) };
78
79
  };
79
80
  export const readConcurrency = () => LIMITS.readConcurrency;
81
+ /**
82
+ * Read one repo live: a snapshot when the endpoint has them and the caller
83
+ * wants one, a GraphQL reconstruction otherwise — and a reconstruction when
84
+ * the snapshot read fails.
85
+ *
86
+ * The fallback is the point. Pipefy's storage answered every signed snapshot
87
+ * URL with HTTP 500 for days while creation and the UPLOADED status were
88
+ * fine, and every command that read live through a snapshot — verify, diff
89
+ * --against live, a cross-repo diff, export — failed outright, after `pull`
90
+ * had already learned to carry on. Reconstruction says per group what it
91
+ * could not see, so falling back never makes absence look like deletion.
92
+ * `--no-fallback` keeps the old refusal.
93
+ */
94
+ export const readLive = async (net, p, repoId, opts = {}) => {
95
+ const caps = await net.capabilities();
96
+ const reconstruct = () => net.reconstructor.reconstruct(repoId, { organizationId: opts.organizationId ?? net.organizationId, noInternal: opts.noInternal });
97
+ if (!caps.snapshots || opts.preferSnapshot === false)
98
+ return reconstruct();
99
+ try {
100
+ const { payload, versionId } = await net.snapshots.pull(repoId, opts.label);
101
+ return {
102
+ source: 'snapshot',
103
+ versionId,
104
+ readAt: new Date().toISOString(),
105
+ repoKind: opts.kind ?? 'pipe',
106
+ coverage: fullCoverage('complete'),
107
+ notes: [],
108
+ payload,
109
+ };
110
+ }
111
+ catch (e) {
112
+ if (bool(p, 'no-fallback'))
113
+ throw e;
114
+ warn(`snapshot read failed for repo ${repoId}: ${e.message} — reading it over GraphQL instead (--no-fallback to stop here)`);
115
+ return reconstruct();
116
+ }
117
+ };
@@ -6,10 +6,10 @@ import { UserError, bold, dim, info, ok, out, warn } from "../util/log.js";
6
6
  import { unpack } from "../codec/unpack.js";
7
7
  import { readTree } from "../workspace/read.js";
8
8
  import { baselineFile, repoDir } from "../workspace/layout.js";
9
- import { diffTrees } from "../diff/diff.js";
9
+ import { diffTrees, suppressedEdits } from "../diff/diff.js";
10
10
  import { renderChangeSet, toJson } from "../diff/render.js";
11
11
  import { renderHtml } from "../diff/html.js";
12
- import { connect, openWorkspace } from "./context.js";
12
+ import { connect, openWorkspace, readLive } from "./context.js";
13
13
  import { diffFlowsFor } from "./flows.js";
14
14
  import { renderFlowChanges } from "../diff/flow.js";
15
15
  import { diffAgentsFor } from "./agents.js";
@@ -43,9 +43,7 @@ export const run = async (p) => {
43
43
  const net = await connect(p);
44
44
  const caps = await net.capabilities();
45
45
  info(dim(`re-reading pipe ${repo.id} (${caps.snapshots ? 'snapshot' : 'GraphQL reconstruction'})`));
46
- const env = caps.snapshots
47
- ? await liveSnapshot(net, repo.id)
48
- : await net.reconstructor.reconstruct(repo.id, { organizationId: lock.organizationId });
46
+ const env = await readLive(net, p, repo.id, { kind: repo.kind, organizationId: lock.organizationId });
49
47
  baseline = unpack(env);
50
48
  }
51
49
  else if (against !== 'baseline') {
@@ -61,9 +59,9 @@ export const run = async (p) => {
61
59
  }
62
60
  else {
63
61
  const net = await connect(p);
64
- const caps = await net.capabilities();
65
62
  info(dim(`reading pipe ${otherId} live for a cross-repo diff`));
66
- const env = caps.snapshots ? await liveSnapshot(net, otherId) : await net.reconstructor.reconstruct(otherId, { organizationId: lock.organizationId });
63
+ // The other pipe's own organization, not this workspace's (readLive -> reconstruct).
64
+ const env = await readLive(net, p, otherId, { organizationId: null });
67
65
  baseline = unpack(env);
68
66
  }
69
67
  }
@@ -219,6 +217,23 @@ export const run = async (p) => {
219
217
  * report does not: a report keeps the payload verbatim so the run folder is a
220
218
  * usable copy of the pipe, not a reshaped one.
221
219
  */
220
+ /**
221
+ * Refuse a plan that would apply only part of what the workspace asks for.
222
+ * See suppressedEdits. `--force` keeps its meaning elsewhere — apply what can
223
+ * be applied — and says out loud what it is leaving behind.
224
+ */
225
+ export const refuseSuppressedEdits = (cs, force) => {
226
+ const left = suppressedEdits(cs);
227
+ if (!left.length)
228
+ return;
229
+ const list = left.map((s) => ` ${s.op} ${s.entity}: ${s.label}`).join('\n');
230
+ if (force) {
231
+ warn(`--force: ${left.length} create/delete(s) in the workspace are left out of this plan:\n${list}`);
232
+ return;
233
+ }
234
+ throw new UserError(`${left.length} create/delete(s) in the workspace cannot be planned, because the baseline read those rows only partially:\n${list}\n` +
235
+ 'Planning the rest without them would apply part of what was asked for.', 're-pull to refresh the baseline, or pass --force to apply the rest anyway');
236
+ };
222
237
  export const loadBaselineEnvelope = async (root, repoId) => {
223
238
  try {
224
239
  return await readJson(baselineFile(root, repoId));
@@ -246,19 +261,6 @@ export const loadBaselineQuiet = async (root, repoId) => {
246
261
  return null;
247
262
  }
248
263
  };
249
- const liveSnapshot = async (net, repoId) => {
250
- const { payload, versionId } = await net.snapshots.pull(repoId);
251
- const { fullCoverage } = await import("../model/payload.js");
252
- return {
253
- source: 'snapshot',
254
- versionId,
255
- readAt: new Date().toISOString(),
256
- repoKind: 'pipe',
257
- coverage: fullCoverage('complete'),
258
- notes: [],
259
- payload,
260
- };
261
- };
262
264
  export const writeHtml = async (path, changeSet, workspace, plan) => {
263
265
  const html = renderHtml({ changeSet, plan, workspace, generatedAt: new Date().toISOString() });
264
266
  await writeFile(path, html);
@@ -13,7 +13,7 @@ import { RECONSTRUCTION_GAPS } from "../pipefy/reconstruct.js";
13
13
  import { readTree } from "../workspace/read.js";
14
14
  import { registryRows } from "../apply/registry.js";
15
15
  import { validateTree, renderFindings, blockingErrors } from "../validate/index.js";
16
- import { connect, openWorkspace } from "./context.js";
16
+ import { connect, openWorkspace, readLive } from "./context.js";
17
17
  // ── login ────────────────────────────────────────────────────────────────────
18
18
  export const loginHelp = `pipe login [options]
19
19
 
@@ -167,10 +167,13 @@ export const doctor = async (p) => {
167
167
  if (repoId) {
168
168
  out('');
169
169
  out(bold(`probing repo ${repoId}`));
170
- const jumps = await net.internal.getPhaseSettings(await firstPhaseId(net, repoId));
170
+ const repo = await probeRepo(net, repoId);
171
+ const jumps = await net.internal.getPhaseSettings(repo.firstPhaseId);
171
172
  out(` internal settings endpoint ${jumps ? green('answers') : red('no answer')}`);
172
- const autos = net.organizationId ? await net.internal.listAutomations(net.organizationId, repoId) : null;
173
- out(` internal automations query ${autos ? green(`answers (${autos.length} automations)`) : net.organizationId ? red('no answer') : yellow('needs --org')}`);
173
+ // The repo's own organization, as a pull uses — not the saved default.
174
+ const orgId = repo.organizationId ?? net.organizationId;
175
+ const autos = orgId ? await net.internal.listAutomations(orgId, repoId) : null;
176
+ out(` internal automations query ${autos ? green(`answers (${autos.length} automations)`) : orgId ? red('no answer') : yellow('needs --org')}`);
174
177
  }
175
178
  out('');
176
179
  out(bold('internal API'));
@@ -203,9 +206,12 @@ export const doctor = async (p) => {
203
206
  }
204
207
  return 0;
205
208
  };
206
- const firstPhaseId = async (net, repoId) => {
207
- const data = await net.client.tryRaw('query($id: ID!) { pipe(id: $id) { phases { id } } }', { id: repoId }, 'first phase');
208
- return data?.pipe?.phases?.[0]?.id ?? '0';
209
+ const probeRepo = async (net, repoId) => {
210
+ const data = await net.client.tryRaw('query($id: ID!) { pipe(id: $id) { organizationId phases { id } } }', { id: repoId }, 'probe repo');
211
+ return {
212
+ firstPhaseId: data?.pipe?.phases?.[0]?.id ?? '0',
213
+ organizationId: data?.pipe?.organizationId ? String(data.pipe.organizationId) : null,
214
+ };
209
215
  };
210
216
  // ── export ───────────────────────────────────────────────────────────────────
211
217
  export const exportHelp = `pipe export <pipeId> [options]
@@ -225,26 +231,7 @@ export const exportCmd = async (p) => {
225
231
  const net = await connect(p);
226
232
  const caps = await net.capabilities();
227
233
  const useSnapshot = caps.snapshots && !bool(p, 'no-snapshot');
228
- let env;
229
- if (useSnapshot) {
230
- const { payload, versionId } = await net.snapshots.pull(repoId);
231
- const { fullCoverage } = await import("../model/payload.js");
232
- env = {
233
- source: 'snapshot',
234
- versionId,
235
- readAt: new Date().toISOString(),
236
- repoKind: 'pipe',
237
- coverage: fullCoverage('complete'),
238
- notes: [],
239
- payload,
240
- };
241
- }
242
- else {
243
- env = await net.reconstructor.reconstruct(repoId, {
244
- organizationId: net.organizationId,
245
- noInternal: bool(p, 'no-internal'),
246
- });
247
- }
234
+ const env = await readLive(net, p, repoId, { preferSnapshot: useSnapshot, noInternal: bool(p, 'no-internal') });
248
235
  /**
249
236
  * A bare payload has no provenance, and for a reconstructed read that matters.
250
237
  *
@@ -10,7 +10,7 @@ import { compile } from "../apply/compile.js";
10
10
  import { readAutomationCatalogue } from "../pipefy/automations.js";
11
11
  import { validateTree, renderFindings, blockingErrors } from "../validate/index.js";
12
12
  import { connect, openWorkspace } from "./context.js";
13
- import { loadBaseline, writeHtml, openViewer } from "./diff.js";
13
+ import { loadBaseline, writeHtml, openViewer, refuseSuppressedEdits } from "./diff.js";
14
14
  export const help = `pipe plan [workspace] [options]
15
15
 
16
16
  --repo <id> which repo in the workspace (default: the root pipe)
@@ -115,6 +115,8 @@ export const run = async (p) => {
115
115
  throw new UserError(`${blocking.length} validation error${blocking.length === 1 ? '' : 's'} — fix these before planning`);
116
116
  }
117
117
  const changeSet = diffTrees(baseline, edited);
118
+ // Before the empty check: edits that were all left out are not "nothing to do".
119
+ refuseSuppressedEdits(changeSet, bool(p, 'force'));
118
120
  if (!changeSet.changes.length) {
119
121
  /**
120
122
  * The pipe half is clean, which is not the same as nothing to do: a flow-only
@@ -196,14 +196,23 @@ export const run = async (p) => {
196
196
  * the next diff had no baseline. This used to finish with exit code 0 and a
197
197
  * "next: pipe validate", which scripts and agents believed.
198
198
  */
199
- if (!(await readRepo(rootRef))) {
199
+ const rootEnvelope = await readRepo(rootRef);
200
+ if (!rootEnvelope) {
200
201
  ui.stop();
201
202
  throw new UserError(`could not read the root repo ${rootRepoId}${rootRef.name ? ` (${rootRef.name})` : ''} — nothing was written to ${outDir}`, useSnapshots && noFallback
202
203
  ? 'drop --no-fallback to read it over GraphQL, or run with --verbose for the cause'
203
204
  : 'run with --verbose for the cause');
204
205
  }
205
206
  await mkdirp(outDir);
206
- const lock = emptyLock(rootRepoId, net.organizationId, depth);
207
+ /**
208
+ * The workspace's organization is the root pipe's, not the credential's
209
+ * default: `plan` hands this to the compiler, and a token can belong to
210
+ * several organizations. The default is only the fallback for a payload
211
+ * that does not say.
212
+ */
213
+ const rootOrg = rootEnvelope.payload['organization_id'];
214
+ const organizationId = rootOrg ? String(rootOrg) : net.organizationId;
215
+ const lock = emptyLock(rootRepoId, organizationId, depth);
207
216
  lock.edges = discovery.edges;
208
217
  const trees = [];
209
218
  const sources = new Set();
@@ -308,7 +317,8 @@ export const run = async (p) => {
308
317
  const detail = ref.kind === 'table'
309
318
  ? `${fieldCount} fields, ${Math.max(0, tree.phases.length - 1)} record statuses, ${written.length} files`
310
319
  : `${tree.phases.length} phases, ${fieldCount} fields, ` +
311
- `${tree.automations.length} automations, ${written.length} files`;
320
+ // "0 automations" beside an unknown group read as "this pipe has none".
321
+ `${env.coverage.automations === 'unknown' ? 'automations unknown' : `${tree.automations.length} automations`}, ${written.length} files`;
312
322
  ui.update(task, {
313
323
  state: 'done',
314
324
  detail: detail + (gaps.length ? `, ${gaps.length} groups partial or unknown` : ''),
@@ -319,7 +329,7 @@ export const run = async (p) => {
319
329
  name: rootRef.name ?? String(rootRepoId),
320
330
  createdAt: new Date().toISOString(),
321
331
  rootRepoId,
322
- organizationId: net.organizationId,
332
+ organizationId,
323
333
  depth,
324
334
  lastPullSource: sources.size > 1 ? 'mixed' : (sources.values().next().value ?? 'reconstructed'),
325
335
  tool: { ...TOOL },
@@ -1,13 +1,12 @@
1
1
  import { bool } from "../cli/args.js";
2
2
  import { writeJson } from "../util/fsx.js";
3
3
  import { dim, info, ok, out, stripAnsi, warn } from "../util/log.js";
4
- import { fullCoverage } from "../model/payload.js";
5
4
  import { unpack } from "../codec/unpack.js";
6
5
  import { readTree } from "../workspace/read.js";
7
6
  import { baselineFile } from "../workspace/layout.js";
8
7
  import { diffTrees } from "../diff/diff.js";
9
8
  import { renderChangeSet } from "../diff/render.js";
10
- import { connect, openWorkspace } from "./context.js";
9
+ import { connect, openWorkspace, readLive } from "./context.js";
11
10
  import { RunRecorder, invocation } from "../report/run.js";
12
11
  export const help = `pipe verify [workspace] [options]
13
12
 
@@ -40,20 +39,7 @@ export const run = async (p) => {
40
39
  const { writeTree } = await import("../workspace/write.js");
41
40
  const caps = await net.capabilities();
42
41
  info(dim(`reading pipe ${repo.id} to match against (${caps.snapshots ? 'snapshot' : 'GraphQL reconstruction'})`));
43
- const envelope = caps.snapshots
44
- ? await (async () => {
45
- const { payload, versionId } = await net.snapshots.pull(repo.id);
46
- return {
47
- source: 'snapshot',
48
- versionId,
49
- readAt: new Date().toISOString(),
50
- repoKind: repo.kind,
51
- coverage: fullCoverage('complete'),
52
- notes: [],
53
- payload,
54
- };
55
- })()
56
- : await net.reconstructor.reconstruct(repo.id, { organizationId: lock.organizationId });
42
+ const envelope = await readLive(net, p, repo.id, { kind: repo.kind, organizationId: lock.organizationId });
57
43
  const authored = await readTree(dir);
58
44
  const { loadBaselineEnvelope } = await import("./diff.js");
59
45
  const previous = await loadBaselineEnvelope(root, repo.id);
@@ -88,7 +74,7 @@ export const run = async (p) => {
88
74
  return 0;
89
75
  }
90
76
  const rec = recorderFor(p, root, repo, 'verify');
91
- const result = await verifyRepo({ net, root, lock, repo, expectedTreeDir: dir });
77
+ const result = await verifyRepo({ net, parsed: p, root, lock, repo, expectedTreeDir: dir });
92
78
  rec?.after(result.envelope);
93
79
  rec?.verified(result.ok, result.ok ? undefined : `${result.changeCount} difference(s) remain`);
94
80
  if (bool(p, 'json')) {
@@ -143,6 +129,8 @@ const write = async (rec, outcome, summary) => {
143
129
  else if (res)
144
130
  info(dim(` run recorded in runs/${rec.id}/`));
145
131
  };
132
+ /** No flags: a caller that passes none gets the defaults, fallback included. */
133
+ const EMPTY_ARGS = { command: 'verify', positionals: [], flags: {} };
146
134
  export const verifyRepo = async (args) => {
147
135
  const { net, lock, repo, expectedTreeDir, idMap, preferReconstruct } = args;
148
136
  const caps = await net.capabilities();
@@ -159,21 +147,12 @@ export const verifyRepo = async (args) => {
159
147
  * comparison and a verified-clean apply proves less than it appears to.
160
148
  * `preferReconstruct` trades that away deliberately — see its doc comment.
161
149
  */
162
- if (useSnapshot) {
163
- const { payload, versionId } = await net.snapshots.pull(repo.id, `post-apply/${new Date().toISOString()}`);
164
- envelope = {
165
- source: 'snapshot',
166
- versionId,
167
- readAt: new Date().toISOString(),
168
- repoKind: repo.kind,
169
- coverage: fullCoverage('complete'),
170
- notes: [],
171
- payload,
172
- };
173
- }
174
- else {
175
- envelope = await net.reconstructor.reconstruct(repo.id, { organizationId: lock.organizationId });
176
- }
150
+ envelope = await readLive(net, args.parsed ?? EMPTY_ARGS, repo.id, {
151
+ kind: repo.kind,
152
+ label: `post-apply/${new Date().toISOString()}`,
153
+ organizationId: lock.organizationId,
154
+ preferSnapshot: useSnapshot,
155
+ });
177
156
  const live = unpack(envelope);
178
157
  /**
179
158
  * Direction matters: live -> expected. A change in this diff means "the pipe
package/dist/diff/diff.js CHANGED
@@ -378,8 +378,19 @@ export const diffTrees = (before, after, opts = {}) => {
378
378
  * second would be catastrophic.
379
379
  */
380
380
  const skipSet = new Set(opts.skipEntities ?? []);
381
- /** Which sides read this group only partially. See deltasOf. */
381
+ /**
382
+ * Which sides read this group's *columns* only partially. See deltasOf.
383
+ * `meta.columns` when the read recorded it, otherwise the group's coverage —
384
+ * what a snapshot, and every workspace pulled before the two were split,
385
+ * carry.
386
+ */
387
+ const columnCoverage = (tree, entity) => tree.meta.columns?.[entity] ?? tree.meta.coverage?.[entity] ?? 'unknown';
382
388
  const partialSides = (entity) => ({
389
+ before: columnCoverage(before, entity) === 'partial',
390
+ after: columnCoverage(after, entity) === 'partial',
391
+ });
392
+ /** Which sides read this group's *rows* only partially — the row rule below. */
393
+ const partialRows = (entity) => ({
383
394
  before: (before.meta.coverage?.[entity] ?? 'unknown') === 'partial',
384
395
  after: (after.meta.coverage?.[entity] ?? 'unknown') === 'partial',
385
396
  });
@@ -392,12 +403,14 @@ export const diffTrees = (before, after, opts = {}) => {
392
403
  * "deletes" — and an apply would have deleted two live automations because one
393
404
  * read path could not see them.
394
405
  *
395
- * So on a partial side, absence means unseen: a delete is suppressed when the
396
- * *after* side is partial, and a create when the *before* side is partial.
406
+ * So on a side whose *rows* are partial, absence means unseen: a delete is
407
+ * suppressed when the *after* side is partial, and a create when the *before*
408
+ * side is partial. Missing columns say nothing about rows, so they do not
409
+ * enter into this (see Coverage on TreeMeta.columns).
397
410
  */
398
411
  const suppressed = [];
399
412
  const rowAbsenceIsUnknown = (entity, op, label) => {
400
- const sides = partialSides(entity);
413
+ const sides = partialRows(entity);
401
414
  const unknown = op === 'delete' ? sides.after : sides.before;
402
415
  if (unknown)
403
416
  suppressed.push({ entity, op, label });
@@ -963,6 +976,7 @@ export const diffTrees = (before, after, opts = {}) => {
963
976
  entity: s.entity,
964
977
  reason: `"${s.label}" would read as a ${s.op}, but that side's coverage is partial — ` +
965
978
  `absence there is unseen, not gone. Re-read from a snapshot to decide.`,
979
+ suppressed: { op: s.op, label: s.label },
966
980
  });
967
981
  }
968
982
  return {
@@ -975,6 +989,16 @@ export const diffTrees = (before, after, opts = {}) => {
975
989
  };
976
990
  };
977
991
  export const isEmpty = (cs) => cs.changes.length === 0;
992
+ /**
993
+ * The creates and deletes this change set had to leave out.
994
+ *
995
+ * Between a workspace and its own baseline every one of these is an edit
996
+ * someone made, and planning the rest without them is applying half of what
997
+ * was asked for. Found in a field run: 20 updates would have gone through and
998
+ * the 2 phase creates they pointed at would not, leaving a pipe with no final
999
+ * phase. `plan` and `apply` refuse on these unless told otherwise.
1000
+ */
1001
+ export const suppressedEdits = (cs) => cs.skipped.flatMap((s) => (s.suppressed ? [{ entity: s.entity, ...s.suppressed }] : []));
978
1002
  export const countByOp = (cs) => {
979
1003
  const out = { create: 0, update: 0, delete: 0, reorder: 0, archive: 0, unarchive: 0 };
980
1004
  for (const c of cs.changes)
@@ -132,7 +132,11 @@ export class InternalApi {
132
132
  this.status = 'unavailable';
133
133
  if (!this.warned) {
134
134
  this.warned = true;
135
- warn(`internal API unavailable (${reason}). Automation details and phase jumps ` +
135
+ warn(
136
+ // "Unavailable" read as an outage for answers like "Permission denied"
137
+ // or "Couldn't find Repo", which are the endpoint answering about this
138
+ // pipe and this credential. The reason says which.
139
+ `internal API call failed (${reason}). Automation details and phase jumps ` +
136
140
  `will be reported as unknown rather than empty — see PLAN.md §2.1.`);
137
141
  }
138
142
  }
@@ -188,8 +192,19 @@ export class InternalApi {
188
192
  this.status = results.every((r) => r.ok) ? 'ok' : 'incompatible';
189
193
  return results;
190
194
  }
191
- /** GraphQL over the internal endpoint. Returns null when the endpoint fails. */
192
- async gql(query, variables, label) {
195
+ /** The message of the last failed `gql` call, for a caller that reports it itself. */
196
+ lastError = null;
197
+ /**
198
+ * GraphQL over the internal endpoint. Returns null when the endpoint fails.
199
+ *
200
+ * `soft` is for a call whose failure is an answer about this pipe rather than
201
+ * about the endpoint — the iPaaS token, refused with "iPaaS is disabled for
202
+ * your organization". It used to mark the adapter unavailable and print that
203
+ * automation details and phase jumps would be unknown, on a pull where both
204
+ * had just been read. The caller reports it, from `lastError`.
205
+ */
206
+ async gql(query, variables, label, opts = {}) {
207
+ this.lastError = null;
193
208
  try {
194
209
  const data = await retry(async () => {
195
210
  this.requestCount++;
@@ -225,6 +240,9 @@ export class InternalApi {
225
240
  }
226
241
  catch (e) {
227
242
  debug(`internal gql ${label} failed: ${e.message}`);
243
+ this.lastError = e.message;
244
+ if (opts.soft && !e.invalidQuery)
245
+ return null;
228
246
  if (e.invalidQuery)
229
247
  this.markIncompatible(label, e.message);
230
248
  else
@@ -86,11 +86,12 @@ export class IpaasClient {
86
86
  return hit.session;
87
87
  const minted = await this.internal.gql(`query GetAdvancedAutomationsToken($repoId: ID!) {
88
88
  advancedAutomationsToken(repoId: $repoId) { token }
89
- }`, { repoId: String(pipeId) }, 'GetAdvancedAutomationsToken');
89
+ }`, { repoId: String(pipeId) }, 'GetAdvancedAutomationsToken', { soft: true });
90
90
  const token = minted?.advancedAutomationsToken?.token;
91
91
  if (!token) {
92
- throw new IpaasUnavailable(pipeId, `no advanced-automations token for pipe ${pipeId} — iPaaS may be off for this organization, ` +
93
- 'or this credential may not be allowed to create automations on the pipe');
92
+ const said = this.internal.lastError;
93
+ throw new IpaasUnavailable(pipeId, `no advanced-automations token for pipe ${pipeId} — ` +
94
+ (said ?? 'iPaaS may be off for this organization, or this credential may not be allowed to create automations on the pipe'));
94
95
  }
95
96
  const res = await fetch(`${this.base}/managed-authn/external-token`, {
96
97
  method: 'POST',
@@ -236,7 +236,8 @@ export class Reconstructor {
236
236
  p['index'] = i;
237
237
  });
238
238
  relations.phases = phaseRows;
239
- mark('phases', phase.missing ? 'partial' : 'complete');
239
+ // Every phase comes back; what can be missing is columns, recorded below.
240
+ mark('phases', 'complete');
240
241
  /**
241
242
  * `Phase.next_phase_ids` is public, but it is **not** the payload's
242
243
  * `phase_jumps`. Checked against a snapshot of the same pipe:
@@ -291,7 +292,7 @@ export class Reconstructor {
291
292
  for (const p of gqlPhases)
292
293
  pushFields(p['fields'] ?? [], numOrNull(p['id']));
293
294
  relations.fields = fieldRows;
294
- mark('fields', field.missing ? 'partial' : 'complete');
295
+ mark('fields', 'complete');
295
296
  if (field.missing)
296
297
  notes.push(`field columns not in this schema: ${field.skipped.join(', ')}`);
297
298
  // ── labels ──────────────────────────────────────────────────────────────
@@ -303,9 +304,9 @@ export class Reconstructor {
303
304
  color: l['color'],
304
305
  repo_id: root['id'],
305
306
  }));
306
- // No `uuid` over the public API, and uuid is how labels are identified in
307
- // a snapshot — so this is partial, not complete.
308
- mark('labels', 'partial');
307
+ // Every label comes back. No `uuid` over the public API — and uuid is how
308
+ // a snapshot identifies labels — but that is a column, recorded below.
309
+ mark('labels', 'complete');
309
310
  }
310
311
  else {
311
312
  notes.push('labels: not readable over this endpoint');
@@ -329,27 +330,115 @@ export class Reconstructor {
329
330
  notes.push('webhooks: not readable over this endpoint');
330
331
  }
331
332
  // ── pipe_relations ──────────────────────────────────────────────────────
332
- const prSel = await this.client.prune('Pipe', [
333
- ['pipe_relations', '{ id name parent { id } child { id } canCreateNewItems canConnectExistingItems canConnectMultipleItems childMustExistToMoveParent childMustExistToFinishParent allChildrenMustBeDoneToMoveParent allChildrenMustBeDoneToFinishParent }'],
334
- ]);
335
- if (prSel) {
336
- const pr = await this.client.tryRaw(`query($id: ID!) { pipe(id: $id) { ${prSel} } }`, { id: String(repoId) }, 'pipe_relations');
337
- if (pr?.pipe?.pipe_relations) {
338
- relations.pipe_relations = pr.pipe.pipe_relations.map((r) => ({
339
- id: numOrNull(r['id']),
340
- name: r['name'],
341
- parent_id: numOrNull(firstId(r['parent'])),
342
- child_id: numOrNull(firstId(r['child'])),
343
- can_create_new_items: r['canCreateNewItems'],
344
- can_connect_existing_items: r['canConnectExistingItems'],
345
- can_connect_multiple_items: r['canConnectMultipleItems'],
346
- child_must_exist_to_move_parent: r['childMustExistToMoveParent'],
347
- child_must_exist_to_finish_parent: r['childMustExistToFinishParent'],
348
- all_children_must_be_done_to_move_parent: r['allChildrenMustBeDoneToMoveParent'],
349
- all_children_must_be_done_to_finish_parent: r['allChildrenMustBeDoneToFinishParent'],
350
- }));
351
- mark('pipe_relations', 'partial');
352
- }
333
+ /**
334
+ * `Pipe.pipe_relations` does not exist — this used to ask for it, the prune
335
+ * dropped it, and every reconstructed pull had pipe_relations `unknown`.
336
+ * The public names are `parentsRelations` and `childrenRelations`, and their
337
+ * `parent` / `child` are a union (Pipe | Table), so `{ id }` is refused
338
+ * until it is spread per type. Both directions are read and merged by id:
339
+ * which of them a snapshot carries has not been measured, and a superset
340
+ * can only add a row a snapshot lacks, never hide one this pipe has.
341
+ */
342
+ const REL = `{ id name
343
+ parent { ... on Pipe { id } ... on Table { id } }
344
+ child { ... on Pipe { id } ... on Table { id } }
345
+ canCreateNewItems canConnectExistingItems canConnectMultipleItems
346
+ childMustExistToMoveParent childMustExistToFinishParent
347
+ allChildrenMustBeDoneToMoveParent allChildrenMustBeDoneToFinishParent }`;
348
+ const pr = await this.client.tryRaw(`query($id: ID!) { pipe(id: $id) { parentsRelations ${REL} childrenRelations ${REL} } }`, { id: String(repoId) }, 'pipe_relations');
349
+ if (pr?.pipe && Array.isArray(pr.pipe.parentsRelations) && Array.isArray(pr.pipe.childrenRelations)) {
350
+ const byId = new Map();
351
+ for (const r of [...pr.pipe.parentsRelations, ...pr.pipe.childrenRelations])
352
+ byId.set(String(r['id']), r);
353
+ relations.pipe_relations = [...byId.values()].map((r) => ({
354
+ id: numOrNull(r['id']),
355
+ name: r['name'],
356
+ parent_id: numOrNull(firstId(r['parent'])),
357
+ child_id: numOrNull(firstId(r['child'])),
358
+ can_create_new_items: r['canCreateNewItems'],
359
+ can_connect_existing_items: r['canConnectExistingItems'],
360
+ can_connect_multiple_items: r['canConnectMultipleItems'],
361
+ child_must_exist_to_move_parent: r['childMustExistToMoveParent'],
362
+ child_must_exist_to_finish_parent: r['childMustExistToFinishParent'],
363
+ all_children_must_be_done_to_move_parent: r['allChildrenMustBeDoneToMoveParent'],
364
+ all_children_must_be_done_to_finish_parent: r['allChildrenMustBeDoneToFinishParent'],
365
+ }));
366
+ mark('pipe_relations', 'complete');
367
+ }
368
+ else {
369
+ notes.push('pipe_relations: parentsRelations / childrenRelations did not answer');
370
+ }
371
+ // ── phase team members ──────────────────────────────────────────────────
372
+ // `Phase.teamMembers` is public; the payload holds one row per member.
373
+ const tm = await this.client.tryRaw(`query($id: ID!) { pipe(id: $id) { phases { id teamMembers { id } } } }`, { id: String(repoId) }, 'phase team members');
374
+ if (tm?.pipe?.phases?.every((ph) => Array.isArray(ph.teamMembers))) {
375
+ relations.phase_team_memberships = tm.pipe.phases.flatMap((ph) => (ph.teamMembers ?? []).map((u) => ({ phase_id: numOrNull(ph.id), user_id: numOrNull(u['id']) })));
376
+ mark('phase_team_memberships', 'complete');
377
+ }
378
+ else {
379
+ notes.push('phase_team_memberships: Phase.teamMembers did not answer');
380
+ }
381
+ // ── repo preferences and the public form ────────────────────────────────
382
+ /**
383
+ * One row each, both public. Fewer columns than a snapshot's (no id, no
384
+ * custom sorting, no guests settings) — columns are partial, as for every
385
+ * reconstructed group — but every column the compiler can write is here.
386
+ * `submitterEmailCollectionMethod` is the payload's `submitter_email_required`:
387
+ * both read "required" on the same pipe.
388
+ */
389
+ const prefs = await this.client.tryRaw(`query($id: ID!) { pipe(id: $id) { preferences {
390
+ aiAgentsEnabled aiCopilotEnabled findable hiddenStartFormAttributes
391
+ hiddenTopButtons inboxEmailEnabled mainTabViews startFormTitle } } }`, { id: String(repoId) }, 'repo_preferences');
392
+ if (prefs?.pipe) {
393
+ const r = prefs.pipe.preferences;
394
+ relations.repo_preferences = r
395
+ ? [
396
+ {
397
+ repo_id: root['id'],
398
+ ai_agents_enabled: r['aiAgentsEnabled'],
399
+ ai_copilot_enabled: r['aiCopilotEnabled'],
400
+ findable: r['findable'],
401
+ hidden_start_form_attributes: r['hiddenStartFormAttributes'] ?? [],
402
+ hidden_top_buttons: r['hiddenTopButtons'] ?? [],
403
+ inbox_email_enabled: r['inboxEmailEnabled'],
404
+ main_tab_views: r['mainTabViews'] ?? [],
405
+ start_form_title: r['startFormTitle'] ?? null,
406
+ },
407
+ ]
408
+ : [];
409
+ mark('repo_preferences', 'complete');
410
+ }
411
+ const pf = await this.client.tryRaw(`query($id: ID!) { pipe(id: $id) { publicForm {
412
+ id active title description submitButtonText afterSubmitMessage brandColor
413
+ backgroundColor backgroundImage logo displayPipefyLogo showSubmitAnotherResponseButton
414
+ reuseLastSubmissionResponse submitterEmailCollectionEnabled
415
+ submitterEmailCollectionMethod submitterEmailField { id } } } }`, { id: String(repoId) }, 'public_forms');
416
+ if (pf?.pipe) {
417
+ const f = pf.pipe.publicForm;
418
+ relations.public_forms = f
419
+ ? [
420
+ {
421
+ id: numOrNull(f['id']),
422
+ repo_id: root['id'],
423
+ active: f['active'],
424
+ title: f['title'] ?? null,
425
+ description: f['description'] ?? null,
426
+ submit_button_text: f['submitButtonText'] ?? null,
427
+ after_submit_message: f['afterSubmitMessage'] ?? null,
428
+ brand_color: f['brandColor'] ?? null,
429
+ background_color: f['backgroundColor'] ?? null,
430
+ background_image: f['backgroundImage'] ?? null,
431
+ logo: f['logo'] ?? null,
432
+ display_pipefy_logo: f['displayPipefyLogo'],
433
+ show_submit_another_response_button: f['showSubmitAnotherResponseButton'],
434
+ reuse_last_submission_response: f['reuseLastSubmissionResponse'],
435
+ submitter_email_collection_enabled: f['submitterEmailCollectionEnabled'],
436
+ submitter_email_required: f['submitterEmailCollectionMethod'] ?? null,
437
+ submitter_email_field_id: numOrNull(firstId(f['submitterEmailField'])),
438
+ },
439
+ ]
440
+ : [];
441
+ mark('public_forms', 'complete');
353
442
  }
354
443
  // ── field conditions (+ actions, condition, expressions) ────────────────
355
444
  await this.readFieldConditions(repoId, root, relations, mark, notes);
@@ -367,19 +456,27 @@ export class Reconstructor {
367
456
  }
368
457
  }
369
458
  /**
370
- * No reconstructed group is ever 'complete'.
459
+ * No reconstructed group has every *column*; most have every *row*.
371
460
  *
372
- * Row coverage and *column* coverage are different things: the public API
461
+ * Row coverage and column coverage are different things: the public API
373
462
  * returns every phase, but not `color_uuid`, `only_admin_can_move_to_previous`
374
- * or `repo_id`. Marking such a group complete made a snapshot-vs-reconstruction
375
- * diff report ten phantom changes per phase, each one a proposal to null a
376
- * column that had simply not been read. 'partial' is the honest ceiling, and it
377
- * is what tells the diff to treat absence here as no information.
463
+ * or `repo_id`. Marking such a group's columns complete made a
464
+ * snapshot-vs-reconstruction diff report ten phantom changes per phase, each
465
+ * one a proposal to null a column that had simply not been read — so every
466
+ * group that was read has its columns marked partial.
467
+ *
468
+ * Rows are another matter, and they used to be marked partial too. That
469
+ * made the diff refuse every create and delete on a reconstructed pull,
470
+ * which was every pull while the snapshot download answered 500: a new
471
+ * phase could not be created because the read had not seen `color_uuid`.
472
+ * `coverage` now says what was measured about rows — complete where the
473
+ * API returns the whole list, partial where it is known not to
474
+ * (phase_jumps read forward-only, pipe_relations never compared against a
475
+ * snapshot, automations that would not load).
378
476
  */
379
- for (const name of ENTITY_NAMES) {
380
- if (coverage[name] === 'complete')
381
- coverage[name] = 'partial';
382
- }
477
+ const columns = {};
478
+ for (const name of ENTITY_NAMES)
479
+ columns[name] = coverage[name] === 'unknown' ? 'unknown' : 'partial';
383
480
  const payload = { ...root, id: Number(root['id']), relations };
384
481
  return {
385
482
  source: 'reconstructed',
@@ -387,6 +484,7 @@ export class Reconstructor {
387
484
  readAt: new Date().toISOString(),
388
485
  repoKind: 'pipe',
389
486
  coverage,
487
+ columns,
390
488
  repo_columns: pipe.missing ? 'partial' : 'complete',
391
489
  notes,
392
490
  payload,
@@ -472,7 +570,16 @@ export class Reconstructor {
472
570
  * shortcut — this is N+1 by necessity, bounded by readConcurrency.
473
571
  */
474
572
  async readAutomations(repoId, root, relations, mark, notes, organizationId) {
475
- const orgId = organizationId ?? (root['organization_id'] ? String(root['organization_id']) : null);
573
+ /**
574
+ * The repo's own organization first. The one passed in is the caller's
575
+ * default — `--org`, or whatever `pipe login --token` saved — and a token
576
+ * that belongs to several organizations pulls pipes from all of them. Sent
577
+ * a pipe from another organization, the internal query answers "Couldn't
578
+ * find Repo with 'id'=…" about a pipe that plainly exists, and every
579
+ * automation is left unknown. Seen in a field run, reproduced here with
580
+ * one pipe under its own organization (32 automations) and under another.
581
+ */
582
+ const orgId = (root['organization_id'] ? String(root['organization_id']) : null) ?? organizationId;
476
583
  if (!orgId) {
477
584
  notes.push('automations: no organization id available for the internal automations query');
478
585
  return;
@@ -664,8 +771,7 @@ const snakeParams = (v) => {
664
771
  };
665
772
  /** Reported by `pipe doctor` so an FDE knows what a reconstructed pull cannot see. */
666
773
  export const RECONSTRUCTION_GAPS = [
667
- 'public_forms — no public read path',
668
- 'repo_preferences — no public read path',
774
+ 'public_forms / repo_preferences — read, with fewer columns than a snapshot (no ids for preferences, no custom sorting)',
669
775
  'visibilities — no public read path (read-only anyway)',
670
776
  'email_templates / email_inboxes — no public read path (read-only anyway)',
671
777
  'labels.uuid — not exposed publicly, so labels are matched by id and name',
package/dist/util/log.js CHANGED
@@ -51,13 +51,23 @@ export const step = (s) => {
51
51
  process.stderr.write(blue('> ') + s + '\n');
52
52
  };
53
53
  const CLEAR_LINE = `\r${ESC}2K`;
54
- /** Single-line progress that collapses to nothing on a non-tty. */
54
+ /**
55
+ * Single-line progress that collapses to nothing on a non-tty.
56
+ *
57
+ * Dropped while a display holds the screen, like `step`: it wrote beneath the
58
+ * live frame without the frame knowing, and a long automation name wrapped to
59
+ * a second row, so the next repaint landed a row short. Clipped to the
60
+ * terminal otherwise, because a `\r` rewrite only reaches back one row too.
61
+ */
55
62
  export const progress = (s) => {
56
- if (!process.stderr.isTTY)
63
+ if (!process.stderr.isTTY || sink)
57
64
  return;
58
- process.stderr.write(CLEAR_LINE + dim(' ' + s));
65
+ const line = ' ' + s;
66
+ const width = process.stderr.columns;
67
+ const cut = width && [...line].length > width - 1 ? [...line].slice(0, width - 2).join('') + '…' : line;
68
+ process.stderr.write(CLEAR_LINE + dim(cut));
59
69
  };
60
- export const progressDone = () => { if (process.stderr.isTTY)
70
+ export const progressDone = () => { if (process.stderr.isTTY && !sink)
61
71
  process.stderr.write(CLEAR_LINE); };
62
72
  export class UserError extends Error {
63
73
  hint;
package/dist/util/tui.js CHANGED
@@ -17,6 +17,42 @@ const HIDE_CURSOR = `${ESC}?25l`;
17
17
  const SHOW_CURSOR = `${ESC}?25h`;
18
18
  const CLEAR_LINE = `${ESC}2K`;
19
19
  const cursorUp = (n) => (n > 0 ? `${ESC}${n}A` : '');
20
+ /**
21
+ * Cut a line to `max` visible columns, keeping its colour codes intact.
22
+ *
23
+ * The repaint moves up as many rows as it drew last time, counting one row per
24
+ * line. A line wider than the terminal wraps onto two, the cursor then lands
25
+ * one row short, and every frame leaves its top line behind — on a narrow
26
+ * PowerShell window a pull scrolled hundreds of copies of its header and cut
27
+ * the repo line off mid-word. Clipping is what keeps the count true. Every
28
+ * code point is taken as one column, which holds for everything this tool
29
+ * prints (the spinner, the dashes, Portuguese pipe names).
30
+ */
31
+ const ANSI = /\x1b\[[0-9;?]*[A-Za-z]/y;
32
+ export const clipToWidth = (line, max) => {
33
+ if (max <= 0)
34
+ return '';
35
+ if ([...line.replace(new RegExp(ANSI.source, 'g'), '')].length <= max)
36
+ return line;
37
+ let out = '';
38
+ let visible = 0;
39
+ for (let i = 0; i < line.length;) {
40
+ ANSI.lastIndex = i;
41
+ const m = ANSI.exec(line);
42
+ if (m) {
43
+ out += m[0];
44
+ i += m[0].length;
45
+ continue;
46
+ }
47
+ if (visible === max - 1)
48
+ break;
49
+ const ch = String.fromCodePoint(line.codePointAt(i));
50
+ out += ch;
51
+ visible++;
52
+ i += ch.length;
53
+ }
54
+ return `${out}…${ESC}0m`;
55
+ };
20
56
  /** Braille spinner: 8 frames, reads as motion even at 80ms. */
21
57
  const FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧'];
22
58
  const MARK = {
@@ -256,8 +292,13 @@ export class Progress {
256
292
  ...(hiddenBelow > 0 ? [dim(` … ${hiddenBelow} more`)] : []),
257
293
  ...this.notes.map((n) => ` ${n}`),
258
294
  ];
259
- // Repaint in place: up N lines, then clear-and-write each one.
260
- const out = cursorUp(this.linesDrawn) + body.map((l) => CLEAR_LINE + l).join('\n') + '\n';
295
+ // Repaint in place: up N lines, then clear-and-write each one. Each line is
296
+ // clipped to the terminal so it occupies exactly one row (see clipToWidth);
297
+ // the final frame is not, since nothing repaints over it and a warning
298
+ // should be readable in full.
299
+ const width = this.stream.columns;
300
+ const fit = (l) => (final || !width ? l : clipToWidth(l, width - 1));
301
+ const out = cursorUp(this.linesDrawn) + body.map((l) => CLEAR_LINE + fit(l)).join('\n') + '\n';
261
302
  this.stream.write(out);
262
303
  this.linesDrawn = final ? 0 : body.length;
263
304
  }
@@ -146,6 +146,7 @@ export const readTree = async (dir) => {
146
146
  readAt: meta.readAt ?? new Date().toISOString(),
147
147
  repoKind: meta.repoKind ?? 'pipe',
148
148
  coverage: meta.coverage ?? fullCoverage('unknown'),
149
+ ...(meta.columns ? { columns: meta.columns } : {}),
149
150
  repo_columns: meta.repo_columns ?? 'complete',
150
151
  notes: meta.notes ?? [],
151
152
  },
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@pipefy/pipefy-process-coder",
3
- "version": "0.1.2",
3
+ "version": "0.1.3",
4
4
  "description": "Pull a Pipefy pipe to disk as readable JSON, edit it with an agent, diff it, push it back.",
5
5
  "type": "module",
6
6
  "license": "Apache-2.0",