@datarails-ai/dr-cli 0.0.1-locked → 1.6.71

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (58) hide show
  1. package/README.md +608 -0
  2. package/dist/chunk-DWMCBGQH.js +242 -0
  3. package/dist/cli.js +64468 -0
  4. package/dist/kb-6UWI7Q32.js +21 -0
  5. package/dist/scopes-map.json +4412 -0
  6. package/dist/skills/connector-sandbox/SKILL.md +95 -0
  7. package/dist/skills/datarails-connect-workato/SKILL.md +140 -0
  8. package/dist/skills/datarails-connectors-onboard-connector/SKILL.md +813 -0
  9. package/dist/skills/dr-cli/SKILL.md +659 -0
  10. package/dist/skills/dr-cli/reference/admin.md +796 -0
  11. package/dist/skills/dr-cli/reference/budget-builder.md +151 -0
  12. package/dist/skills/dr-cli/reference/calc-vals.md +73 -0
  13. package/dist/skills/dr-cli/reference/calcmodels.md +76 -0
  14. package/dist/skills/dr-cli/reference/cash.md +61 -0
  15. package/dist/skills/dr-cli/reference/dashboards-widgets.md +85 -0
  16. package/dist/skills/dr-cli/reference/datamappers.md +150 -0
  17. package/dist/skills/dr-cli/reference/desk.md +472 -0
  18. package/dist/skills/dr-cli/reference/filebox-documents.md +373 -0
  19. package/dist/skills/dr-cli/reference/forecast.md +51 -0
  20. package/dist/skills/dr-cli/reference/integrations.md +57 -0
  21. package/dist/skills/dr-cli/reference/lookups.md +168 -0
  22. package/dist/skills/dr-cli/reference/metrics.md +512 -0
  23. package/dist/skills/dr-cli/reference/multi-product.md +180 -0
  24. package/dist/skills/dr-cli/reference/reconciliation-mec.md +76 -0
  25. package/dist/skills/dr-cli/reference/rh-config.md +84 -0
  26. package/dist/skills/dr-cli/reference/safety.md +143 -0
  27. package/dist/skills/dr-cli/reference/spend-control.md +206 -0
  28. package/dist/skills/dr-cli/reference/templates.md +163 -0
  29. package/dist/skills/dr-cli/reference/workflows.md +137 -0
  30. package/dist/skills/dr-error-enrichment/SKILL.md +80 -0
  31. package/dist/skills/dr-error-enrichment/reference/core.md +257 -0
  32. package/dist/skills/dr-error-enrichment/reference/groundcover.md +84 -0
  33. package/dist/skills/dr-error-enrichment/reference/jira-bugs.md +219 -0
  34. package/dist/skills/dr-error-enrichment/reference/safety-backup.md +122 -0
  35. package/dist/skills/dr-error-enrichment-author/SKILL.md +219 -0
  36. package/dist/skills/dr-error-enrichment-improve/SKILL.md +218 -0
  37. package/dist/skills/dr-error-enrichment-maintain/SKILL.md +237 -0
  38. package/dist/skills/dr-error-enrichment-reconcile/SKILL.md +167 -0
  39. package/dist/skills/dr-filebox-api-setup/SKILL.md +170 -0
  40. package/dist/skills/dr-switch-user/SKILL.md +395 -0
  41. package/dist/skills/dr-validate/SKILL.md +392 -0
  42. package/dist/skills/dr-validate/report-template.html +950 -0
  43. package/dist/skills/fx-conversion-matrix/SKILL.md +763 -0
  44. package/dist/skills/fx-conversion-matrix/scripts/_args.mjs +52 -0
  45. package/dist/skills/fx-conversion-matrix/scripts/_xlsx.mjs +241 -0
  46. package/dist/skills/fx-conversion-matrix/scripts/build-matrix.mjs +366 -0
  47. package/dist/skills/fx-conversion-matrix/scripts/gen-mappers.mjs +171 -0
  48. package/dist/skills/fx-conversion-matrix/scripts/rates-to-xlsx.mjs +172 -0
  49. package/dist/skills/fx-onboard/SKILL.md +325 -0
  50. package/dist/skills/mec-table-validation/SKILL.md +383 -0
  51. package/dist/skills/plugins.json +24 -0
  52. package/dist/skills/validate-workato-migration/SKILL.md +167 -0
  53. package/dist/skills/validate-workato-migration/reference/gotchas.md +16 -0
  54. package/dist/skills/validate-workato-migration/reference/staging-orgs.md +26 -0
  55. package/dist/skills/validate-workato-migration/scripts/compare_files.py +255 -0
  56. package/dist/zendeskKb-JPSHVOPH.js +243 -0
  57. package/package.json +56 -9
  58. package/scripts/postinstall.mjs +91 -0
@@ -0,0 +1,257 @@
1
+ # Core — what enrichment rules are, the command surface, how matching works
2
+
3
+ ## What a rule is
4
+
5
+ When an integration sync fails, Datarails matches the raw vendor error against a
6
+ table of **enrichment rules**. A rule maps a regex to a readable `description`
7
+ plus `resolution_steps`, and collects thumbs up/down feedback.
8
+
9
+ Rules live **per server** in each region's own database — there is no global
10
+ table and no cross-server sync, so the same rule can exist on `US` and be
11
+ missing (or wrong) on `UK`. Rule `id`s are per-server database ids: the same
12
+ logical rule has a different id on every server.
13
+
14
+ ## Permissions
15
+
16
+ The admin API is restricted to **Datarails admins** and members of the
17
+ **`Integrations Team`** permission group on the parent org. A `403`/`405` on any
18
+ subcommand means the logged-in identity lacks that — confirm with
19
+ `dr whoami --server <S>` and escalate rather than switching to another
20
+ identity's credentials.
21
+
22
+ ## Command surface
23
+
24
+ ```
25
+ dr error-enrichments list [--server S | --all-servers | --servers A,B] [--include-test]
26
+ [--integration-type T]... [--classification C]... [--approved true|false]
27
+ [--search Q] [--json]
28
+ dr error-enrichments stats (same server + filter flags) [--json]
29
+ dr error-enrichments get <ids...> --server S [--json] [--timeout SECS]
30
+ dr error-enrichments create --server S --integration-type T --rank N --pattern RE --description D
31
+ [--resolution-step S]... [--example E]... [--classification C] [--approved true|false]
32
+ dr error-enrichments create --server S --payload-file f.json # one object OR an array of rule objects
33
+ dr error-enrichments update [ids...] --server S (same field flags) # PATCH, same patch applied to every id
34
+ dr error-enrichments update --server S --payload-file patches.json # array of {id, ...fields}, no positional ids
35
+ dr error-enrichments delete <ids...> --server S
36
+ dr error-enrichments test-regex --server S --pattern RE (--sample S... | --samples-file f)
37
+ dr error-enrichments export [--server S | --all-servers | --servers A,B] [--include-test]
38
+ [--out PATH] # PATH is a DIRECTORY when sweeping
39
+ dr error-enrichments import --server S --file backup.csv # DESTRUCTIVE — see safety-backup.md
40
+ ```
41
+
42
+ An array `--payload-file` on `create` / `update` takes neither positional ids nor
43
+ field flags — array mode carries every field inside each entry, and the command
44
+ rejects the combination rather than guessing which one wins.
45
+ `create` / `update` / `delete` also take `--timeout <seconds>` for the
46
+ whole-command budget (see *Execution semantics*).
47
+
48
+ Exact flags come from `dr error-enrichments <sub> --help` — the binary is
49
+ authoritative. Reads accept `--all-servers` / `--servers`; **writes never do**
50
+ (one `--server` per mutation, by design). `export` is a read, so it sweeps too —
51
+ that is the whole backup invariant in one command, see
52
+ [safety-backup.md](safety-backup.md).
53
+
54
+ ### Bulk forms
55
+
56
+ `get`, `update`, and `delete` take a variadic id list; `create` takes an array
57
+ payload. Two ways to update many rules:
58
+
59
+ - **Shared patch** — `update 7 12 30 --approved true --server <S>` applies the
60
+ *same* field changes to every id. Use it for a uniform flip (approving a batch,
61
+ re-classifying a vendor's rules).
62
+ - **Per-rule patches** — `update --payload-file patches.json --server <S>` where
63
+ the file is an array of objects each carrying its own `id` plus the fields to
64
+ change. Use it for a reconcile run where every rule gets different text. Do not
65
+ pass positional ids in this mode.
66
+
67
+ ```json
68
+ [
69
+ {"id": 7, "approved": true},
70
+ {"id": 12, "description": "NetSuite rejected the login.", "rank": 3}
71
+ ]
72
+ ```
73
+
74
+ `create --payload-file` accepts one object (as before) or an array of rule
75
+ objects for a batch create.
76
+
77
+ Array entries are type-checked before anything is sent, and the message names the
78
+ offending position (`entry 3: expected a JSON object, got a string`). Fix the
79
+ entry the message names — the index is 1-based and matches file order.
80
+
81
+ **`get --json` changes shape with the id count.** One id returns a **bare
82
+ object**; two or more return an **array**. `get 7 --json | jq '.approved'` works,
83
+ `get 7 12 --json | jq '.approved'` returns `null` — use `jq '.[].approved'` (or
84
+ `.[] | select(.id == 7)`) for a batch. The array holds only the rules that were
85
+ readable: a failed id is **not** represented in the JSON at all, and if no id was
86
+ readable stdout stays **empty** rather than printing `[]`. Failures come back as
87
+ `warning: rule <id>: …` lines on **stderr** plus a non-zero exit. So when you
88
+ parse a batch `get`, check the exit code and compare the returned ids against the
89
+ ids you asked for — never infer "the rule is gone" from a short array, and never
90
+ read empty stdout as "no such rules".
91
+
92
+ **"Batch" means two different things in this file, and they draw the line in
93
+ different places.** For *execution semantics* (below) a batch starts at **two
94
+ targets**: a run with one target — one positional id, a single-object
95
+ `--payload-file`, **or a one-entry array `--payload-file`** — behaves exactly
96
+ like the pre-batch single-rule command. It prints the one-line success message
97
+ (`Updated error-enrichment rule 7.`), not a summary table, and a failure
98
+ propagates as the server's own error rather than a one-row `FAILED` table. So
99
+ never tell a user (or yourself) to "read the summary table afterwards" without
100
+ checking the target count first: on a one-entry `proposals.json` there is no
101
+ table, and the bare error *is* the report. For *confirm-token binding* (further
102
+ below) the line is elsewhere: an array `--payload-file` of **any** length,
103
+ including one entry, binds as a batch.
104
+
105
+ **Execution semantics (two or more targets).** Batch requests are sent sequentially, and the run gets a
106
+ timeout scaled to the number of targets (the flat single-call budget plus a
107
+ per-target allowance, capped) instead of the single-call default, so a long batch
108
+ does not abort just for being long; `--timeout <seconds>` on
109
+ `get`/`create`/`update`/`delete` overrides it, and one target keeps exactly the
110
+ budget it had before batches existed. A per-item failure is recorded and the run
111
+ **continues**; afterwards a summary table names what succeeded (with the new ids
112
+ for `create`) and what failed, and the command exits non-zero. So a partial batch
113
+ is normal and visible — read the summary before retrying, and retry only the ids
114
+ that failed.
115
+
116
+ If the run *times out*, the summary marks the untouched targets
117
+ `not attempted — run timed out` and the thrown error says so explicitly. Treat it
118
+ as neither "done" nor "nothing happened": a request already in flight may have
119
+ landed. Re-read the affected rules with `get` before re-running, and raise
120
+ `--timeout` rather than re-issuing the same batch blind.
121
+
122
+ **Confirm-token binding — read this before reusing a token.** What the token
123
+ binds depends on how many targets the run has, and the two halves behave
124
+ differently:
125
+
126
+ - **Single target** — one positional id with flag-built fields, or a
127
+ `--payload-file` holding a single *object*. The token binds
128
+ `(resource, id, verb)` and **nothing else**: `error-enrichment` + the rule id
129
+ (for `create`, the `integration_type:rank` pair) + the verb. **Field values are
130
+ not bound.** Hand-edit a `--description`, a `--pattern`, `--approved`, or the
131
+ contents of that object file and re-run with the same live token, and it still
132
+ redeems — what executes is then no longer what the plan the user approved
133
+ showed. If anything but the target changes after the plan, throw the token
134
+ away, re-run the plan step, and relay the new plan.
135
+ - **Batch** — two or more targets, **or an array `--payload-file` of any length,
136
+ including one entry**. The token binds `error-enrichment-batch` + a sha256
137
+ prefix over a canonical form of the whole change: the targets **and**, for
138
+ `create` / `update`, the field values being written. (`delete` has no field
139
+ values, so its hash covers the sorted, deduped ids alone.) Editing the payload
140
+ file or the id list between plan and confirm invalidates the token — the re-run
141
+ fails the token check and prints a fresh plan instead of writing. Reordering
142
+ ids, or a formatter reordering JSON keys, does *not* invalidate it: the
143
+ canonical form is sorted at both levels.
144
+
145
+ One confirm covers the whole set either way — the batch hash pins the payload, it
146
+ does not add a second gate.
147
+
148
+ ## PATCH-omit semantics
149
+
150
+ `update` is a PATCH — it changes only the fields you pass. Two consequences:
151
+ omitting a field leaves it alone (so you can't accidentally blank a description),
152
+ and the repeatable list flags can't *empty* a list, because "no `--resolution-step`
153
+ given" and "an empty list" look identical on the command line. To clear
154
+ `resolution_steps` or `examples`, pass the empty list explicitly:
155
+
156
+ ```bash
157
+ echo '{"resolution_steps": []}' > /tmp/clear.json
158
+ dr error-enrichments update 7 --payload-file /tmp/clear.json --server <S>
159
+ ```
160
+
161
+ In array mode the same rule applies per entry:
162
+ `[{"id": 7, "resolution_steps": []}]`.
163
+
164
+ ## How matching actually works (get this wrong and rules silently do nothing)
165
+
166
+ Verified against `DataIntegrationErrorDescription.get_err_description`:
167
+
168
+ 1. Rules are scoped **per `integration_type`** and scanned in **`rank` order**
169
+ (rank 1 first). A rule only ever sees errors from its own vendor.
170
+ 2. Matching uses Python **`re.match`**, which anchors at the **start** of the
171
+ string. `INVALID_LOGIN` does *not* match `Error: INVALID_LOGIN` — write
172
+ `.*INVALID_LOGIN.*`. This is the single most common authoring mistake.
173
+ 3. Newlines are stripped from the error first (`"".join(err.splitlines())`), so
174
+ a multi-line error is matched as one long line. Don't write patterns
175
+ expecting `\n`.
176
+ 4. **A draft (unapproved) rule steals the hit counter but shows nothing.** The
177
+ highest-ranked matching rule gets the hit whether or not it is approved;
178
+ scanning then continues for an *approved* match. If only drafts matched, the
179
+ user sees **no explanation at all** while the hit count climbs. High hits +
180
+ `approved: false` is therefore an active user-facing gap, not a curiosity —
181
+ it is what `stats`' `needs_attention` list surfaces.
182
+ 5. `test-regex` runs the same `re.match` + newline-stripping server-side. Trust
183
+ it over local regex reasoning, and never hand-verify a pattern you could test.
184
+
185
+ ## Raw error sources on a tenant
186
+
187
+ Three different `dr integrations` surfaces, and they are not interchangeable:
188
+
189
+ | Command | Gives you |
190
+ |---|---|
191
+ | `integrations get <id> --server S` | the integration-level raw `error_description` |
192
+ | `integrations query-status --query-id <qid>... --json --server S` | per-query `last_sync_error` — takes query ids, not an integration id (at least one `--query-id` is required), and **use `--json`**, the table truncates the error to 60 chars |
193
+ | `integrations errors <id> --server S` | the *enriched* message the user currently sees (matched rule's description + steps) |
194
+
195
+ `integrations status <id>` shows only status and last-run time — no error text;
196
+ don't reach for it here.
197
+
198
+ `integrations errors` printing *"no canned error explanation available"* is a
199
+ direct signal that **no approved rule matched** for that integration — the
200
+ single fastest per-tenant confirmation that a rule is missing.
201
+
202
+ ## Guard rails
203
+
204
+ - **Reads are free; writes need the user.** Never run a mutation the user hasn't
205
+ approved in this conversation, and never on a prod server without walking the
206
+ plan output through them first.
207
+ - **`import` is not an edit tool.** It replaces the whole table and zeroes every
208
+ counter. Fleet changes go through `update` / `create`; see
209
+ [safety-backup.md](safety-backup.md) for the policy.
210
+ - **One server per mutation.** A batch is many rules on **one** server; there is
211
+ still no bulk write across servers — a fleet change is N confirmed runs, each
212
+ visible to the user.
213
+ - **One confirm covers the whole batch.** The plan output is your last chance to
214
+ check the id list — read every id in it before handing back the token, because
215
+ nothing will stop at item three to ask again. A batch `delete` plan prints one
216
+ compact line per rule — `rule <id> (<type> r<rank>) hits N up N down N — <description>`
217
+ — so the whole blast radius, counters included, fits above the single Tier-A
218
+ challenge word instead of pushing it off the dialog.
219
+ - **A token is not a licence to change the payload.** On a single-target write the
220
+ token binds the target only, so an edited field flag still redeems — see the
221
+ binding rules above and re-run the plan whenever the values move.
222
+ - **A batch is not all-or-nothing.** It continues past a failed item and exits
223
+ non-zero; the summary table is the record of what actually landed. Never report
224
+ "done" from the exit code alone, and never re-run the whole batch blindly after
225
+ a partial failure.
226
+ - **Ids are per-server.** An id list copied from another server's output points
227
+ at unrelated rules. Re-derive ids from a `list`/`get` against the server you
228
+ are about to write to.
229
+ - **Never match rules across servers by `id`.** Key on
230
+ `(integration_type, re_pattern)`.
231
+ - **Don't invent flags or subcommands.** Run `--help`; if it isn't there, it
232
+ doesn't exist.
233
+ - **`--json` for anything you parse**, never the human tables.
234
+ - **User-facing text is the product.** `description` and `resolution_steps` are
235
+ read by customers — get wording approved, and keep internal identifiers,
236
+ stack traces, and vendor jargon out of them.
237
+ - **Credentials stay in the keychain.** Never echo JWTs or session ids.
238
+
239
+ ## Common mistakes
240
+
241
+ | Mistake | Fix |
242
+ |---|---|
243
+ | Pattern without a leading `.*` | `re.match` anchors at the start — `.*NEEDLE.*` |
244
+ | Assuming a draft rule is harmless | It steals the hit and shows the user nothing; approve it or lower its rank |
245
+ | A general rule ranked above a specific one | The general one matches first and hides the better message — rank specific first |
246
+ | Reverting with `import` | Wipes all rules and zeroes counters; revert per-rule with `create`/`update`/`delete` |
247
+ | Matching rules across servers by `id` | Key on `(integration_type, re_pattern)` |
248
+ | Reusing an id list from another server's output | Ids are per-server database ids — re-derive them against the target server |
249
+ | Treating a non-zero batch exit as "nothing happened" | The run continues past failures; read the summary table for what landed |
250
+ | Waiting for a summary table on a **one-entry** array payload | One target prints the single-rule success line and propagates its error verbatim — the table starts at two targets, even though the token binds as a batch |
251
+ | Skimming the plan on a batch write | One confirm covers every id in it — verify the list, not the count |
252
+ | Passing positional ids **or field flags** alongside an array `--payload-file` | Array mode carries every field inside each entry; pick one mode |
253
+ | Parsing a batch `get --json` as an array of the ids you asked for | One id is a bare object, and failed ids are absent — read stderr + the exit code |
254
+ | Reusing a live token after editing a field flag on a **single**-target write | Single targets bind `(resource, id, verb)` only; re-run the plan |
255
+ | Expecting `--resolution-step` with no value to clear the steps | PATCH-omit semantics — clear with `--payload-file {"resolution_steps": []}` |
256
+ | Writing a rule on one server and stopping | Reconcile the fleet — per-server drift is the problem this tooling exists to fix |
257
+ | Skipping the backup because "it's one small change" | The backup is also the diff baseline; it costs one command per server |
@@ -0,0 +1,84 @@
1
+ # Groundcover — mining errors that nothing matches
2
+
3
+ The monolith logs every unmatched resolve with the marker
4
+ `[ErrorEnrichment] No matching rule`. Read `get_gcql_reference` before composing
5
+ queries — gcQL is not LogQL and not shell.
6
+
7
+ ## Find the lines
8
+
9
+ Use a bare free-text keyword (it hits the full-text index;
10
+ `content:*ErrorEnrichment*` forces a full scan and will time out on a busy env):
11
+
12
+ ```
13
+ ErrorEnrichment | fields _time, content | limit 20
14
+ ErrorEnrichment | stats count()
15
+ ```
16
+
17
+ ## The marker's payload is text, not fields
18
+
19
+ The line carries `vendor_name`, `datasource_id`, `query_id`, `err_len`, and
20
+ `err_sha256_prefix`, but they are printf-formatted **inside the log message** —
21
+ they are not indexed log attributes. `stats by (vendor_name)` will not error, it
22
+ will silently return one empty group. Extract them first:
23
+
24
+ ```
25
+ ErrorEnrichment | extract_regexp "vendor_name=(?P<vendor>[^ ]+)" from content
26
+ | stats by (vendor) count() as unmatched | sort by (unmatched desc) | limit 20
27
+
28
+ ErrorEnrichment | extract_regexp "err_sha256_prefix=(?P<sig>[0-9a-f]+)" from content
29
+ | stats by (sig) count() as hits | sort by (hits desc) | limit 50
30
+ ```
31
+
32
+ Group by `vendor` to see which connector has the biggest unmatched volume, and by
33
+ `sig` to count *distinct* unmatched errors and rank the repeat offenders —
34
+ identical prefixes are the same normalised error, so the top `sig` rows are your
35
+ highest-value new rules. Widen the window with `_time:7d` only after a tight
36
+ window proves the predicate returns rows.
37
+
38
+ ## The hash is deliberate
39
+
40
+ `err_sha256_prefix` + `err_len` exist precisely so the raw error text stays out of
41
+ the log — vendor errors can embed customer paths and tokens. Don't expect to
42
+ recover the message from the marker.
43
+
44
+ Get the **raw text** from a different source: the vendor sync error logs around
45
+ the same timestamp / `datasource_id`, or the tenant itself via the surfaces in
46
+ [core.md](core.md) (`integrations get <id>`, or
47
+ `integrations query-status --query-id <qid> --json` for the `query_id` the marker
48
+ names — that subcommand keys off query ids, not the integration id).
49
+
50
+ Treat raw error text as customer data — use it to build the rule, keep it out of
51
+ chat summaries beyond what the rule needs, and prefer sanitised text in
52
+ `examples`.
53
+
54
+ ## Environment mapping
55
+
56
+ Groundcover envs are not dr-cli server keys. Envs are `prod-us`, `prod-east-us`,
57
+ `prod-west-us-2`, `prod-demo`, `prod-uk`, `prod-uk-south`,
58
+ `prod-canada-central`.
59
+
60
+ | dr-cli server | Groundcover env |
61
+ |---|---|
62
+ | `UK` | `prod-uk*` |
63
+ | `CA` | `prod-canada-central` |
64
+ | `DEMO` | `prod-demo` |
65
+ | `US` / `US2` | **ambiguous** across `prod-us`, `prod-east-us`, `prod-west-us-2` |
66
+ | `EU` | **unmapped** — resolve it before mining (see below) |
67
+
68
+ `EU` was added to the registry after this table was written and its
69
+ groundcover env has not been confirmed. Resolve it the same way as the US
70
+ ambiguity before mining, and record the answer here once known.
71
+
72
+ Resolve the US ambiguity with
73
+ `ErrorEnrichment | stats by (env) count() | limit 100`, or ask. Never mine one
74
+ region's logs to justify a rule you then only write to another.
75
+
76
+ ## Prove the pipeline before concluding zero
77
+
78
+ An empty gcQL result is indistinguishable from a wrong env, too-narrow window, or
79
+ misspelled field — none of those error, they all return nothing. Run a query you
80
+ expect to return rows (`ErrorEnrichment | stats count()`) before reporting zero
81
+ unmatched errors.
82
+
83
+ Errors that already match a rule can be mined the same way, for `examples` and
84
+ for validating a pattern change against real traffic.
@@ -0,0 +1,219 @@
1
+ # Known bugs — is this error even the customer's problem?
2
+
3
+ An enrichment rule is a message *to a customer* about something they can act on.
4
+ Point one at a defect of ours and it does two kinds of damage: the customer is
5
+ told to go fix something they cannot fix, and the defect stops looking like a
6
+ defect — once an error reads as "explained", nobody escalates it again. A rule
7
+ over our own bug is worse than no rule.
8
+
9
+ So every candidate the improve pass ranks gets one question before it becomes a
10
+ proposal: **is this error ours?** Jira is the check.
11
+
12
+ ## The field
13
+
14
+ DR bugs carry a single-select **`System area`** — note the lowercase `a` —
15
+ `customfield_10717`. Integration work is the option **`Integration`**, id
16
+ `20748`, **singular**. There is no `Integrations` option, and a non-existent
17
+ option value in JQL **returns zero rows instead of erroring** — so
18
+ `"System Area" = "Integrations"` is indistinguishable from "no known bugs".
19
+ Same discipline as gcQL: prove a query returns rows before believing a clean
20
+ result.
21
+
22
+ Three near-identically named dead fields exist — `System Area old`
23
+ (`customfield_10654`), `System Area till April 22` (`customfield_10671`),
24
+ `System Area for SF (Child Area)` (`customfield_10765`). Write `cf[10717]`; it is
25
+ the one unambiguous form.
26
+
27
+ Site: `datarailsteam.atlassian.net`, cloudId
28
+ `a84c2098-d520-4c84-a5cf-a576c83ede6e`.
29
+
30
+ ## The sweeps
31
+
32
+ **Open integration bugs** — the exclusion list (~20 rows at any time):
33
+
34
+ ```jql
35
+ project = DR AND issuetype = Bug AND cf[10717] = "Integration"
36
+ AND statusCategory != Done ORDER BY created DESC
37
+ ```
38
+
39
+ **Recently fixed ones** — just as disqualifying, and easier to miss:
40
+
41
+ ```jql
42
+ project = DR AND issuetype = Bug AND cf[10717] = "Integration"
43
+ AND resolved >= -60d ORDER BY resolved DESC
44
+ ```
45
+
46
+ A signature whose volume stops the week a fix shipped needs no rule at all.
47
+ Compare the bug's `resolved` date against the last `_time` of the signature
48
+ before writing anything.
49
+
50
+ **Adjacent areas.** Sync failures also get filed under `Fileboxes` (`22505`) and
51
+ `Data Mapper` (`20746`). Widen when the vendor or symptom points that way:
52
+ `cf[10717] IN ("Integration", "Fileboxes", "Data Mapper")`.
53
+
54
+ **Free-text pass — do not skip it.** `System area` is *optional* on the Bug
55
+ create screen, so filtering on it under-counts by design: dozens of DR bugs a
56
+ year are filed with the wrong area or none at all (20 with no area at all in the
57
+ last 180 days). Before concluding a signature is unticketed, search the text with
58
+ no area filter:
59
+
60
+ ```jql
61
+ project = DR AND issuetype = Bug AND text ~ "connection was forcibly closed"
62
+ ORDER BY created DESC
63
+ ```
64
+
65
+ `text ~` spans summary, description, comments **and** the `Bug Description`
66
+ custom field, which is where CS-raised bugs actually keep the error body
67
+ (`description` is usually `null` on them — see below). Strip ids, paths, numbers
68
+ and quotes out of the phrase first; a phrase carrying a tenant-specific id
69
+ matches nothing and reads as "no bug".
70
+
71
+ Fetch light on all of these:
72
+ `fields: ["key","summary","status","priority","created","resolutiondate"]`. A
73
+ `*all` search over 20 DR bugs is mostly Salesforce account metadata and will
74
+ blow the response limit.
75
+
76
+ ## Matching a signature to a bug
77
+
78
+ Bugs record no error regexes, so this is judgement, not a join. Four signals:
79
+
80
+ - **Vendor** — the connector named in the bug against the marker's
81
+ `vendor_name`.
82
+ - **Symptom** — the bug's own error text. On CS-raised bugs it lives in
83
+ **`Bug Description` (`customfield_10720`)**, not `description`, which is
84
+ `null`; read both.
85
+ - **Time** — first appearance of the `err_sha256_prefix` against the bug's
86
+ `created`. A signature that starts the week a bug was filed is very likely that
87
+ bug.
88
+ - **Tenant** — the marker's `datasource_id` → org, against the bug's
89
+ `Customer Name` / `Business details`. A signature confined to one tenant that
90
+ already has a bug open is that bug; one spread across tenants usually is not.
91
+
92
+ State which signal you matched on. "Probably DR-51612" with no reason is not a
93
+ triage result.
94
+
95
+ ## The four verdicts
96
+
97
+ | Verdict | Looks like | Do |
98
+ |---|---|---|
99
+ | **Customer- or vendor-side** | expired or revoked credentials, permission denied, deleted object, vendor rate limit, vendor outage, an ODBC/DSN condition on their machine | Propose a rule — the normal improve flow |
100
+ | **Our defect, ticket exists** | matches an open or recently-fixed integration bug | **No rule.** Report the signature with the bug key and its volume — the volume is priority evidence the bug does not have yet |
101
+ | **Our defect, no ticket** | our code raised it — see below | **No rule.** Flag it and offer to open a bug |
102
+ | **Unclear whose fault** | plausible either way and no bug matches | **No rule.** Say what is ambiguous and what would settle it |
103
+
104
+ **Whose code raised it is the whole distinction**, and error *shape* does not
105
+ settle it. A Python traceback, an exception class, `NoneType` / `KeyError` /
106
+ `IndexError`, a raw un-enriched exception body, a 5xx, a timeout — every one of
107
+ these reaches us from vendors and from customer-side components too: a vendor API
108
+ returning its own stack trace in the response body, an ODBC driver or the Database
109
+ Sync Agent failing on the customer's own machine, a vendor SDK exception
110
+ surfacing through our process. Those shapes are **clues that raise the question**,
111
+ not the answer to it.
112
+
113
+ Answer it from where the error was raised, not from what it looks like:
114
+
115
+ - **The log source.** Which service and which module emitted the line, and is the
116
+ exception raised in our frame or caught from a vendor call?
117
+ - **The host.** Our infrastructure, the vendor's, or the customer's machine — the
118
+ Sync Agent and ODBC/DSN conditions are the customer's.
119
+ - **The endpoint.** A 500 from a vendor endpoint is *their* outage and a fine
120
+ thing to enrich; a 500 from a Datarails endpoint is ours.
121
+ - **Internal ids or our own identifiers in customer-visible text** — that text was
122
+ composed by us, so the leak is ours regardless of what triggered it.
123
+
124
+ If those signals don't converge, the verdict is **Unclear**, not "ours" — and not
125
+ a rule either. Guessing "ours" suppresses a valid vendor-side rule; guessing
126
+ "theirs" buries one of our defects.
127
+
128
+ Two consequences worth saying out loud to the user: the unmatched-error pass is a
129
+ **defect detector** at least as much as a rule generator, and a run that
130
+ surfaces three bugs and no rules is a successful run, not an empty one.
131
+
132
+ ## Offering to open a bug
133
+
134
+ Filing a Jira issue is outward-facing work. **Ask first, show the exact summary
135
+ and body you intend to file, and wait for a yes.** Never file one per signature
136
+ in a loop — related signatures are usually one bug, and one bug with four
137
+ signatures listed is more useful to whoever picks it up than four thin tickets.
138
+
139
+ `createJiraIssue` with `projectKey: "DR"`, `issueTypeName: "Bug"`, and
140
+ `additional_fields`:
141
+
142
+ | Field | Value |
143
+ |---|---|
144
+ | `customfield_10717` | `{"id": "20748"}` — **System area = Integration.** Optional on the create screen, which is why it gets forgotten: a bug filed without it is invisible to every sweep above, including the next improve run's own sweep |
145
+ | `customfield_10683` | Environment DR — `{"id": "14663"}` Production (Development `14662`, Testing `14664`) |
146
+ | `customfield_10682` | Severity — Minor `14658`, Medium `14659`, Major `14660`, Critical `14661` |
147
+ | `priority` | `{"name": "P2"}` — **ask, do not pick.** Volume is evidence for a priority, not a priority |
148
+ | `customfield_10720` | `Bug Description` — the body every DR bug is actually read from. Put the detail here, not only in `description` |
149
+
150
+ Only `project`, `issuetype` and `summary` are enforced; everything else silently
151
+ defaults to empty.
152
+
153
+ Body content: vendor, groundcover env and window, the `err_sha256_prefix`, the
154
+ unmatched volume, how many distinct `datasource_id`s are affected, first and last
155
+ occurrence, and redacted error text.
156
+
157
+ ### Redaction is a step, not an adjective
158
+
159
+ Nothing in the CLI sanitises this for you — you are pasting text into a ticket by
160
+ hand, so the redaction happens in the draft you show the user, before
161
+ `createJiraIssue` is called at all. Jira is a **wider** audience than this
162
+ conversation, not a narrower one: the ticket is readable by the whole company and
163
+ outlives the run, and the hash-only log marker
164
+ ([groundcover.md](groundcover.md)) exists precisely so this text does not spread.
165
+
166
+ Walk the error text and remove every one of these before it goes in the body or
167
+ `Bug Description`. If you cannot tell what a value is, it is customer data:
168
+
169
+ | Remove | Replace with |
170
+ |---|---|
171
+ | Tokens, keys, passwords, connection strings, `Authorization` / cookie values, SAS URLs, anything after `pwd=` / `password=` / `apikey=` | `<redacted>` |
172
+ | Usernames, email addresses, personal names | `<user>` |
173
+ | Host names, IPs, DSNs, database and schema names, file paths, share and bucket names | `<host>` / `<path>` |
174
+ | Customer identifiers — org and tenant names, account codes, GL account names and numbers, vendor object ids, invoice or entity numbers | `<id>` |
175
+ | Row-level financial data of any kind — amounts, balances, dates tied to a customer's records | drop the fragment entirely |
176
+
177
+ What survives is the part that is about *our code*: the exception class, the
178
+ message template, the module and line, the vendor's own error code and its
179
+ generic text. That is what the fix needs; the rest only widens exposure. Keep the
180
+ `err_sha256_prefix` in the ticket — it is the stable handle that ties the ticket
181
+ back to the signature without carrying any of this.
182
+
183
+ Two identifiers that are deliberately fine to include, because they are ours and
184
+ not customer content: the `err_sha256_prefix` and the `datasource_id`. Prefer a
185
+ count of affected `datasource_id`s to a list of tenant names.
186
+
187
+ `@mentions` need ADF. A markdown `@Name` renders as plain text and notifies
188
+ nobody — pass `contentFormat: "adf"` with a real
189
+ `{"type":"mention","attrs":{"id":"<accountId>"}}` node, or mention people in a
190
+ follow-up comment.
191
+
192
+ Then report the new key in the improve run's output, so the next run's sweep
193
+ excludes the signature instead of re-proposing it.
194
+
195
+ ## No Jira access is not a clean sweep
196
+
197
+ If the Atlassian tools are unavailable in the session, say so and mark the
198
+ proposals **un-triaged**. Silently skipping the sweep and presenting a ranked
199
+ proposal list is the failure this file exists to prevent — the list looks
200
+ identical either way.
201
+
202
+ ## Pitfalls
203
+
204
+ | Mistake | Fix |
205
+ |---|---|
206
+ | `cf[10717] = "Integrations"` | The option is singular `Integration`; a bad value returns **zero rows, no error** |
207
+ | `"System Area" = …` | Three stale look-alike fields exist — use `cf[10717]` |
208
+ | Only sweeping open bugs | A fix that shipped last week disqualifies the rule too — sweep `resolved >= -60d` |
209
+ | Trusting the `System area` filter alone | It is optional on create and often wrong or empty — add a `text ~` pass with no area filter |
210
+ | Searching a phrase that carries ids or paths | Strips to nothing useful — search the stable words only |
211
+ | Reading `description` for the error text | CS-raised bugs leave it `null`; the body is `Bug Description` (`customfield_10720`) |
212
+ | `fields: ["*all"]` on a bug sweep | Mostly Salesforce metadata, and it exceeds the response limit |
213
+ | Calling it ours because it *looks* like a traceback, or theirs because it looks like a vendor message | Shape is a clue, not the verdict — settle it on the log source, the host and whose endpoint answered, and say **unclear** when they don't converge |
214
+ | Pasting the error body into the ticket as-is | Redact first, in the draft you show the user — nothing in the CLI does it for you, and a Jira ticket is a wider audience than this conversation |
215
+ | Writing a rule "so the customer sees something" while the bug is open | It buries the defect and blames the customer — report the signature against the bug instead |
216
+ | Filing a bug without `System area` | The next sweep cannot see it; set `customfield_10717` |
217
+ | Filing one bug per signature | Group related signatures into one, and only with the user's yes |
218
+ | Pasting the raw error body into the ticket | Sanitise it — Jira is a wider audience than this chat |
219
+ | Reporting "no unmatched errors worth a rule" after an un-run sweep | Say the sweep did not run; un-triaged is a different answer from clean |