@dooer/dooer-test-env 1.0.0

package/lib/engine/GUIDE.md

@@ -0,0 +1,176 @@
+# Seed a test account — how to run it
+
+Copies all of one organization's data into another (a test entity). See **PROCESS.md** for how it
+works; this is the run cheatsheet.
+
+## Prerequisites
+
+- **VPN** connected (the script talks to the `dooer-database` ClusterIP directly).
+- **kubectl** access to the namespaces you use (`dooer-staging`, and `dooer-production` for real runs).
+- **Node 18+** and a one-time `npm install` in this folder (installs `pg` + `pg-copy-streams`).
+
+```bash
+cd _wip/_jimmy/seed-test-account
+npm install
+```
+
+Run it via **`./seed-test-account.sh`** (not `node …` directly): the wrapper raises Node's heap so a
+large org's uuid set fits in memory. Override the heap with `SEED_MAX_HEAP_MB` (default 8192).
+
+Check VPN reachability first:
+```bash
+IP=$(kubectl get svc dooer-database -n dooer-staging -o jsonpath='{.spec.clusterIP}')
+nc -z -w4 "$IP" 5432 && echo reachable
+```
+
+## The golden rule: dry-run first
+
+The tool is **dry-run by default** ("dummy" mode): it prints exactly what it would do and writes
+nothing. Add `--execute` only once the plan looks right.
+
+```bash
+# 1) DRY-RUN — see the plan, no writes
+./seed-test-account.sh --source <SOURCE_ORG> --target <TARGET_ORG> --source-namespace dooer-staging
+
+# 2) EXECUTE — actually copy (same namespace)
+./seed-test-account.sh --source <SOURCE_ORG> --target <TARGET_ORG> --source-namespace dooer-staging --execute
+```
+
+## Options
+
+| flag | meaning |
+|---|---|
+| `--source <uuid>` | org to copy FROM (required) |
+| `--target <uuid>` | existing org to copy INTO |
+| `--source-namespace <ns>` | default `dooer-staging` |
+| `--target-namespace <ns>` | default = source; set differently to copy across clusters |
+| `--execute` | actually write (omit = dry-run) |
+| `--skip-files` | skip the S3 file copy (DB rows only). Default: files ARE copied. |
+| `--skip-users` | skip copying referenced users missing from the target. Default: they ARE copied (customer-role emails anonymized). Same-namespace copies are a no-op regardless. |
+| `--confirm-production` | REQUIRED to write when `--target-namespace dooer-production` |
+| `--email <addr>` | scrub address (default `testcustomer@dooer.com`) |
+| `--if-target-nonempty refuse\|insert` | if a copy table already has target rows (default `refuse`) |
+| `--map <path>` | schema map (default `./schema-map.json`) |
+
+## Common recipes
+
+**Create a NEW target org and copy into it** (no pre-created target needed). The org is cloned from the
+source's `companies` row (fresh id, your name, a unique short_name) and the owner is linked with role
+`Owner`. The owner **user must already exist** in the target namespace — the tool never creates users.
+It prints the new org id.
+```bash
+./seed-test-account.sh \
+  --source <SOURCE_ORG> --name "My Test Copy" --owner-user <EXISTING_USER_UUID> \
+  --source-namespace dooer-staging --execute
+```
+
+**Copy a production org into a brand-new STAGING test org** (safest — never writes to prod; the owner
+must be a user that exists on staging):
+```bash
+./seed-test-account.sh \
+  --source <PROD_SOURCE_ORG> --source-namespace dooer-production \
+  --name "Large (copy of prod)" --owner-user <STAGING_USER_UUID> --target-namespace dooer-staging \
+  --execute
+```
+
+**Copy production data into an existing staging test org:**
+```bash
+./seed-test-account.sh \
+  --source <PROD_SOURCE_ORG> --source-namespace dooer-production \
+  --target <STAGING_TARGET_ORG> --target-namespace dooer-staging --execute
+```
+
+**Populate the production test entities (prod → prod, writes to production):**
+```bash
+# ALWAYS dry-run first:
+./seed-test-account.sh --source <PROD_SOURCE> --target <PROD_TEST_ENTITY> --source-namespace dooer-production
+# then, deliberately:
+./seed-test-account.sh --source <PROD_SOURCE> --target <PROD_TEST_ENTITY> \
+  --source-namespace dooer-production --execute --confirm-production
+```
+
+### The 2026-08 Dennis task (Large / Medium / Small)
+
+Source = the real companies; target = the empty test entities Dennis created. **Confirm the direction
+with a dry-run before executing** (the counts should show the source populated, target ~empty).
+
+| size | `--source` (real) | `--target` (test entity) |
+|---|---|---|
+| Large  | `14015598-3a63-4d10-9bf6-588ec365ec2e` | `4131681b-1099-49f8-930f-cb057310bb35` |
+| Medium | `f8284a0f-82d0-41eb-aaa5-29b78db89b06` | `dcd126fb-0516-4ea4-a461-78a676d38524` |
+| Small  | `c67577e9-2ea1-4fe5-bf27-867e28d67274` | `8205b113-a039-441c-9ddd-a137d73717d4` |
+
+```bash
+# dry-run each, e.g. Small:
+./seed-test-account.sh --source c67577e9-2ea1-4fe5-bf27-867e28d67274 \
+  --target 8205b113-a039-441c-9ddd-a137d73717d4 --source-namespace dooer-production
+# execute (prod write) when the plan is right:
+./seed-test-account.sh --source c67577e9-2ea1-4fe5-bf27-867e28d67274 \
+  --target 8205b113-a039-441c-9ddd-a137d73717d4 --source-namespace dooer-production \
+  --execute --confirm-production
+```
+
+## Running over a flaky link (VPN) — resilience & resuming
+
+The tool assumes a **stable, low-latency path to both databases**. A big copy holds connections open for a
+long time; over a VPN that drops, expect trouble. Built-in resilience + how to survive the rest:
+
+- **Reads auto-retry & reconnect.** Pass-1, the user discovery, the strategy/guard reads and the S3 file-id
+  read all reconnect a dropped connection and retry (exponential backoff). A flaky link no longer kills a read.
+- **The write phase is still ONE transaction by default** — a mid-copy drop rolls the whole thing back cleanly
+  (no partial org). That's intentional, but it means a very large org may never finish in one unbroken window.
+- **The remap is deterministic.** The id remap salt is derived from `(source, target)`, so **re-running the
+  exact same copy produces identical target ids**. That's what makes resuming/chunking safe.
+- **Chunk a big org with `--only` / `--skip-tables`.** Pass-1 still scans every table (so cross-references
+  remap), but only the named tables are written. Re-run with the **same `--target`** and
+  `--if-target-nonempty insert` to copy the next chunk; ids line up because the salt is stable.
+
+  ```bash
+  # chunk A (creates the org — note the id it prints), then chunk B into the SAME --target:
+  ./seed-test-account.sh --source <S> --name "Big copy" --owner-user <U> \
+    --source-namespace dooer-production --target-namespace dooer-staging \
+    --only service_ledger.voucherJournal,service_ledger.voucherJournalLine --skip-files --execute
+  ./seed-test-account.sh --source <S> --target <NEW_ORG_ID> \
+    --source-namespace dooer-production --target-namespace dooer-staging \
+    --skip-tables service_ledger.voucherJournal,service_ledger.voucherJournalLine \
+    --if-target-nonempty insert --execute
+  ```
+
+  **Keep FK-linked integer-key tables in the SAME chunk.** Almost every id is a deterministic uuid, but a few
+  unique **integer** keys are assigned per run. Only one is cross-referenced: `service_core_objects.transactions`
+  (its `transactions_pk` twin is referenced by `documents2transactions`, `payments2transactions`,
+  `reports2transactions`, `transactions2transactions`, `transactions_fetch_logs`). Copy `transactions` **together
+  with those five** in one invocation (don't split them across chunks). Everything else is safe to split freely.
+
+## Verify a copy
+
+After executing, spot-check that counts match and references are intact:
+```sql
+-- source vs target should match (read via the -1 replica pod is fine):
+SELECT (SELECT count(*) FROM service_ledger."voucherJournal" WHERE "organizationId"='<SOURCE>'),
+       (SELECT count(*) FROM service_ledger."voucherJournal" WHERE "organizationId"='<TARGET>');
+-- emails scrubbed:
+SELECT count(*) FROM service_employees.employee WHERE "organizationId"='<TARGET>'
+  AND email <> 'testcustomer@dooer.com';   -- expect 0
+```
+
+## Notes & gotchas
+
+- **Reads go to the primary via the ClusterIP;** the `dooer-database-1` pod is a **read-only replica**,
+  so don't try to write through `kubectl exec` into it — the script connects to the write endpoint.
+- **A big org takes a few minutes**, dominated by pass 1 (reading source ids + building the uuid set).
+  Writes use `COPY` and are fast. Watch `[pass1 n/N]` then `[copy n/N]` on stderr; redirect with
+  `2>&1 | tee /tmp/run.out` rather than piping through `tail` (which buffers).
+- **Memory:** run via `./seed-test-account.sh` (raised heap). A very large org can still need more — bump
+  `SEED_MAX_HEAP_MB` (e.g. `SEED_MAX_HEAP_MB=16384 ./seed-test-account.sh …`).
+- **Files (S3):** copied by default *after* the DB commit — reads the source env's Ceph RGW and writes
+  the target env's (needs kubectl to both `service-file-storage` pods; VPN to the RGW ClusterIPs). It is
+  best-effort/non-transactional: if interrupted, the DB is already committed — just **re-run** (idempotent
+  overwrite) to finish the files, or run with `--skip-files` to skip them. `missing-in-source` counts are
+  files that no longer exist in either source bucket (deleted); `failed` should be 0.
+- **Change a copy/ignore or a uniqueness decision** by editing `schema-map.json` (`action`, or a
+  column's `strategy`) — no code change. Re-measure + `node gen-schema-map.js` after a schema change.
+- **Creating a brand-new target** (`--name` / `--owner-user`) works: it clones the source `companies`
+  row and links the owner (role `Owner`). The owner user must already exist in the target namespace.
+- The tool **never deletes** and **never copies users**. To reset a throwaway *test* target you made,
+  delete its rows manually (see the cleanup snippet in the repo history) — never against a real org.