procesio-cli 0.1.1__py3-none-any.whl

procesio_cli/agents/procesio/PROCESIO-BEST-PRACTICES.md

@@ -0,0 +1,316 @@
+# PROCESIO implementation best practices (distilled)
+
+Agent-scoped guidance: the standards a build must meet, not tool mechanics. The
+agent applies these when designing and reviewing any process, form, or use case.
+
+**Source:** Google Doc "Best practices for implementing with PROCESIO"
+(`docId 1t0nqrVKEJwkuubhiIDpRcXd3OubClmfmmbfav953dZo`, captured 2026-06-25).
+Captured here per Hard rule 6 so the standard survives chat compaction. Re-pull the
+doc if it changes; this is a distillation, the doc is the source of truth.
+Platform tricks: https://docs.procesio.com/platform-tricks
+
+The source doc has 6 image-only examples (bad vs good parameter handling; the
+5-Decisional ~2s vs single-Decisional <200ms speed comparison; the JSON-to-Data-Model
+extraction diagram) that carry no alt text and are not reproduced here - open the doc
+for the visuals. The principles they illustrate are captured in sections 2, 9 and 10.
+
+---
+
+## 1. Frame the work
+
+- One measurable KPI per process (e.g. "reduce onboarding cycle time by 30%").
+- Map as-is -> to-be: triggers, systems, data contracts, SLAs, exceptions.
+- Define RACI early: business owner, builder(s), reviewer/QA, approver, operator.
+  Use the `bpof-expert` skill for process mapping and RACI.
+
+## 2. Solution design
+
+- **Modularize:** one main process + small reusable subprocesses (auth, paging,
+  validation, notifications). Trade-off both ways: too many subprocesses = harder
+  debugging + extra execution time; none = a monolith that re-runs steps it should
+  not. Choose deliberately.
+- **Stable contracts:** standardize inputs/outputs as DTO-style variables. Do not
+  pass raw API payloads between steps.
+- **Idempotency:** a retry or a double-trigger must not create duplicates. Use
+  external IDs and dedupe/once-only keys. Triggering twice must not change records.
+- **Error ports:** wire them for error capture and error-handling logic.
+- **Stateless core:** anything that must survive across runs (checkpoints, last-sync
+  time, job status, idempotency keys, approvals) lives in a durable system of record
+  (CRM/ERP/ticketing or a DB you own). PROCESIO variables are ephemeral - use them
+  only within a single execution (pass data between steps, hold tokens, page
+  cursors, correlation IDs).
+- **Perceived speed is non-negotiable.** Users must feel the UI helps them, not that
+  they wait on PROCESIO. A slow form reads to the client as "PROCESIO is slow". Do
+  not let this happen. See sections 9 and 10.
+- **Secrets:** never in processes. Credential Manager only.
+- **Centralize parameters:** a value used in many processes (and that will change)
+  is set in ONE place and shared, so one change propagates. Never hardcode it in
+  many places. Hardcoding + scattering always creates future effort.
+
+## 3. Integrations (target 1-2 days for the first system)
+
+- Wrap each external API in reusable subprocesses: `auth-get-token`,
+  `<system>-get-by-id`, `<system>-search`, `<system>-upsert`, plus a pagination
+  helper and a rate-limit handler.
+- Exponential backoff with jitter. Example attempt -> wait: 1 -> ~5s, 2 -> ~10s,
+  3 -> ~30s, 4 -> ~1min.
+- Idempotency keys so the same logical action runs once even on retry/double-trigger.
+- Standard error object across integrations.
+
+## 4. Data contracts
+
+- Define DTO variables (Customer, OrderLine, Address).
+- Reuse data-model children of the same data model where appropriate.
+
+## 5. Robustness
+
+Global try/catch via Error Port + Decisional action. Classify and handle:
+- **Retryable:** retry(max=3, backoff=2^n).
+- **Business:** send to queue + notify owner.
+- **Hard fail:** alert, capture context, stop.
+
+## 6. Observability
+
+- Generate a correlation ID at start; pass it to all logs/requests.
+- Store the main process instance ID; set an explicit status (success/fail/custom).
+- Alert on thresholds when something fails repeatedly.
+
+## 7. Document & handover
+
+- Document each process. Map the order processes trigger in (visual map if needed).
+- Write SOPs so someone without prior context can debug or follow the logic.
+
+## 8. DB actions (SQL Server / MySQL) + Data Store
+
+The DB credential (catalog type **`SQL`**) is now multi-engine: it carries a **server
+type** (`DbClientType` — `MSSQL=1`, `MYSQL=2`), so the same `Execute Query` / `Execute
+Command` actions run against **MySQL** by binding a MySQL-typed DB credential (PRC-3696).
+**Redis** shipped as its own credential type + connector. No new action families — pick
+the engine on the credential. Outbound DB hosts are SSRF-guarded by a host blacklist, so
+a private/blacklisted host is rejected at connect time. Bind values as **parameters**,
+never inline — the injection-safe `@param` map is engine-agnostic. **Data Store** (the
+tenant table feature) is separate — see the `datastore` guidance topic; scheduling a
+process on a cron is the `scheduling` topic.
+
+Use the `sql-server-optimizer` skill for any SQL in a SQL action. Core rules:
+- Inject data via action parameters; centralize all params at the top of the
+  statement for visibility.
+- Parameters go on the RIGHT-hand side of an operation only. `WHERE I.Id =
+  @Param_Id` good; `SELECT *, @Param_Id` or `SET @Parameter = @SQLVariable` errors.
+- Param names may match internal SQL variable names (the compiler resolves safely);
+  use each parameter once. After injection, clean the SQL variables holding param
+  data before the rest of the statement runs.
+- FROM/JOIN: smallest record set first, largest last.
+- Join conditions: most restrictive first. INNER joins before LEFT joins.
+- Max 5-6 joins. Beyond that the engine cannot compute an optimal plan (5!=120,
+  6!=720, 7!=5040) and falls back to running it as written.
+- WHERE: most restrictive condition first.
+- Pre-calculate values; do not call functions per row (no `ISNULL(...)`/`CASE` in a
+  per-row WHERE/SELECT when it can be computed once into a variable).
+- Filters activate only when set: `(@p IS NULL OR (<col> LIKE '%'+@p+'%'))`, not a
+  `CASE` that always evaluates.
+
+## 9. Forms & Tasks - UX timing (outcome rules)
+
+- **>=90% of clicks under 1s.** Limit calls to processes (Forms have features that
+  avoid most calls). When you must call a process, keep it to a minimum number of
+  actions (1 ideal, 3 still ok, 5 bad, >5 really bad) and optimize it for speed
+  (section 10). Use scoped loading visuals for the affected section, not a full
+  screen. Paginate and filter tables that can grow large.
+- **<=10% of clicks in the 1-3s band.** Break a long task into steps each under 3s,
+  updating the UI as each finishes, so the UI feels faster.
+- **No click ever over 3s.** Use an async mechanism: let the user continue and
+  notify them when done.
+
+## 10. Optimize for speed
+
+- Every action costs ~10ms. Some are natively slower (Decisional), some faster
+  (numeric Add). An action gets slower when it parses/injects large data, or talks
+  to an external system (Call API, SQL, FTP).
+- Levers: reduce action count at all costs; use scripting actions effectively
+  (do more per action); optimize the external systems (SQL statement + DB; use
+  Call API/FTP only where needed when form-call latency matters).
+- Example: 5 Decisional actions ~2s -> a single Decisional <200ms -> best, a single
+  scripting action that does the whole thing.
+- Do not extract from JSON with chains of actions. Model the JSON into a Data Model
+  and use the variable as `<DataModel>`; then read attributes directly, often with
+  no action at all.
+- A Node that builds an LLM prompt should put the static rules FIRST and the per-run
+  data (history, records, a language hint) LAST. Provider prompt caches key on a
+  byte-identical prefix (OpenAI: >= 1024 tokens), so per-run data at the top makes
+  every call a cache miss. Order only, wording unchanged; diff the line multiset
+  before the PUT.
+
+---
+
+## Name every Call/Trigger Subprocess node suggestively (never leave the generic label)
+
+When a flow contains a `Call Subprocess` / `Trigger Subprocess`, set the node's **`name`**
+(the builder accepts `name` on any action) to describe **what that call does in this flow**,
+or the **called process's name** — whichever reads clearer for the context. Never ship the
+generic "Call Subprocess" label: on a canvas with several subprocess calls they become
+indistinguishable and the flow is unreadable. Examples: `"Citește Contacte proprietari
+(read-range)"`, `"Adaugă rândul în Colectare acte (append-row)"`, `"Creează folder Ap.<nr>
+(create-folder)"`. The right name varies by context — pick the one that makes the canvas
+self-explanatory.
+
+## Form: pre-fill controls from URL query params (validated live 2026-07-22)
+
+A published PROCESIO form pre-fills a control from a URL query parameter **whose key
+equals the control's binding `name`**. Open `https://forms.procesio.app/{tinyUrl}?<name>=<value>`
+and the control renders with `<value>`. The match is on the config `name` (field/binding
+name), NOT the DOM `id`/`name` (empty on the rendered input). Unmatched params are ignored.
+Undocumented by PROCESIO, but confirmed live (form with `cod`+`apartament` controls opened
+at `?cod=TEST123&apartament=27` → both pre-filled).
+
+Use it to **seed a form from a per-record link**: give each record a stable, URL-safe token,
+name a control after the query key, distribute `…?token=<value>`. Then a form event (e.g.
+`on: open → do: process`) can read that seed control and fetch the rest. This is the mechanism
+behind the Uranus 100 „colectare acte" access model (unique link per apartment, `?cod=`).
+To make a form public + get the link: `isPrivate:false` + publish, then
+`POST /api/CustomUrl/FormTemplate {Type:3,EntityType:1,EntityId:<formTemplateId>,Url:<slug>}`
+→ `tinyUrl` → `forms.procesio.app/{tinyUrl}`.
+
+## Recipe: a dynamic, TYPED PROCESIO document (AAT_ Company Brief — live)
+A branded PDF whose repeating table + scalars are filled from a live API call:
+1. **Table rows <- `Call API` -> a typed `list<item>` model.** Point the Call API Response
+   Body at e.g. `AAT_GoogleResponse {items: list<AAT_SearchHit>}`; PROCESIO deserializes by
+   `jsonProperty`. docMap the doc's `hits` list `<- {var: googleResponse, path:[items_attr]}`.
+   (The Javascript action canNOT build a typed list — see tool PROCESIO-API-NOTES.)
+2. **Scalars** — use the **Node** scripting action (raw output) and map straight onto the doc
+   vars; OR with the **Javascript** action route through an envelope model
+   `{result:{searchTerm, generatedOn, resultCount, heroImageUrl}}` + docMap `path:[result,
+   field]`. Build the envelope model via the attribute-endpoint create flow so it COMPILES.
+3. **Never** feed a Javascript-action output downstream as a query/filename — the `{result}`
+   wrapper leaks. Drive the Call API query + file name from the ORIGINAL form inputs.
+4. Build every data model via the attribute-endpoint create flow (empty model -> POST each
+   attribute) so it's COMPILED — else the document renders "Unknown".
+
+## Email a generated file (Send Email)
+- **Map attachment is `list<File>`** — declare the file input `isList:true`; a single File
+  -> designer "data type mismatch" + NO attachment (run still status 50, so green != attached).
+  Pass `[fileDTO]`. Form file-upload controls already hold a list.
+- **To** can be a variable for a dynamic recipient (`<%0%>` + Variable[recipientEmail]).
+- The Body can come from a document template via Generate Document's "HTML string" output,
+  or a templated HTML string.
+
+## Form: an "Email me" button enabled only once the file exists
+Conflict: btnSearch must run with no file yet; btnEmail only once the file is present.
+Resolve via FIELD VALIDITY, not JS `.disabled` toggling:
+- `briefFile` (file-upload) **`required:true`** -> the form is invalid until the PDF lands.
+- `btnEmail` **`disabledIfFormIsInvalid:true`** -> enabled only when ALL required (company +
+  valid email + file) are present.
+- `btnSearch` **`disabledIfFormIsInvalid:false`** -> always clickable (it GENERATES the file).
+- email field: `required:true` + `regex` = email pattern; file-upload `readonly:true` for a
+  download-only control.
+
+## Type-matching across resources + Required inputs (LOAD-BEARING)
+
+**Variable TYPES must match EXACTLY for anything that passes between Process / Document /
+Form.** An input's type, an output's type, and a binding's type must be identical, or the
+consumer rejects it. Real failure (AAT_ v2): Send Email "Map attachment" is `List<File>`, so
+Process 3's input was made `List<File>` — but the FORM sends a single `File`, so the form
+**could not START the process** (a silent type mismatch between the form output and the
+process input). Always check the chain: Form control type → process INPUT type → action
+PARAMETER type → next resource. One mismatch anywhere breaks the hand-off.
+
+**Bridge a single `T` → `List<T>` INSIDE the process** (do NOT change the upstream resource):
+1. Keep the process INPUT as the single `T` (so it matches what the form/webhook sends).
+2. Add a **Node** action: `return [<%0%>];` (Node returns RAW — no wrapper) bound to its
+   **List Result** output → a `List<T>` process var. **Node REQUIRES `Timeout` > 0** (it
+   defaults to `00:00:00`, which errors `value must be greater than 00:00:00`); set `Timeout: 60`.
+3. Feed that `List<T>` var to the action (e.g. Send Email Map attachment ← briefFileList).
+   (Use **Single Result** instead when the script returns a non-list value.)
+
+**Required inputs block the start.** A process input marked `Required` that the trigger does
+NOT supply prevents the process from starting at all. Ensure every Required input is provided
+by the caller — the form button's inputMap, the webhook payload, or the run payload — or do
+not mark it Required. Audit the form↔process input map against the process's Required vars.
+
+
+## Creating nodes/actions in a flow via API (WORKS - userpass; verified 2026-07-05)
+
+Adding a NEW node to a flow via `PUT /api/Projects` works with the default userpass `account` profile.
+No designer UI needed. The one trap: do NOT copy an existing node to make a new one - a copied node
+carries the source flow's stale variable bindings in `customData.configuration`, and the validator then
+throws a diagnostic-free `HTTP 500 "Object reference not set to an instance of an object"`. Instead:
+
+1. Fetch the CLEAN action template: `GET /api/Actions?getFullAction=true`, find by `actionId`/`name`.
+   `copy.deepcopy(template.configuration)` and fill only your setting values. A clean config makes the
+   validator return REAL errors (141/142) instead of the 500.
+2. A JS "Node" (templateId e0fc20c3): settings are Code (1e6a5523) = JS + input `<%N%>` bound in the
+   Code param's `variable` list; **Single Result** (51b9fcdc) = the JS `return` output (NOT Error
+   9e0110c9); List Result (60237e88); Error (9e0110c9) = error port; Timeout (d3e52aab).
+3. New variables use key **`dataType`** (a type id) + `type` (10 input / 20 process / 30 output / 40 sys).
+   Missing type -> `statusCode 141 "type cannot be empty"`. Param/var type must MATCH the setting's
+   expected `dataTypeId` -> else `statusCode 142 data type mismatch`. Primitive ids: `...1211` number,
+   `...1214` string, `...1221` any/object; file `10c6ac59-...-121212121219` = FileDataModel
+   `{name,mimeType,path,size,id,hash}` (blob reference, NOT inline content).
+4. Files from text: a Node cannot return a FileModel (no inline content). Use **`Base64 To File`**
+   (actionId 6fad5599; settings Base64 String, File Name -> File output) to turn a base64 string into a
+   stored FileModel; then Call API `/media` upload (FORM_DATA `file`=FILE type, `type`=mime).
+5. Validate via `POST /api/Projects/validate` (empty body = valid) before PUT.
+
+Best path: `tools/procesio/dto/process/builder.py` already builds correct DTOs data-driven from the live
+catalog (`_action_node` = clean template config + values; `_edit` rebuilds a whole flow and PUTs). Prefer
+it for full-flow builds; replicate `_action_node`'s recipe for a surgical single-node insert into a
+fetched (lowercase-keyed) flow. Worked example: added Build VCF (Node) + Base64 To File to Chat Flow/Send
+Export to produce a `.vcf` bulk export (scratchpad build_vcf_flow3.py pattern).
+
+
+## Field learnings 2026-07-05 (designer review + real-corpus E2E) — AUTHORITATIVE
+
+1. **Decisional icon**: when creating a Decisional via API set `customData.icon = "call_split"` (the
+   designer standard). `icon-decisional` or anything else renders NO icon in the run view. Same rule
+   class: give decisional CASES meaningful `name`s — the designer shows them as chips on the canvas.
+2. **Scripting (Node) actions FAIL SILENTLY**: a JS code error does NOT raise / does NOT take the
+   type-1 error port. The action reports SUCCESS and writes the error text to its **Error output**
+   setting (9e0110c9). ALWAYS: (a) bind Error output to a string var on every Node; (b) on hot paths,
+   follow the Node with a Decisional `errVar IS_NOT_EMPTY -> error join`; (c) compose the user-facing
+   error message from the error vars in one downstream Node (values from the FAILED action are unset —
+   never map them into the notice directly). Proven: 6 silently-dying turns became 6 successes.
+3. **Huge values (base64 media) are NOT an engine limit** (corrected 2026-09-21). A controlled probe through
+   `/run` carried a 4 MB string intact through a Node `<%N%>` injection, a Decisional `IS_NOT_EMPTY`, the native
+   **Length** action (template 057d754a: Input String, Result), Map Data, subprocess maps and SQL parameters, on a
+   plain variable and on a data-model attribute alike. The earlier note that 100 KB+ values "materialize as EMPTY"
+   in a Node or Decisional was an artefact of the middleware that fed the flow (it capped its own payload), not of
+   PROCESIO. Length is still the cheapest way to test a big value's presence; it is not a workaround.
+4. **For Each via API** (template dbef0804-66a9-4f8f-872c-ece1b89b8fdb, shape `area`): params =
+   For Each Item (57b5b1eb…, the per-iteration var) + In List (2d5f230f…) + timeout + 2 ignore
+   literals. The FE node has TWO type-0 ports: first -> loop-body entry child, second -> the after-loop
+   node. Children carry `parentId` = FE id; the LAST child ports BACK to the FE (loop-back edge).
+5. **Workspace header is mandatory on run/launch**: `ProcesioClient(workspace_id=…)`. Without it,
+   `/api/Webhooks/launch` and `run-process` bind LARGE attribute values as EMPTY (looks like an engine
+   bug, is actually your missing header), and admin flows 400/502.
+6. **WhatsApp bulk contact export**: document sends get the MIME's extension APPENDED by WhatsApp
+   (filename `Contacts.vcf` + text/plain ⇒ phone receives `Contacts.vcf.txt`, unimportable). No allowed
+   /media MIME maps to .vcf. Correct approach: native `type:"contacts"` messages (array, chunk ~15 per
+   message, For Each over chunks) — renders as a tappable contact stack ("X and N other contacts").
+
+
+### Validation: trust the designer's Process Errors (2026-07-06, corrected)
+**A 2026-07-05 note wrongly dismissed the designer's Process Errors as stale phantoms. They are REAL
+and block SAVE. Trust them.** There are two layers:
+- **Runtime** — `POST /api/Projects/validate` (the only server validate endpoint): checks
+  `parameters[]`, returns EMPTY when valid. Does NOT catch designer-layer problems — a flow can pass
+  it yet be unsavable.
+- **Designer** — the client-side "Process Errors" panel blocks SAVE; it validates `customData`
+  (designer maps + code chips). No server endpoint exists for it.
+- **Replicate the designer check offline with `procesio flow-lint --id <flow> --workspace-id <ws>`**
+  (a Node binding an error-scope/undeclared variable in code; a required subprocess input left
+  unmapped or with a dead source in the `customData` process-inputs map). Never call a flow "done"
+  on `/validate` alone — always run flow-lint too, and if you can, confirm SAVE in the designer.
+- **The recurring cause:** cloning a subprocess-call node copies its `customData` process-inputs, so
+  the clone keeps the SOURCE's subprocess-input ids and the real required input reads as "missing"
+  (the panel shows the stale ids as raw GUIDs). Fix: regenerate `customData` process-inputs from the
+  correct runtime map. Never bind an `isError` variable (an action's error-port output) into Node code.
+
+### How to actually find a designer validation error (2026-07-06)
+When the designer's "Process Errors" flags a node but the data "looks correct", DON'T reverse-engineer
+the rule from that one node (I guessed wrong twice). Instead COMPARE THREE THINGS: the FLAGGED node,
+a HEALTHY node of the SAME actionTemplateName that the designer accepts, and the CLEAN template from
+`GET /api/Actions?getFullAction=true`. The one field that differs between flagged and healthy is the
+cause. That method found it in one shot: flagged subprocess nodes had a stale side-pannel setting id
+(`1da555da`) vs the template's `5456caf0`; the designer couldn't find the mapping. Then encode the rule
+into `procesio flow-lint` so it's caught mechanically next time — never call a flow done on
+`/Projects/validate` alone.

procesio_cli/agents/procesio/PROCESIO-BUILD-AND-TEST-PLAYBOOK.md

@@ -0,0 +1,217 @@
+# PROCESIO build-and-test playbook (agent operating procedure)
+
+How the PROCESIO agent works when it creates or edits any resource or use case. The
+rule: a build is not done when it is created. It is done when it has been run,
+validated, and verified against real behavior, the learnings captured, and the
+defects fixed. Building without testing is not allowed.
+
+This is agent behavior. Tool mechanics live in `tools/procesio/` - this playbook
+points at them, it does not duplicate them.
+
+---
+
+## The loop (run on every build/edit)
+
+1. **Frame** - state the outcome and KPI in one sentence, the scope, the
+   constraints, the RACI. (best practices section 1)
+2. **Design** - pick the approach; where it matters, weigh 2-3 with trade-offs.
+   Apply best practices up front: modular, DTO contracts, idempotent, stateless,
+   secrets in Credential Manager, centralized parameters.
+3. **Build programmatically** - via the `procesio` tool DTO sub-tools. Create data
+   models complete with all attributes up front (the runtime compiled-model cache
+   will not see attributes added by a later edit - PHASE4-E2E learning 12).
+4. **Self-test rigorously** - the checklist below. Non-negotiable.
+5. **Capture learnings** - write every discovery to the right durable note (routing
+   below). Compounding, not chat-only.
+6. **Fix** - bugs, inefficiencies, UX gaps, and any suggestion surfaced. Structural
+   defects get fixed in the builder/tool code, not just patched on the live
+   resource, so a fresh session builds it correctly.
+7. **Re-verify** - re-run the relevant checklist items on what changed.
+8. **Document & handover** - SOP, trigger order, resource ids. (best practices 7)
+
+## Definition of Done (outranks your own judgement)
+
+Measured failure (a GPT-driven session, 2026-09-05): the agent said "verified / works
+/ done" over and over on FALSE evidence - a PDF existed, a DTO field existed, a call
+returned `ok:true` - while the cache never cached, the button never fired, and the
+report was empty. Nine hours, thirty user turns, because the human was the only oracle.
+Do not repeat it.
+
+- **A build is done only when `verify --process-id <id> --run` returns a passing
+  verdict** - not when you inspected it, not when a file was produced, not when the API
+  accepted the write. `ok:true` proves *acceptance*, never *effect*. If `verify` cannot
+  cover a step (forms/browser/E2E), it lists it in `manual_checks`; run those, do not
+  wave them through.
+- **Clarify before you build.** In the Frame step, ask the questions that pin scope and
+  the load-bearing decisions first - the data source, and especially the **cache
+  mechanism** (native Data Store vs SQL; a cache MUST be a real Data Store / SQL action,
+  never a Call API dressed with a cache icon - `audit` now hard-fails that masquerade).
+  A task you *could* finish without asking finishes better, with fewer wrong-path loops,
+  when the spec is pinned up front.
+- **A process is called from a form ONLY through a native `RUN_PROCESS` event**, never
+  by injected JS. Verify the event is actually bound (`form-get-element-events`), not
+  just that JS "should" trigger it.
+- **Author processes through `process-create` / `process-edit`, never a raw
+  `PUT /api/Projects`.** The builder sets validity and the designer-config correctly; a
+  hand-authored flow JSON pushed via `request PUT /api/Projects` lands `isValid:false`
+  and the process will not launch (and hand-setting `isValid:true` only lies about it -
+  `verify --run` still fails because the flow itself is invalid). Build the flow through the
+  builder's structured config, never by hand-authoring node/edge DTOs: Data Store read/write
+  nodes via `dsWhere`/`dsMap`, and Decisional routing via the `branches` config - the builder
+  emits the correct `Ports[].Config` (the default branch is the string `"default"`, not a
+  boolean `isDefault`, which the BE rejects with `unexpected character 't'`). Guessing a
+  port/edge DTO is the same trap as guessing a node DTO.
+- **Fix the cause in the generic tool, never patch the use case in the repo.** If a
+  capability is missing (e.g. the builder could not author a Data Store node), EXTEND
+  the builder with tests - the way `process-create` gained native Data Store authoring -
+  never drop a `patch_<usecase>.py` into `scripts/`. Use-case artefacts live in a
+  scratch dir; the guard `tests/test_no_usecase_scripts.py` fails the build otherwise
+  (CLAUDE.md Hard rules 5 + 10).
+
+## Self-test checklist - what "tested" actually means
+
+Each item names the tool and the verified mechanic. Do not skip a layer because the
+one above passed.
+
+**The agent enforces the automatable layers.** Run them instead of doing them by hand:
+- `python scripts/run-agent.py procesio verify --process-id <id> [--run]` - validates
+  the process (real PROCESIO validator), audits designer-vs-runtime parity, and with
+  `--run` executes it and reads the real instance status. Returns a `verdict`
+  (pass/warn/fail) and a `manual_checks` list of the steps it cannot run.
+- `python scripts/run-agent.py procesio audit --process-id <id>` - static
+  best-practice + correctness smells (action count, slow actions, error handling,
+  inline secrets, parity).
+- `python scripts/run-agent.py procesio checklist` - the full structured checklist.
+
+`verify` cannot drive a real browser, so the forms/webhook/E2E steps below stay
+manual - it names them in `manual_checks` precisely so they are not forgotten.
+
+### A. Static validation (before running anything)
+- Validate the process: `procesio` -> `POST /api/Projects/validate`.
+- Test credentials: `POST /api/Credentials/test` (returns a real upstream body when
+  the placement is right - PHASE4-E2E learnings 3-4).
+- Test actions: `POST /api/Actions/test`.
+- Designer-vs-runtime parity: every action's runtime `Parameters[]` must be mirrored
+  into the designer `CustomData.configuration` or the designer shows "not
+  configured" and Validate fails even when the process runs. The builder enforces
+  this with a build audit; if you hand-edit a DTO, re-check it. Bespoke shapes
+  (document-mapper, decisional-case) differ between runtime and designer - PHASE4-E2E
+  "CRITICAL process-builder fix" + rounds 3 and 5.
+
+### B. Run + instance verification
+- Run synchronously: `run-process --synchronous`; read output vars under
+  `result.variable`.
+- Read the instance status, do not assume success: 50 = finished, 40 = ran with
+  errors. Check per-action `errorMessage` via
+  `GET /api/Projects/instances/{iid}/status?flowTemplateId={pid}` (PHASE4-E2E 6, 15).
+- Webhook-launched runs are async (launch returns an empty 200) - then poll
+  `GET /api/Projects/{id}/instances`.
+- Download and open any generated file: `GET /api/File/download` with headers
+  `uploadFilePath`, `variableId`, `instanceId`, `flowTemplateId`.
+
+### C. Forms - render AND behavior, with diagnostics (MANDATORY before "done")
+A form is NOT done until EVERY tab, EVERY control, and EVERY event/action has been
+exercised live and the runtime diagnostics are clean. Screenshots alone are not enough
+(they hide JS errors, failed launches, flicker). Steps:
+
+1. **Publish + URL:** publish, create a CustomUrl (`POST /api/CustomUrl/FormTemplate`),
+   open `https://forms.procesio.app/{tinyUrl}`. SPA needs ~15-20s to render; `wait`.
+2. **Drive every control/action with the `web` tool** (`run` steps: click tabs, `fill`
+   inputs, click buttons, open selects/dropdowns/side-panels, click menu items). The
+   CURRENT ui-builder forms (forms.procesio.app, pds/BootstrapVue design system) WORK
+   HEADLESS - verified 2026-06-29: `fill` updates the value, `RUN_JAVASCRIPT` form JS
+   fires, `RUN_PROCESS` triggers flow end-to-end (output mapped back into the field).
+   (The old "headless can't test forms / Angular model" caveat applied to the legacy
+   engine; it is superseded for these forms.)
+3. **Read `diagnostics` from the web-tool result** (added 2026-06-29). It returns
+   `{console, page_errors, failed_requests, bad_responses}`. **A build with any
+   page_error, any console error, or any 4xx/5xx on the form's OWN calls is NOT done** -
+   that is how a real failure shows (e.g. the RUN_PROCESS launch 400, a JS exception,
+   the icon-font break). Fix, re-run, confirm clean.
+4. **Assert the expected EFFECT per capability**, not just "it rendered": typed value
+   reflected, computed/JS output appears, dropdown opens un-clipped, process result
+   fills the target field, side-panel content shows, no flicker/scroll-trap.
+5. **Test the NEGATIVE / dismiss path of every stateful interaction**, not just the happy
+   click: open a dropdown/menu then **click OUTSIDE** it → only the menu should close,
+   nothing else should fire (a form-level click handler that substring-matches a menu label
+   up the ancestor chain will clear/trigger on outside-click — match exactly ONE label; see
+   form `description.md`). For a **chat reply via a process**, send a message and confirm a
+   bot reply renders (wire `messagesent`, NOT `messagesfetch`; produce the array with a
+   **Node** action `return …`→`List Result`, not JS `setOutput` which wraps `{result}`).
+- **Process triggers:** RUN_PROCESS in/out maps must put the PROCESS variable GUID on
+  the LEFT (resolved from name->id) and the form value-path on the RIGHT, in BOTH maps;
+  a name or a form-path on the left = the designer can't render it / launch 400s
+  (2026-06-29). The builder resolves this (prepare_ctx); guarded by `test_form_parity.py`.
+- Form data model: the field-value path is `root.<fields-const>.<elementId>.
+  <valueConfigId>`, attr ids must equal the element config ids, and the dataModel
+  root id must equal the form handle variable id - else the designer shows "Unknown"
+  and values do not flow. The builder handles this; verify after any manual change.
+- **Structural parity is auto-tested:** `pytest tools/procesio/tests/test_form_parity.py`
+  (+ `test_form_sync`) assert the builder's DTO matches real exports. Run before shipping.
+- **CSS/JS gotcha:** form CSS/JS lives in encrypted `Data.code`; never force a font on
+  `#app *` (breaks Material-Icons glyphs); the form JS runs in `iframe.trigger-sandbox`
+  (use `window.parent.document`). See `tools/procesio/dto/form/FORM-STYLING-NOTES.md`.
+- **Theme + alignment (new):** if the form uses dark mode, toggle it and confirm the
+  16-var palette applies; if it uses element alignment, confirm the `--fd/--jc/--ai`
+  vars lay the container out — both only in a REAL browser (`form-theme-render`).
+- **Data Store trigger (new):** a `RUN_DATA_STORE_OPERATION` control must run its
+  READ/ADD/UPDATE/DELETE against the store and map the result back
+  (`form-datastore-trigger`). See the `datastore` guidance topic.
+
+### C2. Data Store (standalone)
+- Round-trip a store: add → filtered read → update-by-PK → delete, asserting
+  `affectedRows` and that the filtered read returns exactly the expected rows
+  (`datastore-roundtrip`). Rows are keyed by column DISPLAY name; reads go through
+  `POST …/rows/filter`. Full guidance: the `datastore` topic.
+
+### C3. Scheduling
+- Preview any crontab with `validate-crontab` before creating the schedule, then
+  `get-schedule` to confirm the stored recurrence (`schedule-cron`). See the
+  `scheduling` topic.
+
+### D. Webhooks
+- Trigger the webhook with a real payload; confirm an async instance launches and
+  reaches status 50. The webhook body binds as a WHOLE to a model-typed variable
+  (not per-field) - bind it to a primitive and you get raw JSON (PHASE4-E2E 1).
+
+### E. End-to-end
+- Exercise the use case as a user would: form -> process(es) -> document/email/SQL ->
+  external system. Verify the real artifact: open the file, confirm the email
+  delivered, confirm the record in the system of record.
+- Idempotency: trigger twice, confirm no duplicate side effects (best practices 2).
+- Timing: check against the forms UX rules (best practices 9) and the
+  optimize-for-speed rules (best practices 10 - action count, ~10ms each).
+
+## Implement the learnings (this is where it compounds)
+
+Testing that does not change the build is wasted. After each test pass:
+
+- **Fix every defect found.** Bugs; inefficiencies (too many actions, slow SQL,
+  unscoped loading screens, an avoidable process call from a form); UX gaps; and any
+  improvement you spotted. Address it, do not just note it.
+- **Structural/tool defects go into the builder code + a regression test**, not only
+  the live resource. PHASE4-E2E is the proof of this discipline: every fix landed in
+  the builder so a fresh session's create/edit is already correct, backed by tests.
+- **Platform limits you cannot fix** get documented so the next build avoids them
+  (e.g. JS output is always `{result:v}` and will not fill a typed model - use the
+  HTML-string shell pattern; PHASE4-E2E rounds 4 and 8-10).
+- **Route the learning:**
+  - tool / platform API / DTO quirk / builder behavior -> `tools/procesio/*-NOTES.md`
+    (or the relevant tool folder).
+  - methodology / how to build well -> this folder (`agents/procesio/`).
+  - cross-cutting standing preference -> user memory.
+
+
+## Tools the agent drives
+
+The registry is the source of truth - `python scripts/list-tools.py`, never a
+hardcoded list. Core for building in PROCESIO:
+- `procesio` - create/edit resources, validate, run, read instances.
+- `web` - real-browser form render + behavior testing, screenshots, UX timing.
+
+## Read these (do not duplicate)
+
+- Tool mechanics + verified gotchas: `tools/procesio/PHASE4-E2E-NOTES.md`,
+  `PROCESIO-API-NOTES.md`, `PROCESIO-AUTH-NOTES.md`, `DTO-SUBTOOLS-NOTE.md`,
+  `PROCESIO-SEND-EMAIL-NOTES.md`.
+- Standards the build must meet: `PROCESIO-BEST-PRACTICES.md` (this folder).

procesio_cli/agents/procesio/PROCESIO-DATASTORE.md

@@ -0,0 +1,72 @@
+# PROCESIO Data Store — build & use
+
+**Data Store** is PROCESIO's user-defined dynamic-table feature: a tenant defines a
+typed table (columns backed by a data model) and reads/writes its rows. It is
+API-first, it surfaces as a **Process Designer action**, and it can be triggered from a
+**form** — the same operation reachable three ways over one service.
+
+## The model (know this before building)
+
+- A data store = **metadata** (name + typed columns) + **rows**. Rows are JSON objects
+  keyed by **column display name**, never the internal alias. Values are arbitrary JSON.
+- **Primary-key columns** identify a row for update/delete. Four system columns are
+  read-only and managed by the platform (`CREATED_ON`, `UPDATED_ON`, and the created/
+  updated user ids).
+- Two **permission entities** gate it: `DataStoreSchema` (create/alter/drop the table)
+  and `DataStoreRows` (read/write rows + CSV). A user may hold one without the other.
+
+**When to use it.** Reach for a Data Store when a process needs a small, tenant-owned,
+queryable table it can also read/write from a form (a lookup list, a queue, a
+submissions log). Prefer a **data model** (data type) for the *shape* of a variable, and
+an **external DB action** (Execute Query/Command over a DB credential) when the data
+already lives in a real database you own.
+
+## Tool actions (`procesio datastore-*`)
+
+Metadata: `datastore-create` / `-update` / `-modify-column` / `-delete` / `-get` /
+`-list` / `-list-restricted` / `-from-data-model` / `-from-json` / `-get-data-model`.
+Rows: `datastore-get-rows` (paged read with filters/sort), `-add-rows`, `-update-row`,
+`-delete-rows`. CSV jobs: `datastore-export-start` / `-export-download` /
+`-import-start` / `-import-failures`.
+
+- **Reads: `POST …/rows/filter`** — **pagination on the query string**
+  (`?pageNumber=&pageItemCount=`), body a RECURSIVE filter tree + sort:
+  `{filter:{id,logic(0/1/2),items:[{id,type(1 condition|2 group),logic,condition:{id,column,
+  operator,value,auxValue?}}]}, sort:[{column,direction(1 asc|2 desc)}]}` (no body = read
+  all). `datastore-get-rows` builds this from `--filters '[{column,operator,value}]'`
+  (+`--logic and|or`) or takes a raw `--filter`/`--body`. Operators are `QueryOperators`
+  (0–20); pass a name (`Equals`,`Contains`,`Between`,`In`,`IsNull`,…) or number.
+- **Update = `{values:{…}, filter:<tree>}`; delete = `{filter:<tree>}`** — a **non-empty
+  filter is mandatory** on both (there is no key-array form). `datastore-update-row`
+  takes `--values` + `--filter/--filters`; `datastore-delete-rows` takes `--filter/--filters`.
+- Row dicts are keyed by **display name** everywhere (add/update/delete).
+
+## Process Designer — the "Data Store" action
+
+A native **`Data Store`** node runs one operation against a store. Operation is
+`DataStoreOperationType` shown as **SELECT / INSERT / UPDATE / DELETE**
+(SelectRows=1 / InsertRows=2 / UpdateRows=3 / DeleteRows=4). Its inputs use a dedicated
+FE mapper component (`Data_Store_Mapper`) to bind the store id, the operation, and the
+Set-Values / Match-Where maps. It is a platform (native) action dispatched by
+`IsProcesio`, so there is **no code-visible action GUID** — resolve it by name from the
+live action catalog (`procesio list-actions-catalog`) when authoring a flow, and refresh
+the tool's bundled catalog from `/api/Actions` after a platform release.
+
+## Forms — the Data Store trigger
+
+A form element can trigger a Data Store operation the same way it triggers a process
+(RUN_PROCESS). The event action is **`RUN_DATA_STORE_OPERATION`**; wire it surgically
+with `procesio form-set-element-event --action RUN_DATA_STORE_OPERATION --data-store-id
+<id> --operation READ|ADD|UPDATE|DELETE` (or in a full form rebuild via a control event
+`do: datastore`). Config shape: `{dataStoreId, operation, inputMap, outputMap, filters?,
+areFiltersConfigured?}`. **Filters apply to every operation except ADD.** The form
+runtime calls the anonymous `api/Form/dataStore/{id}/rows(/filter)` endpoints with the
+form context in headers — the form does not need the tenant's DataStore permissions.
+
+## Verify (never call it done on compile)
+
+Round-trip a store end to end: create (from-json is quickest), add rows, read them back
+with a filter, update one by its PK, delete it — asserting `affectedRows` each step and
+that the filtered read returns exactly the rows you expect. Then build a process with a
+Data Store SELECT node and run it, and (if a form is in scope) render the form and fire
+the trigger. See the build-and-test playbook, forms section.