@jigx/core-sdk 1.1.0 → 1.2.0-rc

package/docs/pattern-datasources.md

@@ -0,0 +1,459 @@
+# Pattern: Datasources
+
+This is the standard approach unless the user instructs otherwise.
+
+## Rules
+
+1. **Always select `id` and `data` only** — never use `json_extract` in SELECT columns
+2. **Add `jsonProperties: ['data']`** — Jigx parses the JSON column into an object automatically
+3. **Use `json_extract` only in WHERE, JOIN, ORDER BY** — when you need to filter or sort by a specific field
+4. **Access fields via the parsed `data` object** — `@ctx.datasources.name.data.FieldName.value`
+
+## Examples
+
+```typescript
+// App-scoped datasource
+app.addDatasource
+  .sqlite({ datasourceId: 'data-select-customers', provider: 'dynamic' })
+  .entity('default/customers')
+  .query('SELECT id, data FROM [default/customers]')
+  .jsonProperties('data')
+
+// Screen-scoped datasource with WHERE filter (json_extract only in WHERE)
+screen.addDatasource
+  .sqlite({ datasourceId: 'data-select-contacts', provider: 'dynamic' })
+  .entity('default/contacts')
+  .query(`SELECT id, data FROM [default/contacts] WHERE json_extract(data, '$.customerId') = @customerId`)
+  .queryParameter('customerId', '=@ctx.jig.inputs.customerId')
+  .jsonProperties('data')
+
+// Single document with isDocument
+screen.addDatasource
+  .sqlite({ datasourceId: 'data-select-customer', provider: 'dynamic' })
+  .entity('default/customers')
+  .query('SELECT id, data FROM [default/customers] WHERE id = @customerId')
+  .queryParameter('customerId', '=@ctx.jig.inputs.customerId')
+  .jsonProperties('data')
+  .isDocument(true)
+```
+
+## Global vs screen-scoped datasources
+
+Datasources that reference `@ctx.jig.state`, `@ctx.jig.inputs`, or other jig-level context **must be screen-scoped** (defined inline on the screen). Global datasources (defined in `datasources.ts` at the app level) do not have access to jig state.
+
+```typescript
+// WRONG — global datasource cannot access @ctx.jig.state.searchText
+app.addDatasource.sqlite({ datasourceId: 'data-select-customers', provider: 'local' })
+  .query('... WHERE ... @search ...')
+  .queryParameter('search', '=@ctx.jig.state.searchText')  // Error: no jig context
+
+// CORRECT — screen-scoped datasource has jig context
+screen.addDatasource.sqlite({ datasourceId: 'data-select-customers', provider: 'local' })
+  .query('... WHERE ... @search ...')
+  .queryParameter('search', '=@ctx.jig.state.searchText')  // Works: jig context available
+```
+
+**Rule**: If a datasource references `@ctx.jig.*`, define it on the screen. If it only references `@ctx.datasources.*` or static values, it can be global.
+
+## Accessing fields in expressions
+
+With `jsonProperties: ['data']`, the `data` column is auto-parsed. Field access follows the table schema:
+
+```typescript
+// Acumatica schema (nested with .value wrappers)
+'=@ctx.datasources.data-select-customer.data.CustomerName.value'
+'=@ctx.datasources.data-select-customer.data.MainContact.Email.value'
+
+// Flat schema (no .value wrappers)
+'=@ctx.current.item.data.firstName'
+
+// In list items (current item context)
+'=@ctx.current.item.data.CustomerName.value'
+```
+
+### Field filters with `jsonProperties` — `data.` prefix required
+
+When you filter a datasource by field using JSONata `[field = value]`, you MUST use
+the `data.` prefix, because `jsonProperties('data')` leaves field values under a
+`data` sub-object:
+
+```typescript
+// WRONG — filter never matches, silently returns empty
+'=$exists(@ctx.datasources.checklistAnswers[itemID = @ctx.current.item.id])'
+
+// RIGHT — data.itemID, not itemID
+'=$exists(@ctx.datasources.checklistAnswers[data.itemID = @ctx.current.item.id])'
+```
+
+The only time you drop the `data.` prefix is when the SELECT columns are explicit
+(not `SELECT id, data`), like:
+
+```sql
+SELECT id, json_extract(data, '$.itemSection') as itemSection FROM [...]
+```
+
+In that case `itemSection` is a top-level column (no `data.` wrapper), so filters
+like `[itemSection = "foo"]` work without the prefix.
+
+**Rule of thumb**: match your JSONata filter to the SELECT shape. If the SELECT has
+bare `data`, you need `data.field` filters. If the SELECT aliases fields as top-level
+columns, you don't.
+
+## isDocument for single-row datasources
+
+When a datasource always returns one row (config table, detail lookup by ID), set `isDocument(true)`. This returns an object instead of an array, so you access fields directly without `[0]`:
+
+```typescript
+// With isDocument: true
+'=@ctx.datasources.data-select-config.data.acumaticaURL'
+
+// Without isDocument (array) — would require [0]
+'=@ctx.datasources.data-select-config[0].data.acumaticaURL'
+```
+
+## Seeded startup rows and solution state
+
+When a standalone app seeds local data that the landing screen immediately displays
+or uses to set `@ctx.solution.state`, do not rely on a datasource reading the seeded
+row during the same first render. The screen may focus before the app `onLoad` save
+has refreshed the datasource, which can leave the first screen blank until the user
+navigates away and back.
+
+Use this pattern:
+
+- Keep the seed row in one shared constant/helper so app `onLoad`, `onRefresh`, and
+  landing-screen `onFocus` write the exact same values.
+- On app `onLoad`, save the seed row and then set required solution-state keys from
+  the seed values directly, not from a datasource that may not have refreshed yet.
+- On the landing screen `onFocus`, upsert the same seed row first, then set solution
+  state from the seed values directly.
+- If downstream screens need seeded identifiers, persist them into the quote/order
+  row when creating it instead of reintroducing hardcoded constants in submit actions.
+
+This is especially important for mock/standalone appointment screens that will later
+be replaced by a parent app's real appointment context.
+
+## Cross-provider JOINs: registering dynamic tables for local queries
+
+When a datasource with `provider: local` JOINs a `default/` dynamic data table, the dynamic table must be **registered** separately — otherwise Jigx won't create it on the device and the query fails with `no such table`.
+
+A datasource's `entities` list can only contain entities of the **same provider type**. You cannot mix local and dynamic entities in one datasource.
+
+**Fix**: Add a separate global datasource (or sync entity call) with `provider: dynamic` that lists the dynamic table as its entity. This subscribes the table so Jigx creates it on the device. The datasource doesn't need to be actively used — just declared.
+
+```typescript
+// Register a dynamic table so it exists on device for JOINs from local datasources
+app.addDatasource
+  .sqlite({ datasourceId: 'data-register-panel-mapping', provider: 'dynamic' })
+  .entity('default/panelDesignMapping')
+  .query('SELECT id, data FROM [default/panelDesignMapping]')
+  .jsonProperties('data')
+
+// Local datasource that JOINs the dynamic table — now works because the table is registered
+screen.addDatasource
+  .sqlite({ datasourceId: 'panelDesigns', provider: 'local' })
+  .entity('AttributeDetails')
+  .query(`
+    SELECT ad.id, ad.description, pm.model, pm.windows
+    FROM [AttributeDetails] ad
+    JOIN [default/panelDesignMapping] pm ON ad.valueID = pm.panelDesignID
+    WHERE ad.attributeID = 'PANELDES' AND pm.model = @model
+  `)
+```
+
+**Rule**: If a dynamic data table (`default/`) is referenced in a query but not listed under `entities` on any dynamic-provider datasource, the Jigx runtime won't subscribe to it and the table won't exist on the device. Always ensure every dynamic table used in queries is registered via at least one dynamic-provider datasource or sync entity call.
+
+## Local vs dynamic tables
+
+| Provider | Entity format | Database file | Use case |
+| --- | --- | --- | --- |
+| `local` | `Customers` (no `default/` prefix) | NOT registered | User-created data, offline-first, synced to remote via REST |
+| `dynamic` | `default/config` (with prefix) | Registered in `database.ts` | Cloud-synced data managed via Jigx management portal |
+
+**Only dynamic tables go in `database.ts`.** Local tables are created automatically when referenced in datasources or operations — do not add them to the database file.
+
+```typescript
+// database.ts — only dynamic tables
+app.addDatabase.default.table('config')
+// Do NOT add Customers, Contacts, or error tables here — they are local
+
+// Local table datasource — entity name only, capitalize
+app.addDatasource
+  .sqlite({ datasourceId: 'data-select-customers', provider: 'local' })
+  .entity('Customers')
+  .query('SELECT id, data FROM [Customers]')
+
+// Dynamic table datasource — default/ prefix
+app.addDatasource
+  .sqlite({ datasourceId: 'data-select-config', provider: 'dynamic' })
+  .entity('default/config')
+  .query('SELECT id, data FROM [default/config]')
+```
+
+## List search support
+
+When a list screen has `isSearchable: true`, the user's search text is available via `@ctx.jig.state.searchText`. Pass it as a query parameter and use `LIKE` in the WHERE clause:
+
+```typescript
+// Must be screen-scoped (uses @ctx.jig.state.searchText)
+screen.addDatasource
+  .sqlite({ datasourceId: 'data-select-customers', provider: 'local' })
+  .entity('Customers')
+  .query(`
+    SELECT id, data FROM [Customers]
+    WHERE (
+      json_extract(data, '$.CustomerName.value') LIKE '%' || @search || '%'
+      OR json_extract(data, '$.CustomerID.value') LIKE '%' || @search || '%'
+      OR @search IS NULL
+    )
+    ORDER BY json_extract(data, '$.LastModifiedDateTime.value') DESC
+  `)
+  .queryParameter('search', '=@ctx.jig.state.searchText')
+  .jsonProperties('data')
+```
+
+Key points:
+- **Must be screen-scoped** — uses `@ctx.jig.state.searchText` which requires jig context (not available on global datasources)
+- `@search IS NULL` ensures all records show when the search box is empty
+- Search common fields like name and ID — check with the solution builder if other fields should be searchable
+- `json_extract` is used in the WHERE clause (not SELECT) — consistent with our datasource pattern
+- The `searchText` state is managed automatically by the list screen when `isSearchable: true`
+
+## Ordering by last modified date
+
+For list screens showing entities, order by a last-modified timestamp descending so the most recently modified/created records appear first:
+
+```sql
+SELECT id, data FROM [Customers]
+ORDER BY json_extract(data, '$.LastModifiedDateTime.value') DESC
+```
+
+When creating new records locally, set the timestamp to `$now()` so they sort to the top:
+
+```jsonata
+"LastModifiedDateTime": {"value": $now()},
+"Remote": "new",
+```
+
+This timestamp is replaced by the remote system's actual timestamp when the record syncs.
+
+**Acumatica-specific**: The field is called `LastModifiedDateTime` with a `{ "value": ... }` wrapper. Other systems may use a different field name (e.g., `updatedAt`, `modified_date`) and format — adapt the field path in the ORDER BY and save expression accordingly.
+
+## SQLite expression indexes for JSON fields
+
+Jigx stores local and Dynamic Data rows as JSON in the `data` column. Any query that repeatedly filters, joins, or sorts with `json_extract(data, '$.field')` can become slow once the table grows. For large local lookup tables, matrix tables, sync queues, or frequently joined support tables, create SQLite expression indexes after the data sync completes.
+
+Use an idempotent `action.execute-sql` action at the end of the initial sync action list:
+
+```typescript
+const createIndexes = app.addAction.list({
+  actionId: 'act-create-local-indexes',
+  isSequential: true,
+})
+
+createIndexes.addAction.executeSql({
+  entities: ['default/items', 'default/itemDetails'],
+  statements: [
+    {
+      statement: `
+        CREATE INDEX IF NOT EXISTS idx_items_customer_status
+        ON [default/items] (
+          trim(json_extract(data, '$.customerId')),
+          trim(json_extract(data, '$.status'))
+        )
+      `,
+    },
+    {
+      statement: `
+        CREATE INDEX IF NOT EXISTS idx_itemdetails_parent_sort
+        ON [default/itemDetails] (
+          json_extract(data, '$.parentId'),
+          json_extract(data, '$.sortOrder')
+        )
+      `,
+    },
+  ],
+})
+```
+
+Index only fields that are used in expensive `WHERE`, `JOIN`, or `ORDER BY` clauses. Match the index expression exactly to the query expression, including `trim(...)`, `cast(...)`, casing, and JSON path. Composite indexes should follow the query prefix order: if dropdowns filter by `chainId`, then `v1`, then `v2`, build indexes like `(chainId)`, `(chainId, v1)`, `(chainId, v1, v2)` for the prefixes the app actually uses.
+
+Run the index action after the tables have been synced or seeded. `CREATE INDEX IF NOT EXISTS` makes the action safe to rerun on app load, app restart, or force sync. For diagnostic screens, query `sqlite_master` to confirm the index exists on the device:
+
+```sql
+SELECT name, tbl_name, sql
+FROM sqlite_master
+WHERE type = 'index'
+  AND name NOT LIKE 'sqlite_autoindex%'
+ORDER BY lower(tbl_name), lower(name)
+```
+
+Do not create indexes blindly for every JSON field. Indexes improve reads but add write cost and storage. Start with the fields used by high-cardinality lookups, large matrix filtering, parent-child joins, and list sorting.
+
+## Config table
+
+A `config` table stores integration settings (API base URLs, feature flags, etc.). Always create it in `database.ts` and add an app-scoped datasource with `isDocument: true`.
+
+```typescript
+// database.ts — only config (dynamic)
+app.addDatabase.default.table('config')
+
+// datasources.ts
+app.addDatasource
+  .sqlite({ datasourceId: 'data-select-config', provider: 'dynamic' })
+  .entity('default/config')
+  .query('SELECT id, data FROM [default/config]')
+  .jsonProperties('data')
+  .isDocument(true)
+```
+
+## Table schemas
+
+Always create a `schemas/` folder in the project — even in YAML projects (Jigx doesn't have this in its object model, but it helps track what the data looks like).
+
+Each schema file documents the fields stored by the save action. Infer from:
+- The JSONata save expression (for complex nested saves)
+- The `.data()` record on executeEntity (for simple flat saves)
+- The external API schema if syncing from a remote source
+
+If the user doesn't provide schema guidance, infer it from the form controls and call it out.
+
+```yaml
+# schemas/customers.yaml
+id: string
+
+data:
+  Remote: boolean
+  CustomerName:
+    value: string
+  MainContact:
+    Email:
+      value: string
+    Phone1:
+      value: string
+```
+
+## Signature and media storage — URI vs base64
+
+`signatureField` and `media` components emit a **local file URI** (e.g. `file:///...`). When saving to a dynamic table via `executeEntity`, you have two options:
+
+### Option 1 — store the URI (simple, preferred for device-local flows)
+
+```typescript
+saveActions
+  .executeEntity({ instanceId: 'save-signature' })
+  .dynamicData(ENTITY.FORM_SIGNATURES.FQN, 'save')
+  .data({
+    id: '=$uuid()',
+    signatureImage: '=@ctx.components.mySignature.state.value', // local URI
+  })
+```
+
+Use this when:
+
+- The signature/image stays on the device (PDF generation reads the URI and embeds the file).
+- No cloud sync of the raw image is needed.
+- You control the quote/form lifecycle (no app reinstall risk during an active quote).
+
+### Option 2 — convert to base64/data-uri (portable, heavier)
+
+```typescript
+saveActions
+  .executeEntity({
+    instanceId: 'save-signature',
+    conversions: [{ property: 'signatureImage', from: 'local-uri', to: 'base64' }],
+  })
+  .dynamicData(ENTITY.FORM_SIGNATURES.FQN, 'save')
+  .data({
+    id: '=$uuid()',
+    signatureImage: '=@ctx.components.mySignature.state.value',
+  })
+```
+
+Use this when:
+
+- Signatures/images sync to a remote system (Acumatica attachment, server-side rendering, email).
+- Cross-device rendering is required — a `file://` URI is meaningless on another device/server.
+- You need self-contained rows (base64 travels in JSON, survives file-system changes).
+
+### Option 3 — dual-field storage for forms plus generated PDFs
+
+When a signature field must reload in a Jigx form and also render in a generated
+PDF that may be regenerated on another device, store both values:
+
+- `signatureImage`: the original local URI, used only by `signatureField.initialValue`.
+- `signatureImagePortable`: converted to `data-uri`, used only by the PDF HTML.
+
+```typescript
+saveActions
+  .executeEntity({
+    instanceId: 'save-signature',
+    conversions: [{ property: 'signatureImagePortable', from: 'local-uri', to: 'data-uri' }],
+  })
+  .dynamicData('default/formSignatures', 'save')
+  .data({
+    id: '=$uuid()',
+    signatureImage: '=@ctx.components.mySignature.state.value',
+    signatureImagePortable: '=@ctx.components.mySignature.state.value',
+  })
+```
+
+This avoids the signature field asymmetry trap while keeping generated PDFs
+independent of stale device sandbox paths.
+
+### `conversions` schema
+
+```typescript
+conversions: [
+  { property: 'fieldName', from: 'local-uri' | 'base64' | 'data-uri' | 'buffer', to: 'base64' | 'data-uri' | 'buffer' | 'local-uri' }
+]
+```
+
+When converting back to `local-uri`, you can also pass `localFilename` to control the written file name, and `convertHeicToJpg: true` to transcode iPhone HEIC photos.
+
+### Our convention
+
+**Default to local `file://` URI.** The Jigx PDF renderer supports `file://` sources
+in `<img src>`, and the `signatureField` re-renders its own URI natively on reload.
+No conversion, symmetric save/load, lightweight rows.
+
+Only add `conversions` if the image/signature needs to cross a network boundary
+(Acumatica attachment, server-side rendering, email) where the device-local file
+path is meaningless.
+
+### Signature field asymmetry trap
+
+`signatureField.initialValue` accepts `"base64 or URL"` according to the SDK types,
+but a **raw base64 string** (without the `data:image/png;base64,` prefix) does **not**
+render. The field needs either a file URL (`file://...`) or a data URI
+(`data:image/png;base64,...`).
+
+This creates a trap if you store raw base64 via `conversions: [{from: 'local-uri', to: 'base64'}]`:
+
+1. User draws signature → component emits `file://...`
+2. Save converts to raw base64 `iVBOR...` → stored in row
+3. Re-open form → `initialValue` loads `iVBOR...` → signature field can't render it → blank
+
+You might think "just prepend `data:image/png;base64,` on load". But that creates an
+asymmetry: the component now has a **data URI** in its state, while the row has
+**raw base64**. The save's `when` guard compares them and they're never equal — so the
+save fires every time, and the conversion `{from: 'local-uri', to: 'base64'}` tries to
+convert a data URI as if it were a file URI and either fails or stores garbage.
+
+**Correct approach for device-local forms:** don't convert the field bound back
+into `signatureField.initialValue`. Store the `file://` URI directly. The signature
+field re-renders its own URI natively on reload. If generated PDFs must survive
+cross-device sync, add a separate portable field as described above. For PDF
+generation, use a JS helper (`html.js`) that handles portable values first, then
+current-device `file://` and legacy base64 formats via a simple startsWith check:
+
+```javascript
+const img = sig.signatureImage
+if (img.startsWith('file://') || img.startsWith('http://') || img.startsWith('https://')) {
+  return img
+}
+return `data:image/png;base64,${img}` // legacy fallback
+```
+
+The same trap applies to `media` fields and `avatarField` which use the same
+"base64 or URL" convention.