ecoportal-api-graphql 1.3.5 → 1.3.9

data/.ai-assistance/code/ecoPortal_architecture/06_search_and_filters.md

@@ -0,0 +1,228 @@
+# EcoPortal Search and Filters
+
+---
+
+## Two Search Modalities
+
+### 1. Org Search — `currentOrganization { pages(...) }`
+
+**Data source:** PostgreSQL/MongoDB — full DB render
+**Returns:** `PageUnion` (BasicPage | PhasedPage) — complete data
+**Performance:** ~3 seconds per page (API-layer assembly)
+**Supports archived:** Yes (via `searchConf` state filter)
+**Has field IDs:** Yes — use for pre-update fetch
+**patchVer:** Yes — required for mutations
+
+Use when you need full field data to build mutation inputs.
+
+**Arguments:**
+- `searchConf: Search` — filter/sort/query
+- `templatesOnly: Boolean` — restrict to templates
+- `showHiddenData: Boolean` — include hidden fields
+- `first/last/after/before` — cursor pagination
+
+**Key `searchConf` filter for register-scoping within org search:**
+```json
+{ "operation": "register_filter", "params": { "ids": ["REGISTER_ID"] } }
+```
+This scopes the org search to a register without switching to `previewPages`.
+
+### 2. Register Search — `currentOrganization { register(id:) { previewPages(...) } }`
+
+**Data source:** Elasticsearch — search index snapshot
+**Returns:** `PreviewPage` — metadata only (id, name, state, locations, dataFields as previews)
+**Performance:** Sub-second (ES-powered)
+**Supports archived:** Yes — `includeArchived: Boolean` parameter (default false)
+**Has field IDs:** NO — cannot build mutation inputs from this
+**patchVer:** NO
+
+Use for fast metadata lookup (find page IDs, check existence, list pages).
+
+**Additional arguments:**
+- `presetViewId: ID` — apply a register PageView preset configuration
+- `searchConf: Search`
+- `includeArchived: Boolean`
+
+**Field name:** `previewPages` (not `pages`) — common confusion point.
+
+---
+
+## SearchConf Structure
+
+`searchConf` is a `Hash` type (`Search` in the schema) — NOT a typed GraphQL input.
+Engineering has backend classes but they are not exposed as GraphQL types.
+
+```json
+{
+  "filters": [ <filter_object>, ... ],
+  "sorters": { "key": "created_at", "direction": "asc" },
+  "query":   "optional full-text search string"
+}
+```
+
+**Note on sorters:** Can be a single object OR an array. Key field is `key` (not `fieldName`).
+
+---
+
+## Filter Operations
+
+**ALL operation names are snake_case strings.** If CamelCase is used (e.g. `ExactFilter`),
+the server returns NO error AND NO results — completely silent failure. This is one of
+the hardest bugs for customers to diagnose.
+
+**`filters` should always be an Array.** The server normalises a single object to an
+array internally, but always use array form for consistency.
+
+### `exact_filter`
+
+Equality match on a field.
+
+```json
+{ "operation": "exact_filter", "params": { "key": "external_id", "value": "EXT-001" } }
+```
+
+Common keys: `external_id`, `state`, `creator_id`, `template_id`
+
+### `date_filter`
+
+Date range filter.
+
+```json
+{
+  "operation": "date_filter",
+  "params": {
+    "key":       "updated_at",
+    "gte":       "2025-01-01T00:00:00Z",
+    "lte":       "2025-12-31T23:59:59Z",
+    "time_zone": "UTC"
+  }
+}
+```
+
+`gte` = lower bound, `lte` = upper bound. Both optional.
+
+**v2 format note:** APIv2 used `lbound`/`ubound` instead of `gte`/`lte`. The
+eco-helpers compat layer translates these automatically.
+
+### `register_filter`
+
+Scope org search to specific registers (org search only — not applicable to `previewPages`).
+
+```json
+{ "operation": "register_filter", "params": { "ids": ["REGISTER_ID"] } }
+```
+
+### `and_filter`
+
+Explicit AND group (top-level `filters` array is already implicit AND).
+
+```json
+{
+  "operation": "and_filter",
+  "params": {
+    "filters": [
+      { "operation": "exact_filter", "params": { "key": "external_id", "value": "X" } },
+      { "operation": "date_filter",  "params": { "key": "updated_at",  "gte": "..." } }
+    ]
+  }
+}
+```
+
+### `or_filter`
+
+OR group.
+
+```json
+{
+  "operation": "or_filter",
+  "params": {
+    "filters": [
+      { "operation": "exact_filter", "params": { "key": "external_id", "value": "A" } },
+      { "operation": "exact_filter", "params": { "key": "external_id", "value": "B" } }
+    ]
+  }
+}
+```
+
+### `tag_filter` (v2 pass-through)
+
+Filter by location tags. Used by APIv2 eco-helpers scripts. Passes through to the ES
+backend unchanged. Same ES backend as GraphQL register search.
+
+```json
+{
+  "type": "tag_filter",
+  "tags": ["Auckland", "Wellington"],
+  "mode": "any",
+  "negate": false,
+  "key": "tags"
+}
+```
+
+---
+
+## SearchConf Builder (Gem)
+
+The gem provides `Input::SearchConf` for building filters programmatically:
+
+```ruby
+# Find by externalId
+conf = SearchConf.new.filter(SearchConf::Exact.new(:external_id, 'EXT-001'))
+
+# Register-scoped date range
+conf = SearchConf.new
+  .filter(SearchConf::Register.new('REG_ID'))
+  .filter(SearchConf::DateRange.new(:updated_at, gte: '2025-01-01', time_zone: 'UTC'))
+  .sort(:created_at, :asc)
+
+# OR filter (multiple externalIds)
+conf = SearchConf.new.filter(
+  SearchConf::Or.new(
+    SearchConf::Exact.new(:external_id, 'EXT-001'),
+    SearchConf::Exact.new(:external_id, 'EXT-002')
+  )
+)
+
+# Parametrize (same search, different value)
+base = SearchConf.new.filter(SearchConf::Exact.new(:external_id, 'X'))
+copy = base.with(external_id: 'Y')  # original unchanged
+
+# Use with queries
+org.pages(searchConf: conf.to_h, first: 10)
+```
+
+**Filter classes:** `SearchConf::Exact`, `SearchConf::DateRange`, `SearchConf::Register`,
+`SearchConf::And`, `SearchConf::Or`
+
+---
+
+## Known Filterable Field Keys (confirmed from production queries)
+
+| key | Type | Operations |
+|-----|------|------------|
+| `external_id` | string | `exact_filter` |
+| `updated_at` | datetime | `date_filter` |
+| `created_at` | datetime | `date_filter` (likely) |
+| `state` | enum | `exact_filter` (key: active/archived/draft) |
+| `location_ids` | ID array | likely `in_filter` or similar |
+| `creator_id` | ID | `exact_filter` |
+| `template_id` | ID | `exact_filter` |
+
+**Note:** The `fieldName` convention in some Confluence docs uses this same key concept.
+The keys are snake_case corresponding to indexed ES document fields — NOT necessarily
+the GraphQL field names.
+
+---
+
+## Future: Fluent DSL (Planned)
+
+A more ergonomic API using Ruby's `&` (AND) and `|` (OR) operators is planned:
+
+```ruby
+Search[:external_id].eq('EXT-001') & Search.in_register('REG_ID')
+Search[:state].is(:active) | Search[:state].is(:draft)
+(Search[:state].is(:active) | Search[:state].is(:draft)) & Search.in_register('REG_ID')
+```
+
+This is operator-based (familiar to Arel/Sequel users), avoids ambiguity in chaining,
+and will support AI-assisted filter generation. See `search-filter-builder/` project.

data/.ai-assistance/code/ecoPortal_architecture/07_data_fields.md

@@ -0,0 +1,197 @@
+# EcoPortal Data Fields
+
+---
+
+## Overview
+
+Data fields are the form inputs within a page. Each field is an INSTANCE of a field
+definition from the template. Field instances have:
+- A MongoDB ObjectId (`id`) assigned on page creation — NOT predictable from template
+- The field type (`__typename` in GraphQL)
+- Label (the question/field name shown to users)
+- Value(s) — the user's response
+- `ref` — the ES index key (Field ID / System ID future)
+- `deindex: Boolean` — if true, field is excluded from Elasticsearch index
+- Linked field configuration (cross-reference from another register)
+
+---
+
+## All Field Types (`DataFieldUnion`)
+
+The GraphQL `DataFieldUnion` has 20+ concrete types:
+
+| GraphQL type | eco-helpers type | Description |
+|---|---|---|
+| `PlainText` | `:plain_text` | Single or multi-line text |
+| `RichText` | `:rich_text` | HTML rich text editor |
+| `Date` | `:date` | Date/datetime picker |
+| `Number` | `:number` | Numeric input |
+| `Gauge` | `:gauge` | Numeric with min/max (risk matrix output) |
+| `Select` | `:select` | Single or multi-select dropdown |
+| `Checklist` | `:checklist` | Checkbox list (each item: label + checked) |
+| `Signature` | `:signature` | Digital signature capture |
+| `TagField` | `:tag_field` | Location tag selector |
+| `People` | `:people` | Person selector (linked to PersonMember) |
+| `ContractorEntities` | `:contractor_entities` | Contractor selector |
+| `CrossReference` | `:cross_reference` | Reference to pages in another register |
+| `Geo` | `:geo` | Geographic coordinates + address |
+| `ImageGallery` | `:image_gallery` | Image upload gallery |
+| `File` | `:file` | File attachment |
+| `Mailbox` | `:mailbox` | Email inbox linked to the field |
+| `ActionsList` | `:actions_list` | Embedded actions list |
+| `Law` | `:law` | Legal obligation snippets |
+| `AiSummary` | `:ai_summary` | AI-generated summary field |
+| `EmbeddedStructure` | `:embedded_structure` | Embedded reporting structure |
+
+---
+
+## DataFieldInput Structure
+
+When updating or adding field values, each field uses a typed input within `DataFieldInput`:
+
+```json
+{
+  "plainText":   { "id": "FIELD_ID", "value": "Job description text" },
+  "richText":    { "id": "FIELD_ID", "content": "<p>HTML content</p>" },
+  "date":        { "id": "FIELD_ID", "value": "2024-11-25T08:30:00+13:00" },
+  "number":      { "id": "FIELD_ID", "value": 42.5 },
+  "gauge":       { "id": "FIELD_ID", "value": 14.0 },
+  "select":      { "id": "FIELD_ID", "options": [{ "id": "OPT_ID", "selected": true }] },
+  "people":      { "id": "FIELD_ID", "peopleIds": ["PERSON_ID_1", "PERSON_ID_2"] },
+  "geo":         { "id": "FIELD_ID", "address": "...", "coordinates": { "lat": -36.86, "lon": 174.76 } },
+  "checklist":   { "id": "FIELD_ID", "items": [{ "id": "ITEM_ID", "checked": true }] },
+  "locationField": { "id": "FIELD_ID", "locationIds": ["LOC_ID"] }
+}
+```
+
+Only ONE field type key is used per `DataFieldInput` object.
+
+---
+
+## DataFieldOneToManyInput
+
+Used in both `createPageFromTemplate` and `updatePage`:
+
+```json
+{
+  "additions": [<DataFieldInput>, ...],
+  "updates":   [<DataFieldInput>, ...],
+  "deletions": ["FIELD_ID_TO_DELETE", ...]
+}
+```
+
+- **`updates`** — change values of existing fields (require their current `id`)
+- **`additions`** — add new field data (for fields that exist in template but have no instance data yet)
+- **`deletions`** — remove field instances
+
+**For `createFromTemplate`:** Use `updates` — the `buildFromTemplate` response gives
+instance field IDs, and `updates` sets their values on creation.
+
+---
+
+## Field Access in the Gem
+
+### Getting fields from a page
+
+```ruby
+page = graphql.pages.get(page_id)  # fetches CommonPageUnion with $fields: true
+
+# As DataField::Collection
+fields = page.components  # or page.field_collection
+
+# Filter by type
+plain_texts = fields.get_by_type(:plain_text)
+selects     = fields.get_by_type('Select')
+
+# Find by label
+field = fields.get_by_name('Summary')
+
+# Access raw docs (v2 compat)
+fields.doc  # → [raw_hash, ...]
+```
+
+### Modifying fields
+
+```ruby
+# PlainText
+field = page.components.get_by_name('Summary')
+field.value = 'New summary text'
+
+# Select — by option name, value, or id
+field = page.components.get_by_name('Status')
+field.select_option('Active')  # deselects all others
+
+# People
+field = page.components.get_by_name('Assigned To')
+field.people_ids = ['person_id_1', 'person_id_2']
+
+# Date (ISO8601 string)
+field = page.components.get_by_name('Due Date')
+field.value = '2025-06-30T09:00:00+12:00'
+
+# Checklist
+field = page.components.get_by_name('Checklist')
+field.check('Step A')
+field.check('Step B', checked: false)
+
+# Rich text (HTML)
+field = page.components.get_by_name('Description')
+field.content = '<p>Updated description</p>'
+```
+
+### Adding a new field (additions)
+
+```ruby
+# When a field exists in the template but has no data in this instance:
+draft = graphql.pages.get_new(template_id)
+draft.components.add(id: 'f1_from_draft', type: 'PlainText') { |f| f.value = 'X' }
+```
+
+---
+
+## Stage-Level Field Access
+
+For PhasedPage, fields are within stages → sections → fields:
+
+```ruby
+# Get stage by name
+stage = page.stages['Risk Assessment']
+
+# Stage sections
+stage.sections.each do |section|
+  puts "Section: #{section.heading}"
+  section.components.each { |f| puts "  Field: #{f.label}" }
+end
+
+# All fields in stage (flat)
+stage.components.get_by_name('Hazard Description').value = 'Exposure to...'
+```
+
+---
+
+## Field Visibility: deindex and Hidden
+
+- `deindex: Boolean` — if true, field value is NOT sent to Elasticsearch. Used for
+  sensitive data. Affects: search, filters, analytics, exports.
+- Fields can be hidden (template configuration). `showHiddenData: true` in queries
+  includes hidden fields in the response.
+
+---
+
+## Field Reference (ref) — ES Index Key
+
+The `ref` property identifies a field in the search index (ES). It is:
+- Composed of: `<type_shorthand>.<hash_of_label>`
+- Used in: sorters (`key:` field), filters, chart configurations, linked field config
+- Different from the MongoDB instance `id`
+- Breaks if the field label changes (hash of label changes)
+
+**Formula:** `Ecoportal::API::Common::Content::StringDigest` in `ecoportal-api-v2`
+
+**Examples:**
+- Plain text field "Job Number": `plain_text.job_nu` (approximate hash)
+- Date field "Due Date": `date.due_da`
+
+**Known issue:** If a field was added via a tech script without copying from the template,
+its `ref` may be unique per instance rather than consistent across all instances of the
+same template. This causes analytics issues (missing data, duplicate columns in exports).

data/.ai-assistance/code/ecoPortal_architecture/08_stages_sections.md

@@ -0,0 +1,243 @@
+# EcoPortal Stages and Sections
+
+---
+
+## Stage Lifecycle
+
+### Stage States (`StageStateEnum`)
+
+```
+pending → inprogress → complete
+```
+
+- **`pending`** — Stage not yet started. No fill-in task exists yet.
+- **`inprogress`** — Stage is active. A fill-in task (`CompletePage`) exists and is
+  assigned to someone (per workflow rules). The assignee can enter data.
+- **`complete`** — All required tasks are done. Stage is locked.
+
+**Transition triggers:**
+- `pending → inprogress`: Previous stage completes (its task is done/signed off)
+- `inprogress → complete`: Fill-in task completed, OR review task signed off (if review configured)
+
+### Active vs Started
+
+- `active: Boolean` — Stage is currently active for input (the "current" stage)
+- `started: Boolean` — Stage has had at least some input (includes all past stages)
+
+For a PhasedPage: typically ONE stage is `active` at a time. Previous stages are
+`started` but not `active`. Future stages are neither.
+
+### Stage-Specific Submit
+
+When submitting a stage via GraphQL, ALWAYS include `stageId`:
+
+```graphql
+mutation SubmitStage($input: UpdatePageInput!) {
+  updatePage(input: $input) {
+    item { ...PhasedPageFields }
+    errors { details fullMessages }
+  }
+}
+```
+
+```json
+{
+  "input": {
+    "id": "PAGE_ID",
+    "patchVer": 5,
+    "stageId": "STAGE_ID",
+    "submit": true,
+    "dataFields": {
+      "updates": [
+        { "plainText": { "id": "FIELD_ID", "value": "Final value" } }
+      ]
+    }
+  }
+}
+```
+
+Without `stageId`, the submit may not correctly associate with the right stage and
+may not trigger the expected task assignments in the next stage.
+
+**Why a stage is mandatory (verified — Oscar, 2026-06-30):** stage tasks, and the
+permissions granted through them, are **stage-scoped**. A people field can grant the user
+attached to it permission to *only the stage where they were attached* — so a save / submit /
+sign-off MUST be executed against that specific stage as the target. The web/mobile front-ends
+always operate within a stage for this reason. Inferring a "current/active" stage from the
+page is therefore not just unreliable, it is semantically wrong: on a multi-stage page it may
+name a different stage than the one the work was done on. The submit must carry the stage the
+caller actually worked on. ⇒ **submit / sign_off without a target stage is invalid.**
+
+### Gem enforcement & stageId resolution (compat layer)
+
+The gem treats "a submit must name its stage" as a hard invariant:
+
+- `Input::Page::Update.from_model` **raises** (`ArgumentError`) when `submit: true` or a stage
+  `task` is requested with no `stage_id`. This replaced a silent failure: the operation hash's
+  nil-guard used to *drop* a nil `stageId`, so a stage-less submit went out unnoticed (the
+  act-gov TOOCS bug, 2026-06-30). A plain field update / publish needs no stage and is exempt.
+- `Compat::Pages#compat_stage_id` resolves the target most-explicit-first:
+  1. **pinned** — `submit!(stage_id:)` / `sign_off!(stage_id:)` from the script;
+  2. **active** — `page._compat_active_stage_id`, recorded when a `Compat::StageView` is taken
+     over the page (`pages.get(id, stage_id:)` or eco-helpers `OozeRedirect#stage`). This is the
+     reliable source on the OozeSamples path, where the base loop enqueues and updates the FULL
+     page (first-wins by id — the StageView is never the queued object), so the chosen stage must
+     live on the page itself;
+  3. **current** — `Phased#current_stage_id` server-state inference (`state=='inprogress'` /
+     `active==true`). Last resort only, and guarded: if it yields nil the `from_model` guard fires.
+
+So the canonical way to submit a specific stage from a script is to work *through a stage view*
+(`with_stage` → `stage()` → `StageView`), which records the stage; the later `target.submit!`
+then resolves to it automatically.
+
+---
+
+## Stage Permissions
+
+Each stage can have restricted access:
+- **User group restrictions** — only members of specified groups can edit the stage
+- **People field access** — if a stage has a people field configured to "grant stage access",
+  the people assigned to that field get editing access to that stage only
+
+**Key insight:** The configuration enum on a people field determines access scope:
+- `stage_only` — grant access only to the stage where the people field is embedded
+- `page_wide` — grant access to the entire page
+
+**GraphQL behaviour:** If a user doesn't have access to a stage, the stage is simply
+ABSENT from the response (`stages`, `stagesIndex`, `activeStages`). No error is thrown.
+This is different from APIv2 which returned explicit access errors per stage.
+
+**Scripting implication:** Scripts should check whether an expected stage is present
+in the response rather than catching errors. If a stage is missing, the authenticated
+user may not have access to it.
+
+---
+
+## Section Types (HasSectionsInterface)
+
+Both BasicPage (directly) and Stage (within PhasedPage) implement `HasSectionsInterface`.
+
+### `ContentSection`
+
+Standard section — one column of fields:
+```graphql
+fragment on ContentSection {
+  id
+  heading
+  dataFields { ...dataField }
+}
+```
+
+### `SplitSection`
+
+Two-column section (left + right panels):
+```graphql
+fragment on SplitSection {
+  id
+  heading
+  leftDataFields  { ...dataField }
+  rightDataFields { ...dataField }
+}
+```
+
+When accessing a split section's fields, always merge left + right:
+```ruby
+section.components  # → DataField::Collection merging left + right
+```
+
+---
+
+## Stage-Level Field Access (PhasedPage)
+
+In GraphQL, all stage data is returned in one request via `CommonPageUnion`.
+The response has stages → sections → fields.
+
+```ruby
+# Gem access pattern
+page = graphql.pages.get(page_id)
+
+# All stages
+page.stages                          # → StageCollection
+page.stages['Risk Assessment']       # → Stage (by name or ID)
+page.stages.ordered                  # → [Stage, ...] sorted by ordering
+
+# Stage contents
+stage = page.stages['Risk Assessment']
+stage.sections                       # → SectionCollection
+stage.components                     # → DataField::Collection (all fields in stage)
+stage.sections.get_by_type(:content_section)
+stage.sections.get_by_heading('Details')
+
+# Field access in a stage
+stage.components.get_by_name('Hazard Description').value = 'Exposure to...'
+stage.components.get_by_type(:select).find { |f| f.label == 'Risk Level' }.select_option('High')
+```
+
+---
+
+## APIv2 vs GraphQL Stage Rendering Comparison
+
+| Concern | APIv2 | GraphQL |
+|---------|-------|---------|
+| Request count for PhasedPage | One per stage | One for entire page |
+| Stage data isolation | Strict — each request scoped to one stage | Full page returned, client filters |
+| Stage-specific mutations | Via stage_id on URL | Via `stageId:` in `UpdatePageInput` |
+| Permission handling | Access error per forbidden stage | Forbidden stages absent from response |
+| Template stages | Full render | Full render |
+
+**Compat layer approach:**
+`graphql.pages.get(page_id, stage_id: 's1')` returns a `StageView` — a client-side
+wrapper that shows only the specified stage's sections and components, while delegating
+all other page accessors (id, name, patchVer, as_update, etc.) to the full page.
+This matches what v2 scripts expect from a per-stage fetch.
+
+---
+
+## StageIndex (Lightweight Stage Summary)
+
+For displaying workflow progress without loading full stage content:
+
+```graphql
+fragment PhasedPageFields on PhasedPage {
+  stagesIndex {
+    id
+    name
+    ordering
+    state
+    started
+    hasCompleteTask
+    hasReviewTask
+  }
+}
+```
+
+`hasCompleteTask` — whether a fill-in task is configured for this stage
+`hasReviewTask` — whether a review/sign-off task is configured for this stage
+
+---
+
+## Task Types on Stages
+
+| Task type | GraphQL type | Trigger | When complete |
+|-----------|-------------|---------|---------------|
+| Fill-in | `CompletePage` | Stage becomes active | Moves to review (if configured) or completes stage |
+| Review/sign-off | `ReviewPage` | Fill-in task completed | Completes stage → activates next stage |
+
+Task fields: `id`, `complete`, `completedAt`, `completedBy`, `due`, `overdue`, `users`
+
+**Stage close-out sequence:**
+1. Fill-in task assigned to user(s) (per workflow rules)
+2. User submits: `updatePage(stageId:, submit: true)` triggers `CompletePage` completion
+3. If review configured: `ReviewPage` task created and assigned
+4. Review approved: stage state → `complete`, next stage activated
+5. Next fill-in task created per workflow rules
+
+**Scripting a stage submit:**
+```ruby
+input = Input::Page::Update.from_model(page,
+  stage_id: 'STAGE_ID',
+  submit:   true
+)
+# Input will include stageId + submit even if no page/field changes
+Builder::Page.new(client).update(input: input)
+```