cognite-data-quality 0.0.1b1__tar.gz

cognite_data_quality-0.0.1b1/.cursor/skills/merge-ready-pr-gate/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: merge-ready-pr-gate
+description: >-
+  Mandatory pre-push checklist for data-quality-validation PRs to avoid CI and
+  Gemini review failures. Use before opening or updating any pull request, after
+  feature work on handlers, orchestrator, deploy, ADRs, or skills.
+disable-model-invocation: true
+---
+
+# Merge-Ready PR Gate
+
+## See also
+
+- CI parity: `release.md` — run `uv run pre-commit run --all-files` before push
+- Lifecycle semantics: `../validation-lifecycle-baseline-incremental/SKILL.md`
+- Pagination invariants: `../rule-engine-sync-state-watermarks/SKILL.md`
+
+## Purpose
+
+Run this gate **before every push** that opens or updates a PR. Gemini and CI catch the same classes of issues; fixing after review wastes time.
+
+## Required commands (same as CI)
+
+From repo root:
+
+```bash
+uv run pre-commit run --all-files
+uv run pytest tests/ -v --no-cov
+```
+
+For focused changes, at minimum:
+
+```bash
+uv run pytest tests/test_<touched_module>.py -v --no-cov
+uv run mypy cognite_data_quality/_function_code/
+```
+
+**Do not push** if pre-commit or mypy fails.
+
+## Code checklist (handlers / orchestrator)
+
+- [ ] **Mypy clean** on all touched files under `cognite_data_quality/_function_code/`
+- [ ] **Pagination resume:** mid-batch timeout must **not** advance cursor/processed count (no silent skip)
+- [ ] **Idempotent re-invoke:** completed stateful runs return early unless `force_rerun: true`
+- [ ] **CDF string parsing:** `space/externalId` uses `split("/", 1)` not `rsplit` (external IDs may contain `/`)
+- [ ] **Regression tests** for every Gemini/ review edge case (timeout, resume, early return)
+- [ ] **Deploy bundle:** new handler modules added to `deploy.py` `include_files` / embedded file list
+- [ ] **Handler dispatch:** `validation_type` in `handler.py` + `tests/test_handler_routing.py`
+
+## ADR / docs checklist
+
+- [ ] ADR status matches reality (Open PR vs Merged)
+- [ ] No stale "interim / planned / remaining work" sections after implementation
+- [ ] Broken anchor links removed or updated
+- [ ] Delivery plan references correct PR numbers
+
+## Review self-pass (Gemini-class questions)
+
+Ask before push:
+
+1. What happens on **Function timeout mid-page**?
+2. What happens on **duplicate invoke** with same `job_run_id`?
+3. Does **mypy** accept all SDK calls (especially `instances.list` kwargs)?
+4. Are **Records** posted with the same contract as steady-state?
+5. Did I run **pre-commit**, not just pytest?
+
+## Anti-patterns that caused PR #290 review pain
+
+| Mistake | Prevention |
+|---------|------------|
+| Pushed without `pre-commit` | Run full pre-commit before every push |
+| Mid-page timeout advanced cursor | Break before `processed_total` / `list_cursor` update |
+| `rsplit` on edge type URI | `split("/", 1)` at first slash |
+| ADR marked "Merged" before merge | Match delivery plan to PR state |
+| No test for review finding | Add test in same commit as fix |
+
+## When updating skills or ADRs
+
+Skills and ADRs are part of the PR — run the same gate. Cross-link new skills from related skills (`See also`).

cognite_data_quality-0.0.1b1/.cursor/skills/pydantic-first-coding/SKILL.md

@@ -0,0 +1,63 @@
+---
+name: pydantic-first-coding
+description: Enforce Pydantic-first implementation patterns across the repository. Use when adding or modifying Python code, handler payload contracts, helper data structures, or validation logic.
+disable-model-invocation: true
+---
+
+# Pydantic-First Coding
+
+## Core rule
+
+Treat Pydantic as the default contract layer for all Python code in this repository.
+
+- Use `BaseModel` for request/response payloads, config objects, derived results, and nested structures.
+- Prefer typed models over raw dictionaries and untyped containers.
+- Use `Field(...)` aliases/defaults/constraints to encode contract details.
+
+## Required patterns
+
+- Add type hints to all function parameters and return values.
+- Use concrete types (`str`, `int`, `list[MyModel]`, `dict[str, int]`) instead of `Any` unless unavoidable.
+- Model nested payloads as nested Pydantic models.
+- Validate inputs at boundaries with `MyModel.model_validate(...)`.
+- Return serialized model payloads with `model_dump(mode="json", by_alias=True)` when producing JSON-like responses.
+
+## Avoid
+
+- `dict[str, Any]` for complex payload contracts.
+- `list[dict[str, Any]]` for structured collections.
+- Ad-hoc key probing when a model can represent the structure.
+- Duplicated validation logic that can be expressed as model field constraints.
+
+## Preferred workflow
+
+1. Define/extend the relevant Pydantic model(s).
+2. Parse raw input once at boundary (`model_validate`).
+3. Use typed attributes in logic.
+4. Build outputs as models.
+5. Serialize at the final boundary only.
+
+## Example direction
+
+Prefer this:
+
+```python
+class ViewRef(BaseModel):
+    space: str
+    external_id: str = Field(alias="externalId")
+```
+
+Instead of this:
+
+```python
+view: dict[str, str]
+space = view.get("space")
+```
+
+## Review checklist
+
+- [ ] Is every complex payload represented by a `BaseModel`?
+- [ ] Are nested objects typed as models instead of nested dicts?
+- [ ] Are aliases/constraints/defaults expressed with `Field(...)`?
+- [ ] Is `Any` minimized and justified?
+- [ ] Are outputs serialized from models at boundaries?

cognite_data_quality-0.0.1b1/.cursor/skills/rule-engine-focus-node-filtering/SKILL.md

@@ -0,0 +1,105 @@
+---
+name: rule-engine-focus-node-filtering
+description: Build RuleEngineResult filters for focusNode and focusNodeInstance safely, including direct-relation normalization for CDF Records API queries. Use when implementing or reviewing rule_engine_result_sync filters, focus node criteria, or focus node instance matching.
+disable-model-invocation: true
+---
+
+# Rule Engine Focus Node Filtering
+
+## See also
+
+- For lifecycle-level idempotent publish semantics (RuleSet/DataProduct), deploy preflight checks, and resolved-version logging patterns, see `../ruleset-lifecycle-state/SKILL.md` (including `## Recent changes`).
+
+## Purpose
+
+Use this skill when changing filter logic for `RuleEngineResult` reads in:
+
+- `cognite_data_quality/_function_code/common/rule_engine_result_sync.py`
+- deploy/config surfaces that feed `filter.focus_node_instances`
+
+Goal: always generate CDF-valid filter payloads for both scalar and list input, including direct-relation values.
+
+## Canonical filter fields
+
+For `RuleEngineResult` listener filters, map config keys to container properties:
+
+- `focus_nodes` -> `focusNode`
+- `focus_node_instances` -> `focusNodeInstance`
+
+Always include `hasData` for the configured records container.
+
+## Required focusNodeInstance normalization
+
+`focusNodeInstance` is a direct relation. Normalize values before building `in` filters:
+
+1. Accept scalar or list input.
+2. Accept either string refs (`space:externalId`) or object refs.
+3. Accept object refs with either key style:
+   - `externalId` (preferred)
+   - `external_id` (normalize to `externalId`)
+4. Emit normalized values as node refs:
+   - `{ "space": "...", "externalId": "..." }`
+
+If a string value is not parseable as `space:externalId`, pass it through unchanged and let higher-level validation/tests catch it.
+
+## Implementation checklist
+
+- [ ] `_append_in_filter(...)` wraps scalar values into lists.
+- [ ] `focusNodeInstance` path calls normalization helper.
+- [ ] Normalizer handles:
+  - [ ] `"space:externalId"` string
+  - [ ] `{"space": "...", "externalId": "..."}`
+  - [ ] `{"space": "...", "external_id": "..."}`
+- [ ] Generated filter uses `in.property = [records_space, records_container, "focusNodeInstance"]`
+
+## Test checklist
+
+In `tests/test_rule_engine_result_sync.py`, ensure coverage for:
+
+- scalar-to-list behavior
+- string direct-relation normalization
+- dict alias normalization (`external_id` -> `externalId`)
+- mixed value lists (string + dict) normalize consistently
+
+Use mock `client.post` assertions against final request body:
+
+- `json["filter"]["and"][...]["in"]["values"]` contains normalized node refs.
+
+Also verify when combined with incremental sync:
+
+- focus-node filters remain structurally stable across cursor pages within a run
+- only allowed dynamic fields change between pages (`cursor`, response-derived state), not filter criteria
+- when no cursor is returned, watermark-only pagination may advance `lastUpdatedTime.gt` between pages
+
+## Review checklist for filter changes
+
+- Confirm output uses only canonical relation object keys (`space`, `externalId`).
+- Confirm snake_case aliases are accepted on input but never emitted.
+- Confirm direct relation values are normalized before request body construction.
+- Add a regression test whenever a new accepted input shape is introduced.
+
+## Do and don't
+
+- Do keep behavior backward compatible for existing YAML patterns.
+- Do keep naming consistent with deploy/config model fields (`focus_node_instances`).
+- Don't send raw non-normalized direct-relation strings to CDF when they can be normalized.
+- Don't introduce alternative key names in output; always emit `externalId`.
+
+## Recurring failure patterns (must not regress)
+
+- **Direct relation sent as raw string**:
+  - `focusNodeInstance` requires node refs, not plain strings
+  - fix: normalize `space:externalId` and dict inputs before request assembly
+- **Input shape drift between deploy config and runtime filter builder**:
+  - fix: keep deploy field typing and runtime normalization intentionally aligned
+- **Falsy scalar values skipped by generic truthiness checks**:
+  - fix: use explicit emptiness checks so valid values are not dropped
+- **Cursor pagination mutating filter boundaries**:
+  - fix: keep filter criteria stable across cursor pages within a run
+
+## Gate before merge
+
+- [ ] Direct relation filters serialize to canonical node refs only
+- [ ] `external_id` accepted on input and normalized to `externalId`
+- [ ] Scalar, list, mixed-shape inputs tested
+- [ ] Falsy-but-valid scalar handling tested where relevant

cognite_data_quality-0.0.1b1/.cursor/skills/rule-engine-listener-blueprint/SKILL.md

@@ -0,0 +1,167 @@
+---
+name: rule-engine-listener-blueprint
+description: Design, implement, and review robust incremental RuleEngineResult listeners end-to-end, including payload parsing, filter construction, state persistence, pagination invariants, runtime safety, testing, and live probe behavior.
+disable-model-invocation: true
+---
+
+# Rule Engine Listener Blueprint
+
+## See also
+
+- For lifecycle-level idempotent publish semantics (RuleSet/DataProduct), deploy preflight checks, and resolved-version logging patterns, see `../ruleset-lifecycle-state/SKILL.md` (including `## Recent changes`).
+- For `test_and_deploy` expectations around resolved RuleSet/DataProduct version propagation, see `../ruleset-lifecycle-state/SKILL.md` (`## test_and_deploy alignment requirements`).
+
+## Purpose
+
+Use this skill when creating or modifying listeners based on incremental RuleEngineResult consumption, especially in:
+
+- `cognite_data_quality/_function_code/handlers/rule_engine_result_sync.py`
+- `cognite_data_quality/_function_code/common/rule_engine_result_sync.py`
+- `tests/test_rule_engine_result_sync.py`
+- `test_and_deploy/test_rule_engine_result_sync_chaining.py`
+
+Goal: build listeners that are deterministic, safe across retries, and production-ready.
+
+## Design contract
+
+A good listener has:
+
+1. **Stable input contract**: robust payload parsing and clear errors.
+2. **Correct incremental semantics**: cursor/watermark progression without data loss or duplicates.
+3. **Safe state persistence**: backward-compatible read/write behavior.
+4. **Runtime control**: bounded pagination and timeout headroom.
+5. **Operational confidence**: targeted tests plus live probe guidance.
+
+## Handler contract (input and validation)
+
+In handler entrypoints:
+
+- Accept both modern and compatibility fields where needed:
+  - `max_records` and fallback `limit`
+  - `source_mode` (`sync` or `filter`)
+  - `initialize_cursor` for first sync page
+- Parse integer-like values defensively:
+  - `page_limit`, `max_records`, initial watermark values
+  - return explicit error payload on invalid input (do not crash)
+- Keep defaults explicit and conservative:
+  - deterministic default page size
+  - deterministic default max records
+
+## Filter construction contract
+
+- Always include `hasData` for the source container.
+- Keep filter-generation centralized in common module helpers.
+- Normalize direct relation fields before request construction:
+  - for `focusNodeInstance`, emit canonical node refs with `space` + `externalId`
+  - accept common input variants (`space:externalId`, `external_id` alias), but normalize output
+
+## Stateful incremental contract
+
+`RuleEngineSyncState`:
+
+- `sync_cursor`: opaque cursor from API
+- `watermark_ms`: logical last-processed timestamp in ms
+
+State persistence in `FunctionValidationState`:
+
+- persist cursor in `syncCursor`
+- persist watermark using overflow-safe encoding in `processedCount` (seconds)
+- decode legacy ms values with backward compatibility
+- treat missing/invalid persisted values as zero
+
+## Pagination invariants (critical)
+
+Within a single paginated run:
+
+1. Append fetched items before stop checks.
+2. Stop only when no progression (cursor unchanged and no watermark advance in watermark-only mode).
+3. Allow empty pages when progression exists.
+4. Preserve query invariants with cursor:
+   - while paging with cursor in `filter` mode, keep `lastUpdatedTime.gt` anchored to run-start watermark.
+5. Support watermark-only progression:
+   - when no cursor is returned, advance query watermark between loop iterations.
+6. Never regress accumulated watermark.
+
+## Runtime and failure handling
+
+- Enforce runtime headroom in loops (leave time for teardown/state save).
+- Use specific exceptions or controlled error returns.
+- Keep behavior idempotent on re-runs.
+- Avoid hidden mutable global state.
+
+## Known Environment Constraints
+
+Known constraints observed in live environments should be treated as first-class listener requirements:
+
+- Filter endpoint may require `lastUpdatedTime` in every `/records/filter` request.
+- Some environments enforce max filter interval windows (for example, 7 days).
+- Unbounded first-run filter bootstrap (watermark `0`) can fail with API `400`.
+
+Implementation guidance:
+
+- In `filter` mode, always send `lastUpdatedTime`.
+- Use bounded bootstrap watermark defaults for first run when no persisted state exists.
+- Keep bootstrap windows explicit in handler and `test_and_deploy` scripts so local/live behavior matches.
+
+## Live probe and retry rules
+
+For `test_and_deploy` scripts invoking stateful listeners:
+
+- Do **not** retry stateful listener consumption in tight loops.
+- Add short indexing delay after ingestion before first listener call.
+- If retries are needed, retry read-only verification queries, not listener state advancement.
+
+## Test checklist (required)
+
+Ensure targeted tests cover:
+
+- input parsing robustness for integer-like fields and watermark initialization
+- watermark monotonicity and no-regression behavior
+- cursor paging progression and terminal no-progress stop
+- empty-page-with-progress continuation
+- cursor-mode anchored query watermark in `filter` pagination
+- watermark-only pagination updates query watermark between pages
+- direct relation filter normalization (`focusNodeInstance`) for string and dict forms
+- persisted state encode/decode compatibility (seconds + legacy ms)
+
+## Review checklist before merge
+
+- [ ] Query invariants preserved across cursor pages
+- [ ] Watermark accumulation still advances correctly
+- [ ] No stateful retry anti-patterns in live scripts
+- [ ] Regression tests added for each fixed edge case
+- [ ] Lint/type checks pass for changed files
+- [ ] Naming and structure align with existing Rule Engine modules
+
+## Implementation do and don't
+
+- Do keep state transition logic explicit and localized.
+- Do separate request-query state from accumulated returned state where needed.
+- Do write tests that assert exact outgoing request bodies for filter correctness.
+- Don't mutate query boundaries mid-cursor page sequence.
+- Don't rely on eventual consistency without explicit indexing delay in live probes.
+- Don't resolve review comments without locking the behavior with regression tests.
+
+## Prevent recurrence checklist
+
+Before closing review threads on listeners, confirm all of these:
+
+- **Cursor lifecycle**
+  - terminal response without cursor clears persisted cursor
+  - cursor->none transition does not count as progression
+- **Mode-aware progression**
+  - `sync` mode progression is cursor-only
+  - `filter` mode progression can use watermark in no-cursor path
+- **Filter request validity**
+  - `lastUpdatedTime` always present for filter requests
+  - first-run filter bootstrap is bounded for environment constraints
+- **Input parsing correctness**
+  - numeric defaults use `is None`, not `or`
+  - explicit `0` survives parsing and reaches validation
+- **Filter value handling**
+  - explicit empty checks in filter helpers (avoid dropping falsy valid values)
+  - direct relation normalization is validated by request-body assertions
+- **Live script alignment**
+  - `test_and_deploy` payload semantics match handler semantics
+  - stateful listener calls are not retried in tight loops
+  - mode-specific options are applied conditionally (`initialize_cursor` for sync only)