@groupby/ai-dev 0.5.13 → 0.5.15

package/teams/fhr-andromeda/instructions/modules/workflow-scripts.md

@@ -0,0 +1,119 @@
+# CLAUDE.md - workflow-scripts
+
+**Agent type:** `Python Dev - workflow-scripts`
+
+This file owns module-specific context for Python workflow-step logic. Read it after the root
+harness files. Inspect current source for exact scripts, utilities, and Docker layout.
+
+---
+
+## What This Module Is
+
+`workflow-scripts` contains Python code executed by Argo workflow steps. Scripts prepare ETL
+jobs, coordinate runtime execution, move data, publish results, report metrics, and integrate
+with DQSys.
+
+Workflow definitions and cluster templates are in `andromeda-argo-deployments`.
+
+---
+
+## Architecture Constraints
+
+- Each script must have one responsibility and behave as a workflow-step entrypoint.
+- Configuration comes from environment variables or from `hop_job_config` via
+  `apply_hop_job_config()` from `arg_utils.py`.
+- Logging must use `logger_config.py`; never use `print()` in production code.
+- Argument parsing must use `arg_utils.py`; never parse raw command-line arguments directly.
+- Scripts that receive `hop_job_config` must call `apply_hop_job_config()` to merge its fields
+  onto the argument namespace — they do not re-derive identity, paths, or config from scratch.
+- Fatal failures must be explicit, logged with `exc_info=True`, and exit non-zero.
+- Scripts must not silently skip external-system failures.
+- Do not encode workflow-template assumptions that belong in `andromeda-argo-deployments`.
+
+### Mandatory Script Structure
+
+```python
+# imports
+# logger = setup_logging(...)
+# constants
+# helper functions
+def main() -> None: ...
+if __name__ == "__main__":
+    try:
+        main()
+    except Exception:
+        logger.error("Fatal error", exc_info=True)
+        sys.exit(1)
+```
+
+---
+
+## Shared Utilities
+
+All scripts use these utilities from the same directory:
+
+| Utility | Purpose |
+|---|---|
+| `arg_utils.py` | CLI argument parsing and `apply_hop_job_config()` |
+| `logger_config.py` | Structured logger setup |
+| `db_utils.py` | Database connections, DQ data models, credential helpers |
+| `directory_utils.py` | S3 path and local directory helpers |
+| `file_utils.py` | File I/O helpers |
+
+---
+
+## Integration Boundaries
+
+This module integrates with:
+
+- S3 for ETL inputs, outputs, and status artefacts;
+- Kafka for completion signals and streaming dispatch;
+- Kubernetes for runtime orchestration;
+- `data-quality` REST API (`dq_execution.py`, `dq_custom_checks.py`);
+- DQSys PostgreSQL database (`populate_dashboard.py`, `update_etl_dashboard.py`);
+- `custom-etl` GitHub repository (`deploy_custom_etl_package.py`).
+
+When changing script arguments, environment variables, API payloads, or output artefacts,
+identify workflow-template and consumer impact explicitly.
+
+---
+
+## Testing Expectations
+
+- Tests live in `src/test/python/test_<script_name>.py`.
+- Mock all external systems (S3, Kafka, Kubernetes, PostgreSQL); no real connections in tests.
+- Cover required configuration validation, happy path, external failures, and exit behaviour.
+- New scripts need tests for argument/env validation and fatal-error handling.
+
+---
+
+### Grafana dashboard dependency
+
+`realtime_metrics_daemon.py` and `metrics_exporter.py` push `argo_etl_*` metrics to the
+Prometheus Pushgateway. These metrics are consumed by the Grafana ETL dashboard defined in the
+**`andromeda-argo-deployments`** repo:
+
+```
+misc/attraqt-grafana-dashboards/fms-dev/etl2-overview-dashboard.yaml
+```
+
+**Any change to a metric name, label key, label value, or aggregation in either script must be
+accompanied by a corresponding update to that dashboard in the same cross-repo set of PRs.**
+Specifically check:
+- PromQL `expr` lines that filter on `step_name=`, `workflow_suffix=`, or use `max by (...)` on
+  label dimensions you have changed.
+- `legendFormat` strings referencing those labels.
+- Column-visibility settings (`"<label>": true/false`) in table panels.
+- Template variables whose `query` or `options` reference label values that have changed.
+
+---
+## Context to Confirm When Needed
+
+Stop and ask, or inspect current source, when a task depends on:
+
+- exact environment variable names;
+- S3 bucket or key conventions;
+- Kafka topic conventions;
+- workflow-step input/output contracts;
+- `data-quality` API or database contracts;
+- Docker image ownership for a script.

package/teams/fhr-andromeda/skills/fhr-pr/SKILL.md

@@ -0,0 +1,71 @@
+---
+name: fhr-pr
+description: Generate a filled PR description for any branch in the fhr-apache-hop repository using the project's PULL_REQUEST_TEMPLATE.md and AI harness context.
+---
+
+# fhr-pr — Generate PR Description
+
+Generate a filled PR description for any branch in the fhr-apache-hop repository using the project's PULL_REQUEST_TEMPLATE.md and AI harness context.
+
+## When to invoke
+
+When the user says "write a PR", "create a PR description", "fill in the PR template", or similar, and they are working in the fhr-apache-hop repo.
+
+## Instructions
+
+### Step 1 — Gather context
+
+Read the following in order:
+1. `.github/PULL_REQUEST_TEMPLATE.md` — the template to fill
+2. `CLAUDE.md` (root) — to understand the repo and module map
+3. `.ai/ai-instructions.md` — for TDD phase definitions referenced in the template
+
+Then collect the changes to describe:
+- Run `git log origin/master..HEAD --oneline` to list commits on the branch
+- Run `git diff origin/master --stat` to list changed files with their modules
+- Read each changed file (or its diff via `git diff origin/master -- <file>`) to understand what changed and why
+
+If a Jira ticket ID is mentioned by the user or visible in the branch name (`git rev-parse --abbrev-ref HEAD`), extract it.
+
+### Step 2 — Determine agent types
+
+From the changed file paths, identify which modules are in scope and look up the agent type from the relevant module `CLAUDE.md` header. Modules and their agent types:
+- `hop-dev-platform/` → `Java Dev - hop-dev-platform`
+- `hop-dev-platform-frontend/` → `Angular Dev - hop-dev-platform-frontend`
+- `data-quality/` → `Java Dev - data-quality`
+- `data-quality-frontend/` → `Angular Dev - data-quality-frontend`
+- `workflow-scripts/` → `Python Dev - workflow-scripts`
+- `fhr-custom-transforms/` → `Java Dev - fhr-custom-transforms`
+
+### Step 3 — Fill the template
+
+Fill every section of `.github/PULL_REQUEST_TEMPLATE.md`:
+
+**AI Generation Info**
+- Jira ticket: extract from branch name (e.g. `FHR-1234-...`) or ask the user
+- Agent type: from Step 2 — list all modules in scope
+- AI tool: `Claude (claude-sonnet-4-6)`
+- Spec referenced: Jira ticket title if known, otherwise "inline"
+
+**TDD Summary**
+- Tests written first: answer honestly based on what you observe (test files vs production file commit order)
+- Test files: list all `*.spec.ts`, `*Test.java`, `*Spec.groovy`, `test_*.py` files changed
+- Production files: all other changed source files
+
+**What was generated**
+Write one concise paragraph or bullet group per logical change area. Group by module. Each entry should say what changed and why (the user-visible outcome or bug fixed), not just which file was touched.
+
+**What was reviewed / changed after generation**
+Note any manual corrections, edge-case decisions, or departures from the generated output. If nothing was changed post-generation, write: "None — generated output accepted as-is"
+
+**Review notes**
+Flag anything that warrants careful human review: async error paths that are fire-and-forget, schema migrations, security surface changes, cross-repo impact, S3/Kafka contracts. If none, write: "None identified"
+
+**Checklist**
+Check each item based on what you observed. Leave unchecked items that the user must verify locally (build success, TDD order).
+
+### Step 4 — Save and present
+
+Write the filled description to `<TICKET-ID>-pr.md` in the repo root (e.g. `FHR-7204-pr.md`). If no ticket ID is known, use `pr-description.md`.
+
+Present the file to the user.

package/teams/fhr-andromeda/skills/fhr-review/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: fhr-review
+description: Review changed files on the current branch against the project's code-review checklist, producing a structured list of BLOCK, WARN, and NOTE findings.
+---
+
+# fhr-review — Code Review Against Project Checklist
+
+Review changed files on the current branch against the project's `.ai/code-review.md` checklist, producing a structured list of BLOCK, WARN, and NOTE findings.
+
+## When to invoke
+
+When the user says "review this", "review the changes", "run a code review", or asks for a checklist-based review of what's on the branch.
+
+## Instructions
+
+### Step 1 — Read the review criteria
+
+Read in order:
+1. `.ai/code-review.md` — the checklist with BLOCK/WARN/NOTE severity rules
+2. `CLAUDE.md` (root) — module map, to route module-specific checks
+3. The relevant module `CLAUDE.md` for each changed module — for module-specific constraints
+
+### Step 2 — Collect the diff
+
+Run `git diff origin/master --stat` to identify changed files and their modules.
+
+For each changed file, read it fully (or use `git diff origin/master -- <file>` for the diff). Group files by module.
+
+### Step 3 — Apply the checklist
+
+Work through `.ai/code-review.md` section by section. Apply:
+- **Universal BLOCK checks** to every file
+- **Universal WARN checks** to every file
+- **Java checks** (and module-specific sub-checks) for Java files
+- **Python checks** for Python files
+- **Angular checks** for TypeScript/HTML/SCSS files in frontend modules
+- **Database and migration checks** for any SQL or migration files
+- **Cross-module impact checks** for any API, schema, or contract changes
+
+For each finding, produce one entry in the format:
+
+```
+[BLOCK|WARN|NOTE] <file>:<line-range>
+  <what the finding is>
+  <why it matters>
+  <concise recommendation>
+```
+
+If a section has no findings, write `✓ <section name> — no issues`.
+
+### Step 4 — Summarise
+
+After all findings, output a summary block:
+
+```
+## Summary
+BLOCK: <n>  WARN: <n>  NOTE: <n>
+
+<One sentence on the overall state — e.g. "Ready to merge after BLOCK items are resolved" or "No blocking issues; WARN items are recommended before merge.">
+```
+
+### Step 5 — Cross-module impact
+
+Explicitly state whether any changes affect:
+- Backend API consumers (frontend models, workflow scripts)
+- `data-quality` REST API (check `dq_execution.py`, `dq_custom_checks.py`, `populate_dashboard.py` in `workflow-scripts`)
+- Database schema consumed outside one module
+- `andromeda-argo-deployments` (workflow templates, Argo config)
+
+If cross-module impact exists and is not tracked in the PR or a linked ticket, raise it as a **WARN**.
+
+### Notes on severity
+
+- **BLOCK** — must be fixed before merge. Do not soften a BLOCK to a WARN to be polite.
+- **WARN** — should be fixed; flag clearly even if the reviewer may choose to defer.
+- **NOTE** — optional suggestion. Keep these brief; do not pad the review with trivial notes.
+
+If there are zero BLOCKs and zero WARNs, say so explicitly — a clean review is useful information.

package/teams/fhr-andromeda/skills/fhr-spec/SKILL.md

@@ -0,0 +1,96 @@
+---
+name: fhr-spec
+description: Turn a Jira ticket into a structured AI Spec following the format defined in the project harness, ready for an AI coding agent to implement.
+---
+
+# fhr-spec — Generate AI Spec from Jira Ticket
+
+Turn a Jira ticket into a structured AI Spec following the format defined in `.ai/ai-instructions.md §11`, ready for an AI coding agent to implement.
+
+## When to invoke
+
+When the user says "generate a spec", "write an AI spec", "spec out this ticket", or pastes a Jira ticket ID or description and wants it turned into an implementation-ready spec.
+
+## Instructions
+
+### Step 1 — Read the harness
+
+Read in order:
+1. `CLAUDE.md` (root) — repo map and module ownership
+2. `.ai/architecture.md` — system boundaries and constraints
+3. `.ai/domain.md` — business vocabulary
+4. `.ai/coding-guidelines.md` — implementation rules
+5. `.ai/jira-ticket-template.md` — the expected ticket format
+
+### Step 2 — Ingest the ticket
+
+If the user provided a Jira ticket ID and the Jira MCP is available, fetch it with `getJiraIssue`. Otherwise, read whatever the user pasted.
+
+Extract:
+- Title and ticket ID
+- Module (from the ticket's **Module** field or infer from description)
+- Agent type (from the relevant module `CLAUDE.md` header)
+- What to build (objective)
+- Requirements (given/when/then items)
+- Do Not Touch constraints
+- Acceptance criteria
+- Extra context
+
+### Step 3 — Identify gaps
+
+Before writing the spec, check whether the ticket has enough information to produce concrete, testable requirements. Common gaps:
+- Missing field names, endpoint paths, or status codes
+- Ambiguous acceptance criteria that cannot become a unit test
+- Unclear module scope (which module owns the change)
+- Cross-repo impact not declared (does `andromeda-argo-deployments` need to change?)
+- New dependency or framework required
+
+If critical gaps exist, ask the user to fill them before continuing. Keep questions batched and specific.
+
+### Step 4 — Write the AI Spec
+
+Produce the spec in the canonical format from `.ai/ai-instructions.md`:
+
+```markdown
+## AI Spec
+
+**Ticket**: <id>  **Module**: <module>  **Agent**: <type>
+
+### Objective
+One or two sentences — what this achieves and why.
+
+### Out of Scope
+- <item>
+
+### Functional Requirements
+1. Given … when … then …
+2. Given … when … then …  (include error paths and edge cases)
+
+### Technical Constraints
+- Architecture pattern (e.g. synchronous Spring MVC, reactive WebFlux, standalone Angular component)
+- Security requirements (e.g. endpoint must be behind existing OAuth2 filter chain)
+- Cross-repo impact (e.g. `andromeda-argo-deployments` needs a matching PR — yes/no)
+- New dependencies: none | <name + approval required>
+
+### Acceptance Criteria
+- [ ] Given … when … then …
+- [ ] All existing tests pass.
+- [ ] BUILD SUCCESS confirmed locally.
+- [ ] No TODO, FIXME, placeholder comments, or commented-out code.
+```
+
+Rules when writing requirements:
+- Every requirement must be testable — if it can't become a `given/when/then` unit test, rewrite it until it can
+- Include at least one error/edge-case requirement per feature area
+- Use exact field names, endpoint paths, and status codes from the ticket or source code
+- Do not invent technical implementation details — describe observable behaviour only
+
+### Step 5 — Present and optionally post to Jira
+
+Present the spec to the user in the conversation.
+
+If the Jira MCP is available and the user confirms, append the spec to the Jira ticket description using `editJiraIssue` (append only — do not overwrite the original description).
+
+End the spec with:
+
+> **Awaiting spec approval.** Reply with `APPROVED`, `lgtm`, or `proceed` to begin implementation.