groundworkers 0.4.0__tar.gz → 0.6.0__tar.gz

groundworkers-0.6.0/.github/workflows/ci.yml

@@ -0,0 +1,52 @@
+name: CI
+on:
+  pull_request:
+    branches: [main]
+    types: [opened, synchronize, reopened, labeled, unlabeled]
+jobs:
+  label-gate:
+    uses: AustralianCancerDataNetwork/cava-devops/.github/workflows/label-gate.yml@main
+  build-test:
+    uses: AustralianCancerDataNetwork/cava-devops/.github/workflows/build-test.yml@main
+  build-test-postgres:
+    uses: AustralianCancerDataNetwork/cava-devops/.github/workflows/build-test-postgres.yml@main
+    with:
+      ty-src: src
+      postgres-image: pgvector/pgvector:pg18
+      postgres-db: test_db
+      postgres-user: postgres
+      postgres-password: postgres
+      setup-commands: |
+        mkdir -p "$RUNNER_TEMP/omop"
+        cp tests/support/postgres_ci_config.toml "$RUNNER_TEMP/omop/config.toml"
+        export OA_CONFIG_PATH="$RUNNER_TEMP/omop/config.toml"
+        export GROUNDWORKERS_CONFIG_PATH="$RUNNER_TEMP/omop/config.toml"
+        export GROUNDWORKERS_REQUIRE_INTEGRATION=1
+        export GROUNDWORKERS_TEST_DATABASE_URL="postgresql+psycopg://postgres:postgres@localhost:5432/test_db"
+        echo "OA_CONFIG_PATH=$RUNNER_TEMP/omop/config.toml" >> "$GITHUB_ENV"
+        echo "GROUNDWORKERS_CONFIG_PATH=$RUNNER_TEMP/omop/config.toml" >> "$GITHUB_ENV"
+        echo "GROUNDWORKERS_REQUIRE_INTEGRATION=1" >> "$GITHUB_ENV"
+        echo "GROUNDWORKERS_TEST_DATABASE_URL=postgresql+psycopg://postgres:postgres@localhost:5432/test_db" >> "$GITHUB_ENV"
+        uv run omop-config connections add pg_test \
+          --dialect postgresql+psycopg \
+          --host localhost \
+          --port 5432 \
+          --user postgres \
+          --password postgres \
+          --database-name test_db \
+          --test-only false
+        uv run omop-config databases add test_cdm_db \
+          --kind cdm \
+          --connection pg_test \
+          --schema-name public
+        uv run omop-config databases add test_vector_db \
+          --kind generic \
+          --connection pg_test \
+          --schema-name public
+        uv run omop-config vector-stores add test_embeddings \
+          --backend-type pgvector \
+          --database test_vector_db
+        uv run omop-config configure groundworkers \
+          --cdm-db test_cdm_db \
+          --vector-store-name test_embeddings
+        uv run python -m tests.support.seed_postgres_fixture

groundworkers-0.6.0/CHANGELOG.md

@@ -0,0 +1,58 @@
+### CHANGELOG NO LONGER MAINTAINED
+now handled in ci/cd automation consistent the rest of the cava-devops stack
+
+## 0.3.4
+- started building out the configuration TUI 
+- lock upper boundary for oa_configurator compatibility
+
+## 0.3.3
+
+feature: changed the way that knowledge packs are registered, updated baseline knowledge packs (status is very draft for this part - ymmv)
+
+## 0.3.2
+
+fix: repaired integration tests
+
+## 0.3.1
+
+chore: version bump
+
+## 0.3.0
+
+feat: knowledge-base integration, layer cleanup, oa-configurator integration, graph and embedding services
+fix: candidate-bundle metadata, knowledge content-serving, embedding-backend honesty, source-profile signals
+
+- populate identity metadata (`concept_code`, `vocabulary_id`, `domain_id`, `concept_class_id`) on embedding-only candidates in `concept_candidate_bundle` via a single batch `concept_views` backfill (omop-emb never carries these); warn when the graph service is unavailable to enrich them
+- finish the knowledge catalogue: add `KnowledgeCatalogue.get_pack()` and a `knowledge_pack` MCP tool that serve pack `guidance`/`rules`/`examples` content — previously only manifest metadata and filenames were exposed; also collapse the dead tail branch in `PackApplicability.matches`
+- reject an unsupported embedding backend eagerly at build time with an actionable message, clarifying that FAISS is a query-time cache accelerator (`faiss_cache_dir` + the `embedding-faiss` extra), not a standalone backend value
+- record `inferred_vocab` on source-vocabulary columns so the router derives a table-level domain hint from them, and surface the matched source profile's `structural_skip_field_types` and `packed_value_column_hint` on `PreIngestBundle` / the assisted-plan REST response instead of discarding them
+- require `omop-emb>=1.1.1`
+- add knowledge-base catalogue integration and expand the knowledge-facing tool and service surface
+- introduce dedicated graph and grounding services so concept and resolver tools delegate through a cleaner service-layer split
+- integrate `oa-configurator` into application/bootstrap wiring and refresh the example configuration path
+- expand the `omop-graph` and `omop-emb` adapter layer, including parentless domain-constrained grounding support via `omop-graph>=1.3.0`
+- refresh architecture and service documentation, including new graph, grounding, resolver, and source-planning pages
+
+## 0.2.0
+
+feat: add direct Python service layer, mapping workflows, and docs refresh
+
+- add `build_application()` and a shared application container for direct Python consumers
+- add `MappingService` as a reusable service-layer API for mapping-oriented workflows
+- add mapping tools for normalized search, candidate bundles, parent backoff, mapping context, `Maps to value`, mapping-expression resolution, and candidate evaluation
+- keep mapping MCP tools thin by delegating orchestration to the service layer
+- tighten `omop-emb` typing through the adapter and server composition path
+- extend docs to cover MCP and direct-Python integration, layer boundaries, and mapping workflows
+
+## 0.1.1
+
+feat: add chunk coherence pass and review-state transitions
+
+- add coherence reranking based on approved-set distribution
+- infer provisional sets for inferred chunks
+- transition processed chunks to REVIEW and skip them on resume
+- add unit coverage for coherence behavior
+
+## 0.1.0
+
+- alpha release for review

groundworkers-0.6.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.5
+Name: groundworkers
+Version: 0.6.0
+Summary: Groundworkers MCP server — read-only agentive access to OMOP vocabularies, concept graphs, and embeddings.
+Requires-Python: >=3.12
+Requires-Dist: fastapi<1.0,>=0.110
+Requires-Dist: mcp[cli]<2,>=1
+Requires-Dist: numpy<3,>=1.26
+Requires-Dist: oa-configurator<2,>=1.3.0
+Requires-Dist: omop-alchemy<2,>=1.1
+Requires-Dist: omop-emb<3,>=2.1
+Requires-Dist: omop-graph<3,>=2.1
+Requires-Dist: omop-llm<2,>=1
+Requires-Dist: omop-semantics<1,>=0.6
+Requires-Dist: psutil<8,>=5.9
+Requires-Dist: psycopg[binary]<4,>=3.1
+Requires-Dist: pydantic<3,>=2
+Requires-Dist: python-dotenv<2,>=1
+Requires-Dist: pyyaml<7,>=6
+Requires-Dist: sqlalchemy<3,>=2
+Requires-Dist: uvicorn[standard]<1.0,>=0.29
+Provides-Extra: all-source
+Requires-Dist: openpyxl<4,>=3.1; extra == 'all-source'
+Requires-Dist: pdfplumber>=0.10; extra == 'all-source'
+Requires-Dist: python-docx>=1.0; extra == 'all-source'
+Provides-Extra: dev
+Requires-Dist: httpx<1,>=0.27; extra == 'dev'
+Requires-Dist: ipython>=8.0; extra == 'dev'
+Requires-Dist: mkdocs-material>=9.7.1; extra == 'dev'
+Requires-Dist: mkdocs-mermaid2-plugin; extra == 'dev'
+Requires-Dist: mkdocs>=1.6.1; extra == 'dev'
+Requires-Dist: mkdocstrings-python>=2.0.1; extra == 'dev'
+Requires-Dist: mypy>=1.8; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0; extra == 'dev'
+Requires-Dist: pytest>=9.0.3; extra == 'dev'
+Requires-Dist: requests>=2.33.0; extra == 'dev'
+Requires-Dist: ruff>=0.4; extra == 'dev'
+Requires-Dist: textual>=0.60; extra == 'dev'
+Requires-Dist: tornado>=6.5.9; extra == 'dev'
+Requires-Dist: ty>=0.0.59; extra == 'dev'
+Provides-Extra: docx
+Requires-Dist: python-docx>=1.0; extra == 'docx'
+Provides-Extra: embedding-artifacts
+Requires-Dist: h5py<4,>=3.10; extra == 'embedding-artifacts'
+Provides-Extra: embedding-faiss
+Requires-Dist: omop-emb[faiss-cpu]<3,>=2.1; extra == 'embedding-faiss'
+Provides-Extra: embedding-pgvector
+Requires-Dist: omop-emb[pgvector]<3,>=2.1; extra == 'embedding-pgvector'
+Provides-Extra: pdf
+Requires-Dist: pdfplumber>=0.10; extra == 'pdf'
+Provides-Extra: tui
+Requires-Dist: groundskeeping<1,>=0.8.1; extra == 'tui'
+Requires-Dist: rich<15,>=13; extra == 'tui'
+Requires-Dist: textual>=0.60; extra == 'tui'
+Provides-Extra: xlsx
+Requires-Dist: openpyxl<4,>=3.1; extra == 'xlsx'
+Description-Content-Type: text/markdown
+
+# groundworkers
+
+Groundworkers is the reusable, read-only capability layer for OMOP vocabulary lookup, concept mapping, source planning, and related analysis. It combines deterministic services with optional embeddings and model-assisted operations.
+
+It does not own patient-level writes, session state, or job lifecycle. Those concerns belong to the calling application; `groundcrew` is the usual stateful orchestrator.
+
+## Choose an interface
+
+| Interface | Use it when |
+| --- | --- |
+| MCP | An agent needs discoverable tools, prompts, resources, and a shared remote runtime. |
+| REST | An application needs the curated HTTP workflows for candidate bundles or assisted source planning. |
+| Python | A process owns batching, evaluation, or orchestration and can run in-process. |
+
+Start with the [documentation home](docs/index.md), or choose a path directly:
+
+- [Concepts and capability choices](docs/concepts.md) explains retrieval, grounding, mapping, source planning, and projection.
+- [Installation](docs/usage/installation.md) covers package extras and prerequisites.
+- [Initial local setup](docs/from-scratch.md) walks through a fresh CDM-backed deployment.
+- [Integrations](docs/usage/integrations.md) shows MCP, REST, and Python calls.
+- [Configuration](docs/usage/configuration.md) explains shared-stack references and optional capabilities.
+- [Tools overview](docs/tools/overview.md) maps jobs to the discoverable MCP surface.
+
+## Quick start
+
+Install the package and configure the shared OMOP stack:
+
+```bash
+pip install groundworkers
+omop-config configure groundworkers
+```
+
+Inspect the active configuration and MCP surface before starting a service:
+
+```bash
+groundworkers --describe
+```
+
+Run MCP over stdio for a client that starts the process, or over Streamable HTTP for a shared service:
+
+```bash
+groundworkers
+groundworkers --transport streamable-http --host 127.0.0.1 --port 8000
+```
+
+REST is a separate, curated interface:
+
+```bash
+groundworkers --transport rest --host 127.0.0.1 --port 8080
+```
+
+## Direct Python
+
+Build the application once and reuse it. Services are optional when their prerequisites are not configured:
+
+```python
+from groundworkers.app import build_application
+from groundworkers.bootstrap import build_app_config
+
+config = build_app_config()
+app = build_application(config)
+
+bundle = app.services.mapping.concept_candidate_bundle(
+    "type 2 diabetes",
+    domain="Condition",
+)
+```
+
+The `mapping` service in this example requires a configured CDM database. Embedding-backed channels require an embedding model and vector store; text, domain classification, and assisted source planning require a configured chat model. Use `groundworkers --describe` to see what the current deployment exposes.
+
+## Companion repositories
+
+- [groundcrew](https://github.com/AustralianCancerDataNetwork/groundcrew) owns session state and orchestration.
+- [omop-graph](https://australiancancerdatanetwork.github.io/omop-graph/) provides graph-backed OMOP traversal.
+- [omop-emb](https://australiancancerdatanetwork.github.io/omop-emb/) provides embedding stores and indexes.

groundworkers-0.6.0/README.md

@@ -0,0 +1,75 @@
+# groundworkers
+
+Groundworkers is the reusable, read-only capability layer for OMOP vocabulary lookup, concept mapping, source planning, and related analysis. It combines deterministic services with optional embeddings and model-assisted operations.
+
+It does not own patient-level writes, session state, or job lifecycle. Those concerns belong to the calling application; `groundcrew` is the usual stateful orchestrator.
+
+## Choose an interface
+
+| Interface | Use it when |
+| --- | --- |
+| MCP | An agent needs discoverable tools, prompts, resources, and a shared remote runtime. |
+| REST | An application needs the curated HTTP workflows for candidate bundles or assisted source planning. |
+| Python | A process owns batching, evaluation, or orchestration and can run in-process. |
+
+Start with the [documentation home](docs/index.md), or choose a path directly:
+
+- [Concepts and capability choices](docs/concepts.md) explains retrieval, grounding, mapping, source planning, and projection.
+- [Installation](docs/usage/installation.md) covers package extras and prerequisites.
+- [Initial local setup](docs/from-scratch.md) walks through a fresh CDM-backed deployment.
+- [Integrations](docs/usage/integrations.md) shows MCP, REST, and Python calls.
+- [Configuration](docs/usage/configuration.md) explains shared-stack references and optional capabilities.
+- [Tools overview](docs/tools/overview.md) maps jobs to the discoverable MCP surface.
+
+## Quick start
+
+Install the package and configure the shared OMOP stack:
+
+```bash
+pip install groundworkers
+omop-config configure groundworkers
+```
+
+Inspect the active configuration and MCP surface before starting a service:
+
+```bash
+groundworkers --describe
+```
+
+Run MCP over stdio for a client that starts the process, or over Streamable HTTP for a shared service:
+
+```bash
+groundworkers
+groundworkers --transport streamable-http --host 127.0.0.1 --port 8000
+```
+
+REST is a separate, curated interface:
+
+```bash
+groundworkers --transport rest --host 127.0.0.1 --port 8080
+```
+
+## Direct Python
+
+Build the application once and reuse it. Services are optional when their prerequisites are not configured:
+
+```python
+from groundworkers.app import build_application
+from groundworkers.bootstrap import build_app_config
+
+config = build_app_config()
+app = build_application(config)
+
+bundle = app.services.mapping.concept_candidate_bundle(
+    "type 2 diabetes",
+    domain="Condition",
+)
+```
+
+The `mapping` service in this example requires a configured CDM database. Embedding-backed channels require an embedding model and vector store; text, domain classification, and assisted source planning require a configured chat model. Use `groundworkers --describe` to see what the current deployment exposes.
+
+## Companion repositories
+
+- [groundcrew](https://github.com/AustralianCancerDataNetwork/groundcrew) owns session state and orchestration.
+- [omop-graph](https://australiancancerdatanetwork.github.io/omop-graph/) provides graph-backed OMOP traversal.
+- [omop-emb](https://australiancancerdatanetwork.github.io/omop-emb/) provides embedding stores and indexes.

{groundworkers-0.4.0 → groundworkers-0.6.0}/docs/adapters/omop_emb.md

@@ -37,6 +37,7 @@ No live model API is required.
 Requires an embedding client configured through `omop-emb` package settings.
 
 - `embedding_search` encodes a text query on the fly
+- `embedding_search_batch` encodes and searches multiple text queries with one provider/index batch, returning rows aligned to input order
 - `embedding_encode` returns a raw embedding vector
 
 ## Availability model

{groundworkers-0.4.0 → groundworkers-0.6.0}/docs/development/extending.md

@@ -1,6 +1,6 @@
 # Extending groundworkers
 
-This guide is for contributors adding new adapters, services, MCP tools, or REST endpoints. The main rule is simple:
+This guide is for contributors adding new adapters, services, MCP tools, or REST endpoints directly in this repository. If you're adding a self-contained capability as a separate installable package instead — no core file edits — see [Plugin architecture](plugins.md). The main rule here is simple:
 
 **keep business logic above the transport layer and keep dependency details below it.**
 

groundworkers-0.6.0/docs/development/plugins.md

@@ -0,0 +1,195 @@
+# Plugin architecture
+
+## What a plugin is
+
+Groundworkers plugins are independently installed packages that add an adapter, service, and MCP surface without adding plugin-specific imports to Groundworkers. They register one ready-to-use plugin object under the `groundworkers.plugins` entry-point group.
+
+```toml
+[project.entry-points."groundworkers.plugins"]
+my_plugin = "my_package.plugin:plugin"
+```
+
+```mermaid
+flowchart LR
+    subgraph PKG["my_package — a separate installed package"]
+        direction TB
+        CFG["Config class<br/><i>the settings this plugin needs</i>"]
+        OBJ["Plugin object<br/>name &middot; config class &middot; build() &middot; register()"]
+    end
+
+    subgraph HOST["Groundworkers — the host application"]
+        direction TB
+        DISC["Discovery<br/><i>looks up installed plugins</i>"]
+        CTX["Shared connections<br/><i>database, embeddings — already<br/>set up by Groundworkers itself</i>"]
+        SRV["Shared MCP tool server"]
+    end
+
+    OBJ -.->|"found via the environment's<br/>own package listing"| DISC
+    DISC -->|"reads + validates"| CFG
+    CTX -->|"handed to"| OBJ
+    OBJ -->|"register() adds its tools"| SRV
+```
+
+The object implements `GroundworkersPlugin`: a stable `name`, an optional `PackageConfigBase` class, `build(context, config)`, and `register(server, state)`. The plugin name and `config_cls.tool_name` must match. **Installed plugin names must be unique.**
+
+```python
+class MyPlugin:
+    name = "my_plugin"
+    config_cls = ExampleConfig  # or None, if this plugin needs no configuration
+
+    def build(self, context, config):
+        ...  # return a service object, or None if a prerequisite is unavailable
+
+    def register(self, server, state):
+        ...  # add this plugin's MCP tools/resources
+
+
+plugin = MyPlugin()
+```
+
+## Start from the use case
+
+A plugin's configuration should follow from what it needs to be able to complete its task. 
+
+Work through each requirement before writing the class.
+
+For example, a plugin recommending comparator cohorts might reason like this.
+
+**Reuse a resource Groundworkers already resolved.** Nothing to declare — read it straight off `PluginContext` inside `build()`:
+
+| Use case | What you need | How you get it |
+|---|---|---|
+| Look up existing OMOP concepts | Read access to the CDM vocabulary connection Groundworkers already opened | `context.cdm_engine` / `context.resolver` |
+| Rank or filter by similarity to concepts Groundworkers already embedded | The concept embeddings Groundworkers already populated, via omop-emb | `context.vector_store` + `context.embedding_backend_factory` |
+| Ask an LLM to help decide something | The chat model Groundworkers already configured, via omop-llm | `context.chat_backend_factory` |
+
+**Create something this plugin owns.** Each becomes a field on its own `PackageConfigBase` — see [Package configuration](#package-configuration):
+
+| Use case | What you need | Configuration field established | Notes | 
+|---|---|---|---|
+| Store its own distinct non-CDM source tables | A database connection this plugin owns and writes to | `comparator_db: Annotated[str, RefTo(CDMDatabaseConfig)] = "cdm_db"` | Same shape as a core reference, but the operator points it at a new entry when they configure the plugin |
+| Vectorised *arbitrary free text or objects* for similarity search | Its own embedding model | `embedding_model_name: Annotated[str \| None, RefTo(ModelConfig)] = None` | Optional, independent of Groundworkers' own model choice |
+| Store and search those vectors | Its own vector index | `vector_store_name: Annotated[str \| None, RefTo(VectorStoreConfig)] = None` | **Not existing OMOP concepts, see above** |
+| Let operators tune how strict "similar enough" is | Parameter the plugin's logic requires | `min_databases_default: int = Field(default=2, ge=1)` | A plain setting, nothing to resolve |
+
+The concept-embedding row and the free-text-embedding row look similar but are not the same decision: the first reuses a vector store Groundworkers already populated with OMOP concepts; the second stands up an independent embedding model and vector store for data that was never a concept to begin with. Reusing `context.vector_store` for the second case would put unrelated vectors in Groundworkers' own concept index.
+
+### Assumptions this table is making
+
+- **Vocabulary only, so far.** Every example above reads OMOP *vocabulary* i.e. operations over concepts, concept relationships, concept embeddings, or the host's shared grounding cascade. Nothing about `context.cdm_engine` restricts a plugin to vocabulary tables specifically, however. These tables are available over the same CDM connection groundworkers itself uses, and a plugin *could* query clinical/person-level tables (`condition_occurrence`, `drug_exposure`, etc.) through it just as easily, subject to whatever environmental and procedural access controls actually govern that data.
+- **Shared mapping access.** `PluginContext.mapping_context` is an optional,read-only capability façade over the host's mapping and grounding policy. Use its operations when a plugin needs candidate bundles, text grounding, or concept context. `concept_shared_standard_ancestor(concept_ids, max_depth=5)` returns the unique nearest shared standard ancestor when one exists; raise `max_depth` to widen the hierarchy search. The underlying vocabulary, graph, and service objects are deliberately not exposed. Do not construct a second `omop-graph` or vocabulary adapter from `context.cdm_engine`. The field is `None` only when the host did not compose services for the plugin context.
+
+## Build and register a new plugin
+
+`build()` receives `PluginContext`. The context provides the resolved CDM database and engine, vector store, shared lazy model backend factories, the optional read-only `mapping_context`, and a deliberately narrow resolver for independent named OA resources. Return `None` when a prerequisite is unavailable. Groundworkers then keeps serving its core capabilities and reports why the plugin was not activated.
+
+`register()` adds MCP tools, prompts, or resources against the state returned by `build()`. The shared MCP registration wrapper supplies the same safe error translation used by core tools. Code-defined prompts can continue to derive their arguments from the callable signature. A plugin publishing data-defined prompts can pass explicit string metadata with `PromptArgument`; this keeps dynamic pack fields out of Python closure signatures:
+
+```python
+from groundworkers.base import PromptArgument
+
+server.prompt(
+    "my_pack_workflow",
+    title="Run my pack workflow",
+    description="Start the workflow published by my pack.",
+    arguments=[
+        PromptArgument(
+            "request",
+            "The user's starting request.",
+            required=False,
+        )
+    ],
+)(handler)
+```
+
+```mermaid
+sequenceDiagram
+    participant Env as Installed packages
+    participant GW as Groundworkers (host)
+    participant P as my_package (a plugin)
+
+    Note over Env: 1.
+
+    GW->>Env: At startup: list every registered plugin
+    Env-->>GW: my_package's plugin object
+
+    Note over GW,P: 2.
+
+    GW->>P: Resolve its config class into validated settings
+    GW->>P: build(shared connections, those settings)
+    P-->>GW: its own service object (or "not available")
+
+    GW->>P: register(shared MCP server, that service object)
+    P-->>GW: adds its own tools to the shared server
+
+    Note over GW: 3.
+```
+
+1. The plugin's own installer registered one entry: "here is my plugin object."
+2. Groundworkers' code never imports `my_package` by name. It only expects four things on the object (`name`, optional `PackageConfigBase` class, plus `build` and `register` methods, described above).
+3.  The plugin's tools are now callable through the one Groundworkers server.
+
+## Package configuration
+
+A plugin with scalar and `RefTo` fields receives a setup-console page without any Groundskeeping-aware plugin code:
+
+```python
+from typing import Annotated, ClassVar
+
+from oa_configurator import CDMDatabaseConfig, PackageConfigBase, RefTo
+from pydantic import Field
+
+
+class ExampleConfig(PackageConfigBase):
+    tool_name: ClassVar[str] = "my_plugin"
+
+    database: Annotated[str, RefTo(CDMDatabaseConfig)] = "cdm_db"
+    retries: int = Field(default=2, ge=1)
+```
+
+The generic workflow maps supported Pydantic scalars to Groundskeeping fields, offers existing references of the correct concrete type, and can create a new reference chain recursively through oa-configurator's public `plan_configure()` API. Candidates remain private to the Groundworkers provider; review output is redacted and persistence is revision-aware.
+
+Schemas requiring live discovery or a conditional flow (and therefore not fitting the default configuration UX specification) may additionally implement the runtime-checkable `GroundworkersPluginConfigUI` protocol. Its `tui_workflow(operation)` method returns a custom `(ConfigWorkflowSpec, ConfigMutationService)` pair. Keep Groundskeeping in the plugin's optional TUI dependencies so headless plugin imports remain lightweight.
+
+### Sensitive fields
+
+`PackageConfigBase` inherits oa-configurator's `SecretSafeModel`. A plugin developer must mark every plugin-owned secret with `Sensitive()` so model representations, interactive fields, configuration views, and review diffs can redact it by schema declaration. Groundworkers does not infer sensitivity from field names.
+
+```python
+from typing import Annotated, ClassVar
+
+from oa_configurator import PackageConfigBase, Sensitive
+from pydantic import Field
+
+
+class ExampleConfig(PackageConfigBase):
+    tool_name: ClassVar[str] = "my_plugin"
+    api_token: Annotated[str | None, Sensitive()] = Field(default=None)
+```
+
+The `oa_configurator.Secret` alias is equivalent to `Annotated[str | None, Sensitive()]`. Prefer a `RefTo` field when a plugin can reuse an existing database, model, provider, or vector-store entry. The plugin then stores only the logical reference name; sensitivity remains declared on the referenced schema, such as `ConnectionConfig.password` or `ProviderConfig.api_key`, and the generic recursive configuration workflow preserves those declarations.
+
+This protects display and logging surfaces; it does not encrypt persisted configuration. Plugin documentation must describe the storage expectations for any secret it introduces.
+
+## Setup steps
+
+Plugin data preparation belongs in registered setup steps. A plugin may expose a `setup_steps` tuple of `PluginSetupStep` instances. Each step declares a stable key, operator copy, typed arguments, and a `build_plan(values, config_path)` callable that returns a `MaintenancePlan`.
+
+The host provides the common setup-console form and review flow, passes argument values only to that run, and starts the resulting plan through the durable maintenance-run store. Logs, progress, cancellation, retry, and the Runs view are therefore shared by core operations and plugins.
+
+Setup arguments are separate from package configuration: database/model references and policy defaults belong in configuration, while paths to an input export belong to the one-off setup run.
+
+Long-running plugin work should be represented by commands in the plan and run out of process. The command may invoke the plugin's own setup module or console entry point; Groundworkers does not import plugin internals or reproduce their loading logic.
+
+## Readiness verification
+
+A plugin may implement `GroundworkersPluginReadiness.verify_readiness(state)` when operators need to distinguish valid package configuration from an initialised, usable capability.
+
+Verification is read-only and returns a `PluginReadinessResult` containing a stable state, a summary, and `PluginReadinessField` values. Display values must never contain credentials, connection URLs, or raw exception text.
+
+The contract is headless: it contains no Groundskeeping types. Groundworkers uses the same result for generic setup-console rendering, while a plugin may return `result.as_dict()` from its own MCP status tool. Plugins own dependency- specific checks; the host only builds current plugin state, handles unavailable configuration safely, and maps readiness states to existing Groundskeeping status, table, empty, action, and detail contracts.
+
+
+## Lifecycle and diagnostics
+
+Groundworkers discovers the plugin set once per application build, validates its identities, resolves each package configuration, and builds state. MCP registration uses those same discovered objects. `groundworkers --describe` lists active plugins and safe `plugin_issues` for plugins skipped because their configuration, prerequisites, build, or registration failed.