groundworkers 0.5.0__tar.gz → 0.6.0__tar.gz

{groundworkers-0.5.0 → groundworkers-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: groundworkers
-Version: 0.5.0
+Version: 0.6.0
 Summary: Groundworkers MCP server — read-only agentive access to OMOP vocabularies, concept graphs, and embeddings.
 Requires-Python: >=3.12
 Requires-Dist: fastapi<1.0,>=0.110
@@ -36,7 +36,7 @@ Requires-Dist: pytest>=9.0.3; extra == 'dev'
 Requires-Dist: requests>=2.33.0; extra == 'dev'
 Requires-Dist: ruff>=0.4; extra == 'dev'
 Requires-Dist: textual>=0.60; extra == 'dev'
-Requires-Dist: tornado>=6.5.5; extra == 'dev'
+Requires-Dist: tornado>=6.5.9; extra == 'dev'
 Requires-Dist: ty>=0.0.59; extra == 'dev'
 Provides-Extra: docx
 Requires-Dist: python-docx>=1.0; extra == 'docx'

{groundworkers-0.5.0 → groundworkers-0.6.0}/docs/development/plugins.md

@@ -76,12 +76,12 @@ The concept-embedding row and the free-text-embedding row look similar but are n
 
 ### Assumptions this table is making
 
-- **Vocabulary only, so far.** Every example above reads OMOP *vocabulary* i.e. operations over concepts, concept relationships, concept embeddings. Nothing about `context.cdm_engine` restricts a plugin to vocabulary tables specifically, however. These tables are available over the same CDM connection groundworkers itself uses, and a plugin *could* query clinical/person-level tables (`condition_occurrence`, `drug_exposure`, etc.) through it just as easily, subject to whatever environmental and procedural access controls actually govern that data. 
-- **Plugin graph access is TODO.** `PluginContext` does not expose graph traversal at this time, which means that a plugin that wants graph traversal would have to depend on `omop-graph` itself and build its own adapter, reusing `context.cdm_engine` for the connection but not groundworkers' own shared instance. Recommend not doing it that way, however, as it would be easier to expose the core graph the same way `vector_store` and `embedding_backend_factory` already are. 
+- **Vocabulary only, so far.** Every example above reads OMOP *vocabulary* i.e. operations over concepts, concept relationships, concept embeddings, or the host's shared grounding cascade. Nothing about `context.cdm_engine` restricts a plugin to vocabulary tables specifically, however. These tables are available over the same CDM connection groundworkers itself uses, and a plugin *could* query clinical/person-level tables (`condition_occurrence`, `drug_exposure`, etc.) through it just as easily, subject to whatever environmental and procedural access controls actually govern that data.
+- **Shared mapping access.** `PluginContext.mapping_context` is an optional,read-only capability façade over the host's mapping and grounding policy. Use its operations when a plugin needs candidate bundles, text grounding, or concept context. `concept_shared_standard_ancestor(concept_ids, max_depth=5)` returns the unique nearest shared standard ancestor when one exists; raise `max_depth` to widen the hierarchy search. The underlying vocabulary, graph, and service objects are deliberately not exposed. Do not construct a second `omop-graph` or vocabulary adapter from `context.cdm_engine`. The field is `None` only when the host did not compose services for the plugin context.
 
 ## Build and register a new plugin
 
-`build()` receives `PluginContext`. The context provides the resolved CDM database and engine, vector store, shared lazy model backend factories, and a deliberately narrow resolver for independent named OA resources. Return `None` when a prerequisite is unavailable. Groundworkers then keeps serving its core capabilities and reports why the plugin was not activated.
+`build()` receives `PluginContext`. The context provides the resolved CDM database and engine, vector store, shared lazy model backend factories, the optional read-only `mapping_context`, and a deliberately narrow resolver for independent named OA resources. Return `None` when a prerequisite is unavailable. Groundworkers then keeps serving its core capabilities and reports why the plugin was not activated.
 
 `register()` adds MCP tools, prompts, or resources against the state returned by `build()`. The shared MCP registration wrapper supplies the same safe error translation used by core tools. Code-defined prompts can continue to derive their arguments from the callable signature. A plugin publishing data-defined prompts can pass explicit string metadata with `PromptArgument`; this keeps dynamic pack fields out of Python closure signatures:
 

{groundworkers-0.5.0 → groundworkers-0.6.0}/docs/services/vocab.md

@@ -48,11 +48,29 @@ vocab.search_normalized(
     vocabulary_id: str | None = None,
     standard_only: bool = False,
     limit: int = 20,
+    allow_unindexed_scan: bool = False,
 ) -> list[ConceptMatch]
 ```
 
 Normalized search: lowercases and strips punctuation from both the query and concept names before matching. Catches common surface-form differences (abbreviations, spacing, case) that exact search misses without the false-positive risk of full-text search.
 
+#### Why this one needs a selective filter
+
+Comparison is on a *computed* expression, so an ordinary index on `concept_name` cannot serve it. Without a selective filter the planner scans every concept. Measured against a 1.7M-concept CDM:
+
+| Call | Time |
+| --- | --- |
+| unconstrained | 40s |
+| `domain='Drug'` | 25s |
+| `vocabulary_id='MedDRA'` | 0.4s |
+
+So the call is **refused** unless one of the following holds:
+
+- `vocabulary_id` or `parent_ids` is given. Note that `domain` does **not** count — it looks like a narrowing filter but the largest domains hold millions of concepts, which is why `domain='Drug'` still took 25s.
+- `allow_unindexed_scan=True`, which accepts the scan deliberately.
+
+It raises rather than warning because the unindexed query does not fail — it succeeds slowly, so a warning gets absorbed into a caller that merely looks sluggish.
+
 ### `search_fulltext`
 
 ```python
@@ -79,32 +97,32 @@ Synonym FTS is included when the `concept_synonym_name_tsvector` sidecar is also
 ```python
 vocab.navigate_to_standard(
     concept_ids: list[int],
-) -> list[StandardMapping]
+) -> list[ConceptMappingResult]
 ```
 
 Batch navigation: given a list of concept IDs (which may be non-standard), returns their standard OMOP equivalents by following `"Maps to"` relationship edges. Two queries total regardless of input list size.
 
-Concepts that are already standard are returned as self-mappings (`relationship_id="self"`). Concepts with no `"Maps to"` edge return an empty `standard_concepts` list. Concept IDs not found in the vocabulary are silently omitted.
+Concepts that are already standard are returned as self-mappings (`relationship_id="self"`). Concepts with no `"Maps to"` edge have an empty `targets` list. Concept IDs not found in the vocabulary are silently omitted.
 
 ### `navigate_to_value`
 
 ```python
 vocab.navigate_to_value(
     concept_ids: list[int],
-) -> list[RelatedConceptMapping]
+) -> list[ConceptMappingResult]
 ```
 
-Same batch navigation pattern as `navigate_to_standard`, but follows `"Maps to value"` relationship edges. Used for value-domain mapping in OMOP measurement and observation workflows.
+Same batch navigation pattern as `navigate_to_standard`, but follows `"Maps to value"` relationship edges. Used for value-domain mapping in OMOP measurement and observation workflows. Results use the same `ConceptMappingResult` shape, with targets in `targets`.
 
 ### `navigate_to_unit`
 
 ```python
 vocab.navigate_to_unit(
     concept_ids: list[int],
-) -> list[RelatedConceptMapping]
+) -> list[ConceptMappingResult]
 ```
 
-Batch navigation following `"Maps to unit"` relationship edges.
+Batch navigation following `"Maps to unit"` relationship edges. Results use the same `ConceptMappingResult` shape, with targets in `targets`.
 
 ## Return types
 
@@ -126,15 +144,16 @@ class ConceptMatch:
     ts_rank: float | None  # only set by search_fulltext
 ```
 
-### `StandardMapping`
+### `ConceptMappingResult`
 
 ```python
 @dataclass
-class StandardMapping:
+class ConceptMappingResult:
     source_concept_id: int
     source_concept_name: str
     source_standard_concept: bool
-    standard_concepts: list[MappedConcept]
+    source_classification_concept: bool
+    targets: list[MappedConcept]
 ```
 
 ### `MappedConcept`
@@ -142,20 +161,19 @@ class StandardMapping:
 ```python
 @dataclass
 class MappedConcept:
-    concept_id: int
-    concept_name: str
-    vocabulary_id: str
-    domain_id: str
-    concept_class_id: str
-    relationship_id: str  # "Maps to", "Maps to value", "Maps to unit", or "self"
+    concept: dict[str, Any]  # concept payload, including identity and quality fields
+    relationship_id: str     # "Maps to", "Maps to value", "Maps to unit", or "self"
 ```
 
+The direct Python API uses `targets` for all three navigation methods. MCP serialization names that list `standard_concepts` for `navigate_to_standard` and `related_concepts` for value and unit navigation.
+
 ## Serialization helpers
 
 `VocabService` provides `serialise_*` helpers that convert the typed return objects to plain dicts suitable for MCP tool responses:
 
 - `serialise_concept_match(match: ConceptMatch) -> dict`
-- `serialise_standard_mapping(mapping: StandardMapping) -> dict`
+- `serialise_standard_mapping(mapping: ConceptMappingResult) -> dict`
+- `serialise_related_concept_mapping(mapping: ConceptMappingResult) -> dict`
 
 The search MCP tools in `search_tools.py` use these helpers to convert service results into the wire format. Direct Python consumers can use the dataclasses directly without serializing.
 

{groundworkers-0.5.0 → groundworkers-0.6.0}/pyproject.toml

@@ -56,7 +56,7 @@ embedding-faiss = [
 dev = [
   "httpx>=0.27,<1",
   "ipython>=8.0",
-  "tornado>=6.5.5",
+  "tornado>=6.5.9",
   "pytest>=9.0.3",
   "pytest-cov>=4.0",
   "mypy>=1.8",
@@ -89,7 +89,12 @@ packages = ["src/groundworkers"]
 
 [tool.uv]
 cache-keys = [{ file = "pyproject.toml" }, { git = { commit = true, tags = true } }]
-constraint-dependencies = ["setuptools>=83.0.0"]
+constraint-dependencies = [
+  "setuptools>=83.0.0",
+  "httpx2>=2.12.0",
+  "urllib3>=2.8.0",
+  "PyJWT>=2.15.0",
+]
 
 [tool.ruff]
 src = ["src", "tests"]

{groundworkers-0.5.0 → groundworkers-0.6.0}/src/groundworkers/adapters/omop_emb.py

@@ -162,8 +162,15 @@ class OmopEmbAdapter:
         concept_id: int,
         limit: int,
         model_name: str | None,
+        standard_only: bool = True,
+        active_only: bool = True,
     ) -> dict[str, Any]:
-        """Return stored-vector neighbours without calling the model provider."""
+        """Return stored-vector neighbours without calling the model provider.
+
+        The pgvector HNSW index is partial on ``is_standard AND is_valid``. This
+        method defaults to that indexed set. Pass ``standard_only=False`` to include
+        non-standard neighbours, at the cost of a full scan.
+        """
 
         record = self._resolve_model_record(model_name)
         reader = self._build_reader(record)
@@ -183,7 +190,10 @@ class OmopEmbAdapter:
         vector = np.asarray(vectors[concept_id], dtype=float).reshape(1, -1)
         raw = reader.get_nearest_concepts(
             query_embedding=vector,
-            concept_filter=self._build_concept_filter(),
+            concept_filter=self._build_concept_filter(
+                standard_only=standard_only,
+                active_only=active_only,
+            ),
             k=limit + 1,
             faiss_index_config=record.index_config,
         )

{groundworkers-0.5.0 → groundworkers-0.6.0}/src/groundworkers/app.py

@@ -15,6 +15,7 @@ from groundworkers.config import AppConfig
 from groundworkers.plugins import (
     GroundworkersPlugin,
     GroundworkersPluginReadiness,
+    MappingContext,
     PluginConfigResolver,
     PluginContext,
     PluginReadinessResult,
@@ -52,8 +53,12 @@ class Adapters:
     # _build_plugin_context) so a plugin reuses the one backend connection
     # instead of opening a second one. None when the corresponding backend
     # is unconfigured, same gating as the adapters above.
-    embedding_backend_factory: Callable[[], Any] | None = field(default=None, repr=False)
-    embedding_model_backend_factory: Callable[[], Any] | None = field(default=None, repr=False)
+    embedding_backend_factory: Callable[[], Any] | None = field(
+        default=None, repr=False
+    )
+    embedding_model_backend_factory: Callable[[], Any] | None = field(
+        default=None, repr=False
+    )
     chat_backend_factory: Callable[[], Any] | None = field(default=None, repr=False)
 
 
@@ -104,7 +109,9 @@ def build_adapters(config: AppConfig) -> Adapters:
         nonlocal shared_model_backend
         if shared_model_backend is None:
             if resolved_model is None:
-                raise RuntimeError("No embedding model is configured for Groundworkers.")
+                raise RuntimeError(
+                    "No embedding model is configured for Groundworkers."
+                )
             from omop_llm import build_model_backend_from_resolved
 
             shared_model_backend = build_model_backend_from_resolved(resolved_model)
@@ -180,9 +187,14 @@ def build_adapters(config: AppConfig) -> Adapters:
 def build_services(config: AppConfig, adapters: Adapters) -> Services:
     services = Services()
     assisted_classifier = None
-    if adapters.llm is not None and config.groundworkers.source_planning_llm_assisted_enabled:
+    if (
+        adapters.llm is not None
+        and config.groundworkers.source_planning_llm_assisted_enabled
+    ):
         assisted_classifier = AssistedColumnRoleClassifier(adapters.llm)
-    services.source_planning = SourcePlanningService(assisted_classifier=assisted_classifier)
+    services.source_planning = SourcePlanningService(
+        assisted_classifier=assisted_classifier
+    )
     if adapters.omop_graph is not None:
         # cdm is optional; only the classified-edge traversals
         # (concept_associations / concept_extended_inheritance) require it.
@@ -206,7 +218,27 @@ def build_services(config: AppConfig, adapters: Adapters) -> Services:
     return services
 
 
-def _build_plugin_context(config: AppConfig, adapters: Adapters) -> PluginContext:
+def build_mapping_context(
+    config: AppConfig,
+    *,
+    adapters: Adapters | None = None,
+    services: Services | None = None,
+) -> MappingContext:
+    """Build the mapping capabilities shared by hosted and standalone plugins."""
+
+    shared_adapters = adapters or build_adapters(config)
+    shared_services = services or build_services(config, shared_adapters)
+    return MappingContext(
+        mapping_service=shared_services.mapping,
+        grounding_service=shared_services.grounding,
+    )
+
+
+def _build_plugin_context(
+    config: AppConfig,
+    adapters: Adapters,
+    services: Services | None = None,
+) -> PluginContext:
     """Assemble the resolved handles every plugin is given.
 
     `cdm_database`/`vector_store` come straight off `AppConfig`, already
@@ -223,6 +255,11 @@ def _build_plugin_context(config: AppConfig, adapters: Adapters) -> PluginContex
         embedding_backend_factory=adapters.embedding_backend_factory,
         embedding_model_backend_factory=adapters.embedding_model_backend_factory,
         chat_backend_factory=adapters.chat_backend_factory,
+        mapping_context=(
+            build_mapping_context(config, adapters=adapters, services=services)
+            if services is not None
+            else None
+        ),
     )
 
 
@@ -292,7 +329,8 @@ def verify_plugin_readiness(
             summary="This plugin does not expose readiness verification.",
         )
     adapters = build_adapters(config)
-    context = _build_plugin_context(config, adapters)
+    services = build_services(config, adapters)
+    context = _build_plugin_context(config, adapters, services)
     state, issue = _build_plugin(config, context, plugin)
     if state is None:
         return PluginReadinessResult(
@@ -342,13 +380,14 @@ def load_plugin_readiness(
 
 def build_application(config: AppConfig) -> GroundworkersApp:
     adapters = build_adapters(config)
-    context = _build_plugin_context(config, adapters)
+    services = build_services(config, adapters)
+    context = _build_plugin_context(config, adapters, services)
     plugin_definitions = tuple(discover_plugins())
     plugins, plugin_issues = _build_plugins(config, context, plugin_definitions)
     return GroundworkersApp(
         config=config,
         adapters=adapters,
-        services=build_services(config, adapters),
+        services=services,
         plugins=plugins,
         plugin_definitions=plugin_definitions,
         plugin_issues=plugin_issues,

{groundworkers-0.5.0 → groundworkers-0.6.0}/src/groundworkers/application/setup/databases.py

@@ -35,7 +35,10 @@ from groundworkers.application.setup.models import (
     DiagnosticSeverity,
     ResourceDiagnostic,
 )
-from groundworkers.base.sql import quote_identifier
+from groundworkers.base.sql import (
+    normalization_key_expression_sql,
+    quote_identifier,
+)
 from groundworkers.config import GroundworkersConfig
 
 CDM_TABLES = (
@@ -88,6 +91,23 @@ GROUNDWORKERS_TRIGRAM_INDEX_TARGETS = (
         ("idx_concept_synonym_lower_name_trgm",),
     ),
 )
+# (table, column, label, index name). The indexed K expression is not restated
+# here: it is rendered from base.sql.normalization_key_expression, the same
+# builder search_normalized queries with, so it cannot drift.
+GROUNDWORKERS_NORMALIZED_INDEX_TARGETS = (
+    (
+        "concept",
+        "concept_name",
+        "concept normalized name",
+        "idx_concept_name_normkey",
+    ),
+    (
+        "concept_synonym",
+        "concept_synonym_name",
+        "concept_synonym normalized name",
+        "idx_concept_synonym_name_normkey",
+    ),
+)
 # Taken from omop-emb rather than restated: these describe tables omop-emb owns
 # and creates, so a column added there must not need a matching edit here to stay
 # checkable.
@@ -111,6 +131,9 @@ def resolve_database_targets(
     cdm = resolver.resolve_database(groundworkers.cdm_db)
     if not isinstance(cdm, ResolvedCDMDatabase):
         return ()
+    cdm_schema = cdm.schema_name or (
+        "main" if cdm.connection.url.startswith("sqlite") else "public"
+    )
     embedding_target: DatabaseTarget | None = None
     expected_embedding_model_name: str | None = None
     embedding_safe_url: str | None = None
@@ -146,7 +169,7 @@ def resolve_database_targets(
             database_entry_name=cdm.name,
             connection_name=cdm.connection.name,
             safe_url=cdm.connection.safe_url,
-            cdm_schema=cdm.schema_name or "main",
+            cdm_schema=cdm_schema,
             vocabulary_schema=cdm.vocab_schema,
             connection_url=cdm.connection.url,
             role="cdm",
@@ -157,7 +180,7 @@ def resolve_database_targets(
             database_entry_name=cdm.name,
             connection_name=cdm.connection.name,
             safe_url=cdm.connection.safe_url,
-            cdm_schema=cdm.schema_name or "main",
+            cdm_schema=cdm_schema,
             vocabulary_schema=cdm.vocab_schema,
             connection_url=cdm.connection.url,
             role="graph",
@@ -168,7 +191,7 @@ def resolve_database_targets(
             database_entry_name=cdm.name,
             connection_name=cdm.connection.name,
             safe_url=cdm.connection.safe_url,
-            cdm_schema=cdm.schema_name or "main",
+            cdm_schema=cdm_schema,
             vocabulary_schema=cdm.vocab_schema,
             connection_url=cdm.connection.url,
             role="groundworkers",
@@ -437,6 +460,11 @@ def _groundworkers_diagnostics(
                 )
             )
 
+    if connection.dialect.name == "postgresql":
+        diagnostics.extend(
+            _normalized_index_diagnostics(connection, schema, existing_tables)
+        )
+
     diagnostics.extend(
         _groundworkers_embedding_model_diagnostics(
             target,
@@ -478,9 +506,7 @@ def _groundworkers_embedding_model_diagnostics(
                         "Groundworkers grounding model could not be checked because the omop-emb model registry table is missing.",
                     ),
                 )
-            rows = _embedding_registry_rows(
-                connection, schema=target.embedding_schema
-            )
+            rows = _embedding_registry_rows(connection, schema=target.embedding_schema)
     except Exception as exc:
         # Broad except: reported as a redacted warning.
         failure = classify_connection_error(exc)
@@ -779,6 +805,83 @@ def _functional_lower_index_exists(
     return any(index.get("name") in aliases for index in indexes)
 
 
+def _normalized_index_diagnostics(
+    connection: Connection,
+    schema: str | None,
+    existing_tables: set[str],
+) -> tuple[ResourceDiagnostic, ...]:
+    targets = tuple(
+        target
+        for target in GROUNDWORKERS_NORMALIZED_INDEX_TARGETS
+        if target[0] in existing_tables
+    )
+    if not targets:
+        return ()
+    statuses = tuple(
+        (label, normalized_index_status(connection, schema, table, column))
+        for table, column, label, _name in targets
+    )
+    unchecked = tuple(label for label, present in statuses if present is None)
+    if unchecked:
+        return (
+            _info(
+                "normalized_indexes_unchecked",
+                "Normalized-name indexes unchecked because PostgreSQL index listings "
+                f"could not be read: {', '.join(unchecked)}.",
+            ),
+        )
+    missing = tuple(label for label, present in statuses if not present)
+    if missing:
+        return (
+            _warning(
+                "normalized_indexes_missing",
+                "Optional normalized-name indexes are missing or out of date: "
+                f"{', '.join(missing)}. Groundworkers can still run, but unbounded "
+                "normalized searches scan the whole vocabulary table.",
+            ),
+        )
+    return (
+        _info(
+            "normalized_indexes_present",
+            "Optional normalized-name indexes are present: "
+            f"{', '.join(label for _table, _column, label, _name in targets)}.",
+        ),
+    )
+
+
+def normalized_index_matches(
+    connection: Connection,
+    schema: str | None,
+    table_name: str,
+    column_name: str,
+) -> bool:
+    """True when an index on *table_name* covers the shared K expression."""
+    return normalized_index_status(connection, schema, table_name, column_name) is True
+
+
+def normalized_index_status(
+    connection: Connection,
+    schema: str | None,
+    table_name: str,
+    column_name: str,
+) -> bool | None:
+    """Return True/False for the K index, or None when listing failed."""
+    definitions = _postgres_index_definitions(connection, schema, table_name)
+    if definitions is None:
+        return None
+    expected = _squash_index_sql(
+        normalization_key_expression_sql(connection, column_name)
+    )
+    return any(expected in _squash_index_sql(indexdef) for indexdef in definitions)
+
+
+def _squash_index_sql(sql: str) -> str:
+    # pg_get_indexdef adds ::text casts and parentheses the compiled expression
+    # lacks; drop both, plus whitespace and identifier quotes, on both sides.
+    squashed = sql.lower().replace("::text", "")
+    return "".join(ch for ch in squashed if ch not in ' \t\n"()')
+
+
 def _trigram_lower_index_exists(
     connection: Connection,
     inspector: Any,
@@ -811,6 +914,21 @@ def _postgres_lower_index_exists(
     *,
     required_fragment: str | None = None,
 ) -> bool:
+    return any(
+        _index_defines_lower_expression(
+            indexdef,
+            column_name,
+            required_fragment=required_fragment,
+        )
+        for indexdef in (_postgres_index_definitions(connection, schema, table_name) or ())
+    )
+
+
+def _postgres_index_definitions(
+    connection: Connection,
+    schema: str | None,
+    table_name: str,
+) -> tuple[str, ...] | None:
     try:
         rows = connection.execute(
             text(
@@ -824,16 +942,9 @@ def _postgres_lower_index_exists(
             {"schema": schema or "public", "table_name": table_name},
         )
     except Exception:
-        # Broad except: fall back to reflected index names.
-        return False
-    return any(
-        _index_defines_lower_expression(
-            str(row[0]),
-            column_name,
-            required_fragment=required_fragment,
-        )
-        for row in rows
-    )
+        # Broad except: callers fall back to reflected names or report absence.
+        return None
+    return tuple(str(row[0]) for row in rows)
 
 
 def _index_defines_lower_expression(
@@ -903,10 +1014,7 @@ def _column_has_null_values(
     column = quote_identifier(connection, column_name)
     return bool(
         connection.execute(
-            text(
-                f"SELECT EXISTS (SELECT 1 FROM {qualified} "
-                f"WHERE {column} IS NULL)"
-            )
+            text(f"SELECT EXISTS (SELECT 1 FROM {qualified} WHERE {column} IS NULL)")
         ).scalar()
     )