google-cloud-db-context-engineering 0.7.2__tar.gz → 0.7.4__tar.gz

{google_cloud_db_context_engineering-0.7.2/src/google_cloud_db_context_engineering.egg-info → google_cloud_db_context_engineering-0.7.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: google-cloud-db-context-engineering
-Version: 0.7.2
+Version: 0.7.4
 Summary: A FastMCP server for generating natural language to SQL templates from database schemas.
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
@@ -16,19 +16,17 @@ Requires-Dist: pytest; extra == "test"
 Requires-Dist: pytest-asyncio; extra == "test"
 Dynamic: license-file
 
-This is not an officially supported Google product. This project is not eligible for the [Google Open Source Software Vulnerability Rewards Program](https://bughunters.google.com/open-source-security), [Google Cloud Platform/SecOps Terms of Service](https://cloud.google.com/terms), [How Gemini for Google Cloud uses your data](https://cloud.google.com/gemini/docs/discover/data-governance). This tool is provided "as is" without warranty of any kind. Users are solely responsible for understanding and managing the tool's interaction with their databases. Use of this tool constitutes acceptance of all risks associated with database access, reading, usage, and modifications.
-
 # Context Engineering Agent
 
-The **Context Engineering Agent** is an AI coding agent plugin designed to run in developer agent harnesses (such as Claude Code, Antigravity, or Gemini CLI). It generates, evaluates, and iteratively tunes tailored context artifacts (`ContextSets` comprising `Templates`, `Facets`, and `Value Searches`) to enrich database schemas for **Gemini Data Analytics's data agent developer platform tools**, supporting both **relational SQL** and **Graph Query Language (GQL)** across [AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/data-agent-overview), Cloud SQL ([PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/data-agent-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/data-agent-overview)), and [Cloud Spanner (GoogleSQL & Spanner Graph)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/data-agent-overview).
+The **Context Engineering Agent** is an AI coding agent plugin designed to run in developer agent harnesses (such as Claude Code, Antigravity, or Gemini CLI). It generates, evaluates, and iteratively tunes tailored context artifacts (`ContextSets` comprising `Templates`, `Facets`, and `Value Searches`) to enrich database schemas for **Gemini Data Analytics's data agent developer platform tools**, supporting both **relational SQL** and **Graph Query Language (GQL)** across [AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/data-agent-overview), Cloud SQL ([PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/data-agent-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/data-agent-overview)), and [Cloud Spanner (GoogleSQL, Spanner Graph, and PostgreSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/data-agent-overview).
 
 ---
 
 ## Why Context Engineering?
 
-When building data agents and natural language analytics interfaces, accurately translating user intent into database queries—whether relational SQL, pure GQL, or hybrid graph queries—is critical. 
+When building data agents and natural language analytics interfaces, accurately translating user intent into database queries—whether relational SQL (PostgreSQL, GoogleSQL, MySQL), pure GQL, or hybrid graph queries—is critical. 
 
-As outlined in **Build Context with Context Engineering Agent** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli) | [Spanner (GoogleSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli)), by optimizing a `ContextSet` to match your application's expected query stream, the **QueryData API** acts as a data agent tool capable of achieving **~100% NL-to-SQL/GQL translation accuracy with low latency**.
+As outlined in **Build Context with Context Engineering Agent** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli) | [Spanner](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli)), by optimizing a `ContextSet` to match your application's expected query stream, the **QueryData API** acts as a data agent tool capable of achieving **~100% NL-to-SQL/GQL translation accuracy with low latency**.
 
 ---
 
@@ -40,7 +38,7 @@ A `ContextSet` is the central artifact generated and managed by the agent, conta
 * **Facets**: Reusable, modular query fragments (e.g., parameterized `WHERE` clauses, specialized join filters, or graph `MATCH` traversal patterns) linked to domain vocabulary.
 * **Value Searches**: Specialized mapping queries that dynamically resolve user-supplied values (e.g., *"Lndn"*) to database records (*"London"*) via the capabilities of the underlying database, such as embedding search, AI operators, or trigram search on relational and graph property tables.
 
-For full schema details, structure specifications, and dialect-specific JSON representations of `ContextSets`, see the official **Context Sets Overview** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/context-sets-overview) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/context-sets-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/context-sets-overview) | [Spanner (GoogleSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/context-sets-overview)).
+For full schema details, structure specifications, and dialect-specific JSON representations of `ContextSets`, see the official **Context Sets Overview** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/context-sets-overview) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/context-sets-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/context-sets-overview) | [Spanner](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/context-sets-overview)).
 
 ---
 
@@ -49,13 +47,13 @@ For full schema details, structure specifications, and dialect-specific JSON rep
 Before getting started, prepare your GCP environment, required APIs (Data Analytics API, Gemini for Google Cloud API, Dataplex Universal Catalog API), IAM permissions, and database Data API settings.
 
 Follow the step-by-step setup guide in the official documentation:
-👉 **Prepare Your Environment**: ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli#prepare-your-environment) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli#prepare-your-environment) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli#prepare-your-environment) | [Spanner (GoogleSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli#prepare-your-environment))
+👉 **Prepare Your Environment**: ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli#prepare-your-environment) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli#prepare-your-environment) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli#prepare-your-environment) | [Spanner](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli#prepare-your-environment))
 
 ---
 
 ## Primary Workflow Phases
 
-The agent enables you to craft an optimized context for QueryData API through three primary phases:
+The agent enables you to craft an optimized context for QueryData API through three primary phases. Depending on your needs, you may also toggle the agent to skip phases. For example, if you have a dataset already, you can skip directly to context optimization "optimize context using dataset in \<file\>."
 
 ### Phase 1: Artifact Ingestion
 *Why it matters: Without broader context on the application's goals and scope, AI models generate sterile queries based solely on database column names, missing how your users actually ask for information.*
@@ -80,8 +78,7 @@ The optimization loop creates an initial `ContextSet` and then iteratively refin
 1. **Bootstrap**: Generate an initial baseline context.
 2. **Evaluate**: Measure context effectiveness against a golden dataset.
 3. **Hill-Climbing**: Perform gap analysis on failures and generate automated fixes.
-4. **Iterate**: Apply the improved context and re-run evaluation to continuously improve quality.
-5. **Final Validation** (Optional): Verify mutations against a separated test set to ensure generalization and prevent overfitting.
+4. **Iterate**: Apply the improved context and re-run evaluation to continuously improve quality until we reach an optimal point.
 
 *Note: While there is a typical ordering for these CUJs, the agent is flexible in how you want to execute. You can run the full pipeline end-to-end, trigger any individual phase, or ask for targeted changes to the `ContextSet`.*
 

{google_cloud_db_context_engineering-0.7.2 → google_cloud_db_context_engineering-0.7.4}/README.md

@@ -1,16 +1,14 @@
-This is not an officially supported Google product. This project is not eligible for the [Google Open Source Software Vulnerability Rewards Program](https://bughunters.google.com/open-source-security), [Google Cloud Platform/SecOps Terms of Service](https://cloud.google.com/terms), [How Gemini for Google Cloud uses your data](https://cloud.google.com/gemini/docs/discover/data-governance). This tool is provided "as is" without warranty of any kind. Users are solely responsible for understanding and managing the tool's interaction with their databases. Use of this tool constitutes acceptance of all risks associated with database access, reading, usage, and modifications.
-
 # Context Engineering Agent
 
-The **Context Engineering Agent** is an AI coding agent plugin designed to run in developer agent harnesses (such as Claude Code, Antigravity, or Gemini CLI). It generates, evaluates, and iteratively tunes tailored context artifacts (`ContextSets` comprising `Templates`, `Facets`, and `Value Searches`) to enrich database schemas for **Gemini Data Analytics's data agent developer platform tools**, supporting both **relational SQL** and **Graph Query Language (GQL)** across [AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/data-agent-overview), Cloud SQL ([PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/data-agent-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/data-agent-overview)), and [Cloud Spanner (GoogleSQL & Spanner Graph)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/data-agent-overview).
+The **Context Engineering Agent** is an AI coding agent plugin designed to run in developer agent harnesses (such as Claude Code, Antigravity, or Gemini CLI). It generates, evaluates, and iteratively tunes tailored context artifacts (`ContextSets` comprising `Templates`, `Facets`, and `Value Searches`) to enrich database schemas for **Gemini Data Analytics's data agent developer platform tools**, supporting both **relational SQL** and **Graph Query Language (GQL)** across [AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/data-agent-overview), Cloud SQL ([PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/data-agent-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/data-agent-overview)), and [Cloud Spanner (GoogleSQL, Spanner Graph, and PostgreSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/data-agent-overview).
 
 ---
 
 ## Why Context Engineering?
 
-When building data agents and natural language analytics interfaces, accurately translating user intent into database queries—whether relational SQL, pure GQL, or hybrid graph queries—is critical. 
+When building data agents and natural language analytics interfaces, accurately translating user intent into database queries—whether relational SQL (PostgreSQL, GoogleSQL, MySQL), pure GQL, or hybrid graph queries—is critical. 
 
-As outlined in **Build Context with Context Engineering Agent** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli) | [Spanner (GoogleSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli)), by optimizing a `ContextSet` to match your application's expected query stream, the **QueryData API** acts as a data agent tool capable of achieving **~100% NL-to-SQL/GQL translation accuracy with low latency**.
+As outlined in **Build Context with Context Engineering Agent** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli) | [Spanner](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli)), by optimizing a `ContextSet` to match your application's expected query stream, the **QueryData API** acts as a data agent tool capable of achieving **~100% NL-to-SQL/GQL translation accuracy with low latency**.
 
 ---
 
@@ -22,7 +20,7 @@ A `ContextSet` is the central artifact generated and managed by the agent, conta
 * **Facets**: Reusable, modular query fragments (e.g., parameterized `WHERE` clauses, specialized join filters, or graph `MATCH` traversal patterns) linked to domain vocabulary.
 * **Value Searches**: Specialized mapping queries that dynamically resolve user-supplied values (e.g., *"Lndn"*) to database records (*"London"*) via the capabilities of the underlying database, such as embedding search, AI operators, or trigram search on relational and graph property tables.
 
-For full schema details, structure specifications, and dialect-specific JSON representations of `ContextSets`, see the official **Context Sets Overview** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/context-sets-overview) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/context-sets-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/context-sets-overview) | [Spanner (GoogleSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/context-sets-overview)).
+For full schema details, structure specifications, and dialect-specific JSON representations of `ContextSets`, see the official **Context Sets Overview** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/context-sets-overview) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/context-sets-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/context-sets-overview) | [Spanner](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/context-sets-overview)).
 
 ---
 
@@ -31,13 +29,13 @@ For full schema details, structure specifications, and dialect-specific JSON rep
 Before getting started, prepare your GCP environment, required APIs (Data Analytics API, Gemini for Google Cloud API, Dataplex Universal Catalog API), IAM permissions, and database Data API settings.
 
 Follow the step-by-step setup guide in the official documentation:
-👉 **Prepare Your Environment**: ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli#prepare-your-environment) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli#prepare-your-environment) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli#prepare-your-environment) | [Spanner (GoogleSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli#prepare-your-environment))
+👉 **Prepare Your Environment**: ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli#prepare-your-environment) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli#prepare-your-environment) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli#prepare-your-environment) | [Spanner](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli#prepare-your-environment))
 
 ---
 
 ## Primary Workflow Phases
 
-The agent enables you to craft an optimized context for QueryData API through three primary phases:
+The agent enables you to craft an optimized context for QueryData API through three primary phases. Depending on your needs, you may also toggle the agent to skip phases. For example, if you have a dataset already, you can skip directly to context optimization "optimize context using dataset in \<file\>."
 
 ### Phase 1: Artifact Ingestion
 *Why it matters: Without broader context on the application's goals and scope, AI models generate sterile queries based solely on database column names, missing how your users actually ask for information.*
@@ -62,8 +60,7 @@ The optimization loop creates an initial `ContextSet` and then iteratively refin
 1. **Bootstrap**: Generate an initial baseline context.
 2. **Evaluate**: Measure context effectiveness against a golden dataset.
 3. **Hill-Climbing**: Perform gap analysis on failures and generate automated fixes.
-4. **Iterate**: Apply the improved context and re-run evaluation to continuously improve quality.
-5. **Final Validation** (Optional): Verify mutations against a separated test set to ensure generalization and prevent overfitting.
+4. **Iterate**: Apply the improved context and re-run evaluation to continuously improve quality until we reach an optimal point.
 
 *Note: While there is a typical ordering for these CUJs, the agent is flexible in how you want to execute. You can run the full pipeline end-to-end, trigger any individual phase, or ask for targeted changes to the `ContextSet`.*
 

{google_cloud_db_context_engineering-0.7.2 → google_cloud_db_context_engineering-0.7.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "google-cloud-db-context-engineering"
-version = "0.7.2"
+version = "0.7.4"
 description = "A FastMCP server for generating natural language to SQL templates from database schemas."
 readme = "README.md"
 requires-python = ">=3.12"
@@ -45,8 +45,8 @@ dev = [
 ]
 
 [tool.db-context-engineering]
-toolbox_version = "1.4.0"
-evalbench_version = "1.12.0"
+toolbox_version = "1.10.0"
+evalbench_version = "1.17.0"
 
 [tool.ruff]
 line-length = 88

{google_cloud_db_context_engineering-0.7.2 → google_cloud_db_context_engineering-0.7.4}/src/google/cloud/db_context_enrichment/common/context_mutator.py

@@ -48,7 +48,7 @@ def mutate_context_set(file_path: str, mutations: list[Mutation]) -> None:
         context_set = context.ContextSet()
     else:
         try:
-            with open(file_path) as f:
+            with open(file_path, encoding="utf-8") as f:
                 raw_data = json.load(f)
             context_set = context.ContextSet.model_validate(raw_data)
         except ValidationError as e:
@@ -126,7 +126,7 @@ def mutate_context_set(file_path: str, mutations: list[Mutation]) -> None:
     # 4. Save validated ContextSet
     try:
         os.makedirs(os.path.dirname(os.path.abspath(file_path)), exist_ok=True)
-        with open(file_path, "w") as f:
+        with open(file_path, "w", encoding="utf-8") as f:
             f.write(context_set.model_dump_json(indent=2, exclude_none=True))
     except OSError as e:
         raise RuntimeError(f"Error saving ContextSet to {file_path}: {e}") from e

google_cloud_db_context_engineering-0.7.4/src/google/cloud/db_context_enrichment/common/context_validator.py

@@ -0,0 +1,188 @@
+import json
+from typing import Any
+
+from pydantic import ValidationError
+
+from google.cloud.db_context_enrichment.model import context
+
+_ATTR_TO_TYPE = {
+    "templates": "template",
+    "facets": "facet",
+    "value_searches": "value_search",
+}
+
+# ContextSet accepts camelCase aliases (Context Store returns them on download)
+# and the deprecated "fragments" name. Map them back to the canonical attribute
+# so every check below only has to handle one spelling.
+_ALIAS_TO_ATTR = {
+    "valueSearches": "value_searches",
+    "fragments": "facets",
+}
+
+
+def validate_context_set(file_path: str) -> dict[str, Any]:
+    """Validate a ContextSet file and return a structured report of issues.
+
+    Always returns a dict of the shape:
+      {
+        "valid": bool,
+        "issues": [
+          {
+            "location": {"type": str, "index": int} | None,
+            "message": str,
+          },
+          ...
+        ],
+      }
+
+    File access failures (missing path, permission denied, path is a directory,
+    etc.) are surfaced as a single issue rather than raised.
+    """
+    try:
+        with open(file_path, encoding="utf-8") as f:
+            text = f.read()
+    except OSError as e:
+        return {
+            "valid": False,
+            "issues": [
+                _make_issue(f"Could not read file {file_path}: {type(e).__name__}: {e}")
+            ],
+        }
+
+    if text.strip() == "":
+        return {"valid": True, "issues": []}
+
+    try:
+        raw = json.loads(text)
+    except json.JSONDecodeError as e:
+        snippet = e.doc[max(0, e.pos - 30) : e.pos + 30]
+        return {
+            "valid": False,
+            "issues": [_make_issue(f"File is not valid JSON: {e}. Near: {snippet!r}")],
+        }
+
+    if not isinstance(raw, dict):
+        return {
+            "valid": False,
+            "issues": [_make_issue("Top-level value must be a JSON object")],
+        }
+
+    raw = _normalize_aliases(raw)
+
+    issues: list[dict[str, Any]] = []
+    issues.extend(_check_pydantic(raw))
+    issues.extend(_check_duplicates(raw))
+    issues.extend(_check_value_search_value_param(raw))
+
+    return {"valid": len(issues) == 0, "issues": issues}
+
+
+def _make_issue(message: str, location: dict[str, Any] | None = None) -> dict[str, Any]:
+    return {"location": location, "message": message}
+
+
+def _normalize_aliases(raw: dict[str, Any]) -> dict[str, Any]:
+    """Rewrite alias keys to their canonical ContextSet attribute names."""
+    return {_ALIAS_TO_ATTR.get(key, key): value for key, value in raw.items()}
+
+
+def _check_pydantic(raw: dict[str, Any]) -> list[dict[str, Any]]:
+    issues: list[dict[str, Any]] = []
+    try:
+        context.ContextSet.model_validate(raw)
+        return issues
+    except ValidationError as e:
+        for err in e.errors():
+            loc = err.get("loc", ())
+            msg = err.get("msg", "validation error")
+            location: dict[str, Any] | None = None
+            descriptor = ""
+
+            if len(loc) >= 1 and loc[0] in _ATTR_TO_TYPE:
+                item_type = _ATTR_TO_TYPE[loc[0]]
+                if len(loc) >= 2 and isinstance(loc[1], int):
+                    item_index = loc[1]
+                    location = {"type": item_type, "index": item_index}
+                    items = raw.get(loc[0])
+                    if isinstance(items, list) and 0 <= item_index < len(items):
+                        descriptor = _describe(item_type, items[item_index])
+
+            field_path = ".".join(str(p) for p in loc) if loc else ""
+            parts = [msg]
+            if field_path:
+                parts.append(f"at {field_path}")
+            if descriptor:
+                parts.append(descriptor)
+            full_msg = parts[0] + (
+                " (" + "; ".join(parts[1:]) + ")" if len(parts) > 1 else ""
+            )
+
+            issues.append(_make_issue(full_msg, location=location))
+    return issues
+
+
+def _check_duplicates(raw: dict[str, Any]) -> list[dict[str, Any]]:
+    issues: list[dict[str, Any]] = []
+    for attr, item_type in _ATTR_TO_TYPE.items():
+        items = raw.get(attr)
+        if not isinstance(items, list):
+            continue
+        seen: dict[str, int] = {}
+        for idx, item in enumerate(items):
+            try:
+                key = _canonical(item)
+            except (TypeError, ValueError):
+                continue
+            if key in seen:
+                descriptor = _describe(item_type, item)
+                suffix = f" ({descriptor})" if descriptor else ""
+                issues.append(
+                    _make_issue(
+                        f"Exact duplicate of {item_type} at index {seen[key]}{suffix}",
+                        location={"type": item_type, "index": idx},
+                    )
+                )
+            else:
+                seen[key] = idx
+    return issues
+
+
+def _check_value_search_value_param(raw: dict[str, Any]) -> list[dict[str, Any]]:
+    issues: list[dict[str, Any]] = []
+    items = raw.get("value_searches")
+    if not isinstance(items, list):
+        return issues
+    for idx, item in enumerate(items):
+        if not isinstance(item, dict):
+            continue
+        query = item.get("query")
+        if isinstance(query, str) and "$value" not in query:
+            descriptor = _describe("value_search", item)
+            suffix = f" ({descriptor})" if descriptor else ""
+            issues.append(
+                _make_issue(
+                    f"value_search query must reference the $value parameter{suffix}",
+                    location={"type": "value_search", "index": idx},
+                )
+            )
+    return issues
+
+
+def _describe(item_type: str, item: Any) -> str:
+    """Short human-readable identifier for an item, e.g. "intent: 'active users'"."""
+    if not isinstance(item, dict):
+        return ""
+    if item_type == "template":
+        nl = item.get("nl_query")
+        return f"nl_query: {nl!r}" if nl else ""
+    if item_type == "facet":
+        intent = item.get("intent")
+        return f"intent: {intent!r}" if intent else ""
+    if item_type == "value_search":
+        ct = item.get("concept_type")
+        return f"concept_type: {ct!r}" if ct else ""
+    return ""
+
+
+def _canonical(obj: Any) -> str:
+    return json.dumps(obj, sort_keys=True, ensure_ascii=False)

google_cloud_db_context_engineering-0.7.4/src/google/cloud/db_context_enrichment/evaluate/db_generators/__init__.py

@@ -0,0 +1,15 @@
+from .alloydb import AlloyDBConfigGenerator
+from .base import BaseDBConfigGenerator
+from .custom import CustomDBConfigGenerator
+from .mysql import MySQLConfigGenerator
+from .postgres import PostgresConfigGenerator
+from .spanner import SpannerConfigGenerator
+
+__all__ = [
+    "BaseDBConfigGenerator",
+    "AlloyDBConfigGenerator",
+    "CustomDBConfigGenerator",
+    "MySQLConfigGenerator",
+    "PostgresConfigGenerator",
+    "SpannerConfigGenerator",
+]

google_cloud_db_context_engineering-0.7.4/src/google/cloud/db_context_enrichment/evaluate/db_generators/bigtable.py

@@ -0,0 +1,44 @@
+from typing import Any
+
+import yaml
+
+from .base import BaseDBConfigGenerator
+
+
+class BigtableConfigGenerator(BaseDBConfigGenerator):
+    """
+    Dedicated generator mapping properties to explicit Bigtable configuration
+    topologies. Bypasses the GDA Python SDK to construct the model configuration
+    directly, as the SDK may lack native BigtableReference definitions.
+    """
+
+    SOURCE_TYPE = "bigtable"
+    DIALECT = "bigtable"
+    REQUIRED_FIELDS = {"project", "instance"}
+
+    def generate_db_config(self) -> str:
+        db_config = {
+            "db_type": "bigtable",
+            "dialect": self.DIALECT,
+            "database_name": self.params.get("instance"),
+            "database_path": f"projects/{self.params.get('project')}/instances/{self.params.get('instance')}",
+            "instance_id": self.params.get("instance"),
+            "gcp_project_id": self.params.get("project"),
+            "max_executions_per_minute": 100,
+        }
+        return yaml.safe_dump(
+            db_config, sort_keys=False, default_flow_style=False
+        ).strip()
+
+    def build_datasource_reference(self, context_set_id: str) -> dict[str, Any]:
+        return {
+            "bigtable_reference": {
+                "database_reference": {
+                    "project_id": self.params.get("project"),
+                    "instance_id": self.params.get("instance"),
+                },
+                "agent_context_reference": {
+                    "context_set_id": context_set_id,
+                },
+            }
+        }

google_cloud_db_context_engineering-0.7.4/src/google/cloud/db_context_enrichment/evaluate/db_generators/custom.py

@@ -0,0 +1,104 @@
+import os
+from typing import Any
+
+import yaml
+
+from .base import BaseDBConfigGenerator
+
+
+class CustomDBConfigGenerator(BaseDBConfigGenerator):
+    """
+    Pluggable generator mapping properties to custom evaluation configurations.
+    Enables arbitrary third-party or proprietary internal engines to be plugged
+    into the evaluation framework via dynamic connector and generator SPIs
+    without exposing engine-specific code or configuration schemas.
+    """
+
+    SOURCE_TYPE = "custom"
+    DIALECT = "sql"
+    REQUIRED_FIELDS = set()
+
+    def __init__(self, params: dict[str, Any]):
+        self.params = params
+        if "project" not in self.params:
+            project_id = os.environ.get("GOOGLE_CLOUD_PROJECT") or os.environ.get(
+                "GCP_PROJECT"
+            )
+            if project_id:
+                self.params["project"] = project_id
+        self.connector_class = params.get("connector_class", "")
+        self.generator_class = params.get("generator_class", "")
+        self.DIALECT = params.get("dialect", self.DIALECT)
+        self.validate()
+
+    def validate(self) -> None:
+        """
+        Validates that either connector_class or generator_class is specified.
+        """
+        if not self.connector_class and not self.generator_class:
+            raise ValueError(
+                "Custom source configuration must specify at least 'connector_class' or 'generator_class'."
+            )
+
+    def generate_db_config(self) -> str:
+        """
+        Generates the db_config.yaml payload for custom connectors.
+        """
+        db_config: dict[str, Any] = {
+            "db_type": self.params.get("db_type", "custom"),
+            "dialect": self.DIALECT,
+        }
+        if self.connector_class:
+            db_config["connector_class"] = self.connector_class
+
+        # Forward all non-meta parameters from tools.yaml source block
+        excluded_keys = {
+            "kind",
+            "name",
+            "type",
+            "connector_class",
+            "generator_class",
+            "dialect",
+            "db_type",
+        }
+        for key, value in self.params.items():
+            if key not in excluded_keys:
+                db_config[key] = value
+
+        return yaml.safe_dump(
+            db_config, sort_keys=False, default_flow_style=False
+        ).strip()
+
+    def build_datasource_reference(self, context_set_id: str) -> dict[str, Any]:
+        """
+        Datasource reference dictionary. Custom engines manage their own
+        schema references or context binding.
+        """
+        return {}
+
+    def generate_model_config(self, context_set_id: str) -> str:
+        """
+        Generates model_config.yaml. If generator_class is specified, produces
+        a custom generator configuration for dynamic instantiation.
+        """
+        if self.generator_class:
+            model_config: dict[str, Any] = {
+                "generator": "custom",
+                "generator_class": self.generator_class,
+                "context_set_id": context_set_id,
+            }
+            excluded_keys = {
+                "kind",
+                "name",
+                "type",
+                "connector_class",
+                "generator_class",
+            }
+            for key, value in self.params.items():
+                if key not in excluded_keys:
+                    model_config[key] = value
+            return yaml.safe_dump(
+                model_config, sort_keys=False, default_flow_style=False
+            ).strip()
+
+        return super().generate_model_config(context_set_id)

google_cloud_db_context_engineering-0.7.4/src/google/cloud/db_context_enrichment/evaluate/db_generators/firestore.py

@@ -0,0 +1,60 @@
+from typing import Any
+
+import yaml
+
+from .base import BaseDBConfigGenerator
+
+
+class FirestoreConfigGenerator(BaseDBConfigGenerator):
+    """
+    Dedicated generator mapping properties to Firestore (MongoDB query dialect)
+    topologies utilized by both EvalBench binaries and GDA REST API.
+    """
+
+    SOURCE_TYPE = "firestore"
+    DIALECT = "mongodb"
+    REQUIRED_FIELDS = BaseDBConfigGenerator.REQUIRED_FIELDS | {
+        "project",
+        "database",
+    }
+
+    def __init__(self, params: dict[str, Any]):
+        super().__init__(params)
+        self.project = params.get("project")
+        self.database = params.get("database")
+        self.connection_string = params.get("connection_string")
+        self.collection_ids = params.get("collection_ids") or params.get("table_ids")
+
+    def generate_db_config(self) -> str:
+        db_type = "mongodb"
+
+        db_config = {
+            "db_type": db_type,
+            "dialect": self.DIALECT,
+            "database_name": self.database,
+            "database_path": "",
+            "firestore_database": f"projects/{self.project}/databases/{self.database}",
+            "max_executions_per_minute": 120,
+        }
+        if self.connection_string:
+            db_config["connection_string"] = self.connection_string
+
+        return yaml.safe_dump(
+            db_config, sort_keys=False, default_flow_style=False
+        ).strip()
+
+    def build_datasource_reference(self, context_set_id: str) -> dict[str, Any]:
+        db_ref: dict[str, Any] = {
+            "project_id": self.project,
+            "database_id": self.database,
+        }
+        if self.collection_ids:
+            db_ref["collection_ids"] = self.collection_ids
+
+        ref: dict[str, Any] = {"firestore_reference": {"database_reference": db_ref}}
+        if context_set_id:
+            ref["firestore_reference"]["agent_context_reference"] = {
+                "context_set_id": context_set_id
+            }
+
+        return ref

{google_cloud_db_context_engineering-0.7.2 → google_cloud_db_context_engineering-0.7.4}/src/google/cloud/db_context_enrichment/evaluate/db_generators/spanner.py

@@ -9,10 +9,10 @@ class SpannerConfigGenerator(BaseDBConfigGenerator):
     """
     Dedicated generator mapping properties to explicit Spanner configuration
     topologies utilized by both EvalBench binaries and GDA Context objects.
+    Supports both GoogleSQL and PostgreSQL dialects.
     """
 
     SOURCE_TYPE = "spanner"
-    DIALECT = "spanner_gsql"
     REQUIRED_FIELDS = BaseDBConfigGenerator.REQUIRED_FIELDS | {
         "project",
         "instance",
@@ -25,6 +25,40 @@ class SpannerConfigGenerator(BaseDBConfigGenerator):
         self.instance = params.get("instance")
         self.database = params.get("database")
 
+        raw_dialect = (
+            params.get("dialect")
+            or params.get("engine")
+            or params.get("database_dialect")
+        )
+        if not raw_dialect and params.get("type") in ("spanner-postgres", "spanner-pg"):
+            raw_dialect = "POSTGRESQL"
+
+        if raw_dialect:
+            normalized = str(raw_dialect).strip().lower().replace("-", "_")
+            if normalized in (
+                "postgresql",
+                "postgres",
+                "spanner_pg",
+                "pg",
+                "spanner_postgres",
+            ):
+                self.engine = "POSTGRESQL"
+                self._dialect = "spanner_pg"
+            elif normalized in ("google_sql", "googlesql", "spanner_gsql", "gsql"):
+                self.engine = "GOOGLE_SQL"
+                self._dialect = "spanner_gsql"
+            else:
+                raise ValueError(
+                    f"Unsupported Spanner dialect/engine: '{raw_dialect}'. Must be 'GOOGLE_SQL' or 'POSTGRESQL'."
+                )
+        else:
+            self.engine = "GOOGLE_SQL"
+            self._dialect = "spanner_gsql"
+
+    @property
+    def DIALECT(self) -> str:
+        return self._dialect
+
     def generate_db_config(self) -> str:
         db_type = "spanner"
         db_path = f"projects/{self.project}/instances/{self.instance}/databases/{self.database}"
@@ -44,12 +78,16 @@ class SpannerConfigGenerator(BaseDBConfigGenerator):
 
     def build_datasource_reference(self, context_set_id: str) -> dict[str, Any]:
         database_ref: dict[str, Any] = {
-            "engine": "GOOGLE_SQL",
+            "engine": self.engine,
             "project_id": self.project,
             "instance_id": self.instance,
             "database_id": self.database,
         }
         if graph_ids := self.params.get("graph_ids"):
+            if self.engine == "POSTGRESQL":
+                raise ValueError(
+                    "graph_ids is not supported for Spanner PostgreSQL dialect"
+                )
             if not isinstance(graph_ids, list) or not all(
                 isinstance(g, str) for g in graph_ids
             ):

{google_cloud_db_context_engineering-0.7.2 → google_cloud_db_context_engineering-0.7.4}/src/google/cloud/db_context_enrichment/evaluate/evaluate_generator.py

@@ -1,7 +1,11 @@
+import importlib
 import json
+import logging
 import os
 import re
+import sys
 import textwrap
+import threading
 from typing import Any
 
 import yaml
@@ -10,10 +14,15 @@ from google.cloud.db_context_enrichment.common import config
 
 from .db_generators.alloydb import AlloyDBConfigGenerator
 from .db_generators.base import BaseDBConfigGenerator
+from .db_generators.bigtable import BigtableConfigGenerator
+from .db_generators.custom import CustomDBConfigGenerator
+from .db_generators.firestore import FirestoreConfigGenerator
 from .db_generators.mysql import MySQLConfigGenerator
 from .db_generators.postgres import PostgresConfigGenerator
 from .db_generators.spanner import SpannerConfigGenerator
 
+logger = logging.getLogger(__name__)
+
 # Constants for EvalBench configuration filenames
 DB_CONFIG_NAME = "db_config.yaml"
 MODEL_CONFIG_NAME = "model_config.yaml"
@@ -76,10 +85,10 @@ def _extract_toolbox_params(
         with open(toolbox_config_path) as f:
             content = f.read()
             interpolated = _interpolate_env_vars(content)
-            docs = yaml.safe_load_all(interpolated)
+            docs = [doc for doc in yaml.safe_load_all(interpolated) if doc]
+
+            source_doc = None
             for doc in docs:
-                if not doc:
-                    continue
                 if (
                     doc.get("kind") == "source"
                     and doc.get("name") == toolbox_source_name
@@ -88,11 +97,24 @@ def _extract_toolbox_params(
                         raise ValueError(
                             f"Selected source '{toolbox_source_name}' is missing the 'type' field."
                         )
-                    return doc
+                    source_doc = doc
+                    break
 
-            raise ValueError(
-                f"Could not find a 'kind: source' named '{toolbox_source_name}' in {toolbox_config_path}"
-            )
+            if not source_doc:
+                raise ValueError(
+                    f"Could not find a 'kind: source' named '{toolbox_source_name}' in {toolbox_config_path}"
+                )
+
+            # For Spanner sources, state.md is the authoritative single source of truth for graph_ids
+            # (QueryData API requires explicit graph_ids in model_config.yaml, whereas tools.yaml
+            # only configures MCP Toolbox runtime tools and parameters).
+            if source_doc.get("type") == "spanner":
+                state_md_dir = os.path.dirname(toolbox_config_path)
+                state_md_path = os.path.join(state_md_dir, "state.md")
+                if graph_ids := _parse_graph_ids_from_state_md(state_md_path):
+                    source_doc["graph_ids"] = graph_ids
+
+            return source_doc
 
     except FileNotFoundError:
         raise ValueError(f"Config file not found: {toolbox_config_path}")
@@ -104,6 +126,72 @@ def _extract_toolbox_params(
         raise ValueError(f"Failed to parse {toolbox_config_path} as YAML: {e}")
 
 
+def _parse_graph_ids_from_state_md(state_md_path: str) -> list[str] | None:
+    """Parses graph_ids from state.md.
+
+    state.md is the authoritative source of truth for the database and graph scope
+    because QueryData API requires explicit graph_ids in model_config.yaml to evaluate
+    property graphs, whereas tools.yaml only configures MCP Toolbox runtime tools.
+    """
+    if not os.path.exists(state_md_path):
+        return None
+    with open(state_md_path, encoding="utf-8") as f:
+        content = f.read()
+    match = re.search(
+        r"(?:^[ \t]*[-*][ \t]*)?\*\*Graph\s+Ids?:?\*\*:?[ \t]*([^\n]*)",
+        content,
+        re.MULTILINE | re.IGNORECASE,
+    )
+    if not match:
+        return None
+
+    val_str = match.group(1).split("#")[0].strip()
+
+    # If empty on the same line, check for multiline sub-bullets
+    if not val_str:
+        after_match = content[match.end() :]
+        bullet_items = []
+        for line in after_match.splitlines():
+            line_stripped = line.strip()
+            if not line_stripped:
+                continue
+            if line_stripped.startswith(("-", "*")) and not re.match(
+                r"^[-*]\s*\*\*", line_stripped
+            ):
+                item = line_stripped.lstrip("-* ").split("#")[0].strip().strip("'\"`")
+                if item and item.lower() not in (
+                    "none",
+                    "n/a",
+                    "null",
+                    "nil",
+                    "-",
+                ):
+                    bullet_items.append(item)
+            elif (
+                line_stripped.startswith("#")
+                or line_stripped.startswith("- **")
+                or line_stripped.startswith("* **")
+            ):
+                break
+            else:
+                break
+        return bullet_items if bullet_items else None
+
+    # Handle explicit empty / none indicators
+    if val_str.lower() in ("none", "n/a", "null", "nil", "-", "[]", ""):
+        return None
+
+    if val_str.startswith("[") and val_str.endswith("]"):
+        val_str = val_str[1:-1]
+    graphs = [
+        g.strip().strip("'\"`")
+        for g in val_str.split(",")
+        if g.strip().strip("'\"`")
+        and g.strip().strip("'\"`").lower() not in ("none", "n/a", "null", "nil", "-")
+    ]
+    return graphs if graphs else None
+
+
 def _interpolate_env_vars(raw_yaml: str) -> str:
     """Replaces ${ENV_NAME} or ${ENV_NAME:default_value} with environment variables."""
     # Matches ${VAR_NAME} or ${VAR_NAME:fallback}
@@ -124,17 +212,93 @@ def _interpolate_env_vars(raw_yaml: str) -> str:
     return pattern.sub(replacer, raw_yaml)
 
 
+_SYS_PATH_LOCK = threading.Lock()
+
+
+def _import_with_cwd_fallback(mod_name: str):
+    """Imports mod_name, falling back to appending os.getcwd() to sys.path."""
+    try:
+        return importlib.import_module(mod_name)
+    except ModuleNotFoundError as e:
+        if e.name is None or not (
+            mod_name == e.name or mod_name.startswith(e.name + ".")
+        ):
+            raise
+        cwd = os.path.abspath(os.getcwd())
+        with _SYS_PATH_LOCK:
+            if not any(os.path.abspath(p or cwd) == cwd for p in sys.path):
+                # Keep cwd at the end of sys.path so lazy runtime imports work
+                # without shadowing standard library or installed packages.
+                sys.path.append(cwd)
+                importlib.invalidate_caches()
+        return importlib.import_module(mod_name)
+
+
 def _get_db_generator(params: dict[str, Any]) -> BaseDBConfigGenerator:
     """Factory function to build the correct Evaluation Generator."""
     source_type = params.get("type", "").lower()
 
     generators = {
         AlloyDBConfigGenerator.SOURCE_TYPE: AlloyDBConfigGenerator,
+        BigtableConfigGenerator.SOURCE_TYPE: BigtableConfigGenerator,
         PostgresConfigGenerator.SOURCE_TYPE: PostgresConfigGenerator,
         MySQLConfigGenerator.SOURCE_TYPE: MySQLConfigGenerator,
         SpannerConfigGenerator.SOURCE_TYPE: SpannerConfigGenerator,
+        "spanner-postgres": SpannerConfigGenerator,
+        "spanner-pg": SpannerConfigGenerator,
+        FirestoreConfigGenerator.SOURCE_TYPE: FirestoreConfigGenerator,
+        CustomDBConfigGenerator.SOURCE_TYPE: CustomDBConfigGenerator,
     }
 
+    # Dynamically register external custom database configuration generators.
+    # AUTOCTX_CUSTOM_GENERATORS holds a dotted Python module import path
+    # (e.g., "my_package.custom_generators") that exposes a
+    # CUSTOM_GENERATORS: dict[str, type[BaseDBConfigGenerator]] mapping
+    # custom tools.yaml source types to BaseDBConfigGenerator subclasses.
+    custom_gens = {}
+    custom_plugin = os.environ.get("AUTOCTX_CUSTOM_GENERATORS")
+    if custom_plugin:
+        try:
+            mod = _import_with_cwd_fallback(custom_plugin)
+            custom_gens = getattr(mod, "CUSTOM_GENERATORS", None)
+            if custom_gens is None:
+                raise AttributeError(
+                    f"Custom generator module '{custom_plugin}' must define 'CUSTOM_GENERATORS' dict."
+                )
+            if not isinstance(custom_gens, dict):
+                raise TypeError(
+                    f"CUSTOM_GENERATORS in '{custom_plugin}' must be a dictionary, "
+                    f"got {type(custom_gens).__name__}."
+                )
+            generators.update(custom_gens)
+        except Exception as e:
+            logger.error(
+                "Failed to load custom generators from plugin module '%s': %s",
+                custom_plugin,
+                e,
+            )
+            raise RuntimeError(
+                f"Failed to load custom generators from plugin module '{custom_plugin}': {e}"
+            ) from e
+
+    # If the source type is not handled by an external AUTOCTX_CUSTOM_GENERATORS
+    # plugin, allow inline connector_class / generator_class in tools.yaml to
+    # route directly to CustomDBConfigGenerator.
+    if source_type not in custom_gens and (
+        "connector_class" in params or "generator_class" in params
+    ):
+        if (
+            source_type in generators
+            and source_type != CustomDBConfigGenerator.SOURCE_TYPE
+        ):
+            logger.warning(
+                "Source type '%s' is being overridden by custom connector_class '%s' / generator_class '%s'.",
+                source_type,
+                params.get("connector_class"),
+                params.get("generator_class"),
+            )
+        return CustomDBConfigGenerator(params)
+
     if source_type not in generators:
         supported = ", ".join(generators.keys())
         raise ValueError(

{google_cloud_db_context_engineering-0.7.2 → google_cloud_db_context_engineering-0.7.4}/src/google/cloud/db_context_enrichment/main.py

@@ -6,6 +6,7 @@ from fastmcp import FastMCP
 from google.cloud.db_context_enrichment.common import (
     context_mutator,
     context_store_client,
+    context_validator,
 )
 from google.cloud.db_context_enrichment.dataset import dataset_generator
 from google.cloud.db_context_enrichment.evaluate import (
@@ -73,7 +74,7 @@ def generate_evalbench_configs(
         dataset_path: The absolute path to the golden dataset file in the simplified user-facing format (JSON list of objects with keys: "id", "database", "nlq", "golden_sql").
         context_set_id: Full ContextSet resource name to evaluate against.
         toolbox_config_path: The absolute path to the tools.yaml configuration file.
-        toolbox_source_name: The name of the database source to use inside tools.yaml. The underlying source block must use a supported 'type' (cloud-sql-postgres, cloud-sql-mysql, spanner, alloydb-postgres).
+        toolbox_source_name: The name of the database source to use inside tools.yaml. The underlying source block must use a supported 'type' (cloud-sql-postgres, cloud-sql-mysql, spanner, alloydb-postgres, or custom).
 
     Returns:
         A message indicating that the configuration files were successfully created.
@@ -102,13 +103,14 @@ def generate_upload_url(
 
     Args:
         db_engine: The database engine. Accepted values are 'alloydb',
-                 'cloudsql', or 'spanner'. This can be derived from the 'kind'
-                 field in the tools.yaml file. For example, 'alloydb-postgres'
-                 becomes 'alloydb', and 'cloud-sql-postgres' becomes 'cloudsql'.
+                 'cloudsql', 'spanner', or 'bigtable'. This can be derived from
+                 the 'kind' field in the tools.yaml file. For example,
+                 'alloydb-postgres' becomes 'alloydb', 'cloud-sql-postgres'
+                 becomes 'cloudsql', and 'bigtable' becomes 'bigtable'.
         project_id: The Google Cloud project ID.
         location: The location of the AlloyDB cluster.
         cluster_id: The ID of the AlloyDB cluster.
-        instance_id: The ID of the Cloud SQL or Spanner instance.
+        instance_id: The ID of the Cloud SQL, Spanner, or Bigtable instance.
         database_id: The ID of the Spanner database.
 
     Returns:
@@ -129,8 +131,13 @@ def generate_upload_url(
             return f"https://console.cloud.google.com/spanner/instances/{instance_id}/databases/{database_id}/details/query?project={project_id}"
         else:
             return "Error: Missing instance_id, database_id, or project_id for spanner."
+    elif db_engine == "bigtable":
+        if instance_id and project_id:
+            return f"https://console.cloud.google.com/bigtable/instances/{instance_id}/overview?project={project_id}"
+        else:
+            return "Error: Missing instance_id or project_id for bigtable."
     else:
-        return "Error: Invalid db_engine. Must be one of 'alloydb', 'cloudsql', or 'spanner'."
+        return "Error: Invalid db_engine. Must be one of 'alloydb', 'cloudsql', 'spanner', or 'bigtable'."
 
 
 # NOTE: `@mcp.tool` is intentionally NOT applied to upload_context_set /
@@ -258,6 +265,30 @@ def mutate_context_set(
         return f"Error applying mutations: {str(e)}"
 
 
+@mcp.tool
+def validate_context_set(file_path: str) -> str:
+    """
+    Validate a ContextSet JSON file for structural and convention issues. Reports issues only — does not fix them. The caller (agent) is expected to apply fixes via `mutate_context_set`, then re-run validation until `valid` is true.
+
+    Args:
+        file_path: Absolute path to the ContextSet file.
+
+    Returns:
+        A JSON string of the shape:
+          {
+            "valid": bool,
+            "issues": [
+              {
+                "location": {"type": "template" | "facet" | "value_search", "index": int} | null,
+                "message": str
+              },
+              ...
+            ]
+          }
+    """
+    return json.dumps(context_validator.validate_context_set(file_path), indent=2)
+
+
 @mcp.tool
 async def read_evaluation_result(
     run_folder_path: str, offset: int = 0, batch_size: int = 10

{google_cloud_db_context_engineering-0.7.2 → google_cloud_db_context_engineering-0.7.4/src/google_cloud_db_context_engineering.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: google-cloud-db-context-engineering
-Version: 0.7.2
+Version: 0.7.4
 Summary: A FastMCP server for generating natural language to SQL templates from database schemas.
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
@@ -16,19 +16,17 @@ Requires-Dist: pytest; extra == "test"
 Requires-Dist: pytest-asyncio; extra == "test"
 Dynamic: license-file
 
-This is not an officially supported Google product. This project is not eligible for the [Google Open Source Software Vulnerability Rewards Program](https://bughunters.google.com/open-source-security), [Google Cloud Platform/SecOps Terms of Service](https://cloud.google.com/terms), [How Gemini for Google Cloud uses your data](https://cloud.google.com/gemini/docs/discover/data-governance). This tool is provided "as is" without warranty of any kind. Users are solely responsible for understanding and managing the tool's interaction with their databases. Use of this tool constitutes acceptance of all risks associated with database access, reading, usage, and modifications.
-
 # Context Engineering Agent
 
-The **Context Engineering Agent** is an AI coding agent plugin designed to run in developer agent harnesses (such as Claude Code, Antigravity, or Gemini CLI). It generates, evaluates, and iteratively tunes tailored context artifacts (`ContextSets` comprising `Templates`, `Facets`, and `Value Searches`) to enrich database schemas for **Gemini Data Analytics's data agent developer platform tools**, supporting both **relational SQL** and **Graph Query Language (GQL)** across [AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/data-agent-overview), Cloud SQL ([PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/data-agent-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/data-agent-overview)), and [Cloud Spanner (GoogleSQL & Spanner Graph)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/data-agent-overview).
+The **Context Engineering Agent** is an AI coding agent plugin designed to run in developer agent harnesses (such as Claude Code, Antigravity, or Gemini CLI). It generates, evaluates, and iteratively tunes tailored context artifacts (`ContextSets` comprising `Templates`, `Facets`, and `Value Searches`) to enrich database schemas for **Gemini Data Analytics's data agent developer platform tools**, supporting both **relational SQL** and **Graph Query Language (GQL)** across [AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/data-agent-overview), Cloud SQL ([PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/data-agent-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/data-agent-overview)), and [Cloud Spanner (GoogleSQL, Spanner Graph, and PostgreSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/data-agent-overview).
 
 ---
 
 ## Why Context Engineering?
 
-When building data agents and natural language analytics interfaces, accurately translating user intent into database queries—whether relational SQL, pure GQL, or hybrid graph queries—is critical. 
+When building data agents and natural language analytics interfaces, accurately translating user intent into database queries—whether relational SQL (PostgreSQL, GoogleSQL, MySQL), pure GQL, or hybrid graph queries—is critical. 
 
-As outlined in **Build Context with Context Engineering Agent** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli) | [Spanner (GoogleSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli)), by optimizing a `ContextSet` to match your application's expected query stream, the **QueryData API** acts as a data agent tool capable of achieving **~100% NL-to-SQL/GQL translation accuracy with low latency**.
+As outlined in **Build Context with Context Engineering Agent** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli) | [Spanner](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli)), by optimizing a `ContextSet` to match your application's expected query stream, the **QueryData API** acts as a data agent tool capable of achieving **~100% NL-to-SQL/GQL translation accuracy with low latency**.
 
 ---
 
@@ -40,7 +38,7 @@ A `ContextSet` is the central artifact generated and managed by the agent, conta
 * **Facets**: Reusable, modular query fragments (e.g., parameterized `WHERE` clauses, specialized join filters, or graph `MATCH` traversal patterns) linked to domain vocabulary.
 * **Value Searches**: Specialized mapping queries that dynamically resolve user-supplied values (e.g., *"Lndn"*) to database records (*"London"*) via the capabilities of the underlying database, such as embedding search, AI operators, or trigram search on relational and graph property tables.
 
-For full schema details, structure specifications, and dialect-specific JSON representations of `ContextSets`, see the official **Context Sets Overview** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/context-sets-overview) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/context-sets-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/context-sets-overview) | [Spanner (GoogleSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/context-sets-overview)).
+For full schema details, structure specifications, and dialect-specific JSON representations of `ContextSets`, see the official **Context Sets Overview** ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/context-sets-overview) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/context-sets-overview) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/context-sets-overview) | [Spanner](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/context-sets-overview)).
 
 ---
 
@@ -49,13 +47,13 @@ For full schema details, structure specifications, and dialect-specific JSON rep
 Before getting started, prepare your GCP environment, required APIs (Data Analytics API, Gemini for Google Cloud API, Dataplex Universal Catalog API), IAM permissions, and database Data API settings.
 
 Follow the step-by-step setup guide in the official documentation:
-👉 **Prepare Your Environment**: ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli#prepare-your-environment) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli#prepare-your-environment) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli#prepare-your-environment) | [Spanner (GoogleSQL)](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli#prepare-your-environment))
+👉 **Prepare Your Environment**: ([AlloyDB](https://docs.cloud.google.com/gemini/data-agents/querydata/alloydb/build-context-gemini-cli#prepare-your-environment) | Cloud SQL: [PostgreSQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-postgres/build-context-gemini-cli#prepare-your-environment) / [MySQL](https://docs.cloud.google.com/gemini/data-agents/querydata/sql-mysql/build-context-gemini-cli#prepare-your-environment) | [Spanner](https://docs.cloud.google.com/gemini/data-agents/querydata/spanner/build-context-gemini-cli#prepare-your-environment))
 
 ---
 
 ## Primary Workflow Phases
 
-The agent enables you to craft an optimized context for QueryData API through three primary phases:
+The agent enables you to craft an optimized context for QueryData API through three primary phases. Depending on your needs, you may also toggle the agent to skip phases. For example, if you have a dataset already, you can skip directly to context optimization "optimize context using dataset in \<file\>."
 
 ### Phase 1: Artifact Ingestion
 *Why it matters: Without broader context on the application's goals and scope, AI models generate sterile queries based solely on database column names, missing how your users actually ask for information.*
@@ -80,8 +78,7 @@ The optimization loop creates an initial `ContextSet` and then iteratively refin
 1. **Bootstrap**: Generate an initial baseline context.
 2. **Evaluate**: Measure context effectiveness against a golden dataset.
 3. **Hill-Climbing**: Perform gap analysis on failures and generate automated fixes.
-4. **Iterate**: Apply the improved context and re-run evaluation to continuously improve quality.
-5. **Final Validation** (Optional): Verify mutations against a separated test set to ensure generalization and prevent overfitting.
+4. **Iterate**: Apply the improved context and re-run evaluation to continuously improve quality until we reach an optimal point.
 
 *Note: While there is a typical ordering for these CUJs, the agent is flexible in how you want to execute. You can run the full pipeline end-to-end, trigger any individual phase, or ask for targeted changes to the `ContextSet`.*
 

{google_cloud_db_context_engineering-0.7.2 → google_cloud_db_context_engineering-0.7.4}/src/google_cloud_db_context_engineering.egg-info/SOURCES.txt

@@ -7,6 +7,7 @@ src/google/cloud/db_context_enrichment/common/__init__.py
 src/google/cloud/db_context_enrichment/common/config.py
 src/google/cloud/db_context_enrichment/common/context_mutator.py
 src/google/cloud/db_context_enrichment/common/context_store_client.py
+src/google/cloud/db_context_enrichment/common/context_validator.py
 src/google/cloud/db_context_enrichment/dataset/__init__.py
 src/google/cloud/db_context_enrichment/dataset/dataset_generator.py
 src/google/cloud/db_context_enrichment/evaluate/__init__.py
@@ -15,6 +16,9 @@ src/google/cloud/db_context_enrichment/evaluate/result_reader.py
 src/google/cloud/db_context_enrichment/evaluate/db_generators/__init__.py
 src/google/cloud/db_context_enrichment/evaluate/db_generators/alloydb.py
 src/google/cloud/db_context_enrichment/evaluate/db_generators/base.py
+src/google/cloud/db_context_enrichment/evaluate/db_generators/bigtable.py
+src/google/cloud/db_context_enrichment/evaluate/db_generators/custom.py
+src/google/cloud/db_context_enrichment/evaluate/db_generators/firestore.py
 src/google/cloud/db_context_enrichment/evaluate/db_generators/mysql.py
 src/google/cloud/db_context_enrichment/evaluate/db_generators/postgres.py
 src/google/cloud/db_context_enrichment/evaluate/db_generators/spanner.py