spark-mcp 0.4.1__tar.gz

spark_mcp-0.4.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Knuckles-Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

spark_mcp-0.4.1/MANIFEST.in

@@ -0,0 +1,3 @@
+include README.md
+include requirements.txt
+recursive-include spark_mcp *.py *.json *.md *.yaml *.yml

spark_mcp-0.4.1/PKG-INFO

@@ -0,0 +1,329 @@
+Metadata-Version: 2.4
+Name: spark-mcp
+Version: 0.4.1
+Summary: Spark Connect (gRPC) API + MCP Server for Agentic AI!
+Author-email: Project Maintainers <maintainers@users.noreply.github.com>
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Environment :: Console
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Requires-Python: <3.15,>=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: agent-connector-sdk[mcp]<1.0.0,>=0.3.6
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: numpy>=2.5.4
+Requires-Dist: pandas>=3.0.6
+Requires-Dist: pyarrow>=26.0.0
+Requires-Dist: pyspark-client<4.2.0,>=4.1.3
+Requires-Dist: requests>=2.34.2
+Provides-Extra: mcp
+Requires-Dist: agent-connector-sdk[mcp]<1.0.0,>=0.3.6; extra == "mcp"
+Provides-Extra: all
+Requires-Dist: agent-connector-sdk[mcp]<1.0.0,>=0.3.6; extra == "all"
+Provides-Extra: test
+Requires-Dist: pytest-xdist>=3.8.0; extra == "test"
+Requires-Dist: pytest>=9.1.1; extra == "test"
+Requires-Dist: pytest-asyncio>=1.4.0; extra == "test"
+Requires-Dist: pytest-cov>=7.1.0; extra == "test"
+Dynamic: license-file
+
+# Spark Mcp
+## CLI or API | MCP
+
+![PyPI - Version](https://img.shields.io/pypi/v/spark-mcp)
+![MCP Server](https://badge.mcpx.dev?type=server 'MCP Server')
+![PyPI - Downloads](https://img.shields.io/pypi/dd/spark-mcp)
+![GitHub Repo stars](https://img.shields.io/github/stars/Knuckles-Team/spark-mcp)
+![GitHub forks](https://img.shields.io/github/forks/Knuckles-Team/spark-mcp)
+![GitHub contributors](https://img.shields.io/github/contributors/Knuckles-Team/spark-mcp)
+![PyPI - License](https://img.shields.io/pypi/l/spark-mcp)
+![GitHub](https://img.shields.io/github/license/Knuckles-Team/spark-mcp)
+![GitHub last commit (by committer)](https://img.shields.io/github/last-commit/Knuckles-Team/spark-mcp)
+![GitHub pull requests](https://img.shields.io/github/issues-pr/Knuckles-Team/spark-mcp)
+![GitHub closed pull requests](https://img.shields.io/github/issues-pr-closed/Knuckles-Team/spark-mcp)
+![GitHub issues](https://img.shields.io/github/issues/Knuckles-Team/spark-mcp)
+![GitHub top language](https://img.shields.io/github/languages/top/Knuckles-Team/spark-mcp)
+![GitHub language count](https://img.shields.io/github/languages/count/Knuckles-Team/spark-mcp)
+![GitHub repo size](https://img.shields.io/github/repo-size/Knuckles-Team/spark-mcp)
+![GitHub repo file count (file type)](https://img.shields.io/github/directory-file-count/Knuckles-Team/spark-mcp)
+![PyPI - Wheel](https://img.shields.io/pypi/wheel/spark-mcp)
+![PyPI - Implementation](https://img.shields.io/pypi/implementation/spark-mcp)
+
+*Version: 0.4.1*
+
+> **Documentation** — Installation, deployment, and usage across the API, CLI, MCP,
+> and A2A agent interfaces are maintained in the
+> [official documentation](https://knuckles-team.github.io/spark-mcp/).
+
+---
+
+## Table of Contents
+- [Overview](#overview)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Architecture](#architecture)
+- [Environment Variables](#environment-variables)
+- [MCP Tools](#mcp-tools)
+- [Documentation](#documentation)
+
+## Overview
+`spark-mcp` exposes a standardized interface to the live Spark Connect deployment
+via the Model Context Protocol — SQL/session execution, versioned Transform
+submission/tracking/rerun against the shared Iceberg `lakehouse` catalog
+(`scope=lakekeeper`, explicit — the same landmine Trino/Lakekeeper hit if dropped),
+namespace/table catalog reads, OpenLineage config/status reporting, and a
+Wire-First `spark_ingest_run` tool that pushes the
+`SparkApplication -> SparkJob -> Transform -> TransformRun -> DatasetVersion` chain
+into the epistemic-graph Knowledge Graph.
+
+**Spark Connect is live**, confirmed 2026-08-26 against
+`spark-connect.apps.svc:15002` with a real `pyspark.sql.connect` session — a query
+against `lakehouse.analytics.trino_verify` returned the same 3 rows Trino's sibling
+lane already proved reading/writing (cross-engine Iceberg interop). See
+[docs/architecture.md](docs/architecture.md) for the full evidence and this
+package's `AGENTS.md` for known gaps (identity-token attach, `ActionSpec`
+typed-Actions deferral).
+
+Talks directly to Spark over `pyspark-client` (the thin Spark Connect client) for SQL, DataFrame and catalog operations, and over the Spark monitoring REST API (`/api/v1` on the driver UI and the History Server) for applications, jobs, stages, executors, storage, environment and SQL executions, never
+`bin/spark-shell --remote` (does not work on the vanilla Spark image) and never a
+`kubectl exec spark-sql` fallback.
+
+## Installation
+
+Pick the extra that matches what the operator want to run:
+
+| Extra | Installs | Use when |
+|-------|----------|----------|
+| `spark-mcp[mcp]` | Connector-focused MCP server (`agent-connector-sdk` — FastMCP/FastAPI + `epistemic-graph[full]`) | The operator only run the **MCP server** (smallest install / image) |
+| `spark-mcp[all]` | Everything (`mcp` + `agent` + `logfire`) | Development / both surfaces |
+
+```bash
+uv pip install "spark-mcp[mcp]"
+```
+
+### Container images (`:mcp`)
+
+One multi-stage `docker/Dockerfile` builds the MCP-only image (`--target mcp`):
+
+```bash
+docker build --target mcp   -t spark-mcp:mcp    .
+```
+
+Both images install a headless JRE at runtime — `pyspark[connect]`'s gRPC client has
+a JVM-side dependency, unlike the fleet's pure-Python REST connectors.
+
+## Usage
+Run the MCP server directly:
+```bash
+python -m spark_mcp
+```
+
+### MCP Configuration Example (stdio)
+
+```json
+{
+  "mcpServers": {
+    "spark-mcp": {
+      "command": "uv",
+      "args": ["run", "spark-mcp"],
+      "env": {
+        "MCP_TOOL_MODE": "intent",
+        "SPARK_CONNECT_URL": "sc://spark-connect.apps.svc:15002",
+        "SPARK_LAKEHOUSE_CATALOG": "lakehouse",
+        "SPARK_LAKEHOUSE_SCOPE": "lakekeeper",
+        "SPARK_ENABLE_TRANSFORMS": "False",
+        "SPARKTOOL": "True"
+      }
+    }
+  }
+}
+```
+
+## Architecture
+
+`api_client.py` wraps `api/api_client_spark.py`'s `SparkApi`, a thin gRPC client
+over `pyspark.sql.connect.session.SparkSession`, behind `api/api_client_base.py`'s
+typed-error layer (`SparkApiError` with a `kind`: `unreachable` / `job_failed` /
+`session_evicted` / `disabled` — never a silent empty result). `auth.py` mints
+Keycloak client-credentials identity tokens and can attach them as real gRPC bearer
+credentials, but the deployed Spark Connect endpoint is plaintext gRPC today, so
+attach is disabled by default (see `docs/architecture.md`).
+
+`mcp_server.py` serves one typed `spark_<operation>` tool per `SparkApi` method
+(`spark_sql`, `spark_describe`, `spark_cancel`, `spark_submit_transform`,
+`spark_rerun_transform`, `spark_list_transform_runs`, `spark_list_lakekeeper_tables`,
+`spark_lineage_config`, `spark_lineage_events`, `spark_application_status`) through one
+`register_tool_surface(...)` call; GET-like reads are read-only and `spark_cancel` is
+destructive. `kg_ingest.py` exposes
+`ingest_entities`/`ingest_run` plus record-mapping helpers around the required
+`native_ingest` (Wire-First) authority.
+
+## Python version
+
+Python 3.14 (floor 3.12) against a Spark 4.1 server. `pyspark-client` 4.1 is the
+thin Spark Connect client; it runs on numpy 2, pandas 3 and pyarrow wheels.
+`pyspark-client` ships only a pure-Python sdist, so it is the one package built
+from source (`[tool.pipelines] allow-source-build = ["pyspark-client"]`).
+
+## Environment Variables
+
+<!-- ENV-VARS-TABLE:START -->
+
+#### Package environment variables
+
+| Variable | Example | Description |
+|----------|---------|-------------|
+| `SPARK_CONNECT_URL` | `sc://spark-connect.apps.svc:15002` |  |
+| `SPARK_UI_URL` | `http://spark-runner.apps.svc:4040` | driver monitoring REST |
+| `SPARK_HISTORY_URL` | `http://spark-history.apps.svc:18080` | History Server REST |
+| `SPARK_REST_TIMEOUT_S` | `30` |  |
+| `SPARK_SUBMIT_TIMEOUT_S` | `300` |  |
+| `SPARK_LAKEHOUSE_CATALOG` | `lakehouse` | document/verify the invariant client-side — same invalid_scope landmine as |
+| `SPARK_LAKEHOUSE_SCOPE` | `lakekeeper` |  |
+| `SPARK_ENABLE_TRANSFORMS` | `False` | egeria-mcp's EGERIA_ENABLE_WRITE) — spark_submit_transform/spark_rerun_transform |
+| `SPARK_CONNECT_ATTACH_IDENTITY_TOKEN` | secret-injected | Service is plaintext gRPC; attaching a bearer token forces a TLS channel the |
+| `SPARK_CONNECT_USE_SSL` | `False` |  |
+| `SPARK_SERVICE_CLIENT_ID` | `spark-service` |  |
+| `SPARK_SERVICE_CLIENT_SECRET` | secret-injected |  |
+| `SPARK_OAUTH_SCOPE` | `lakekeeper` |  |
+| `SPARK_OAUTH_TOKEN_URL` | secret-injected |  |
+| `SPARK_KEYCLOAK_URL` | `http://localhost:8080` |  |
+| `SPARK_KEYCLOAK_REALM` | `homelab` |  |
+| `SPARKTOOL` | `True` |  |
+| `INGESTTOOL` | `True` |  |
+
+#### Inherited agent-connector-sdk variables (apply to every connector)
+
+| Variable | Example | Description |
+|----------|---------|-------------|
+| `TRANSPORT` | `stdio` | MCP transport: `stdio` \| `streamable-http` \| `sse` |
+| `HOST` | `127.0.0.1` | Loopback bind host (set an authenticated ingress explicitly) |
+| `PORT` | `8000` | Bind port (HTTP transports) |
+| `MCP_TOOL_MODE` | `intent` | Tool surface: `intent` \| `condensed` \| `verbose` \| `both` |
+| `MCP_ENABLED_TOOLS` | — | Comma-separated tool allow-list |
+| `MCP_DISABLED_TOOLS` | — | Comma-separated tool deny-list |
+| `MCP_ENABLED_TAGS` | — | Comma-separated tag allow-list |
+| `MCP_DISABLED_TAGS` | — | Comma-separated tag deny-list |
+| `EUNOMIA_TYPE` | `none` | Authorization mode: `none` \| `embedded` \| `remote` |
+| `EUNOMIA_POLICY_FILE` | `mcp_policies.json` | Embedded Eunomia policy file |
+| `EUNOMIA_REMOTE_URL` | — | Remote Eunomia authorization server URL |
+| `ENABLE_OTEL` | `False` | Enable OpenTelemetry export |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | — | OTLP collector endpoint |
+| `MCP_CLIENT_AUTH` | — | Outbound MCP child auth: `oidc-client-credentials` \| `basic` \| `none` |
+| `OIDC_CLIENT_ID` | — | OIDC client id (service-account auth) |
+| `OIDC_CLIENT_SECRET_REF` | `secret://identity/oidc-client-secret` | Runtime secret reference for the OIDC service account |
+| `MCP_BASIC_AUTH_USERNAME` | — | HTTP Basic username (`MCP_CLIENT_AUTH=basic`) |
+| `MCP_BASIC_AUTH_PASSWORD_REF` | `secret://identity/mcp-basic-password` | Runtime secret reference for HTTP Basic auth (`MCP_CLIENT_AUTH=basic`) |
+| `DEBUG` | `False` | Verbose logging |
+| `PYTHONUNBUFFERED` | `1` | Unbuffered stdout (recommended in containers) |
+| `MCP_URL` | `http://localhost:8000/mcp` | URL of the MCP server the agent connects to |
+| `PROVIDER` | `openai` | LLM provider for the agent |
+| `MODEL_ID` | `gpt-4o` | Model id for the agent |
+| `ENABLE_WEB_UI` | `True` | Serve the AG-UI web interface |
+
+_15 package + 24 inherited variable(s). Auto-generated from `.env.example` + the shared agent-connector-sdk set — do not edit._
+<!-- ENV-VARS-TABLE:END -->
+
+
+| Variable | Required | Notes |
+|----------|----------|-------|
+| `SPARK_CONNECT_URL` | recommended | Spark Connect gRPC endpoint. Defaults to `sc://spark-connect.apps.svc:15002`. |
+| `SPARK_SUBMIT_TIMEOUT_S` | optional | Submission timeout budget. Defaults to `300`. |
+| `SPARK_LAKEHOUSE_CATALOG` / `SPARK_LAKEHOUSE_SCOPE` | optional | Defaults to `lakehouse` / `lakekeeper` — never leave scope at the shared-client default `catalog`. |
+| `SPARK_ENABLE_TRANSFORMS` | ✅ for writes | `spark_submit_transform`/`spark_rerun_transform` refuse unless `true`. |
+| `SPARK_CONNECT_ATTACH_IDENTITY_TOKEN` / `SPARK_CONNECT_USE_SSL` | optional | Both must be `true` together — see `docs/architecture.md`. |
+| `SPARK_SERVICE_CLIENT_ID` / `SPARK_SERVICE_CLIENT_SECRET` | for identity attach | Keycloak client-credentials. |
+| `SPARK_OAUTH_SCOPE` / `SPARK_OAUTH_TOKEN_URL` / `SPARK_KEYCLOAK_URL` / `SPARK_KEYCLOAK_REALM` | optional | Token-mint configuration. |
+
+See [.env.example](.env.example) for the full template.
+
+## MCP Tools
+
+| Group | Tools |
+|-------|-------|
+| session | `spark_sql`, `spark_describe`, `spark_cancel` |
+| transforms | `spark_submit_transform`, `spark_list_transform_runs`, `spark_rerun_transform` |
+| catalog | `spark_list_lakekeeper_tables` |
+| lineage | `spark_lineage_config`, `spark_lineage_events` |
+| ops | `spark_application_status` |
+| ingest | `spark_ingest_run` |
+
+## Documentation
+- [Architecture](docs/architecture.md)
+- [Configuration](docs/configuration.md)
+- [AGENTS.md](AGENTS.md) — canonical agent-facing context, live-proof evidence, known gaps
+
+## Available MCP Tools
+
+<!-- MCP-TOOLS-TABLE:START -->
+
+#### Condensed action-routed tools (`MCP_TOOL_MODE=condensed`)
+
+| MCP Tool | Toggle Env Var | Description |
+|----------|----------------|-------------|
+| `spark_ingest_run` | `INGESTTOOL` | Push one TransformRun's SparkApplication->SparkJob->Transform->TransformRun |
+
+#### Verbose 1:1 API-mapped tools (`MCP_TOOL_MODE=verbose` or `both`)
+
+<details>
+<summary>12 per-operation tools — one per public API method (click to expand)</summary>
+
+| MCP Tool | Toggle Env Var | Description |
+|----------|----------------|-------------|
+| `spark_application_status` | `SPARKTOOL` | Best-effort application/session status. |
+| `spark_cancel` | `SPARKTOOL` | Interrupt running operations on this session (all, or one tagged operation). |
+| `spark_close` | `SPARK_APITOOL` | Invoke the close operation. |
+| `spark_describe` | `SPARKTOOL` | Schema + column info for one fully-qualified table. |
+| `spark_get_session` | `SPARK_APITOOL` | Return the cached Spark Connect session, building it on first use. |
+| `spark_lineage_config` | `SPARKTOOL` | Report OpenLineage listener config — already set server-side at |
+| `spark_lineage_events` | `SPARKTOOL` | Named per this lane's tool-group contract; NOT a Kafka consumer. |
+| `spark_list_lakekeeper_tables` | `SPARKTOOL` | Invoke the list_lakekeeper_tables operation. |
+| `spark_list_transform_runs` | `SPARKTOOL` | Invoke the list_transform_runs operation. |
+| `spark_rerun_transform` | `SPARKTOOL` | Deterministically re-execute a prior TransformRun's manifest. |
+| `spark_sql` | `SPARKTOOL` | Execute one SQL statement and return rows as plain dicts. |
+| `spark_submit_transform` | `SPARKTOOL` | Execute one Transform manifest, recording a TransformRun (success or typed failure). |
+
+</details>
+
+_1 action-routed tool(s) · 12 verbose 1:1 tool(s). Each is enabled unless its `<DOMAIN>TOOL` toggle is set false; `MCP_TOOL_MODE` selects the surface (**`intent` default** — the six verb-tools, granular set loaded on demand · `condensed` action-routed · `verbose` 1:1 · `both`). Auto-generated — do not edit._
+<!-- MCP-TOOLS-TABLE:END -->
+
+---
+
+## Repository Owners
+
+<img width="100%" height="180em" src="https://github-readme-stats.vercel.app/api?username=example&show_icons=true&hide_border=true&&count_private=true&include_all_commits=true" />
+
+![GitHub followers](https://img.shields.io/github/followers/example)
+![GitHub User's stars](https://img.shields.io/github/stars/example)
+
+---
+
+## Contribute
+
+Contributions are welcome! Please ensure code quality by running local checks before submitting pull requests:
+- Format code using `ruff format .`
+- Lint code using `ruff check .`
+- Validate type-safety with `mypy .`
+- Run test suites using `pytest`
+
+
+
+<!-- GOVERNED-CAPABILITY:START -->
+## Governed capability contract
+
+This package ships a compact canonical skill surface with specialist procedures
+kept as referenced workflows. The current MCP tools, skill metadata,
+`connector_manifest.yml`, ontology, mappings, shapes, fixtures, migrations,
+tool-schema fingerprints, and certification metadata form one versioned
+capability contract. Validate them together; do not rely on stale tool names or
+historical per-task skill wrappers.
+
+Runtime endpoints, credentials, certificate trust, tenant identity, retention,
+and observability policy are deployment inputs and are never packaged values.
+See [Configuration, trust, and privacy](docs/configuration.md) before enabling a
+network transport, connector ingestion, Arbora delegation, or trace export.
+<!-- GOVERNED-CAPABILITY:END -->

spark_mcp-0.4.1/README.md

@@ -0,0 +1,297 @@
+# Spark Mcp
+## CLI or API | MCP
+
+![PyPI - Version](https://img.shields.io/pypi/v/spark-mcp)
+![MCP Server](https://badge.mcpx.dev?type=server 'MCP Server')
+![PyPI - Downloads](https://img.shields.io/pypi/dd/spark-mcp)
+![GitHub Repo stars](https://img.shields.io/github/stars/Knuckles-Team/spark-mcp)
+![GitHub forks](https://img.shields.io/github/forks/Knuckles-Team/spark-mcp)
+![GitHub contributors](https://img.shields.io/github/contributors/Knuckles-Team/spark-mcp)
+![PyPI - License](https://img.shields.io/pypi/l/spark-mcp)
+![GitHub](https://img.shields.io/github/license/Knuckles-Team/spark-mcp)
+![GitHub last commit (by committer)](https://img.shields.io/github/last-commit/Knuckles-Team/spark-mcp)
+![GitHub pull requests](https://img.shields.io/github/issues-pr/Knuckles-Team/spark-mcp)
+![GitHub closed pull requests](https://img.shields.io/github/issues-pr-closed/Knuckles-Team/spark-mcp)
+![GitHub issues](https://img.shields.io/github/issues/Knuckles-Team/spark-mcp)
+![GitHub top language](https://img.shields.io/github/languages/top/Knuckles-Team/spark-mcp)
+![GitHub language count](https://img.shields.io/github/languages/count/Knuckles-Team/spark-mcp)
+![GitHub repo size](https://img.shields.io/github/repo-size/Knuckles-Team/spark-mcp)
+![GitHub repo file count (file type)](https://img.shields.io/github/directory-file-count/Knuckles-Team/spark-mcp)
+![PyPI - Wheel](https://img.shields.io/pypi/wheel/spark-mcp)
+![PyPI - Implementation](https://img.shields.io/pypi/implementation/spark-mcp)
+
+*Version: 0.4.1*
+
+> **Documentation** — Installation, deployment, and usage across the API, CLI, MCP,
+> and A2A agent interfaces are maintained in the
+> [official documentation](https://knuckles-team.github.io/spark-mcp/).
+
+---
+
+## Table of Contents
+- [Overview](#overview)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Architecture](#architecture)
+- [Environment Variables](#environment-variables)
+- [MCP Tools](#mcp-tools)
+- [Documentation](#documentation)
+
+## Overview
+`spark-mcp` exposes a standardized interface to the live Spark Connect deployment
+via the Model Context Protocol — SQL/session execution, versioned Transform
+submission/tracking/rerun against the shared Iceberg `lakehouse` catalog
+(`scope=lakekeeper`, explicit — the same landmine Trino/Lakekeeper hit if dropped),
+namespace/table catalog reads, OpenLineage config/status reporting, and a
+Wire-First `spark_ingest_run` tool that pushes the
+`SparkApplication -> SparkJob -> Transform -> TransformRun -> DatasetVersion` chain
+into the epistemic-graph Knowledge Graph.
+
+**Spark Connect is live**, confirmed 2026-08-26 against
+`spark-connect.apps.svc:15002` with a real `pyspark.sql.connect` session — a query
+against `lakehouse.analytics.trino_verify` returned the same 3 rows Trino's sibling
+lane already proved reading/writing (cross-engine Iceberg interop). See
+[docs/architecture.md](docs/architecture.md) for the full evidence and this
+package's `AGENTS.md` for known gaps (identity-token attach, `ActionSpec`
+typed-Actions deferral).
+
+Talks directly to Spark over `pyspark-client` (the thin Spark Connect client) for SQL, DataFrame and catalog operations, and over the Spark monitoring REST API (`/api/v1` on the driver UI and the History Server) for applications, jobs, stages, executors, storage, environment and SQL executions, never
+`bin/spark-shell --remote` (does not work on the vanilla Spark image) and never a
+`kubectl exec spark-sql` fallback.
+
+## Installation
+
+Pick the extra that matches what the operator want to run:
+
+| Extra | Installs | Use when |
+|-------|----------|----------|
+| `spark-mcp[mcp]` | Connector-focused MCP server (`agent-connector-sdk` — FastMCP/FastAPI + `epistemic-graph[full]`) | The operator only run the **MCP server** (smallest install / image) |
+| `spark-mcp[all]` | Everything (`mcp` + `agent` + `logfire`) | Development / both surfaces |
+
+```bash
+uv pip install "spark-mcp[mcp]"
+```
+
+### Container images (`:mcp`)
+
+One multi-stage `docker/Dockerfile` builds the MCP-only image (`--target mcp`):
+
+```bash
+docker build --target mcp   -t spark-mcp:mcp    .
+```
+
+Both images install a headless JRE at runtime — `pyspark[connect]`'s gRPC client has
+a JVM-side dependency, unlike the fleet's pure-Python REST connectors.
+
+## Usage
+Run the MCP server directly:
+```bash
+python -m spark_mcp
+```
+
+### MCP Configuration Example (stdio)
+
+```json
+{
+  "mcpServers": {
+    "spark-mcp": {
+      "command": "uv",
+      "args": ["run", "spark-mcp"],
+      "env": {
+        "MCP_TOOL_MODE": "intent",
+        "SPARK_CONNECT_URL": "sc://spark-connect.apps.svc:15002",
+        "SPARK_LAKEHOUSE_CATALOG": "lakehouse",
+        "SPARK_LAKEHOUSE_SCOPE": "lakekeeper",
+        "SPARK_ENABLE_TRANSFORMS": "False",
+        "SPARKTOOL": "True"
+      }
+    }
+  }
+}
+```
+
+## Architecture
+
+`api_client.py` wraps `api/api_client_spark.py`'s `SparkApi`, a thin gRPC client
+over `pyspark.sql.connect.session.SparkSession`, behind `api/api_client_base.py`'s
+typed-error layer (`SparkApiError` with a `kind`: `unreachable` / `job_failed` /
+`session_evicted` / `disabled` — never a silent empty result). `auth.py` mints
+Keycloak client-credentials identity tokens and can attach them as real gRPC bearer
+credentials, but the deployed Spark Connect endpoint is plaintext gRPC today, so
+attach is disabled by default (see `docs/architecture.md`).
+
+`mcp_server.py` serves one typed `spark_<operation>` tool per `SparkApi` method
+(`spark_sql`, `spark_describe`, `spark_cancel`, `spark_submit_transform`,
+`spark_rerun_transform`, `spark_list_transform_runs`, `spark_list_lakekeeper_tables`,
+`spark_lineage_config`, `spark_lineage_events`, `spark_application_status`) through one
+`register_tool_surface(...)` call; GET-like reads are read-only and `spark_cancel` is
+destructive. `kg_ingest.py` exposes
+`ingest_entities`/`ingest_run` plus record-mapping helpers around the required
+`native_ingest` (Wire-First) authority.
+
+## Python version
+
+Python 3.14 (floor 3.12) against a Spark 4.1 server. `pyspark-client` 4.1 is the
+thin Spark Connect client; it runs on numpy 2, pandas 3 and pyarrow wheels.
+`pyspark-client` ships only a pure-Python sdist, so it is the one package built
+from source (`[tool.pipelines] allow-source-build = ["pyspark-client"]`).
+
+## Environment Variables
+
+<!-- ENV-VARS-TABLE:START -->
+
+#### Package environment variables
+
+| Variable | Example | Description |
+|----------|---------|-------------|
+| `SPARK_CONNECT_URL` | `sc://spark-connect.apps.svc:15002` |  |
+| `SPARK_UI_URL` | `http://spark-runner.apps.svc:4040` | driver monitoring REST |
+| `SPARK_HISTORY_URL` | `http://spark-history.apps.svc:18080` | History Server REST |
+| `SPARK_REST_TIMEOUT_S` | `30` |  |
+| `SPARK_SUBMIT_TIMEOUT_S` | `300` |  |
+| `SPARK_LAKEHOUSE_CATALOG` | `lakehouse` | document/verify the invariant client-side — same invalid_scope landmine as |
+| `SPARK_LAKEHOUSE_SCOPE` | `lakekeeper` |  |
+| `SPARK_ENABLE_TRANSFORMS` | `False` | egeria-mcp's EGERIA_ENABLE_WRITE) — spark_submit_transform/spark_rerun_transform |
+| `SPARK_CONNECT_ATTACH_IDENTITY_TOKEN` | secret-injected | Service is plaintext gRPC; attaching a bearer token forces a TLS channel the |
+| `SPARK_CONNECT_USE_SSL` | `False` |  |
+| `SPARK_SERVICE_CLIENT_ID` | `spark-service` |  |
+| `SPARK_SERVICE_CLIENT_SECRET` | secret-injected |  |
+| `SPARK_OAUTH_SCOPE` | `lakekeeper` |  |
+| `SPARK_OAUTH_TOKEN_URL` | secret-injected |  |
+| `SPARK_KEYCLOAK_URL` | `http://localhost:8080` |  |
+| `SPARK_KEYCLOAK_REALM` | `homelab` |  |
+| `SPARKTOOL` | `True` |  |
+| `INGESTTOOL` | `True` |  |
+
+#### Inherited agent-connector-sdk variables (apply to every connector)
+
+| Variable | Example | Description |
+|----------|---------|-------------|
+| `TRANSPORT` | `stdio` | MCP transport: `stdio` \| `streamable-http` \| `sse` |
+| `HOST` | `127.0.0.1` | Loopback bind host (set an authenticated ingress explicitly) |
+| `PORT` | `8000` | Bind port (HTTP transports) |
+| `MCP_TOOL_MODE` | `intent` | Tool surface: `intent` \| `condensed` \| `verbose` \| `both` |
+| `MCP_ENABLED_TOOLS` | — | Comma-separated tool allow-list |
+| `MCP_DISABLED_TOOLS` | — | Comma-separated tool deny-list |
+| `MCP_ENABLED_TAGS` | — | Comma-separated tag allow-list |
+| `MCP_DISABLED_TAGS` | — | Comma-separated tag deny-list |
+| `EUNOMIA_TYPE` | `none` | Authorization mode: `none` \| `embedded` \| `remote` |
+| `EUNOMIA_POLICY_FILE` | `mcp_policies.json` | Embedded Eunomia policy file |
+| `EUNOMIA_REMOTE_URL` | — | Remote Eunomia authorization server URL |
+| `ENABLE_OTEL` | `False` | Enable OpenTelemetry export |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | — | OTLP collector endpoint |
+| `MCP_CLIENT_AUTH` | — | Outbound MCP child auth: `oidc-client-credentials` \| `basic` \| `none` |
+| `OIDC_CLIENT_ID` | — | OIDC client id (service-account auth) |
+| `OIDC_CLIENT_SECRET_REF` | `secret://identity/oidc-client-secret` | Runtime secret reference for the OIDC service account |
+| `MCP_BASIC_AUTH_USERNAME` | — | HTTP Basic username (`MCP_CLIENT_AUTH=basic`) |
+| `MCP_BASIC_AUTH_PASSWORD_REF` | `secret://identity/mcp-basic-password` | Runtime secret reference for HTTP Basic auth (`MCP_CLIENT_AUTH=basic`) |
+| `DEBUG` | `False` | Verbose logging |
+| `PYTHONUNBUFFERED` | `1` | Unbuffered stdout (recommended in containers) |
+| `MCP_URL` | `http://localhost:8000/mcp` | URL of the MCP server the agent connects to |
+| `PROVIDER` | `openai` | LLM provider for the agent |
+| `MODEL_ID` | `gpt-4o` | Model id for the agent |
+| `ENABLE_WEB_UI` | `True` | Serve the AG-UI web interface |
+
+_15 package + 24 inherited variable(s). Auto-generated from `.env.example` + the shared agent-connector-sdk set — do not edit._
+<!-- ENV-VARS-TABLE:END -->
+
+
+| Variable | Required | Notes |
+|----------|----------|-------|
+| `SPARK_CONNECT_URL` | recommended | Spark Connect gRPC endpoint. Defaults to `sc://spark-connect.apps.svc:15002`. |
+| `SPARK_SUBMIT_TIMEOUT_S` | optional | Submission timeout budget. Defaults to `300`. |
+| `SPARK_LAKEHOUSE_CATALOG` / `SPARK_LAKEHOUSE_SCOPE` | optional | Defaults to `lakehouse` / `lakekeeper` — never leave scope at the shared-client default `catalog`. |
+| `SPARK_ENABLE_TRANSFORMS` | ✅ for writes | `spark_submit_transform`/`spark_rerun_transform` refuse unless `true`. |
+| `SPARK_CONNECT_ATTACH_IDENTITY_TOKEN` / `SPARK_CONNECT_USE_SSL` | optional | Both must be `true` together — see `docs/architecture.md`. |
+| `SPARK_SERVICE_CLIENT_ID` / `SPARK_SERVICE_CLIENT_SECRET` | for identity attach | Keycloak client-credentials. |
+| `SPARK_OAUTH_SCOPE` / `SPARK_OAUTH_TOKEN_URL` / `SPARK_KEYCLOAK_URL` / `SPARK_KEYCLOAK_REALM` | optional | Token-mint configuration. |
+
+See [.env.example](.env.example) for the full template.
+
+## MCP Tools
+
+| Group | Tools |
+|-------|-------|
+| session | `spark_sql`, `spark_describe`, `spark_cancel` |
+| transforms | `spark_submit_transform`, `spark_list_transform_runs`, `spark_rerun_transform` |
+| catalog | `spark_list_lakekeeper_tables` |
+| lineage | `spark_lineage_config`, `spark_lineage_events` |
+| ops | `spark_application_status` |
+| ingest | `spark_ingest_run` |
+
+## Documentation
+- [Architecture](docs/architecture.md)
+- [Configuration](docs/configuration.md)
+- [AGENTS.md](AGENTS.md) — canonical agent-facing context, live-proof evidence, known gaps
+
+## Available MCP Tools
+
+<!-- MCP-TOOLS-TABLE:START -->
+
+#### Condensed action-routed tools (`MCP_TOOL_MODE=condensed`)
+
+| MCP Tool | Toggle Env Var | Description |
+|----------|----------------|-------------|
+| `spark_ingest_run` | `INGESTTOOL` | Push one TransformRun's SparkApplication->SparkJob->Transform->TransformRun |
+
+#### Verbose 1:1 API-mapped tools (`MCP_TOOL_MODE=verbose` or `both`)
+
+<details>
+<summary>12 per-operation tools — one per public API method (click to expand)</summary>
+
+| MCP Tool | Toggle Env Var | Description |
+|----------|----------------|-------------|
+| `spark_application_status` | `SPARKTOOL` | Best-effort application/session status. |
+| `spark_cancel` | `SPARKTOOL` | Interrupt running operations on this session (all, or one tagged operation). |
+| `spark_close` | `SPARK_APITOOL` | Invoke the close operation. |
+| `spark_describe` | `SPARKTOOL` | Schema + column info for one fully-qualified table. |
+| `spark_get_session` | `SPARK_APITOOL` | Return the cached Spark Connect session, building it on first use. |
+| `spark_lineage_config` | `SPARKTOOL` | Report OpenLineage listener config — already set server-side at |
+| `spark_lineage_events` | `SPARKTOOL` | Named per this lane's tool-group contract; NOT a Kafka consumer. |
+| `spark_list_lakekeeper_tables` | `SPARKTOOL` | Invoke the list_lakekeeper_tables operation. |
+| `spark_list_transform_runs` | `SPARKTOOL` | Invoke the list_transform_runs operation. |
+| `spark_rerun_transform` | `SPARKTOOL` | Deterministically re-execute a prior TransformRun's manifest. |
+| `spark_sql` | `SPARKTOOL` | Execute one SQL statement and return rows as plain dicts. |
+| `spark_submit_transform` | `SPARKTOOL` | Execute one Transform manifest, recording a TransformRun (success or typed failure). |
+
+</details>
+
+_1 action-routed tool(s) · 12 verbose 1:1 tool(s). Each is enabled unless its `<DOMAIN>TOOL` toggle is set false; `MCP_TOOL_MODE` selects the surface (**`intent` default** — the six verb-tools, granular set loaded on demand · `condensed` action-routed · `verbose` 1:1 · `both`). Auto-generated — do not edit._
+<!-- MCP-TOOLS-TABLE:END -->
+
+---
+
+## Repository Owners
+
+<img width="100%" height="180em" src="https://github-readme-stats.vercel.app/api?username=example&show_icons=true&hide_border=true&&count_private=true&include_all_commits=true" />
+
+![GitHub followers](https://img.shields.io/github/followers/example)
+![GitHub User's stars](https://img.shields.io/github/stars/example)
+
+---
+
+## Contribute
+
+Contributions are welcome! Please ensure code quality by running local checks before submitting pull requests:
+- Format code using `ruff format .`
+- Lint code using `ruff check .`
+- Validate type-safety with `mypy .`
+- Run test suites using `pytest`
+
+
+
+<!-- GOVERNED-CAPABILITY:START -->
+## Governed capability contract
+
+This package ships a compact canonical skill surface with specialist procedures
+kept as referenced workflows. The current MCP tools, skill metadata,
+`connector_manifest.yml`, ontology, mappings, shapes, fixtures, migrations,
+tool-schema fingerprints, and certification metadata form one versioned
+capability contract. Validate them together; do not rely on stale tool names or
+historical per-task skill wrappers.
+
+Runtime endpoints, credentials, certificate trust, tenant identity, retention,
+and observability policy are deployment inputs and are never packaged values.
+See [Configuration, trust, and privacy](docs/configuration.md) before enabling a
+network transport, connector ingestion, Arbora delegation, or trace export.
+<!-- GOVERNED-CAPABILITY:END -->