databricks-mason 0.1.6.dev0__tar.gz → 0.2.0__tar.gz

databricks_mason-0.2.0/PKG-INFO

@@ -0,0 +1,741 @@
+Metadata-Version: 2.5
+Name: databricks-mason
+Version: 0.2.0
+Summary: Databricks integration for Mason
+Author-email: Databricks <agent-feedback@databricks.com>
+License-File: NOTICE
+Requires-Python: >=3.10
+Requires-Dist: click>=8.4
+Requires-Dist: databricks-ai-bridge[memory]>=0.22.0
+Requires-Dist: databricks-sdk>=0.94.0
+Requires-Dist: fastapi>=0.129.0
+Requires-Dist: mlflow-skinny>=3.10.1
+Requires-Dist: psycopg[binary]>=3.1
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.7
+Requires-Dist: sqlalchemy[asyncio]>=2.0
+Requires-Dist: tomli>=2.0
+Requires-Dist: tomlkit>=0.13
+Requires-Dist: uvicorn>=0.20.0
+Provides-Extra: langgraph
+Requires-Dist: databricks-agents>=1.9.3; extra == 'langgraph'
+Requires-Dist: databricks-langchain>=0.17.0; extra == 'langgraph'
+Requires-Dist: langchain-mcp-adapters>=0.2.1; extra == 'langgraph'
+Requires-Dist: langchain>=1.0.0; extra == 'langgraph'
+Requires-Dist: langgraph>=1.1.0; extra == 'langgraph'
+Requires-Dist: mlflow>=3.10.1; extra == 'langgraph'
+Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.25.0; extra == 'langgraph'
+Provides-Extra: openai
+Requires-Dist: databricks-agents>=1.9.3; extra == 'openai'
+Requires-Dist: databricks-openai>=0.13.0; extra == 'openai'
+Requires-Dist: mlflow>=3.10.1; extra == 'openai'
+Requires-Dist: openai-agents>=0.7.0; extra == 'openai'
+Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.25.0; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# `databricks-mason`
+
+Mason is an experimental CLI for Databricks custom agent preview APIs and
+deployments. It manages memory, sessions, tracing, and deployments from one
+authenticated command.
+
+> The underlying APIs are in preview and may need workspace enablement.
+
+## Overview
+
+A managed path from your custom agent code to a production-ready, scalable, durable agent hosted on
+Databricks in minutes - with no server framework to build, no infrastructure to provision, and no
+invocation protocol to design yourself. Bring your own agent, or start from a template.
+
+- **Deployment** - a guided lifecycle (scaffold, run locally, deploy) that turns an agent project
+  into a hosted endpoint. Databricks provisions the compute, the stores your agent binds (session,
+  memory), and the access grants, so you ship application code and get a running endpoint.
+- **Runtime** - a managed HTTP invocation contract (synchronous, streaming, background) plus optional
+  durable execution (persistence, heartbeats, crash recovery) backed by Databricks Lakebase, with no
+  database or job queue to operate. Use the opinionated `AgentApp` server to get it out of the box,
+  or bring your own server for full control.
+
+**Deployment**
+
+![Deployment: from a blank directory to a running service](docs/deployment.svg)
+
+- **Agent project** - `mason init` scaffolds a deployable project from a framework template
+  (LangGraph or OpenAI Agents) with the runtime, tests, and an optional chat UI wired up; you edit
+  the application code (model, tools, prompts).
+- **`agent.toml`** - the declarative source of truth for the Databricks-managed infrastructure your
+  agent depends on: tool bindings (data sandbox, managed MCP services, Unity Catalog functions) and
+  memory, session, and durability resources. `mason deploy` reads it to provision and wire everything
+  up (detailed under [Agent tools](#agent-tools)).
+- **`mason deploy`** - provisions the bound stores, grants the app's service principal access to
+  them, provisions the durable-runtime database when durability is on, configures tracing, and rolls
+  out the app. `mason deployments` covers the lifecycle (list, get, logs, start, stop, delete).
+- **`mason dev`** - runs your agent from the same manifest the deployment uses, so local behavior
+  matches what ships.
+
+**Runtime**
+
+![Runtime: one FastAPI server, run as AgentApp or your own implementation](docs/runtime.svg)
+
+The two ways to run an agent:
+
+- **`AgentApp` - opinionated, batteries included.** Register one handler and get Mason's full
+  invocation contract (synchronous, streaming, background). Enable the durable runtime so
+  long-running and background work survives restarts, redeploys, and crashes. The framework
+  templates are thin layers over `AgentApp` (HTTP contract detailed under [Runtime](#runtime)).
+- **Custom server - generic, full control.** `mason init --server custom` scaffolds a minimal FastAPI
+  server with no `AgentApp`: you define your own endpoints, request/response shapes, and protocol.
+  `mason dev` and `mason deploy` run and ship it the same way.
+
+## Prerequisites
+
+- **Python ≥3.10** — the mason CLI installs and runs on any Python 3.10+. The
+  `memory`, `sessions`, `tracing`, and `tools` commands need nothing else.
+- **[`uv`](https://docs.astral.sh/uv/)** — needed to scaffold, run, and deploy an
+  agent (`mason init` → `mason dev` → `mason deploy`): the scaffolded project builds
+  its environment and launches with `uv run`, both locally and in the deployed Apps
+  runtime. Not needed for the store/session/tracing/tools commands above.
+- **[Databricks CLI](https://docs.databricks.com/dev-tools/cli/)** — needed for
+  browser-based `mason login`. If a profile is already authenticated, Mason uses it
+  directly and the Databricks CLI is optional.
+
+## Installation
+
+From PyPI:
+
+```sh
+pip install databricks-mason
+```
+
+From source:
+
+```sh
+pip install 'git+https://github.com/databricks/databricks-ai-bridge.git#subdirectory=integrations/mason'
+```
+
+The base package includes the CLI, store SDK, and `AgentApp` HTTP runtime. Generated projects
+declare their framework dependencies automatically.
+
+## Shell completion
+Add this to `~/.zshrc`:
+```sh
+eval "$(_MASON_COMPLETE=zsh_source mason)"
+```
+
+## Authentication
+
+Mason uses [Databricks authentication](https://docs.databricks.com/aws/en/dev-tools/cli/authentication).
+Ask Mason to authenticate and remember a named profile:
+
+```sh
+mason login --profile <profile>
+mason sessions stores list
+```
+
+`mason login` validates existing credentials first. If credentials are missing or rejected in
+an interactive terminal, Mason runs `databricks auth login --profile <profile>`, revalidates the
+profile, and stores the selection in `~/.mason/config.json`. This browser-based setup requires
+the Databricks CLI. In non-interactive environments, authenticate the profile before running
+Mason. `mason logout` forgets the saved selection without revoking the underlying credentials.
+
+If Databricks SDK default authentication is already configured, you can skip `mason login`.
+You can also pass the global `--profile/-p` option before an individual command, for example
+`mason --profile <profile> tools list`. Use `--output json` for scripting.
+
+## Quickstart
+
+The shortest path from a blank directory to a running and deployed agent:
+
+```sh
+mason login --profile <profile>
+mason init my-agent
+cd my-agent
+mason dev                 # run locally
+mason deploy my-agent     # deploy to Databricks
+```
+
+`mason dev` runs the agent locally on `http://localhost:8000`, wrapping the Databricks Apps
+local runtime so local behavior matches a deployment.
+
+`mason deploy my-agent` deploys a Databricks App named `agent-mason-my-agent`, provisions the
+stores declared in `agent.toml`, and grants the app's service principal access to them. `mason
+deployments list` shows what you have deployed, and `mason deployments get my-agent` prints its
+URL and status.
+
+`mason init` declares default memory and session stores in `agent.toml`, so the deployed agent has
+long-term memory and durable conversation history. It creates `<name>-<6-letter-token>-memory` and
+`<name>-<6-letter-token>-sessions`, and records both names in `agent.toml`.
+`mason deploy` creates them if they don't exist yet. Point the agent at stores you already have with
+`mason memory bind <name>` / `mason sessions bind <name>`, or scaffold without stores using
+`mason init --server custom` (see [Initialize the chat app demo](#initialize-the-chat-app-demo)).
+
+To exercise the agent — locally under `mason dev` or once deployed — `mason endpoint invoke` sends
+it an HTTP request. MLflow tracing is on by default; `mason tracing list` shows the traces it
+produces.
+
+## Python SDK
+
+`MasonClient` adds a small resource-oriented layer over the Mason API. Pass it an
+authenticated Databricks `WorkspaceClient`, or omit the argument to use the
+Databricks SDK's default authentication resolution:
+
+```python
+from databricks.sdk import WorkspaceClient
+from databricks_mason import MasonClient
+
+mason = MasonClient(WorkspaceClient(profile="my-workspace"))
+
+session_store = mason.session_stores.create("support-agent-sessions")
+session = session_store.add(actor_id="customer-123", session_id="case-456")
+session.append_items(
+    [
+        {"type": "message", "role": "user", "content": "I need help with my cluster."},
+        {"type": "message", "role": "assistant", "content": "Let's take a look."},
+    ]
+)
+
+memory_store = mason.memory_stores.create("coding-agent-memory")
+memory = memory_store.add(
+    actor_id="alice",
+    path="/preferences/style.md",
+    content="The user prefers concise answers.",
+)
+results = memory_store.search(
+    actor_id="alice",
+    query="response preferences",
+    limit=10,
+)
+memory = memory.update(content="The user prefers very concise answers.")
+memory.delete()
+```
+
+The root collections manage stores: `mason.memory_stores.create/get/list` and
+`mason.session_stores.create/get/list`. A returned store owns operations on its
+contents, such as `memory_store.add()`, `memory_store.get("memory-id")`,
+`memory_store.list()`, and `memory_store.search()`, or `session_store.add()`,
+`session_store.get("session-id")`, and `session_store.list()`. Returned memories,
+sessions, and stores own their `update()` and `delete()` operations.
+
+All `list()` methods return iterators that automatically consume server pages. List
+`page_size` and search `limit` values must be between 1 and 100. `session.list_items()`
+also auto-pages. `session.fork(...)` creates an independent copy, optionally through
+a specific item. Deleting a session with descendants requires
+`session.delete(force=True)` to cascade the deletion.
+
+The resource layer intentionally does not mirror every API method. Its private
+transport will be replaced by the generated `WorkspaceClient.mason` service when that
+is released, without changing this public surface. Deployment, sandbox, tracing, and
+the existing CLI commands remain separate.
+
+## Runtime
+
+`AgentApp` runs your agent through one HTTP API for synchronous, streaming, and background
+invocations. Register an `@app.invoke` handler, publish progress with `await context.emit(event)`,
+and return a JSON result. You can also add your own FastAPI endpoints.
+
+Start from a template, edit the agent code in `agent/`, and run it locally before deploying:
+
+```sh
+mason init my-agent --framework langgraph --server mason --profile <profile>
+cd my-agent
+mason dev
+# Stop the local server when ready to deploy.
+mason --profile <profile> deploy my-agent
+```
+
+Use `--framework openai` for OpenAI Agents. Templates keep agent code separate from the runtime
+adapter and declare default Session and Memory Store bindings in `agent.toml`.
+
+Each managed run is an **invocation**. Send a client-generated UUID `id` and your agent's `input`:
+
+```json
+{
+  "id": "550e8400-e29b-41d4-a716-446655440000",
+  "input": {"messages": [{"role": "user", "content": "Hello"}]},
+  "background": true,
+  "stream": true
+}
+```
+
+| Endpoint | Behavior |
+| --- | --- |
+| `POST /api/invocations` | Defaults to synchronous execution: `200` with the result under `output`. `stream: true` returns SSE events. `background: true` returns `202` with a status URL; adding `stream: true` also includes an events URL. |
+| `GET /api/invocations/{id}` | Returns the invocation status and, when completed, its output. |
+| `GET /api/invocations/{id}/events?after={cursor}` | Streams events after the last received event ID, allowing clients to reconnect. |
+
+The UUID also acts as an idempotency key: repeating the same request reuses the existing invocation
+while its record is retained; using the ID for a different request returns `409`.
+
+`mason dev` keeps execution state in process and loses it on restart. For projects with
+`[agent].server = "mason"`, `mason deploy` provisions a persistent Runtime Store for requests,
+status, events, and results. Register `@app.recover` to restart interrupted work after worker
+failures. Recovery is at-least-once, so external side effects must be idempotent. Session and
+Memory Stores separately preserve the state used by your agent.
+
+The managed path uses the internal Runtime Store API to create a dedicated database in the
+workspace's shared Lakebase project and give the app SP ownership. Mason initializes its schema and
+tables; no manual Lakebase grant or Postgres app-resource attachment is needed. Backend selection
+is an internal rollout detail, not a user-facing setting; Mason currently retains the legacy
+per-app Lakebase project by default. Once enabled, redeploy reads the stored backend and verifies
+the app identity, and `mason deployments delete` removes the managed store before deleting the app.
+The switch does not migrate existing deployments between backends. Managed cleanup errors retain
+the app for retry. Direct app deletion bypasses managed store cleanup.
+
+Use `server = "custom"` to deploy your own HTTP server without provisioning a Runtime Store.
+Changing the server type of an existing deployment is not supported. To use a different server,
+scaffold a new project with the desired `mason init --server` option and deploy it under a new name.
+See the [runtime guide](src/databricks_mason/runtime/README.md) for agent hooks, full API examples,
+and recovery behavior.
+
+## Memory and sessions
+
+To hold context, an agent needs two kinds of state: the state of the interaction it is handling right
+now, and the durable knowledge it carries from one conversation to the next. Databricks provides a
+fully managed store for each, both backed by Lakebase and usable from agents built on any framework:
+
+- **Managed agent sessions** store an agent's session state: the state an agent or framework keeps
+  for one interaction. Most commonly this is the conversation history (the ordered transcript of
+  messages, tool calls, and results), but it can be any state a framework persists, such as a
+  LangGraph graph. The agent reads it at the start of a turn and appends to it as the interaction
+  runs.
+- **Managed agent memory** stores durable facts, preferences, and decisions that an agent recalls in
+  later, separate conversations, retrieved by semantic search.
+
+The examples below use the [`MasonClient` Python SDK](#python-sdk); the same operations are available
+as `mason sessions` / `mason memory` CLI commands.
+
+![Sessions and memory: the agent reads and appends one conversation's transcript in the session store, and recalls and saves durable facts in the memory store, which outlive any single conversation.](docs/sessions_and_memory.png)
+
+### Sessions
+
+A **session store** holds **sessions**, and each session holds an ordered list of **session items**. A
+session is one interaction — typically a conversation thread — grouped under an `actor_id` (who it
+belongs to; set this from trusted application context, never a model- or user-supplied value) and
+identified by a caller-chosen `session_id` (the service generates one if you omit it). Each item is an
+opaque, JSON-compatible `data` value — a message, tool call, result, or reasoning block — that
+Databricks stores and returns verbatim, in order, and never mutates once appended.
+
+Create a store, start a session, append the conversation's turns, and read the history back on a later
+request:
+
+```python
+from databricks.sdk import WorkspaceClient
+from databricks_mason import MasonClient
+
+mason = MasonClient(WorkspaceClient())
+
+session_store = mason.session_stores.create("support-agent-sessions")
+session = session_store.add(actor_id="customer-123", session_id="case-456")
+
+session.append_items(
+    [
+        {"type": "message", "role": "user", "content": "I need help with my cluster."},
+        {"type": "message", "role": "assistant", "content": "Let's take a look."},
+    ]
+)
+
+# On a later turn, reload the session and read its full history in order.
+session = session_store.get("case-456")
+history = [item.data for item in session.list_items()]  # list_items auto-pages
+```
+
+A session can be **forked** into an independent branch: a new session seeded with the original's
+history, linked back to its origin by `parent_session_id`. Fork the full history, or only up to a
+specific item, to explore an alternate continuation without disturbing the original thread:
+
+```python
+branch = session.fork(actor_id="customer-123")  # add up_to_item_id=... to branch up to one item
+```
+
+Deleting a session that has such descendants requires `session.delete(force=True)` to cascade.
+
+In a Mason-server agent you don't call these directly — the framework adapter reads and appends
+session state for you. With LangGraph, pass `checkpointer()` when you build the agent and scope each
+run with `thread_config(session_id)`; the OpenAI Agents adapter exposes the same as
+`session_store(session_id)`:
+
+```python
+from databricks_mason.langgraph import checkpointer, thread_config
+
+agent = create_agent(model=..., tools=[...], checkpointer=checkpointer())
+result = await agent.ainvoke(inputs, config=thread_config(session_id))
+```
+
+### Memory
+
+A **memory store** holds **memory entries**. Each entry is a free-form `content` string plus a short
+`description` used for retrieval, keyed by three fields: `actor_id` (whose memory it is — set from
+trusted application context, never a model- or user-supplied value), `path` (a filesystem-like key
+within an actor, such as `/preferences/response-style.md`), and an optional `session_id` (the session
+an entry came from, for provenance). An entry is uniquely identified by its `actor_id`, `path`, and
+optional `session_id`.
+
+Write an entry when the agent learns something durable, then recall it in a later, separate
+conversation with a natural-language search — results are ranked by full-text (BM25) relevance, up to
+100 entries, with no pagination or vector similarity:
+
+```python
+from databricks.sdk import WorkspaceClient
+from databricks_mason import MasonClient
+
+mason = MasonClient(WorkspaceClient())
+
+memory_store = mason.memory_stores.create("support-agent-memory")
+memory_store.add(
+    actor_id="user-123",
+    path="/preferences/communication.md",
+    content="Prefers email over phone. Timezone: PST.",
+    description="User 123 communication preferences",
+)
+
+# In a later, separate conversation, recall what the agent knows about this user.
+results = memory_store.search(actor_id="user-123", query="communication preferences", limit=10)
+```
+
+To browse rather than search, `memory_store.list(actor_id=..., path_prefix=...)` returns entries
+directly.
+
+In a Mason-server agent, add the memory tools so the model can read and write memory during a run.
+`memory_tools(actor)` exposes `remember` and `recall` bound to one actor's partition; it resolves the
+store from the `[memory_store]` binding — carried to the runtime by the `AGENT_MEMORY_STORE` env var
+that `deploy` and `mason dev` inject — and returns no tools when no store is bound, so the agent runs
+unchanged. The OpenAI Agents adapter exposes the same as `memory_tools()`:
+
+```python
+from databricks_mason.langgraph import memory_tools
+
+agent = create_agent(model=..., tools=[*your_tools, *memory_tools(actor)])
+```
+
+> **`actor_id` partitions data; it is not access control.** Both stores are workspace-scoped and
+> authorized at the store level, so any principal that can reach a store can read and write every
+> actor's entries. For strict isolation between tenants or users, use a separate store per boundary.
+> Grant another principal — such as your app's service principal — access with
+> `session_store.grant_permission(principal_id)` or `memory_store.grant_permission(principal_id)`;
+> `mason deploy` does this for the deployed app automatically.
+
+### Declaring and provisioning stores
+
+For a deployed agent, `agent.toml` declares which stores it uses and `mason deploy` provisions them —
+you don't create stores by hand. `mason init` declares a default memory and session store named from
+the project; override those names, point at stores you already have, or let `deploy` create them:
+
+```sh
+# Scaffold a project with default memory and session stores declared in agent.toml.
+mason init my-agent
+
+# Override the declared store names at init time.
+mason init my-agent --memory-store support-agent-memory --session-store support-agent-sessions
+
+# Or point an existing project at specific stores (edits agent.toml only; creates nothing).
+mason sessions bind support-agent-sessions
+mason memory bind support-agent-memory
+
+# deploy creates any declared-but-missing store and grants the app's service principal access.
+mason deploy my-agent
+```
+
+Memory and session stores are independent resources: deleting one never affects the other.
+
+## Commands
+
+For the full command reference - every command, subcommand, argument, and option, in table form -
+see [`cli.md`](cli.md). The tree below is a quick overview.
+
+```text
+mason [-p <profile>] [-o text|json]
+  login        [--profile P]
+  logout
+  init         [--framework openai|langgraph] [--server mason|custom]
+               [--disable-chat-app]
+               [--memory-store NAME] [--session-store NAME]
+               [--profile P] [directory]
+  dev          [--source PATH] [--prepare-environment] [--app-port PORT]
+               [--with-traces C.S]
+  memory
+    bind         STORE [--source PATH]
+    unbind       [--source PATH]
+    stores     create | list | get | update | delete
+    entries    create | get | list | search | update | delete
+  sessions     create | list | get | update | delete | fork
+    bind         STORE [--source PATH]
+    unbind       [--source PATH]
+    stores     create | list | get | update | delete
+    items      list | append | pop | clear
+  tracing
+    configure  [--experiment E] [--source PATH]
+    disable    [--source PATH]
+    list | get
+  tools
+    add sandbox      --scope SCOPE [--scope SCOPE ...] [--source PATH]
+    add mcp          SERVICE [--name NAME] [--source PATH]
+    add uc-function  FUNCTION [--name NAME] [--source PATH]
+    add genie-one    [--name NAME] [--source PATH]
+    add genie-agent  SPACE_ID [--name NAME] [--source PATH]
+    list             [--kind sandbox|mcp|uc-function|genie-one|genie-agent]
+                     [--schema CATALOG.SCHEMA]
+    remove           TOOL_ID [MCP_SERVICE] [--source PATH]
+  deploy       <name> --source PATH [--with-traces C.S] [--instances N]
+  deployments  list | get | logs | start | stop | delete
+  endpoint
+    invoke      [APP] --path PATH [--url URL] [--json JSON] [--sse]
+```
+
+## Invoke HTTP endpoints
+
+`mason endpoint invoke` is a low-level HTTP command. It resolves and authenticates a deployed
+Databricks App, or targets localhost and arbitrary servers through `--url`. It does not assume an
+agent protocol: provide the method, path, query parameters, and complete JSON body required by the
+server.
+
+```sh
+mason --profile <profile> endpoint invoke mason-my-agent \
+  --path /api/invocations \
+  --json '{"id":"00000000-0000-4000-8000-000000000001","input":[{"role":"user","content":"Hello"}]}'
+
+mason endpoint invoke --url http://localhost:8000 \
+  --path /api/invocations \
+  --json '{"id":"00000000-0000-4000-8000-000000000001","input":[{"role":"user","content":"Hello"}]}'
+```
+
+The JSON body remains explicit even for Mason-generated agents. For example, Mason Runtime agents require
+a client-generated invocation ID, and streaming servers require their own streaming field plus
+`--sse` so the CLI consumes the response as Server-Sent Events.
+
+```sh
+INVOCATION_ID=$(uuidgen)
+mason --profile <profile> endpoint invoke mason-my-agent \
+  --path /api/invocations \
+  --json "{\"id\":\"$INVOCATION_ID\",\"input\":[{\"role\":\"user\",\"content\":\"Run the report\"}]}"
+
+mason --profile <profile> endpoint invoke mason-my-agent \
+  --path /api/invocations \
+  --sse \
+  --json "{\"id\":\"$INVOCATION_ID\",\"input\":[{\"role\":\"user\",\"content\":\"Hello\"}],\"stream\":true}"
+```
+
+`--session-id` preserves one application session across calls by setting the Databricks Apps routing
+cookie. This also works with a direct App URL and with the generated runtime on localhost. OAuth and
+session headers are managed by Mason; arbitrary custom request headers are intentionally not exposed
+by this command.
+
+## Command help
+
+Use the conventional help flag at any command level. Every command's help includes runnable
+examples:
+
+```sh
+mason --help
+mason deploy --help
+mason sessions items append --help
+```
+
+## Agent tools
+
+For projects with `[agent].server = "mason"` (the default from `mason init`), `agent.toml` is the
+declarative source of truth for Databricks-managed infrastructure: the Runtime Store, sandbox,
+managed MCP, Genie and Unity Catalog function bindings, plus memory and session resources. `mason tools
+add` updates only this file; direct TOML edits have the same behavior. Both Mason-server framework
+adapters read the managed bindings at runtime without generating or patching agent source:
+
+```sh
+mason tools add sandbox --scope table:samples.nyctaxi.trips
+mason tools add mcp system.ai.web_search
+mason tools add uc-function catalog.schema.lookup_ticket
+mason tools add genie-one
+mason tools add genie-agent SPACE_ID
+mason tools remove mcp system.ai.web_search
+mason tools list
+```
+
+For MCP services, the remove command accepts the same service name as the add command. You can also
+remove any binding by its `id` in `agent.toml`, for example `mason tools remove web_search`.
+Every successful add (including an already-configured no-op) points you to the target project's
+`agent.toml` to review configured managed tools and MCP bindings. With `--source`, the message
+points to that project's file. JSON add output includes its path in `manifest`.
+
+`mason tools list` discovers **available integrations to add**, not configured bindings. By default
+it shows built-in add recipes and caller-visible MCP Services in `system.ai`. A recipe may still
+need your resources: sandbox scopes, a concrete UC function name, or a Genie Space ID. Genie One
+needs no additional argument. `system.ai.sandbox` is represented by its scoped recipe rather than
+a second unscoped add command. The list does not enumerate every workspace schema, individual
+operations inside MCP services, or custom Python tools.
+
+`mason tools add mcp` looks up the service in the selected workspace before writing `agent.toml`.
+Use `mason --profile <profile> tools add mcp <service>` to select a workspace. A missing service or
+failed lookup (including authentication or permission errors) leaves the project unchanged. This
+checks service metadata access, not whether every tool can be executed at runtime. Removing local
+bindings does not require workspace access.
+
+```sh
+mason tools list
+mason tools list --kind mcp
+mason tools list --kind mcp --schema main.tools
+mason tools list --kind sandbox
+mason tools list --kind genie-one
+mason tools list --kind genie-agent
+mason --output json tools list
+```
+
+No agent project is required for discovery. MCP discovery uses your Databricks profile; the
+`sandbox`, `uc-function`, `genie-one`, and `genie-agent` kind filters show local recipes without
+authentication. `--schema` requires `--kind mcp` and replaces the default `system.ai` scope. An
+API/authentication failure returns nonzero and marks discovery incomplete, while retaining local
+recipes; it is not reported as an empty successful discovery. Listing metadata does not verify
+runtime execution permissions.
+
+**Migration:** the former configured `tools list` view and its `--source` option are removed.
+Read `agent.toml` (its `[[tools]]` entries) to inspect configured bindings. Discovery JSON uses
+`schema_version: 2`, with `available_tools` (`name`, `kind`, `add_command`), `mcp_schema` (null for
+local-only recipes), `complete`, and `errors`. Replace old scripts that read configured-list JSON
+with TOML inspection. Replace `mason mcp list [--schema catalog.schema]` with
+`mason tools list --kind mcp [--schema catalog.schema]`; the former command is removed. Use
+`mason tools list --help` for the new discovery contract.
+
+Read-only live discovery can be checked against the installed wheel without creating a project
+or deploying an agent:
+
+```sh
+MASON_E2E_PROFILE=<profile> .venv-functional/bin/pytest tests/e2e/tool_discovery_test.py -v
+```
+
+The live checks compare default and MCP-filtered discovery with the compatibility service list.
+Set `MASON_E2E_SCHEMA=catalog.schema` to exercise an additional schema. The installed CLI's local
+add/review/remove flows and all updated help pages are covered by `tests/functional/cli_smoke_test.py`.
+
+In Mason-server templates, custom Python tools are code-first. Write them with the framework's native
+decorator in `agent/tools/`: LangGraph uses `@tool`, while OpenAI Agents uses `@function_tool`. The
+templates auto-discover decorated tools from that package and add them to the agent; there is no CLI
+command or `agent.toml` entry to keep in sync. Customer-managed MCP servers are likewise ordinary
+code in `agent/mcps.py` and are joined with the managed bindings by `mcp_tools(...)` or
+`mcp_servers(...)`.
+
+Projects created with `--server custom` do not auto-discover `agent/tools/` or load managed tool
+bindings from `agent.toml`, so `mason tools add` rejects those projects. Wire framework-native Python
+tools and MCP servers directly in `agent/agent.py` instead.
+
+If an older Mason-server manifest contains `source = { kind = "python", ... }`, remove that
+`[[tools]]` entry; the decorated tool in `agent/tools/` remains active. `mason dev` and `mason deploy`
+do not generate or patch Python tool code, and do not alter the manifest's `[[tools]]` bindings.
+
+Sandbox scopes default to read-only access. Repeat `--scope` to allow more than one resource, use
+`volume:` or `workspace:` for those resource types, and use `--permission read_write` only when the
+agent needs writes. Every sandbox call carries this fixed downscope in MCP `_meta`, outside the tool
+arguments controlled by the model.
+
+### Genie tools
+
+Genie One and Genie Agent support ship with Mason, but bindings are opt-in, like sandbox tools.
+Installing Mason does not configure a Genie Space ID or enable a Genie binding. Add only the
+capabilities your agent needs:
+
+```sh
+mason tools add genie-one --name genie_one
+mason tools add genie-agent SPACE_ID --name genie_agent
+mason tools list --kind genie-one
+mason tools list --kind genie-agent
+mason tools remove genie_one
+mason tools remove genie_agent
+```
+
+`--name` is optional and defaults to `genie_one` or `genie_agent`, respectively. Both add commands
+and `remove` accept `--source PATH` to select a project instead of the current directory. Discovery
+needs no project; read that project's `agent.toml` to inspect configured bindings. For scripted
+output, put the global `-o json` option before `tools`, as in
+`mason -o json tools add genie-one --source ./my-agent`. Adding a binding is offline: it updates
+`agent.toml` without contacting Genie or checking permissions. The corresponding sources are:
+
+```toml
+[[tools]]
+id = "genie_one"
+source = { kind = "genie_one" }
+
+[[tools]]
+id = "genie_agent"
+source = { kind = "genie_agent", space_id = "<your-space-id>" }
+```
+
+Replace `SPACE_ID` or `<your-space-id>` with an existing space's 32-character lowercase hexadecimal
+ID. `genie-one` connects to the workspace-wide MCP endpoint
+`https://<workspace-hostname>/api/2.0/mcp/genie`, without a space suffix. `genie-agent` uses the
+native Genie **Chat-mode** conversation API through the Databricks SDK, not the streaming
+Agent-mode API or the per-space MCP endpoint.
+
+Each native binding exposes `{id}_ask`, `{id}_poll`, and `{id}_query_result`, where `{id}` is its
+binding name. Ask accepts an optional `conversation_id` for follow-ups. Ask and poll share a
+120-second budget per call, including client setup and submission. If the response is still
+running, they return `timed_out` with the conversation and message IDs so the caller can poll
+again. If submission times out before a message ID is received, ask returns
+`INDETERMINATE_SUBMISSION`: the request may still complete, so do not resubmit automatically.
+`NOT_SUBMITTED` means client setup timed out before sending the question. Query results include
+the first 100 rows, column schema, a truncation indicator, and a deep link to the conversation.
+
+Both framework modules, `databricks_mason.langgraph` and `databricks_mason.openai`, export
+`genie_tools()`. New Mason-server templates use it automatically for native Genie Agent bindings;
+Genie One uses the existing managed MCP helpers. In an existing Mason-server project, import
+`genie_tools` from your framework module and add `*genie_tools()` to the agent's existing tool list.
+The CLI does not patch existing Python code.
+
+Both paths use Mason's existing authentication and routed workspace. Genie One requires the
+Managed MCP Servers workspace preview; delegated access requires the `genie` OAuth scope.
+The effective caller needs access to the data, the SQL warehouse, and the selected Genie space
+where applicable. Mason does not grant permissions or promise a service-principal fallback when
+caller credentials lack access. An offline add succeeding does not establish runtime access.
+
+The opt-in live tests exercise both frameworks against the configured workspace and an existing
+Genie space. From `integrations/mason`, with both framework extras installed:
+
+```sh
+DATABRICKS_CONFIG_PROFILE=my-workspace RUN_MASON_GENIE_TESTS=1 \
+  MASON_GENIE_SPACE_ID=SPACE_ID \
+  uv run pytest tests/integration_tests/genie_tools_test.py
+```
+
+By default they ask for the row count of `samples.nyctaxi.trips`. Set `MASON_GENIE_QUESTION` for
+another dataset and `MASON_GENIE_EXPECTED_VALUE` to assert a known result cell.
+
+## Initialize the chat app demo
+
+The chat app is a LangGraph-specific init overlay, not a command that mutates an existing project.
+It is included by default for `--framework langgraph`; pass `--disable-chat-app` to scaffold the
+API-only backend instead.
+
+```sh
+mason init --framework langgraph \
+  --profile <profile> \
+  ./my-agent
+cd ./my-agent
+mason dev
+```
+
+The chat app includes synchronous, SSE streaming, background polling, Session Store, Memory Store,
+and HITL resume UI. The framework-specific overlay adds `ui/`, `runtime/ui.py`, the UI-enabled
+`runtime/main.py`, and UI tests.
+
+For the full deployed demo, bind both managed stores, then deploy:
+
+```sh
+mason sessions bind mason-demo-sessions
+mason memory bind mason-demo-memory
+mason --profile <profile> deploy mason-agent-demo --source .
+```
+
+(`bind` declares the store name in `agent.toml`; `mason deploy` creates any declared-but-missing
+store and grants the app's service principal access to it. The memory store id flows to the runtime
+via the `AGENT_MEMORY_STORE` env var, injected by `deploy` and `mason dev` — it is not persisted
+in `agent.toml`.)
+
+The chat UI generates a stable application session UUID in browser local storage, places it inside
+the invocation's opaque `input`, and creates a fresh invocation UUID per turn. The
+`__Host-databricks-app-router` cookie remains independent: API clients may reuse it for sticky
+replica routing, but it is neither authentication nor the template's application session state.
+
+The generated `README.md` documents every request the client makes: config discovery, sync and SSE
+invocations, background submission and polling, session transcript loading, HITL resume, and memory
+entry operations. Capability colors are automatic from `/api/demo/config`; only the
+sync/streaming/background transport selector is manual.
+
+## Contributing
+
+Developing Mason itself - the CLI, SDK/runtime, and templates - plus the local dev loop and how to
+test unreleased changes on `mason dev` and `mason deploy`, is covered in
+[CONTRIBUTING.md](CONTRIBUTING.md).