dao-ai 0.1.0__tar.gz → 0.1.11__tar.gz

{dao_ai-0.1.0 → dao_ai-0.1.11}/.gitignore

@@ -276,3 +276,8 @@ mlruns/
 
 # Generated bundle configurations (created from databricks.yaml.template at deployment time)
 databricks.yaml
+
+.cursor/
+
+# Deployment directory for local source code
+deploy/

{dao_ai-0.1.0 → dao_ai-0.1.11}/CHANGELOG.md

@@ -7,6 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- **MCP Tool Filtering**: Control which tools are loaded from MCP servers
+  - `include_tools`: Optional allowlist with glob pattern support (e.g., `["query_*", "list_*"]`)
+  - `exclude_tools`: Optional denylist with glob pattern support (e.g., `["drop_*", "delete_*"]`)
+  - Precedence: exclude always overrides include for maximum security
+  - Pattern syntax: `*` (any chars), `?` (single char), `[abc]` (char set), `[!abc]` (negation)
+  - Use cases: Security (block dangerous operations), performance (reduce context), access control
+  - New example config: `config/examples/02_mcp/filtered_mcp.yaml` with 6 filtering strategies
+  - Comprehensive documentation in configuration reference and MCP README
+
+- **CLI: list-mcp-tools Command**: Discover and inspect MCP tools from configuration
+  - Lists all available tools from configured MCP servers with full details
+  - Shows tool descriptions (no truncation), parameters, types, and requirements
+  - Pretty-printed schemas in readable format (53% more compact than JSON)
+  - Filter statistics: total available, included, and excluded tool counts
+  - `--apply-filters` flag: Show only tools that will be loaded (respects include/exclude)
+  - Aggregated output: Collects all data before display (no logging interference)
+  - Detailed exclusion reasons: Shows why tools are filtered out
+  - Use cases: Discovery, debugging, validation, planning, documentation
+
+- **AnyVariable Support for Additional Fields**: More configuration flexibility
+  - `SchemaModel.catalog_name` and `SchemaModel.schema_name` now support AnyVariable
+  - `DatabricksAppModel.url` now supports AnyVariable
+  - Allows environment variables, Databricks secrets, and fallback chains
+  - Benefits: Environment flexibility, security, portability, backwards compatible
+  - Examples: `{env: CATALOG_NAME}`, `{scope: secrets, secret: url}`, composite fallbacks
+
+### Changed
+- **Refactored Dynamic Prompt Creation**: Simplified and improved `prompts.py`
+  - Consolidated redundant prompt creation logic into single `make_prompt()` function
+  - Removed unused `create_prompt_middleware()` function (dead code)
+  - Cleaner context field handling with generic loop over all context attributes
+  - More maintainable codebase with reduced duplication
+
 ## [0.1.0] - 2025-12-19
 
 ### Added
@@ -21,7 +55,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - `LoggingSummarizationMiddleware`: Extends LangChain's `SummarizationMiddleware` with detailed logging
   - Configurable via `chat_history` in YAML with `max_tokens`, `max_tokens_before_summary`, `max_messages_before_summary`
   - Logs original and summarized message/token counts for observability
-  - New example config: `config/examples/conversation_summarization.yaml`
+  - New example config: `config/examples/04_memory/conversation_summarization.yaml`
 
 - **GEPA-Based Prompt Optimization**: Replaced MLflow optimizer with GEPA (Generative Evolution of Prompts and Agents)
   - `optimize_prompt()` function using DSPy's evolutionary optimization

{dao_ai-0.1.0 → dao_ai-0.1.11}/Makefile

@@ -27,7 +27,7 @@ FIND := $(shell which find)
 RM := rm -rf
 CD := cd
 
-.PHONY: all clean distclean dist check format publish help 
+.PHONY: all clean distclean dist check format publish help test unit integration 
 
 all: dist
 
@@ -66,13 +66,19 @@ schema: depends
 test: 
 	$(PYTEST) -ra --tb=short $(TEST_DIR)
 
+unit: 
+	$(PYTEST) -ra --tb=short -m unit $(TEST_DIR)
+
+integration: 
+	$(PYTEST) -ra --tb=short -m integration $(TEST_DIR)
+
 help:
 	$(info TOP_DIR: $(TOP_DIR))
 	$(info SRC_DIR: $(SRC_DIR))
 	$(info TEST_DIR: $(TEST_DIR))
 	$(info DIST_DIR: $(DIST_DIR))
 	$(info )
-	$(info $$> make [all|dist|install|clean|distclean|format|depends|publish|schema|test|help])
+	$(info $$> make [all|dist|install|clean|distclean|format|depends|publish|schema|test|unit|integration|help])
 	$(info )
 	$(info       all          - build library: [$(LIB)]. This is the default)
 	$(info       dist         - build library: [$(LIB)])
@@ -84,7 +90,9 @@ help:
 	$(info       depends      - installs library dependencies)
 	$(info       publish      - publish library)
 	$(info       schema       - print JSON schema for AppConfig)
-	$(info       test         - run tests)
+	$(info       test         - run all tests)
+	$(info       unit         - run unit tests only)
+	$(info       integration  - run integration tests only)
 	$(info       help         - show this help message)
 	@true
 

dao_ai-0.1.11/PKG-INFO

@@ -0,0 +1,489 @@
+Metadata-Version: 2.4
+Name: dao-ai
+Version: 0.1.11
+Summary: DAO AI: A modular, multi-agent orchestration framework for complex AI workflows. Supports agent handoff, tool integration, and dynamic configuration via YAML.
+Project-URL: Homepage, https://github.com/natefleming/dao-ai
+Project-URL: Documentation, https://natefleming.github.io/dao-ai
+Project-URL: Repository, https://github.com/natefleming/dao-ai
+Project-URL: Issues, https://github.com/natefleming/dao-ai/issues
+Project-URL: Changelog, https://github.com/natefleming/dao-ai/blob/main/CHANGELOG.md
+Author-email: Nate Fleming <nate.fleming@databricks.com>, Nate Fleming <nate.fleming@gmail.com>
+Maintainer-email: Nate Fleming <nate.fleming@databricks.com>
+License: MIT
+License-File: LICENSE
+Keywords: agents,ai,databricks,langchain,langgraph,llm,multi-agent,orchestration,vector-search,workflow
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.11
+Requires-Dist: databricks-agents>=1.9.0
+Requires-Dist: databricks-langchain[memory]>=0.12.1
+Requires-Dist: databricks-mcp>=0.5.0
+Requires-Dist: databricks-sdk[openai]>=0.77.0
+Requires-Dist: ddgs>=9.10.0
+Requires-Dist: dspy>=2.6.27
+Requires-Dist: flashrank>=0.2.10
+Requires-Dist: gepa>=0.0.22
+Requires-Dist: grandalf>=0.8
+Requires-Dist: langchain-community>=0.3.26
+Requires-Dist: langchain-mcp-adapters>=0.2.1
+Requires-Dist: langchain-tavily>=0.2.15
+Requires-Dist: langchain>=1.2.0
+Requires-Dist: langgraph-checkpoint-postgres>=3.0.2
+Requires-Dist: langgraph>=1.0.5
+Requires-Dist: langmem>=0.0.30
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: mcp>=1.24.0
+Requires-Dist: mlflow[databricks]>=3.8.1
+Requires-Dist: nest-asyncio>=1.6.0
+Requires-Dist: openevals>=0.1.3
+Requires-Dist: openpyxl>=3.1.5
+Requires-Dist: psycopg[binary,pool]>=3.3.2
+Requires-Dist: pydantic>=2.12.5
+Requires-Dist: python-dotenv>=1.2.1
+Requires-Dist: pyyaml>=6.0.2
+Requires-Dist: rich>=14.2.0
+Requires-Dist: scipy>=1.14.0
+Requires-Dist: sqlparse>=0.5.4
+Requires-Dist: tomli>=2.3.0
+Requires-Dist: unitycatalog-ai[databricks]>=0.3.2
+Provides-Extra: databricks
+Requires-Dist: databricks-connect>=16.0.0; extra == 'databricks'
+Requires-Dist: databricks-vectorsearch>=0.63; extra == 'databricks'
+Requires-Dist: pyspark>=3.5.0; extra == 'databricks'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.19.1; extra == 'dev'
+Requires-Dist: pre-commit>=4.5.1; extra == 'dev'
+Requires-Dist: pytest>=9.0.2; extra == 'dev'
+Requires-Dist: ruff>=0.14.9; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.7.1; extra == 'docs'
+Requires-Dist: mkdocs>=1.6.1; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=1.0.0; extra == 'docs'
+Provides-Extra: test
+Requires-Dist: pytest-cov>=7.0.0; extra == 'test'
+Requires-Dist: pytest-mock>=3.15.1; extra == 'test'
+Requires-Dist: pytest-rerunfailures>=15.0; extra == 'test'
+Requires-Dist: pytest-xdist>=3.6.1; extra == 'test'
+Requires-Dist: pytest>=9.0.2; extra == 'test'
+Description-Content-Type: text/markdown
+
+# DAO: Declarative Agent Orchestration
+
+[![Version](https://img.shields.io/badge/version-0.1.2-blue.svg)](CHANGELOG.md)
+[![Python](https://img.shields.io/badge/python-3.11+-green.svg)](https://www.python.org/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+**Production-grade AI agents defined in YAML, powered by LangGraph, deployed on Databricks.**
+
+DAO is an **infrastructure-as-code framework** for building, deploying, and managing multi-agent AI systems. Instead of writing boilerplate Python code to wire up agents, tools, and orchestration, you define everything declaratively in YAML configuration files.
+
+```yaml
+# Define an agent in 10 lines of YAML
+agents:
+  product_expert:
+    name: product_expert
+    model: *claude_sonnet
+    tools:
+      - *vector_search_tool
+      - *genie_tool
+    prompt: |
+      You are a product expert. Answer questions about inventory and pricing.
+```
+
+### 🎨 Visual Configuration Studio
+
+Prefer a visual interface? Check out **[DAO AI Builder](https://github.com/natefleming/dao-ai-builder)** — a React-based web application that provides a graphical interface for creating and editing DAO configurations. Perfect for:
+
+- **Exploring** DAO's capabilities through an intuitive UI
+- **Learning** the configuration structure with guided forms
+- **Building** agents visually without writing YAML manually
+- **Importing** and editing existing configurations
+
+DAO AI Builder generates valid YAML configurations that work seamlessly with this framework. Use whichever workflow suits you best — visual builder or direct YAML editing.
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/natefleming/dao-ai-builder/6ca07d2b977d9509b75edfb2e0f45681c840a931/docs/images/dao-ai-builder-screenshot.png" width="700" alt="DAO AI Builder Screenshot">
+</p>
+
+---
+
+## 📚 Documentation
+
+### Getting Started
+- **[Why DAO?](docs/why-dao.md)** - Learn what DAO is and how it compares to other platforms
+- **[Quick Start](#quick-start)** - Build and deploy your first agent in minutes
+- **[Architecture](docs/architecture.md)** - Understand how DAO works under the hood
+
+### Core Concepts
+- **[Key Capabilities](docs/key-capabilities.md)** - Explore 14 powerful features for production agents
+- **[Configuration Reference](docs/configuration-reference.md)** - Complete YAML configuration guide
+- **[Examples](docs/examples.md)** - Ready-to-use example configurations
+
+### Reference
+- **[CLI Reference](docs/cli-reference.md)** - Command-line interface documentation
+- **[Python API](docs/python-api.md)** - Programmatic usage and customization
+- **[FAQ](docs/faq.md)** - Frequently asked questions
+
+### Contributing
+- **[Contributing Guide](docs/contributing.md)** - How to contribute to DAO
+
+---
+
+## Quick Start
+
+### Prerequisites
+
+Before you begin, you'll need:
+
+- **Python 3.11 or newer** installed on your computer ([download here](https://www.python.org/downloads/))
+- **A Databricks workspace** (ask your IT team or see [Databricks docs](https://docs.databricks.com/))
+  - Access to **Unity Catalog** (your organization's data catalog)
+  - **Model Serving** enabled (for deploying AI agents)
+  - *Optional*: Vector Search, Genie (for advanced features)
+
+**Not sure if you have access?** Your Databricks administrator can grant you permissions.
+
+### Installation
+
+**Option 1: Install from PyPI (Recommended)**
+
+The simplest way to get started:
+
+```bash
+# Install directly from PyPI
+pip install dao-ai
+```
+
+**Option 2: For developers familiar with Git**
+
+```bash
+# Clone this repository
+git clone <repo-url>
+cd dao-ai
+
+# Create an isolated Python environment
+uv venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+
+# Install DAO and its dependencies
+make install
+```
+
+**Option 3: For those new to development**
+
+1. Download this project as a ZIP file (click the green "Code" button on GitHub → Download ZIP)
+2. Extract the ZIP file to a folder on your computer
+3. Open a terminal/command prompt and navigate to that folder
+4. Run these commands:
+
+```bash
+# On Mac/Linux:
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e .
+
+# On Windows:
+python -m venv .venv
+.venv\Scripts\activate
+pip install -e .
+```
+
+**Verification:** Run `dao-ai --version` to confirm the installation succeeded.
+
+### Your First Agent
+
+Let's build a simple AI assistant in 4 steps. This agent will use a language model from Databricks to answer questions.
+
+**Step 1: Create a configuration file**
+
+Create a new file called `config/my_agent.yaml` and paste this content:
+
+```yaml
+schemas:
+  my_schema: &my_schema
+    catalog_name: my_catalog        # Replace with your Unity Catalog name
+    schema_name: my_schema          # Replace with your schema name
+
+resources:
+  llms:
+    default_llm: &default_llm
+      name: databricks-meta-llama-3-3-70b-instruct  # The AI model to use
+
+agents:
+  assistant: &assistant
+    name: assistant
+    model: *default_llm
+    prompt: |
+      You are a helpful assistant.
+
+app:
+  name: my_first_agent
+  registered_model:
+    schema: *my_schema
+    name: my_first_agent
+  agents:
+    - *assistant
+  orchestration:
+    swarm:
+      model: *default_llm
+```
+
+**💡 What's happening here?**
+- `schemas`: Points to your Unity Catalog location (where the agent will be registered)
+- `resources`: Defines the AI model (Llama 3.3 70B in this case)
+- `agents`: Describes your assistant agent and its behavior
+- `app`: Configures how the agent is deployed and orchestrated
+
+**Step 2: Validate your configuration**
+
+This checks for errors in your YAML file:
+
+```bash
+dao-ai validate -c config/my_agent.yaml
+```
+
+You should see: ✅ `Configuration is valid!`
+
+**Step 3: Visualize the agent workflow** (optional)
+
+Generate a diagram showing how your agent works:
+
+```bash
+dao-ai graph -c config/my_agent.yaml -o my_agent.png
+```
+
+This creates `my_agent.png` — open it to see a visual representation of your agent.
+
+**Step 4: Deploy to Databricks**
+
+**Option A: Using Python** (programmatic deployment)
+
+```python
+from dao_ai.config import AppConfig
+
+# Load your configuration
+config = AppConfig.from_file("config/my_agent.yaml")
+
+# Package the agent as an MLflow model
+config.create_agent()
+
+# Deploy to Databricks Model Serving
+config.deploy_agent()
+```
+
+**Option B: Using the CLI** (one command)
+
+```bash
+dao-ai bundle --deploy --run -c config/my_agent.yaml
+```
+
+This single command:
+1. Validates your configuration
+2. Packages the agent
+3. Deploys it to Databricks
+4. Creates a serving endpoint
+
+**Deploying to a specific workspace:**
+
+```bash
+# Deploy to AWS workspace
+dao-ai bundle --deploy --run -c config/my_agent.yaml --profile aws-field-eng
+
+# Deploy to Azure workspace
+dao-ai bundle --deploy --run -c config/my_agent.yaml --profile azure-retail
+```
+
+**Step 5: Interact with your agent**
+
+Once deployed, you can chat with your agent using Python:
+
+```python
+from mlflow.deployments import get_deploy_client
+
+# Connect to your Databricks workspace
+client = get_deploy_client("databricks")
+
+# Send a message to your agent
+response = client.predict(
+    endpoint="my_first_agent",
+    inputs={
+        "messages": [{"role": "user", "content": "Hello! What can you help me with?"}],
+        "configurable": {
+            "thread_id": "1",           # Conversation ID
+            "user_id": "demo_user"      # User identifier
+        }
+    }
+)
+
+# Print the agent's response
+print(response["message"]["content"])
+```
+
+**🎉 Congratulations!** You've built and deployed your first AI agent with DAO.
+
+**Next steps:**
+- Explore the [`config/examples/`](config/examples/) folder for more advanced configurations
+- Try the [DAO AI Builder](https://github.com/natefleming/dao-ai-builder) visual interface
+- Learn about [Key Capabilities](docs/key-capabilities.md) to add advanced features
+- Read the [Architecture](docs/architecture.md) documentation to understand how it works
+
+---
+
+## Key Features at a Glance
+
+DAO provides powerful capabilities for building production-ready AI agents:
+
+| Feature | Description |
+|---------|-------------|
+| **Multi-Tool Support** | Python functions, Unity Catalog, MCP, Agent Endpoints |
+| **On-Behalf-Of User** | Per-user permissions and governance |
+| **Advanced Caching** | Two-tier (LRU + Semantic) caching for cost optimization |
+| **Vector Search Reranking** | Improve RAG quality with FlashRank |
+| **Human-in-the-Loop** | Approval workflows for sensitive operations |
+| **Memory & Persistence** | PostgreSQL, Lakebase, or in-memory state |
+| **Prompt Registry** | Version and manage prompts in MLflow |
+| **Prompt Optimization** | Automated tuning with GEPA (Generative Evolution of Prompts and Agents) |
+| **Guardrails** | Content filters, safety checks, validation |
+| **Middleware** | Input validation, logging, performance monitoring, audit trails |
+| **Conversation Summarization** | Handle long conversations automatically |
+| **Structured Output** | JSON schema for predictable responses |
+| **Custom I/O** | Flexible input/output with runtime state |
+| **Hook System** | Lifecycle hooks for initialization and cleanup |
+
+👉 **Learn more:** [Key Capabilities Documentation](docs/key-capabilities.md)
+
+---
+
+## Architecture Overview
+
+```mermaid
+graph TB
+    subgraph yaml["YAML Configuration"]
+        direction LR
+        schemas[Schemas] ~~~ resources[Resources] ~~~ tools[Tools] ~~~ agents[Agents] ~~~ orchestration[Orchestration]
+    end
+    
+    subgraph dao["DAO Framework (Python)"]
+        direction LR
+        config[Config<br/>Loader] ~~~ graph_builder[Graph<br/>Builder] ~~~ nodes[Nodes<br/>Factory] ~~~ tool_factory[Tool<br/>Factory]
+    end
+    
+    subgraph langgraph["LangGraph Runtime"]
+        direction LR
+        msg_hook[Message<br/>Hook] --> supervisor[Supervisor/<br/>Swarm] --> specialized[Specialized<br/>Agents]
+    end
+    
+    subgraph databricks["Databricks Platform"]
+        direction LR
+        model_serving[Model<br/>Serving] ~~~ unity_catalog[Unity<br/>Catalog] ~~~ vector_search[Vector<br/>Search] ~~~ genie_spaces[Genie<br/>Spaces] ~~~ mlflow[MLflow]
+    end
+    
+    yaml ==> dao
+    dao ==> langgraph
+    langgraph ==> databricks
+    
+    style yaml fill:#1B5162,stroke:#618794,stroke-width:3px,color:#fff
+    style dao fill:#FFAB00,stroke:#7D5319,stroke-width:3px,color:#1B3139
+    style langgraph fill:#618794,stroke:#143D4A,stroke-width:3px,color:#fff
+    style databricks fill:#00875C,stroke:#095A35,stroke-width:3px,color:#fff
+```
+
+👉 **Learn more:** [Architecture Documentation](docs/architecture.md)
+
+---
+
+## Example Configurations
+
+The `config/examples/` directory contains ready-to-use configurations organized in a **progressive learning path**:
+
+- `01_getting_started/minimal.yaml` - Simplest possible agent
+- `02_tools/vector_search_with_reranking.yaml` - RAG with improved accuracy
+- `04_genie/genie_semantic_cache.yaml` - NL-to-SQL with two-tier caching
+- `05_memory/conversation_summarization.yaml` - Long conversation handling
+- `06_on_behalf_of_user/obo_basic.yaml` - User-level access control
+- `07_human_in_the_loop/human_in_the_loop.yaml` - Approval workflows
+
+And many more! Follow the numbered path or jump to what you need. See the full guide in [Examples Documentation](docs/examples.md).
+
+---
+
+## CLI Quick Reference
+
+```bash
+# Validate configuration
+dao-ai validate -c config/my_config.yaml
+
+# Generate JSON schema for IDE support
+dao-ai schema > schemas/model_config_schema.json
+
+# Visualize agent workflow
+dao-ai graph -c config/my_config.yaml -o workflow.png
+
+# Deploy with Databricks Asset Bundles
+dao-ai bundle --deploy --run -c config/my_config.yaml
+
+# Deploy to a specific workspace (multi-cloud support)
+dao-ai bundle --deploy -c config/my_config.yaml --profile aws-field-eng
+dao-ai bundle --deploy -c config/my_config.yaml --profile azure-retail
+
+# Interactive chat with agent
+dao-ai chat -c config/my_config.yaml
+```
+
+### Multi-Cloud Deployment
+
+DAO AI supports deploying to Azure, AWS, and GCP workspaces with automatic cloud detection:
+
+```bash
+# Deploy to AWS workspace
+dao-ai bundle --deploy -c config/my_config.yaml --profile aws-prod
+
+# Deploy to Azure workspace
+dao-ai bundle --deploy -c config/my_config.yaml --profile azure-prod
+
+# Deploy to GCP workspace
+dao-ai bundle --deploy -c config/my_config.yaml --profile gcp-prod
+```
+
+The CLI automatically:
+- Detects the cloud provider from your profile's workspace URL
+- Selects appropriate compute node types for each cloud
+- Creates isolated deployment state per profile
+
+👉 **Learn more:** [CLI Reference Documentation](docs/cli-reference.md)
+
+---
+
+## Community & Support
+
+- **Documentation**: [docs/](docs/)
+- **Examples**: [config/examples/](config/examples/)
+- **Issues**: [GitHub Issues](https://github.com/your-org/dao-ai/issues)
+- **Discussions**: [GitHub Discussions](https://github.com/your-org/dao-ai/discussions)
+
+---
+
+## Contributing
+
+We welcome contributions! See the [Contributing Guide](docs/contributing.md) for details on:
+
+- Setting up your development environment
+- Code style and testing guidelines
+- How to submit pull requests
+- Project structure overview
+
+---
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.