dataenginex 0.10.0__tar.gz → 1.0.2__tar.gz

{dataenginex-0.10.0 → dataenginex-1.0.2}/.github/workflows/ci.yml

@@ -7,6 +7,10 @@ on:
     branches: [main, dev]
   workflow_dispatch:
 
+  schedule:
+    # Weekly Python version compatibility check
+    - cron: '0 0 * * 0'
+
 permissions:
   contents: read
 
@@ -26,7 +30,7 @@ jobs:
       - run: uv run poe quality
 
   test:
-    name: Tests
+    name: Tests (Python 3.13)
     runs-on: ubuntu-latest
     needs: quality
     steps:
@@ -45,3 +49,21 @@ jobs:
           fail_ci_if_error: false
         env:
           CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+
+  test-compat:
+    name: Python Compatibility
+    runs-on: ubuntu-latest
+    if: github.event_name == 'schedule'
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v6
+      - uses: astral-sh/setup-uv@v7
+        with:
+          version: "latest"
+          python-version: ${{ matrix.python-version }}
+      - run: uv sync --group ml
+        env:
+          UV_PROJECT_ENVIRONMENT: .venv
+      - run: uv run poe check-all

{dataenginex-0.10.0 → dataenginex-1.0.2}/.github/workflows/docs-notify.yml

@@ -9,7 +9,7 @@ jobs:
   notify:
     runs-on: ubuntu-latest
     steps:
-      - uses: peter-evans/repository-dispatch@v3
+      - uses: peter-evans/repository-dispatch@v4
         with:
           token: ${{ secrets.DOCS_DISPATCH_TOKEN }}
           repository: TheDataEngineX/docs

{dataenginex-0.10.0 → dataenginex-1.0.2}/.gitignore

@@ -90,3 +90,4 @@ Notes.txt
 
 # AI Coding Assistant
 .continue/
+.playwright-mcp/

{dataenginex-0.10.0 → dataenginex-1.0.2}/CHANGELOG.md

@@ -28,6 +28,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.0.0] - 2026-04-07
+
+### Highlights
+
+- **Complete Data + ML + AI Framework**: All phases from the v1.0 master plan implemented — config-driven pipeline via `dex.yaml`, BackendRegistry pattern for swappable backends, unified CLI.
+- **Data Layer**: DuckDB connector (default), CSV connector, PipelineRunner with DAG resolution, transforms (filter, derive, cast, deduplicate), quality gates (completeness, uniqueness), column-level lineage tracking, built-in cron scheduler.
+- **ML Layer**: SQLite-backed experiment tracker, DuckDB feature store, sklearn/xgboost training integration, model registry with versioning (dev → staging → production), built-in model serving via FastAPI, PSI drift detection.
+- **AI Layer**: Built-in ReAct agent runtime, Ollama LLM provider (default), tool registry (sql_query, predict, search), BM25 sparse retrieval (DuckDB FTS), dense vector retrieval (DuckDB VSS HNSW), hybrid retrieval with RRF fusion, agent memory (short-term + episodic).
+- **CLI Commands**: `dex init`, `dex validate`, `dex version`, `dex serve`, `dex run`, `dex train`, `dex agent`, `dex query`.
+- **API**: FastAPI app factory, JWT auth, rate limiting, health endpoints, project CRUD, pipeline run/status, data explorer, ML experiments/models, agent chat/manage, WebSocket for live logs and streaming.
+- **Backend Registry Pattern**: Every subsystem follows ABC + BackendRegistry[T] pattern with built-in implementations and optional extras (Dagster, MLflow, Qdrant, LanceDB, sentence-transformers, PySpark).
+
+### Breaking Changes
+
+- **FastAPI now optional**: Core install (`pip install dataenginex`) includes only lightweight deps. Install `[api]` extra for FastAPI/uvicorn: `pip install dataenginex[api]`
+- **Cloud SDKs now optional**: Core install no longer requires boto3/google-cloud-storage/google-cloud-bigquery. Install `[cloud]` extra: `pip install dataenginex[cloud]`
+- **Routers moved**: API routers moved to application packages. Use `from dataenginex.api import ...` directly (requires `[api]` extra)
+- **Root `__init__.py` slimmed**: Re-exports removed. Import from submodules directly: `from dataenginex.api import HealthChecker` etc.
+
+### Added
+
+- **Full project templates**: `dex init --template [minimal|data-pipeline|ml-project|ai-agent|full-stack|career-intelligence]`
+- **Docker support**: Multi-stage Dockerfile (`ghcr.io/thedataenginex/dex`), docker-compose.yml for full stack
+- **SecOps**: PII scanning in pipelines, masking, audit trail
+- **Quality schema**: Spark audit integration for data quality validation
+- **Examples**: 5 runnable examples in `examples/` directory
+
+### Verification checklist
+
+1. `uv run poe lint` — Ruff checks clean
+2. `uv run poe typecheck` — mypy strict (all modules)
+3. `uv run poe test` — 663 passed, 36 skipped
+4. `pip install dataenginex` — installs successfully
+5. `dex validate dex.yaml` — validates config
+6. `dex version` — shows version
+
+[Unreleased]: https://github.com/TheDataEngineX/DEX/compare/v1.0.0...HEAD
+[1.0.0]: https://github.com/TheDataEngineX/DEX/releases/tag/v1.0.0
+
 ## [0.7.1] - 2026-03-17
 
 ### Fixed

{dataenginex-0.10.0 → dataenginex-1.0.2}/CLAUDE.md

@@ -11,7 +11,7 @@ Goal is to save Claude code tokens for lower cost without loosing quality.
 
 | Package | Location | Purpose |
 |---------|----------|---------|
-| `dataenginex` | `src/dataenginex/` | Core framework — config system, backend registry, CLI, API, ML, AI agents |
+| `dataenginex` | `src/dataenginex/` | Core framework — config, registry, CLI, API, ML, AI (routing, runtime, memory, observability, workflows) |
 
 **Stack:** Python 3.13+ · FastAPI · DuckDB · structlog · Pydantic · Click · Rich · uv · Ruff · mypy strict · pytest
 

dataenginex-1.0.2/CODE_OF_CONDUCT.md

@@ -0,0 +1,57 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment:
+
+- Using welcoming and inclusive language
+- Being respectful of differing viewpoints and experiences
+- Gracefully accepting constructive criticism
+- Focusing on what is best for the community
+- Showing empathy towards other community members
+
+Examples of unacceptable behavior:
+
+- The use of sexualized language or imagery, and sexual attention or advances
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information without explicit permission
+- Other conduct which could reasonably be considered inappropriate
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+**conduct@thedataenginex.dev**.
+
+All complaints will be reviewed and investigated promptly and fairly.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org),
+version 2.1, available at
+<https://www.contributor-covenant.org/version/2/1/code_of_conduct.html>.

{dataenginex-0.10.0 → dataenginex-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataenginex
-Version: 0.10.0
+Version: 1.0.2
 Summary: DataEngineX - Core framework for AI/ML/Data engineering projects
 Author-email: Jay <jayapal.myaka99@gmail.com>
 License: MIT License
@@ -26,29 +26,33 @@ License: MIT License
         SOFTWARE.
 License-File: LICENSE
 Requires-Python: >=3.13
-Requires-Dist: click>=8.3.1
-Requires-Dist: croniter>=6.0.0
-Requires-Dist: duckdb>=1.5.0
+Requires-Dist: click>=8.3.2
+Requires-Dist: croniter>=6.2.2
+Requires-Dist: duckdb>=1.5.2
 Requires-Dist: email-validator>=2.3.0
-Requires-Dist: fastapi>=0.135.1
-Requires-Dist: httpx>=0.28.0
-Requires-Dist: opentelemetry-api>=1.40.0
-Requires-Dist: opentelemetry-exporter-otlp>=1.40.0
-Requires-Dist: opentelemetry-instrumentation-fastapi>=0.61b0
-Requires-Dist: opentelemetry-sdk>=1.40.0
-Requires-Dist: prometheus-client>=0.24.1
+Requires-Dist: fastapi>=0.136.0
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: opentelemetry-api>=1.41.0
+Requires-Dist: opentelemetry-exporter-otlp>=1.41.0
+Requires-Dist: opentelemetry-instrumentation-fastapi>=0.62b0
+Requires-Dist: opentelemetry-sdk>=1.41.0
+Requires-Dist: prometheus-client>=0.25.0
 Requires-Dist: pyarrow>=23.0.1
-Requires-Dist: pydantic>=2.10.0
-Requires-Dist: python-dotenv>=1.2.1
-Requires-Dist: python-json-logger>=4.0.0
+Requires-Dist: pydantic>=2.13.2
+Requires-Dist: python-dotenv>=1.2.2
 Requires-Dist: pyyaml>=6.0.3
-Requires-Dist: rich>=14.3.3
+Requires-Dist: rich>=15.0.0
 Requires-Dist: structlog>=25.5.0
-Requires-Dist: uvicorn>=0.42.0
+Requires-Dist: uvicorn>=0.44.0
+Provides-Extra: auth
+Requires-Dist: ldap3>=2.9.1; extra == 'auth'
+Requires-Dist: pyjwt[crypto]>=2.9.0; extra == 'auth'
 Provides-Extra: cloud
 Requires-Dist: boto3>=1.42.0; extra == 'cloud'
 Requires-Dist: google-cloud-bigquery>=3.40.0; extra == 'cloud'
 Requires-Dist: google-cloud-storage>=3.0.0; extra == 'cloud'
+Provides-Extra: observability
+Requires-Dist: langfuse>=4.3.1; extra == 'observability'
 Description-Content-Type: text/markdown
 
 # dataenginex

{dataenginex-0.10.0 → dataenginex-1.0.2}/README.md

@@ -112,7 +112,7 @@ ______________________________________________________________________
 
 ## Development
 
-See [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) for full setup.
+See [docs/development.md](docs/development.md) for full setup.
 
 ```bash
 uv run poe check-all         # lint + typecheck + tests

dataenginex-1.0.2/SECURITY.md

@@ -0,0 +1,71 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported |
+|---------|-----------|
+| Latest minor release (1.0.x) | ✅ |
+| Previous minor release | ✅ (security fixes only) |
+| Older versions | ❌ |
+
+## Reporting a Vulnerability
+
+**Do NOT open a public issue for security vulnerabilities.**
+
+Instead, please report them via one of these channels:
+
+1. **Email**: security@thedataenginex.dev
+2. **GitHub Security Advisories**: Use the "Report a vulnerability" button on the Security tab
+
+### What to Include
+
+- Description of the vulnerability
+- Steps to reproduce
+- Potential impact assessment
+- Suggested fix (if any)
+
+### Response Timeline
+
+| Stage | Timeline |
+|-------|----------|
+| Acknowledgment | Within 48 hours |
+| Initial assessment | Within 5 business days |
+| Fix development | Within 30 days (critical), 90 days (non-critical) |
+| Public disclosure | After fix is released |
+
+## Disclosure Policy
+
+We follow [coordinated disclosure](https://en.wikipedia.org/wiki/Coordinated_vulnerability_disclosure).
+We will credit reporters in the security advisory unless they prefer to remain anonymous.
+
+## Security Practices
+
+DataEngineX follows these security practices:
+
+- **No hardcoded secrets** — all credentials via environment variables
+- **Parameterized queries** — never SQL concatenation
+- **Input validation** — Pydantic models at API boundaries
+- **Dependency auditing** — automated via `uv run poe security`
+- **Pickle safety** — SafeUnpickler with HMAC verification for model loading
+- **Container security** — non-root users, minimal base images
+- **HTTPS only** — all production traffic encrypted
+- **Least privilege** — minimal permissions for service accounts
+
+## Security-Related Dependencies
+
+| Dependency | Purpose | Security Note |
+|------------|---------|---------------|
+| pydantic | Config validation | Validates all inputs |
+| python-dotenv | Env var loading | Never commit .env files |
+| httpx | HTTP client | Timeout configured |
+| structlog | Logging | No PII in logs by default |
+
+## Auditing
+
+Run security audits locally:
+
+```bash
+uv run poe security  # pip-audit for vulnerabilities
+```
+
+CI runs `pip-audit` and dependency scanning on every PR.

{dataenginex-0.10.0 → dataenginex-1.0.2}/docs/ci-cd.md

@@ -407,4 +407,4 @@ gh workflow run pypi-publish.yml -f tag=v<version>
 
 ______________________________________________________________________
 
-**[← Back to Documentation Hub](docs-hub.md)**
+**[← Back to Documentation](index.md)**

{dataenginex-0.10.0 → dataenginex-1.0.2}/docs/index.md

@@ -4,7 +4,7 @@ Welcome to the DEX docs site.
 
 Start here:
 
-- [Documentation Hub](docs-hub.md)
 - [Development Setup](development.md)
 - [Contributing](contributing.md)
 - [Architecture](architecture.md)
+- [Quickstart](quickstart.md)

{dataenginex-0.10.0 → dataenginex-1.0.2}/docs/observability.md

@@ -795,4 +795,4 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
-**[← Back to Documentation Hub](docs-hub.md)**
+**[← Back to Documentation](index.md)**

{dataenginex-0.10.0 → dataenginex-1.0.2}/docs/sdlc.md

@@ -203,4 +203,4 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
-**[← Back to Documentation Hub](docs-hub.md)**
+**[← Back to Documentation](index.md)**

{dataenginex-0.10.0 → dataenginex-1.0.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataenginex"
-version = "0.10.0"
+version = "1.0.2"
 description = "DataEngineX - Core framework for AI/ML/Data engineering projects"
 authors = [
     {name = "Jay", email = "jayapal.myaka99@gmail.com"}
@@ -9,24 +9,24 @@ readme = "src/dataenginex/README.md"
 requires-python = ">=3.13"
 license = {file = "LICENSE"}
 dependencies = [
-    "pydantic>=2.10.0",
-    "python-dotenv>=1.2.1",
+    "pydantic>=2.13.2",
+    "python-dotenv>=1.2.2",
     "pyyaml>=6.0.3",
-    "click>=8.3.1",
-    "rich>=14.3.3",
-    "duckdb>=1.5.0",
-    "croniter>=6.0.0",
-    "httpx>=0.28.0",
-    "prometheus-client>=0.24.1",
+    "click>=8.3.2",
+    "rich>=15.0.0",
+    "duckdb>=1.5.2",
+    "croniter>=6.2.2",
+    "httpx>=0.28.1",
+    "prometheus-client>=0.25.0",
     # API framework
-    "fastapi>=0.135.1",
-    "uvicorn>=0.42.0",
+    "fastapi>=0.136.0",
+    "uvicorn>=0.44.0",
     "structlog>=25.5.0",
-    "python-json-logger>=4.0.0",
-    "opentelemetry-api>=1.40.0",
-    "opentelemetry-sdk>=1.40.0",
-    "opentelemetry-instrumentation-fastapi>=0.61b0",
-    "opentelemetry-exporter-otlp>=1.40.0",
+    # OpenTelemetry
+    "opentelemetry-api>=1.41.0",
+    "opentelemetry-sdk>=1.41.0",
+    "opentelemetry-instrumentation-fastapi>=0.62b0",
+    "opentelemetry-exporter-otlp>=1.41.0",
     "email-validator>=2.3.0",
     # Data formats
     "pyarrow>=23.0.1",
@@ -41,22 +41,36 @@ cloud = [
     "google-cloud-storage>=3.0.0",
     "google-cloud-bigquery>=3.40.0",
 ]
+auth = [
+    "pyjwt[crypto]>=2.9.0",
+    "ldap3>=2.9.1",
+]
+# NOTE: litellm is NOT declared here because it pins python-dotenv==1.0.1, which
+# conflicts with our python-dotenv>=1.2.1. Install separately for LiteLLMProvider:
+#     pip install 'litellm>=1.83.3' --no-deps
+# or manage via a dedicated venv. The provider is lazy-imported; DEX runs fine
+# without it.
+observability = [
+    "langfuse>=4.3.1",
+]
 
 [dependency-groups]
 dev = [
-    "pytest>=9.0.2",
-    "pytest-cov>=7.0.0",
+    "pytest>=9.0.3",
+    "pytest-cov>=7.1.0",
     "pytest-asyncio>=1.3.0",
-    "ruff>=0.15.6",
-    "mypy>=1.19.1",
+    "ruff>=0.15.11",
+    "mypy>=1.20.1",
     "types-pyyaml>=6.0.12.20250915",
     "types-requests>=2.32.4.20260107",
-    "poethepoet>=0.42.1",
-    "uv>=0.10.0",
-    "mkdocs-material>=9.7.0",
-    "mkdocstrings[python]>=0.29.0",
+    "poethepoet>=0.44.0",
+    "uv>=0.11.6",
+    "mkdocs-material>=9.7.6",
+    "mkdocstrings[python]>=1.0.4",
     "pre-commit>=4.5.1",
     "pip-audit>=2.10.0",
+    "pyjwt[crypto]>=2.12.1",
+    "cryptography>=46.0.7",
 ]
 data = [
     "pyspark>=4.1.1",
@@ -141,6 +155,10 @@ ignore_missing_imports = true
 module = ["sentence_transformers.*"]
 ignore_missing_imports = true
 
+[[tool.mypy.overrides]]
+module = ["litellm.*", "langfuse.*", "ldap3.*"]
+ignore_missing_imports = true
+
 [[tool.mypy.overrides]]
 module = ["pyspark.*"]
 ignore_missing_imports = true
@@ -169,6 +187,8 @@ filterwarnings = [
     "error",                                          # Treat all warnings as errors by default
     "ignore::DeprecationWarning:websockets\\.legacy",  # uvicorn uses deprecated websockets.legacy API
     "ignore::DeprecationWarning:uvicorn\\.protocols",  # uvicorn imports deprecated WebSocketServerProtocol
+    "ignore:unclosed.*socket:ResourceWarning",         # Starlette TestClient GC timing
+    "ignore:unclosed event loop:ResourceWarning",      # Starlette TestClient GC timing
 ]
 
 [tool.coverage.run]

dataenginex-1.0.2/src/dataenginex/ai/__init__.py

@@ -0,0 +1,79 @@
+"""AI layer — agents, retrieval, tools, routing, runtime, memory, observability, workflows.
+
+Public API::
+
+    from dataenginex.ai import (
+        retriever_registry, agent_registry, tool_registry,
+        BuiltinRetriever, BuiltinAgentRuntime,
+        ModelRouter, BaseProvider,
+        Sandbox, SandboxConfig,
+        AuditLog, CostTracker,
+        AgentDAG, Condition,
+        ShortTermMemory, LongTermMemory, EpisodicMemory,
+        CheckpointManager, AgentExecutor, AgentConfig,
+        AgentMetrics,
+    )
+"""
+
+from __future__ import annotations
+
+from dataenginex.ai.agents import agent_registry
+from dataenginex.ai.agents.builtin import BuiltinAgentRuntime
+from dataenginex.ai.memory.base import BaseMemory, MemoryEntry, ShortTermMemory
+from dataenginex.ai.memory.episodic import Episode, EpisodicMemory
+from dataenginex.ai.memory.long_term import LongTermMemory
+from dataenginex.ai.observability.audit import AuditEntry, AuditLog
+from dataenginex.ai.observability.cost import CostTracker, TokenUsage
+from dataenginex.ai.observability.metrics import AgentMetrics
+from dataenginex.ai.retrieval import retriever_registry
+from dataenginex.ai.retrieval.builtin import BuiltinRetriever
+from dataenginex.ai.routing.router import BaseProvider, ModelRouter
+from dataenginex.ai.runtime.checkpoint import Checkpoint, CheckpointManager
+from dataenginex.ai.runtime.executor import AgentConfig, AgentExecutor, AgentResponse
+from dataenginex.ai.runtime.sandbox import Sandbox, SandboxConfig, SandboxResult
+from dataenginex.ai.tools import ToolRegistry, ToolSpec, tool_registry
+from dataenginex.ai.workflows.conditions import Condition
+from dataenginex.ai.workflows.dag import AgentDAG
+from dataenginex.ai.workflows.human_loop import ApprovalGate
+
+__all__ = [
+    # Registries
+    "agent_registry",
+    "retriever_registry",
+    "tool_registry",
+    # Agents
+    "BuiltinAgentRuntime",
+    "BuiltinRetriever",
+    # Tools
+    "ToolRegistry",
+    "ToolSpec",
+    # Memory
+    "BaseMemory",
+    "MemoryEntry",
+    "ShortTermMemory",
+    "LongTermMemory",
+    "EpisodicMemory",
+    "Episode",
+    # Observability
+    "AuditEntry",
+    "AuditLog",
+    "CostTracker",
+    "TokenUsage",
+    "AgentMetrics",
+    # Routing
+    "BaseProvider",
+    "ModelRouter",
+    # Runtime
+    "AgentConfig",
+    "AgentExecutor",
+    "AgentResponse",
+    "Checkpoint",
+    "CheckpointManager",
+    "Sandbox",
+    "SandboxConfig",
+    "SandboxResult",
+    # Workflows
+    "AgentDAG",
+    "ApprovalGate",
+    "Condition",
+]

{dataenginex-0.10.0 → dataenginex-1.0.2}/src/dataenginex/ai/agents/builtin.py

@@ -17,6 +17,10 @@ from dataenginex.ai.agents import agent_registry
 from dataenginex.ai.tools import ToolRegistry, tool_registry
 from dataenginex.ai.tools.builtin import register_builtin_tools
 from dataenginex.core.interfaces import BaseAgentRuntime
+from dataenginex.middleware.domain_metrics import (
+    ai_agent_iterations,
+    ai_tool_calls_total,
+)
 
 logger = structlog.get_logger()
 
@@ -38,12 +42,14 @@ class BuiltinAgentRuntime(BaseAgentRuntime):
         system_prompt: str = "You are a helpful data engineering assistant.",
         tools: ToolRegistry | None = None,
         max_iterations: int = 10,
+        name: str = "builtin",
         **kwargs: Any,
     ) -> None:
         self._llm = llm
         self._system_prompt = system_prompt
         self._tools = tools or tool_registry
         self._max_iterations = max_iterations
+        self._name = name
         self._history: list[dict[str, str]] = []
         register_builtin_tools()
 
@@ -64,6 +70,7 @@ class BuiltinAgentRuntime(BaseAgentRuntime):
             if step_result.get("done", False):
                 response = str(step_result.get("response", ""))
                 self._history.append({"role": "assistant", "content": response})
+                ai_agent_iterations.labels(agent=self._name).observe(iterations)
                 return {"response": response, "iterations": iterations, "tool_calls": tool_calls}
 
             # If tool was called, continue the loop
@@ -73,6 +80,7 @@ class BuiltinAgentRuntime(BaseAgentRuntime):
         # Hit max iterations
         final = "I've reached my reasoning limit. Here's what I have so far."
         self._history.append({"role": "assistant", "content": final})
+        ai_agent_iterations.labels(agent=self._name).observe(self._max_iterations)
         return {"response": final, "iterations": self._max_iterations, "tool_calls": tool_calls}
 
     async def step(self, message: str, **kwargs: Any) -> dict[str, Any]:
@@ -134,8 +142,10 @@ class BuiltinAgentRuntime(BaseAgentRuntime):
         try:
             result = self._tools.call(tool_name, **args)
             observation = f"Tool '{tool_name}' returned: {result}"
+            ai_tool_calls_total.labels(tool=tool_name, status="ok").inc()
         except Exception as e:
             observation = f"Tool '{tool_name}' failed: {e}"
+            ai_tool_calls_total.labels(tool=tool_name, status="error").inc()
 
         self._history.append(
             {"role": "assistant", "content": f"[tool: {tool_name}] {observation}"},

dataenginex-1.0.2/src/dataenginex/ai/memory/__init__.py

@@ -0,0 +1,16 @@
+"""Agent memory — short-term, long-term, and episodic memory."""
+
+from __future__ import annotations
+
+from dataenginex.ai.memory.base import BaseMemory, MemoryEntry, ShortTermMemory
+from dataenginex.ai.memory.episodic import Episode, EpisodicMemory
+from dataenginex.ai.memory.long_term import LongTermMemory
+
+__all__ = [
+    "BaseMemory",
+    "Episode",
+    "EpisodicMemory",
+    "LongTermMemory",
+    "MemoryEntry",
+    "ShortTermMemory",
+]

dataenginex-1.0.2/src/dataenginex/ai/memory/base.py

@@ -0,0 +1,59 @@
+"""Agent memory system — short-term, long-term, and episodic memory."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+from typing import Any
+
+
+@dataclass
+class MemoryEntry:
+    """A single memory entry."""
+
+    content: str
+    role: str  # "user", "assistant", "system", "tool"
+    metadata: dict[str, Any] = field(default_factory=dict)
+    timestamp: float = 0.0
+
+
+class BaseMemory(ABC):
+    """Abstract base class for agent memory."""
+
+    @abstractmethod
+    def add(self, entry: MemoryEntry) -> None:
+        """Add a memory entry."""
+
+    @abstractmethod
+    def search(self, query: str, top_k: int = 5) -> list[MemoryEntry]:
+        """Search memory by semantic similarity."""
+
+    @abstractmethod
+    def recent(self, n: int = 10) -> list[MemoryEntry]:
+        """Get the most recent entries."""
+
+    @abstractmethod
+    def clear(self) -> None:
+        """Clear all memory."""
+
+
+class ShortTermMemory(BaseMemory):
+    """Session/conversation memory — lives in-process, lost on restart."""
+
+    def __init__(self, max_entries: int = 100) -> None:
+        self._entries: list[MemoryEntry] = []
+        self._max = max_entries
+
+    def add(self, entry: MemoryEntry) -> None:
+        self._entries.append(entry)
+        if len(self._entries) > self._max:
+            self._entries.pop(0)
+
+    def search(self, query: str, top_k: int = 5) -> list[MemoryEntry]:
+        return [e for e in self._entries if query.lower() in e.content.lower()][:top_k]
+
+    def recent(self, n: int = 10) -> list[MemoryEntry]:
+        return self._entries[-n:]
+
+    def clear(self) -> None:
+        self._entries.clear()