memframe 0.2.2__tar.gz → 0.4.0__tar.gz

{memframe-0.2.2 → memframe-0.4.0}/.env.test.example

@@ -15,3 +15,6 @@ CLICKHOUSE_UPLOAD_DB_BACKEND=clickhouse
 CLICKHOUSE_UPLOAD_DB_PARAMS='{"backend":"clickhouse", "host": "localhost", "port": xxxx,  "user":"xxxxx", "password": "xxxxx"}'
 CLICKHOUSE_DB_PARAMS='{"backend":"clickhouse", "host": "localhost", "port": xxxx,  "user":"xxxxx", "password": "xxxxx"}'
 
+OPENAI_API_KEY = sk-proj-xxx
+OLLAMA_API_KEY = xxx
+OLLAMA_BASE_URL = xxx

memframe-0.4.0/CHANGELOG.md

@@ -0,0 +1,183 @@
+# Changelog
+
+All notable changes to memFrame are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/), and the project adheres
+to [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.4.0] - 2026-09-08
+
+### Added
+- **Datetime Wave 1 ops** (`ctx.dt.*`): day/month names, two-column `diff`, `to_datetime` (format, epoch units, raise/coerce), `between`/`before`/`after` filters, `select_year`/`select_month`.
+- **Datetime Wave 2**: `resample` moved from inspection to `ctx.dt` with multi-aggregation, `group_by`, and label control; new `asfreq` reindexing with forward/backward fill.
+- **Datetime Wave 3**: calendar-aware `add_offset` (years through days plus Mon–Fri business-day mode).
+- **Plotly figures embedded as PNG in plot PDF reports** via a shared `_plot_pdf` helper (kaleido raster branch with graceful placeholder fallback); kaleido added to dev extras.
+
+### Fixed
+- **Postgres resample/asfreq SQL**: `mean` maps to `AVG` on all backends, `median` to `percentile_cont` on Postgres; asfreq grid uses native `generate_series` on Postgres.
+- **ClickHouse datetime**: day/month names via `DATE_FORMAT`, nullable result column for `to_datetime` coerce, proven `add<Unit>` date-math family plus `IGNORE NULLS` fill windows.
+- **Plot PDF teardown**: report rendering no longer calls matplotlib-only `suptitle` on plotly Figures.
+- **Python 3.10 compat**: `timezone.utc` replaces `datetime.UTC` / `datetime.utcnow`.
+
+### Changed
+- **Datetime core split** into `base` plus per-backend modules (`duckdb`/`postgres`/`clickhouse` + factory), mirroring inspection; SQL fingerprints prove byte-identical output.
+
+## [0.3.1] - 2026-09-02
+
+### Changed
+- **Sync API runs on one shared background event loop**: `async_to_sync` no longer spins a fresh `asyncio.run` per call — the asyncpg pool is no longer torn down/rebuilt on every sync call and the DuckDB connection is used from a single thread. Sync APIs called from a coroutine already running on the shared loop now raise a `RuntimeError` pointing at the async form.
+- **Dataset contexts snapshot the active `data_id` at creation**: a later `set_active()` no longer retargets an existing context mid-flight (upload/`aset_active` contexts already behaved this way).
+
+### Fixed
+- **SQL-injection hardening**: `fillna` constant fill values are escaped, the `iloc` raw-WHERE escape hatch rejects multi-statement strings, `map` placeholder substitution no longer corrupts function names, identifiers are quote-doubled everywhere, identifier validation rejects trailing newlines, agent credentials use `SecretStr`, and the dashboard guardrail page HTML-escapes model output.
+- **Postgres numeric-text casts**: the arithmetic text-cast hook emitted DuckDB-only `TRY_CAST` (syntax error on Postgres); emulated with a numeric-pattern CASE guard, junk text still becomes NULL.
+- **CSV typed-stream retries skip already-flushed rows** instead of re-inserting them (duplicated rows on mid-stream conversion failures).
+- **Stale agent context**: `domain_context` caches are cleared by `invalidate()`/`advance_table()` — the previous table's schema is no longer served after a table switch.
+- **Deep cache**: DataFrame/Series signatures include a content hash (same-shape frames can no longer collide into wrong cache hits).
+- **Plot fetches capped at 10k rows**; AI-plot PNG rendering runs off-loop with a timeout so a hung Chromium cannot freeze the event loop.
+- **Planner `UnexpectedModelBehavior`**: `SubQueryNode.query` coerces dict → string, planner `retries=3`, heuristic fallback for `value counts`/`correlation` (fixes dashboard one-sentence flake on `gpt-oss:120b-cloud`).
+
+### Refactor / CI
+- Remove dead `DB_TO_PANDAS_DTYPE_MAP` 80-line map — zero runtime consumers (`helper.py:75-154`).
+- Dedup CI: delete duplicated `tox.yml` workflow (keep `ci.yml:tox`), shrink `tox.ini:deps`.
+- Remove `chardet`/`kaleido` from core deps — stdlib `utf-8` decode loop + best-effort PNG (`plot.py` already `except Exception: png=None`).
+
+### Internal
+- The four analytix packages (cleaning, arithmetic, selection, stats) are consolidated onto hook bases mirroring `inspection/` — 10.3k → 6.3k lines, with SQL-fingerprint regression harnesses (182 scenarios × 3 backends) proving per-backend SQL unchanged.
+- Ops integration assertions are order-insensitive (result samples come from unordered SELECTs; ClickHouse merges reorder rows under load).
+
+## [0.3.1rc1] - 2026-09-02
+
+### Changed
+- **Sync API runs on one shared background event loop**: `async_to_sync` no longer spins a fresh `asyncio.run` per call — the asyncpg pool is no longer torn down/rebuilt on every sync call and the DuckDB connection is used from a single thread. Sync APIs called from a coroutine already running on the shared loop now raise a `RuntimeError` pointing at the async form.
+- **Dataset contexts snapshot the active `data_id` at creation**: a later `set_active()` no longer retargets an existing context mid-flight (upload/`aset_active` contexts already behaved this way).
+
+### Fixed
+- **SQL-injection hardening**: `fillna` constant fill values are escaped, the `iloc` raw-WHERE escape hatch rejects multi-statement strings, `map` placeholder substitution no longer corrupts function names, identifiers are quote-doubled everywhere, identifier validation rejects trailing newlines, agent credentials use `SecretStr`, and the dashboard guardrail page HTML-escapes model output.
+- **Postgres numeric-text casts**: the arithmetic text-cast hook emitted DuckDB-only `TRY_CAST` (syntax error on Postgres); emulated with a numeric-pattern CASE guard, junk text still becomes NULL.
+- **CSV typed-stream retries skip already-flushed rows** instead of re-inserting them (duplicated rows on mid-stream conversion failures).
+- **Stale agent context**: `domain_context` caches are cleared by `invalidate()`/`advance_table()` — the previous table's schema is no longer served after a table switch.
+- **Deep cache**: DataFrame/Series signatures include a content hash (same-shape frames can no longer collide into wrong cache hits).
+- **Plot fetches capped at 10k rows**; AI-plot PNG rendering runs off-loop with a timeout so a hung Chromium cannot freeze the event loop.
+
+### Internal
+- The four analytix packages (cleaning, arithmetic, selection, stats) are consolidated onto hook bases mirroring `inspection/` — 10.3k → 6.3k lines, with SQL-fingerprint regression harnesses (182 scenarios × 3 backends) proving per-backend SQL unchanged.
+- Ops integration assertions are order-insensitive (result samples come from unordered SELECTs; ClickHouse merges reorder rows under load).
+
+## [0.3.0] - 2026-08-28
+
+### Added
+- **Ollama Cloud support**: the AI gateway now forwards an explicit `base_url`/`api_key` (e.g. `https://ollama.com/v1`) for Ollama Cloud instead of assuming a local daemon; `OLLAMA_BASE_URL` remains a fallback and the `localhost:11434` default is preserved.
+
+### Fixed
+- **Plots and dashboards render in Colab/Jupyter**: figures now embed plotly.js **inline**, so `df.bar`/`df.scatter`/`df.line`/`df.pie`/`df.scatter_3d`/`df.bar_polar` and the dashboard API render interactively without fetching plotly.js from the blocked `cdn.plot.ly` CDN. `smart_show()` no longer calls `fig.show()` inside a notebook; the kernel auto-displays the returned figure.
+- **AI agent dynamic re-enable**: `AnalyticsAgent` rebuilds when memframe AI settings change, preserving chat history.
+
+## [0.3.0rc3] - 2026-08-26
+
+### Changed
+- **Schema/table rename**: the three auto-created namespaces and their inner tables are now prefixed `memframe_` to avoid collisions in shared databases — `upload` → `memframe_upload`, `transient` → `memframe_transient`, `registry` → `memframe_csv_registry`; inner tables `csv_registry` → `memframe_csv_registry` and `transient_registry` → `memframe_transient_registry` (the transient-registry table lives in `memframe_csv_registry` — a schema on DuckDB/PostgreSQL, a database on ClickHouse). Existing databases are not auto-migrated; fresh databases pick up the new names.
+- `aset_active`/`set_active` now return the dataset `ContextManager` (instead of the bare `data_id` string), so activation flows straight into operations: `ctx = mf.set_active(data_id); ctx.select_dtypes(...)`. `get_active_table()` still returns the active `data_id`.
+
+### Performance
+- ClickHouse backend reuses a pooled `httpx` client with connection keep-alive per event loop, replacing per-query client creation that opened a new TCP connection for every statement (much faster integration runs).
+
+## [0.3.0rc2] - 2026-08-24
+
+### Fixed
+- **Dashboard now renders in notebooks (Colab/Jupyter)**: the dashboard was composed as one Plotly figure but displayed via `display(HTML(...))`, whose `<script>` is stripped by notebook sanitization (blank dashboard). It now displays the native `go.Figure` via Plotly's mimebundle in a live kernel. Terminal/browser rendering unchanged.
+
+## [0.3.0rc1] - 2026-08-23
+
+### Added
+- **AI Dashboard**: one-shot natural-language sentence produces an auto dashboard; renders as a single full-screen Plotly canvas with every DataFrame shown as a table (scalar/dict/list sub-query results now render correctly).
+- **Query guardrail**: validates requests and returns a graceful "blocked" message instead of failing on unsupported queries.
+- **Opt-in Logfire observability**: bring-your-own-key tracing across all agents (enable via `logfire_enabled`/`logfire_token`); host metrics via the optional `system-metrics` extra; traces flushed at the end of `achat`/`adashboard`.
+- **SyncDB**: register pre-existing DuckDB/PostgreSQL/ClickHouse tables into the csv_registry as datasets.
+- Unit-test parallelism via `pytest-xdist`.
+
+### Changed
+- Renamed the `memframe_ai` logger name to `memframe.ai`.
+- `memframe_ai` docs reorganized under `docs/memframe-ai/` (agent, dashboard, observability).
+- The `[logfire]` extra now installs the `[ai]` runtime, so `pip install "memframe[logfire]"` pulls in `pydantic-ai` automatically.
+
+### Fixed
+- Logfire configuration is resilient when the `system-metrics` extra is absent (host metrics are skipped instead of disabling all tracing).
+
+## [0.2.2] - 2026-08-19
+
+### Changed
+- Data Quality Reports (missing-values, completeness, numeric summary, profile report) relocated from the cleaning module to the inspect module.
+- Removed bivariate association methods (chi_square, cramers_v, theil_u, mutual_information) and their categorical wrappers.
+- CONTRIBUTING.md: added a distinct `[refactor]` commit tag (pure restructuring, no behavior/public-API change) separate from `[upgrade]`.
+- README and docs: documented that every call compiles to backend-native SQL; added Open-in-Colab badge.
+
+## [0.2.1] - 2026-08-17
+
+### Fixed
+- Postgres `.corr()`/`.cov()` now compute via streaming in-memory numpy for wide
+  feature sets, removing the Postgres aggregate-explosion hang and greatly
+  speeding up wide correlation/covariance matrices.
+- DuckDB `.corr()` recursion-depth failure fixed for large column-pair counts.
+- Arithmetic `add`/`subtract`/`multiply`/`divide` now correctly handle
+  vector-scalar operands (column ± scalar, scalar ± column, negative and float
+  scalars); scalar-scalar operands are now rejected with a clear `OperationError`.
+- Fixed `clip` date parsing bug.
+
+## [0.2.0] - 2026-08-16
+
+### Changed
+- (BREAKING) `df.chat()` / `achat()` return shape changed: `answer` is now a
+  compact `sub_queryN`-style status string; `blocks` / `return_blocks` removed.
+- Operation-result DataFrame is now returned in full (all rows, all columns)
+  via new `result` and `results` keys.
+- Inline notebook display uses pandas/Colab's default truncation instead of
+  forcing every row inline (which hung notebooks on large tables).
+- LLM tool-return payload stays capped (`max_output_rows`) to bound agent
+  context; full data is still returned to the caller.
+
+### Removed
+- `src/memframe_ai/format.py` and the block/`return_blocks` machinery
+  (`analytics._package`, `ContextManager.achat`, `entrypoints`).
+
+### [0.1.3] - 2026-08-15
+
+### Changed
+- Public operation results are now raw values (DataFrame, dict, scalar, or
+  streaming async iterator) instead of the internal response envelope.
+  Operations that fail raise `OperationError` instead of returning an
+  `is_error` dict.
+- All `tests/integration` and `tests/integration/ops` tests updated to the
+  new raw public API.
+
+### Added
+- Unit tests covering the public result boundary (`test_public_results.py`).
+
+### Docs
+- `docs/api/{cleaning,arithmetic,stats,inspect,selection}.md` updated to the
+  raw return types.
+
+## [0.1.1] - 2026-08-13
+
+### Changed
+- Removed two debug `print()` statements from `CleaningOrchestrator`
+  (`Inferred :` and `detected_dtype--------- :`) that were leaking into
+  agent chat output.
+
+### Notes
+- Same `0.1.0` feature set; this is a packaging/cleanup patch.
+
+## [0.1.0] - 2026-08-13
+
+### Added
+- Database-backed DataFrame API across DuckDB, PostgreSQL, and ClickHouse.
+- Async-first surface with sync equivalents for every operation.
+- Upload from CSV, Parquet, or pandas DataFrame.
+- Inspection, selection, cleaning, statistics, arithmetic, Plotly charts.
+- Two-level cache: lineage audit + replayable result tables.
+- Optional AI agent layer (`memframe_ai`) for natural-language data work.
+- Conventional commit tags (`[feat]`, `[fix]`, `[upgrade]`, `[ci]`, `[docs]`,
+  `[add]`, `[remove]`) documented in `CONTRIBUTING.md`.
+- Slim README with Quick Start, AI Agent section, and links to per-domain
+  docs in `docs/api/`.

{memframe-0.2.2 → memframe-0.4.0}/CONTRIBUTING.md

@@ -54,6 +54,7 @@ tidy. The full set:
 | `[docs]`    | Documentation-only changes (README, `docs/`, docstrings).                                |
 | `[add]`     | Adding a supporting file that does not introduce a feature (fixture, config, isolated helper). |
 | `[remove]`  | Removing a tracked file from the repo (untracking / deletion of source, doc, or config).  |
+| `[chore]`   | Routine maintenance that doesn't fit another tag — dependency bumps, build/packaging housekeeping, misc repo upkeep. Use `[upgrade]`/`[docs]`/`[ci]` for behavior/doc/CI changes. |
 
 ### Format
 
@@ -65,7 +66,7 @@ tidy. The full set:
 
 - Tag is lowercase, bracketed, single space, then the summary.
 - Summary is imperative mood, ≤72 chars, no trailing period.
-- The eight tags above are exhaustive. If a change genuinely fits none of them, prefer the closest tag and explain in the body. `[update]` is **not** a valid tag — use `[upgrade]` or `[docs]` instead.
+- The tags above are exhaustive. If a change genuinely fits none of them, prefer the closest tag and explain in the body. `[update]` is **not** a valid tag — use `[upgrade]` or `[docs]` instead.
 
 ### Examples
 
@@ -78,6 +79,7 @@ git commit -m "[ci] Remove integration job from ci workflow"
 git commit -m "[docs] Slim README and add agent section"
 git commit -m "[add] tests/datasets/sample.parquet fixture"
 git commit -m "[remove] deprecated commands.md scratchpad"
+git commit -m "[chore] Bump httpx and widen pyarrow range"
 ```
 
 ### Branch-required changes
@@ -101,7 +103,8 @@ branch + pull request — never commit directly to `main`):
 Direct commits to `main` are reserved for the maintainer's small,
 self-contained changes that don't need another pair of eyes: single-file
 `[fix]`, single-file `[docs]`, single-file `[add]`, single-file
-`[remove]`, small `[refactor]` (no public-API change), and small
+`[remove]`, small `[refactor]` (no public-API change), small `[chore]`
+maintenance (dependency bumps, build/packaging housekeeping), and small
 single-workflow `[ci]` tweaks.
 
 When in doubt, open a branch.
@@ -159,8 +162,8 @@ uv run ruff check src/
 
 1. `scripts/release.sh <version> "<message>" --dry-run` to preview.
 2. `scripts/release.sh <version> "<message>"` — runs the full gate, bumps, tags, and pushes.
-3. Verify the TestPyPI install:
-   `pip install --index-url https://test.pypi.org/simple memframe`
+3. Verify the TestPyPI install (dependencies come from real PyPI via `--extra-index-url`):
+   `pip install --index-url https://test.pypi.org/simple --extra-index-url https://pypi.org/simple "memframe[ai,logfire]"`
 4. Promote to PyPI via the `Release to PyPI` workflow (Actions → workflow_dispatch).
 
 ## Code of Conduct

{memframe-0.2.2 → memframe-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: memframe
-Version: 0.2.2
+Version: 0.4.0
 Summary: memFrame is a Python package for working with database-backed DataFrame operations. It lets you upload CSV files, Parquet files, and pandas DataFrames into a local DuckDB database or a remote PostgreSQL/Clickhouse database, then run DataFrame-style inspection, selection, cleaning, and table-management operations through one consistent API.
 Project-URL: Homepage, https://github.com/Debojit95/memFrame
 Project-URL: Documentation, https://debojit95.github.io/memFrame/
@@ -19,11 +19,9 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.10
 Requires-Dist: asyncpg>=0.30
-Requires-Dist: chardet>=7.4.3
 Requires-Dist: clickhouse-connect>=1.4.2
 Requires-Dist: duckdb>=1.0.0
 Requires-Dist: httpx>=0.28.1
-Requires-Dist: kaleido>=0.2.1
 Requires-Dist: pandas<3.0,>=2.2
 Requires-Dist: plotly>=6.9.0
 Requires-Dist: pyarrow>=24.0.0
@@ -34,13 +32,20 @@ Requires-Dist: pydantic-ai>=2.22.0; extra == 'ai'
 Requires-Dist: pydantic-settings>=2.0; extra == 'ai'
 Provides-Extra: dev
 Requires-Dist: build>=1.0; extra == 'dev'
+Requires-Dist: kaleido>=1.0.0; extra == 'dev'
 Requires-Dist: matplotlib>=3.10.9; extra == 'dev'
 Requires-Dist: mkdocs-material>=9.7.6; extra == 'dev'
 Requires-Dist: mkdocstrings[python]>=1.0.4; extra == 'dev'
 Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest-xdist>=3.0; extra == 'dev'
 Requires-Dist: pytest>=8.0; extra == 'dev'
 Requires-Dist: tox>=4.0; extra == 'dev'
 Requires-Dist: twine>=6.0; extra == 'dev'
+Provides-Extra: logfire
+Requires-Dist: logfire[system-metrics]; extra == 'logfire'
+Requires-Dist: pydantic-ai-harness[codemode]>=0.1; extra == 'logfire'
+Requires-Dist: pydantic-ai>=2.22.0; extra == 'logfire'
+Requires-Dist: pydantic-settings>=2.0; extra == 'logfire'
 Description-Content-Type: text/markdown
 
 <p align="center">
@@ -52,7 +57,7 @@ Description-Content-Type: text/markdown
 [![PyPI - Version](https://img.shields.io/pypi/v/memframe)](https://pypi.org/project/memframe/)
 [![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://github.com/Debojit95/memFrame/blob/main/pyproject.toml)
 [![CI](https://img.shields.io/github/actions/workflow/status/Debojit95/memFrame/ci.yml?label=ci)](https://github.com/Debojit95/memFrame/actions/workflows/ci.yml)
-[![Tox](https://img.shields.io/github/actions/workflow/status/Debojit95/memFrame/tox.yml?label=tox)](https://github.com/Debojit95/memFrame/actions/workflows/tox.yml)
+[![Tox](https://img.shields.io/github/actions/workflow/status/Debojit95/memFrame/ci.yml?label=tox)](https://github.com/Debojit95/memFrame/actions/workflows/ci.yml)
 [![License - AGPL-3.0](https://img.shields.io/github/license/Debojit95/memFrame)](https://github.com/Debojit95/memFrame/blob/main/LICENSE)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/memframe)](https://pypi.org/project/memframe/)
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1UtoPjzTmia4Cr6y2btDmV8mINgXaBPGP?usp=sharing)
@@ -65,17 +70,21 @@ Description-Content-Type: text/markdown
 - Compiles every pandas-style call to backend-native SQL (DuckDB / PostgreSQL / ClickHouse) and runs it in-engine — your data never leaves the database.
 - Async-first surface with sync equivalents for every operation.
 - Upload from CSV, Parquet, or pandas DataFrame.
+- Sync pre-existing DuckDB, PostgreSQL, or ClickHouse tables as datasets — no re-upload.
 - Inspection, selection, cleaning, statistics, arithmetic, Plotly charts.
 - Two-level cache: lineage audit + replayable result tables.
 - Optional AI agent layer (`memframe_ai`) for chatting with your CSV.
+- Optional **Logfire observability** for the AI agent layer — traces every agent run, LLM call, and tool call, plus host metrics. Opt-in via `logfire_enabled` (local console needs no token; bring-your-own-key for the cloud).
 
 ## Installation
 
 ```bash
-pip install memframe              # core
-uv add memframe                   # alt: uv
-pip install "memframe[ai]"        # + Pydantic AI agent layer
-uv add "memframe[ai]"             # alt: uv
+pip install memframe                      # core
+uv add memframe                          # alt: uv
+pip install "memframe[ai]"               # + Pydantic AI agent layer
+uv add "memframe[ai]"                    # alt: uv
+pip install "memframe[logfire]"          # + AI agent layer AND Logfire observability
+pip install "memframe[ai,logfire]"       # explicit equivalent of the above
 ```
 
 Local development from this repository:
@@ -127,6 +136,30 @@ Each upload returns a dataset context; chain inspection, selection, cleaning,
 arithmetic, statistics, and plotting on it. Sync methods drop the `a` prefix
 (`head`, `iloc`, `fillna`, …).
 
+Already have tables in your database? Register them as datasets without
+re-uploading:
+
+```python
+registered = mf.register_tables()
+# {'sales': [{'data_id': 'a1b2c3', 'table_name': 'orders', 'row_count': 100}, ...]}
+```
+
+Registered tables act like uploads — activate with `set_active(data_id)` and
+query via `memFrame()`. Deleting one only removes memFrame's registry entry;
+your real table is left untouched.
+
+## Architecture
+
+Every pandas-style call compiles to backend-native SQL and runs in-engine — your
+data never leaves the database. The call path is a four-layer stack (public API →
+orchestrator → core SQL engine → backend adapter), with connection, ingestion,
+and a two-level cache as cross-cutting subsystems, plus an optional
+[`memframe_ai`](docs/architecture.md#optional-ai-layer-memframe_ai) agent layer on top.
+
+
+See the full [Architecture](docs/architecture.md) page for the component diagram
+and a per-layer breakdown.
+
 ## AI Agent
 
 `memframe_ai` adds a Pydantic AI agent fleet on top of memFrame. After enabling
@@ -187,6 +220,7 @@ Full reference lives in [`docs/`](docs/):
 - [Getting Started](docs/getting-started.md) — connect, upload, first query.
 - [Connector & Connection](docs/api/connector.md) — DuckDB / Postgres / ClickHouse wiring.
 - [Upload Manager](docs/api/upload-manager.md) — CSV / Parquet / DataFrame ingestion.
+- [Sync Existing Tables](docs/api/syncdb.md) — register pre-existing DB tables as datasets.
 - [Dataset Operations](docs/api/database.md) — table and active-dataset management.
 - [Inspection](docs/api/inspect.md) · [Selection](docs/api/selection.md) · [Cleaning](docs/api/cleaning.md)
 - [Statistics](docs/api/stats.md) · [Arithmetic](docs/api/arithmetic.md)

{memframe-0.2.2 → memframe-0.4.0}/README.md

@@ -7,7 +7,7 @@
 [![PyPI - Version](https://img.shields.io/pypi/v/memframe)](https://pypi.org/project/memframe/)
 [![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://github.com/Debojit95/memFrame/blob/main/pyproject.toml)
 [![CI](https://img.shields.io/github/actions/workflow/status/Debojit95/memFrame/ci.yml?label=ci)](https://github.com/Debojit95/memFrame/actions/workflows/ci.yml)
-[![Tox](https://img.shields.io/github/actions/workflow/status/Debojit95/memFrame/tox.yml?label=tox)](https://github.com/Debojit95/memFrame/actions/workflows/tox.yml)
+[![Tox](https://img.shields.io/github/actions/workflow/status/Debojit95/memFrame/ci.yml?label=tox)](https://github.com/Debojit95/memFrame/actions/workflows/ci.yml)
 [![License - AGPL-3.0](https://img.shields.io/github/license/Debojit95/memFrame)](https://github.com/Debojit95/memFrame/blob/main/LICENSE)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/memframe)](https://pypi.org/project/memframe/)
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1UtoPjzTmia4Cr6y2btDmV8mINgXaBPGP?usp=sharing)
@@ -20,17 +20,21 @@
 - Compiles every pandas-style call to backend-native SQL (DuckDB / PostgreSQL / ClickHouse) and runs it in-engine — your data never leaves the database.
 - Async-first surface with sync equivalents for every operation.
 - Upload from CSV, Parquet, or pandas DataFrame.
+- Sync pre-existing DuckDB, PostgreSQL, or ClickHouse tables as datasets — no re-upload.
 - Inspection, selection, cleaning, statistics, arithmetic, Plotly charts.
 - Two-level cache: lineage audit + replayable result tables.
 - Optional AI agent layer (`memframe_ai`) for chatting with your CSV.
+- Optional **Logfire observability** for the AI agent layer — traces every agent run, LLM call, and tool call, plus host metrics. Opt-in via `logfire_enabled` (local console needs no token; bring-your-own-key for the cloud).
 
 ## Installation
 
 ```bash
-pip install memframe              # core
-uv add memframe                   # alt: uv
-pip install "memframe[ai]"        # + Pydantic AI agent layer
-uv add "memframe[ai]"             # alt: uv
+pip install memframe                      # core
+uv add memframe                          # alt: uv
+pip install "memframe[ai]"               # + Pydantic AI agent layer
+uv add "memframe[ai]"                    # alt: uv
+pip install "memframe[logfire]"          # + AI agent layer AND Logfire observability
+pip install "memframe[ai,logfire]"       # explicit equivalent of the above
 ```
 
 Local development from this repository:
@@ -82,6 +86,30 @@ Each upload returns a dataset context; chain inspection, selection, cleaning,
 arithmetic, statistics, and plotting on it. Sync methods drop the `a` prefix
 (`head`, `iloc`, `fillna`, …).
 
+Already have tables in your database? Register them as datasets without
+re-uploading:
+
+```python
+registered = mf.register_tables()
+# {'sales': [{'data_id': 'a1b2c3', 'table_name': 'orders', 'row_count': 100}, ...]}
+```
+
+Registered tables act like uploads — activate with `set_active(data_id)` and
+query via `memFrame()`. Deleting one only removes memFrame's registry entry;
+your real table is left untouched.
+
+## Architecture
+
+Every pandas-style call compiles to backend-native SQL and runs in-engine — your
+data never leaves the database. The call path is a four-layer stack (public API →
+orchestrator → core SQL engine → backend adapter), with connection, ingestion,
+and a two-level cache as cross-cutting subsystems, plus an optional
+[`memframe_ai`](docs/architecture.md#optional-ai-layer-memframe_ai) agent layer on top.
+
+
+See the full [Architecture](docs/architecture.md) page for the component diagram
+and a per-layer breakdown.
+
 ## AI Agent
 
 `memframe_ai` adds a Pydantic AI agent fleet on top of memFrame. After enabling
@@ -142,6 +170,7 @@ Full reference lives in [`docs/`](docs/):
 - [Getting Started](docs/getting-started.md) — connect, upload, first query.
 - [Connector & Connection](docs/api/connector.md) — DuckDB / Postgres / ClickHouse wiring.
 - [Upload Manager](docs/api/upload-manager.md) — CSV / Parquet / DataFrame ingestion.
+- [Sync Existing Tables](docs/api/syncdb.md) — register pre-existing DB tables as datasets.
 - [Dataset Operations](docs/api/database.md) — table and active-dataset management.
 - [Inspection](docs/api/inspect.md) · [Selection](docs/api/selection.md) · [Cleaning](docs/api/cleaning.md)
 - [Statistics](docs/api/stats.md) · [Arithmetic](docs/api/arithmetic.md)

{memframe-0.2.2 → memframe-0.4.0}/docs/CONTRIBUTING.md

@@ -71,8 +71,8 @@ uv run ruff check src/
 
 1. `scripts/release.sh <version> "<message>" --dry-run` to preview.
 2. `scripts/release.sh <version> "<message>"` — runs the full gate, bumps, tags, and pushes.
-3. Verify the TestPyPI install:
-   `pip install --index-url https://test.pypi.org/simple memframe`
+3. Verify the TestPyPI install (dependencies come from real PyPI via `--extra-index-url`):
+   `pip install --index-url https://test.pypi.org/simple --extra-index-url https://pypi.org/simple "memframe[ai,logfire]"`
 4. Promote to PyPI via the `Release to PyPI` workflow (Actions → workflow_dispatch).
 
 ## Code of Conduct

{memframe-0.2.2 → memframe-0.4.0}/docs/api/caching.md

@@ -11,7 +11,7 @@ so repeat work is either recorded (cheap signature logging) or fully reused
 `record_call` is a `CacheManager` instance (`from memframe.cache import
 record_call, CacheManager`). Every analytics orchestrator method (`selection`,
 `inspection`, `cleaning`, `stats`, `arithmetic`) is decorated with it. The
-decorator wraps the method, records the call into the `transient_registry`
+decorator wraps the method, records the call into the `memframe_transient_registry`
 table, and — for deep cache — persists the returned DataFrame as a transient
 table.
 
@@ -37,7 +37,7 @@ requests and disables deep caching for the whole session.
 
 ### Level 1 — signature-only (default)
 
-Every call records its identity into `transient_registry`:
+Every call records its identity into `memframe_transient_registry`:
 
 - `data_id` — the dataset the call ran on
 - `opidx` — monotonically increasing operation index per dataset
@@ -84,13 +84,14 @@ method**:
 ### Table relocation
 
 Methods that generate a table (e.g. a cleaned dataset) normally produce it in
-the upload schema. With deep caching, that table is moved into
-`transient_schema` so it can be reloaded on later calls and cleaned up with
+the memframe_upload schema (DuckDB/PostgreSQL) or database (ClickHouse). With deep
+caching, that table is moved into `transient_schema` (the transient schema on
+DuckDB/PostgreSQL, database on ClickHouse) so it can be reloaded on later calls and cleaned up with
 `aclear_cache`. The relocation mechanism is backend-specific:
 
-- DuckDB — `CREATE TABLE transient AS SELECT * FROM upload...` then `DROP`.
-- PostgreSQL — `ALTER TABLE upload... SET SCHEMA transient`.
-- ClickHouse — `RENAME TABLE upload... TO transient...`.
+- DuckDB — `CREATE TABLE memframe_transient AS SELECT * FROM memframe_upload...` then `DROP`.
+- PostgreSQL — `ALTER TABLE memframe_upload... SET SCHEMA memframe_transient`.
+- ClickHouse — `RENAME TABLE memframe_upload... TO memframe_transient...`.
 
 ### Cache invalidation
 
@@ -99,20 +100,20 @@ clears its registry rows. The original uploaded table is left untouched.
 
 ## Backend registry
 
-The transient registry lives in the backend registry schema:
+The transient registry lives in the backend `memframe_csv_registry` schema (DuckDB/PostgreSQL) or database (ClickHouse):
 
-| Backend | Registry table | Transient schema |
+| Backend | Registry table | Transient schema/database |
 | --- | --- | --- |
-| DuckDB | `registry.transient_registry` | `transient` |
-| PostgreSQL | `registry.transient_registry` | `transient` |
-| ClickHouse | `registry.transient_registry` | `transient` |
+| DuckDB | `memframe_csv_registry.memframe_transient_registry` | `memframe_transient` |
+| PostgreSQL | `memframe_csv_registry.memframe_transient_registry` | `memframe_transient` |
+| ClickHouse | `memframe_csv_registry.memframe_transient_registry` | `memframe_transient` |
 
-Registry and transient schemas are fixed: `registry`, `transient`, and
-`upload`.
+Registry and transient schemas/databases are fixed: `memframe_csv_registry`, `memframe_transient`, and
+`memframe_upload`.
 
 ## Cache-key and registry index
 
-Lookups filter `transient_registry` by `data_id`, `class_name`, `method_name`,
+Lookups filter `memframe_transient_registry` by `data_id`, `class_name`, `method_name`,
 `args`, `kwargs`. DuckDB and PostgreSQL backends install an index on
 `(data_id, class_name, method_name)` so repeated lookups do not scan the whole
 registry.

{memframe-0.2.2 → memframe-0.4.0}/docs/api/cleaning.md

@@ -408,7 +408,7 @@ response envelope still carries `new_table`, column metadata, and fill metrics.
 ## Generated Tables
 
 Most cleaning operations are non-destructive to the source upload table. They
-create a generated table internally, usually in the same active schema.
+create a generated table internally, usually in the same active schema (or database on ClickHouse).
 
 Column-cleaning operations usually:
 

{memframe-0.2.2 → memframe-0.4.0}/docs/api/database.md

@@ -14,7 +14,7 @@ the connection lifecycle lives on its `ConnectorManager`.
 | Synchronous | Asynchronous | Purpose |
 | --- | --- | --- |
 | `list_tables()` | `await alist_tables()` | List successful uploads |
-| `set_active(data_id)` | `await aset_active(data_id)` | Set the active dataset |
+| `set_active(data_id)` | `await aset_active(data_id)` | Set the active dataset (returns a `ContextManager`) |
 | `get_active_table()` | `await aget_active_table()` | Read the active dataset ID |
 | `delete_table(data_id=None, filename=None)` | `await adelete_table(data_id=None, filename=None)` | Delete an uploaded dataset |
 | `list_operations(data_id=None)` | `await alist_operations(data_id=None)` | List operation history |
@@ -46,17 +46,21 @@ Only registry entries whose upload completed successfully are returned.
 ### Select the Active Dataset
 
 ```python
-mf.set_active("a1b2c3")
-active_id = mf.get_active_table()
+# set_active returns a ContextManager bound to the dataset
+ctx = mf.set_active("a1b2c3")
+ctx.select_dtypes(include="numeric")   # operate directly on the returned context
+active_id = mf.get_active_table()      # still returns the active data_id string
 ```
 
 ```python
-await mf.aset_active("a1b2c3")
-active_id = await mf.aget_active_table()
+ctx = await mf.aset_active("a1b2c3")
 ```
 
-Setting an active dataset first verifies that its upload table exists. Methods
-that accept an optional `data_id` use the active dataset when no ID is given.
+Setting an active dataset first verifies that its upload table exists.
+`set_active`/`aset_active` return a `ContextManager` you can use for subsequent
+operations; methods that accept an optional `data_id` use the active dataset
+when no ID is given. `get_active_table()` continues to return the active
+`data_id` string.
 
 ### Delete a Dataset
 
@@ -102,7 +106,7 @@ create generated tables.
 ## Errors
 
 - Operations that require a database raise `RuntimeError` when not connected.
-- Selecting an unknown `data_id` raises `ValueError`.
+- Selecting an unknown `data_id` raises `DataNotFound`.
 - Deletion requires either `data_id` or `filename`.
 - Listing operations without a supplied or active dataset raises `ValueError`.