muscles-data-sql 2.0.0__tar.gz

muscles_data_sql-2.0.0/PKG-INFO

@@ -0,0 +1,223 @@
+Metadata-Version: 2.4
+Name: muscles-data-sql
+Version: 2.0.0
+Summary: Muscles SQL data layer for repositories, migrations and AI-first workflow.
+Author-email: "Denis B." <denis@butko.info>
+License: MIT
+Project-URL: Homepage, https://github.com/butkoden/muscles-data-sql
+Project-URL: Repository, https://github.com/butkoden/muscles-data-sql
+Project-URL: Documentation, https://github.com/butkoden/muscles-data-sql#readme
+Project-URL: Releases, https://github.com/butkoden/muscles-data-sql/releases
+Project-URL: muscles, https://pypi.org/project/muscles/
+Project-URL: muscles-documents, https://pypi.org/project/muscles-documents/
+Project-URL: muscles-ai, https://pypi.org/project/muscles-ai/
+Project-URL: muscles-otel, https://pypi.org/project/muscles-otel/
+Project-URL: muscles-benchmarks, https://pypi.org/project/muscles-benchmarks/
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]<4.0,>=3.1; extra == "postgres"
+Provides-Extra: data
+Requires-Dist: muscles-data<1.0.0,>=0.2.0; extra == "data"
+Provides-Extra: legacy
+Requires-Dist: muscles<2.0.0,>=1.0.0rc1; extra == "legacy"
+Requires-Dist: muscles-data<1.0.0,>=0.2.0; extra == "legacy"
+Requires-Dist: SQLAlchemy<3.0,>=2.0; extra == "legacy"
+Requires-Dist: click<9.0,>=8.1; extra == "legacy"
+Requires-Dist: pydantic<3.0,>=2.7; extra == "legacy"
+Provides-Extra: dev
+Requires-Dist: pytest<9.0,>=8.0; extra == "dev"
+Requires-Dist: muscles-data-sql[legacy]; extra == "dev"
+
+# Muscles Data SQL
+
+Muscles SQL is a data layer package for SQL databases:
+- model-to-table mapping
+- engine and session management
+- repository CRUD and advanced query API (filters/operators/joins/aggregates)
+- transactions and Unit of Work (nested/savepoint/retry helpers)
+- native SQL migrations through the shared `muscles-data` migration registry
+- inspect/doctor support with machine-readable diagnostics
+
+## Related Repositories
+
+- [`muscles`](https://github.com/butkoden/muscles) - core schemas, actions, DI and canonical documentation.
+- [`muscles-documents`](https://github.com/butkoden/muscles-documents) - document metadata/state can use SQL persistence in applications.
+- [`muscles-ai`](https://github.com/butkoden/muscles-ai) - AI/RAG flows can use SQL-backed application state without making SQL generic storage.
+- [`muscles-otel`](https://github.com/butkoden/muscles-otel) - observability hooks around SQL-backed flows.
+- [`muscles-benchmarks`](https://github.com/butkoden/muscles-benchmarks) - SQL transaction and mapping regression checks.
+
+## Quickstart
+
+This low-level API is useful for standalone scripts and libraries that explicitly
+own their database lifetime. Application packages should use the shared
+`app.resource` integration below instead of constructing `SqlDatabase` objects.
+
+The lightweight SQL API has no mandatory third-party dependencies. SQLite uses
+Python's standard library; PostgreSQL requires the `postgres` extra. The
+named-resource integration requires `data`; SQLAlchemy repositories and CLI
+require `legacy`. In this checkout install from the library
+directory using `pip install -e '.[postgres]'`, `.[data]` or `.[legacy]` as
+needed. Publishing the renamed distribution is a separate step.
+
+```python
+from muscles.data.sql import SqlDatabase
+
+db = SqlDatabase.sqlite("app.db")
+try:
+    with db.transaction() as tx:
+        tx.execute("CREATE TABLE IF NOT EXISTS users (id INTEGER, name TEXT)")
+        tx.execute("INSERT INTO users VALUES (?, ?)", (1, "Denis"))
+        users = tx.fetch_all("SELECT id, name FROM users WHERE id = ?", (1,))
+finally:
+    db.close()
+```
+
+`SqlDatabase.postgres(dsn)` uses psycopg and its native parameter syntax. A
+database owns one connection and is not a connection pool. Transactions from
+different threads are serialized on that connection; nested transactions in the
+same thread are rejected. Direct SQL does not enforce data policies;
+the caller must ensure access restrictions. Commit occurs on successful exit,
+and exceptions roll the transaction back.
+
+## `muscles-data` Resource Integration
+
+The application composition root owns the catalog and `DataRuntime`. Register
+`NativeSqlFactory` there once, and load `data.resources` from the central
+application configuration:
+
+```python
+from muscles.data import DataConfig, DataRuntime
+from muscles.data.catalog import DataAdapterCatalog
+from muscles.data.sql.data_adapter import NativeSqlFactory
+
+class App:
+    def __init__(self):
+        # self.config is the application's shared Muscles configuration.
+        catalog = DataAdapterCatalog.with_defaults()
+        catalog.register(NativeSqlFactory())
+        self.resource = DataRuntime(
+            config=DataConfig.from_raw({
+                "resources": self.config.get("data.resources").value() or {},
+            }),
+            catalog=catalog,
+        )
+```
+
+The database resource itself is declared once in the application's central
+configuration, for example with `type: native_sql`, `driver: postgres` and
+`url_env: DATABASE_URL`. A package uses the existing resource and does not
+create another catalog, runtime or connection configuration:
+
+```python
+from muscles.data.ports import SqlTransactionPort
+
+sql = app.resource.connect("database", SqlTransactionPort)
+with sql.transaction() as tx:
+    tx.execute("CREATE TABLE IF NOT EXISTS audit (id INTEGER PRIMARY KEY)")
+```
+
+The adapter opens its native connection lazily on the first transaction and
+closes it with the owning `DataRuntime`. Direct `SqlDatabase` examples above
+are for standalone low-level use; application package code should resolve its
+port through `app.resource`.
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+For local development with the test and legacy dependencies:
+
+```bash
+pip install -e ".[dev]"
+pytest -q
+muscles-data-sql doctor --url sqlite:///./app.db
+```
+
+## Named SQL Connections
+
+`muscles-data-sql` can manage multiple SQL connections without becoming a generic
+storage registry. The registry is SQL-only: it owns SQL connection configs,
+lazy SQLAlchemy `EngineManager` instances, sessions, inspect and doctor reports.
+
+```python
+from muscles.data.sql import SqlConnectionConfig, SqlConnectionRegistry
+
+registry = SqlConnectionRegistry(
+    [
+        SqlConnectionConfig(name="default", url="sqlite:///./app.db"),
+        SqlConnectionConfig(name="analytics", url="sqlite:///./analytics.db", role="read"),
+    ]
+)
+
+session = registry.session("analytics")
+report = registry.inspect("analytics")
+```
+
+CLI diagnostics can read a JSON config:
+
+```json
+{
+  "connections": {
+    "default": "sqlite:///./app.db",
+    "analytics": {"url": "sqlite:///./analytics.db", "role": "read"}
+  }
+}
+```
+
+```bash
+muscles-data-sql inspect --config sql-connections.json --connection analytics
+muscles-data-sql doctor --config sql-connections.json --all
+```
+
+Diagnostic output uses safe URLs and does not print passwords from DSNs.
+
+## Advanced Query Example
+
+```python
+from sqlalchemy import func
+from muscles.data.sql import FilterClause, JoinClause, QuerySpec, SqlRepository
+
+spec = QuerySpec(
+    filters=[FilterClause("status", "eq", "active")],
+    joins=[JoinClause(table=orders, on=users.c.id == orders.c.user_id)],
+    select_columns=[users.c.id, func.count(orders.c.id).label("orders_total")],
+    group_by=[users.c.id],
+    order_by=[users.c.id.asc()],
+)
+rows = SqlRepository(session, users).aggregate(spec)
+```
+
+## Data operation policies
+
+`SqlRepository` accepts the policy pipeline and a request-scoped context from
+`muscles-data`. Policy predicates are combined with caller filters using AND;
+they apply to reads, counts, updates, deletes, creates, bulk creates and upserts.
+For update/delete, predicates are part of the SQL statement. Create and changed
+values are checked before execution. Under an active policy pipeline, upsert
+uses the constrained repository path instead of a native conflict statement.
+
+```python
+from muscles.data import DataOperationContext
+from muscles.data.sql import SqlRepository
+
+repo = SqlRepository(
+    session,
+    documents,
+    policies=data_runtime.policies,
+    context=DataOperationContext(
+        subject_type="account",
+        subject_uid=account_uid,
+        attributes={"workspace_uid": workspace_uid},
+    ),
+)
+rows = repo.find({"owner_uid": account_uid}, limit=50)
+```
+
+The policy contract is backend-neutral; each adapter is responsible for
+translating predicates and enforcing them before pagination or mutation. Direct
+SQLAlchemy session use is outside this enforcement boundary.
+
+Repository: https://github.com/butkoden/muscles-data-sql

muscles_data_sql-2.0.0/README.md

@@ -0,0 +1,192 @@
+# Muscles Data SQL
+
+Muscles SQL is a data layer package for SQL databases:
+- model-to-table mapping
+- engine and session management
+- repository CRUD and advanced query API (filters/operators/joins/aggregates)
+- transactions and Unit of Work (nested/savepoint/retry helpers)
+- native SQL migrations through the shared `muscles-data` migration registry
+- inspect/doctor support with machine-readable diagnostics
+
+## Related Repositories
+
+- [`muscles`](https://github.com/butkoden/muscles) - core schemas, actions, DI and canonical documentation.
+- [`muscles-documents`](https://github.com/butkoden/muscles-documents) - document metadata/state can use SQL persistence in applications.
+- [`muscles-ai`](https://github.com/butkoden/muscles-ai) - AI/RAG flows can use SQL-backed application state without making SQL generic storage.
+- [`muscles-otel`](https://github.com/butkoden/muscles-otel) - observability hooks around SQL-backed flows.
+- [`muscles-benchmarks`](https://github.com/butkoden/muscles-benchmarks) - SQL transaction and mapping regression checks.
+
+## Quickstart
+
+This low-level API is useful for standalone scripts and libraries that explicitly
+own their database lifetime. Application packages should use the shared
+`app.resource` integration below instead of constructing `SqlDatabase` objects.
+
+The lightweight SQL API has no mandatory third-party dependencies. SQLite uses
+Python's standard library; PostgreSQL requires the `postgres` extra. The
+named-resource integration requires `data`; SQLAlchemy repositories and CLI
+require `legacy`. In this checkout install from the library
+directory using `pip install -e '.[postgres]'`, `.[data]` or `.[legacy]` as
+needed. Publishing the renamed distribution is a separate step.
+
+```python
+from muscles.data.sql import SqlDatabase
+
+db = SqlDatabase.sqlite("app.db")
+try:
+    with db.transaction() as tx:
+        tx.execute("CREATE TABLE IF NOT EXISTS users (id INTEGER, name TEXT)")
+        tx.execute("INSERT INTO users VALUES (?, ?)", (1, "Denis"))
+        users = tx.fetch_all("SELECT id, name FROM users WHERE id = ?", (1,))
+finally:
+    db.close()
+```
+
+`SqlDatabase.postgres(dsn)` uses psycopg and its native parameter syntax. A
+database owns one connection and is not a connection pool. Transactions from
+different threads are serialized on that connection; nested transactions in the
+same thread are rejected. Direct SQL does not enforce data policies;
+the caller must ensure access restrictions. Commit occurs on successful exit,
+and exceptions roll the transaction back.
+
+## `muscles-data` Resource Integration
+
+The application composition root owns the catalog and `DataRuntime`. Register
+`NativeSqlFactory` there once, and load `data.resources` from the central
+application configuration:
+
+```python
+from muscles.data import DataConfig, DataRuntime
+from muscles.data.catalog import DataAdapterCatalog
+from muscles.data.sql.data_adapter import NativeSqlFactory
+
+class App:
+    def __init__(self):
+        # self.config is the application's shared Muscles configuration.
+        catalog = DataAdapterCatalog.with_defaults()
+        catalog.register(NativeSqlFactory())
+        self.resource = DataRuntime(
+            config=DataConfig.from_raw({
+                "resources": self.config.get("data.resources").value() or {},
+            }),
+            catalog=catalog,
+        )
+```
+
+The database resource itself is declared once in the application's central
+configuration, for example with `type: native_sql`, `driver: postgres` and
+`url_env: DATABASE_URL`. A package uses the existing resource and does not
+create another catalog, runtime or connection configuration:
+
+```python
+from muscles.data.ports import SqlTransactionPort
+
+sql = app.resource.connect("database", SqlTransactionPort)
+with sql.transaction() as tx:
+    tx.execute("CREATE TABLE IF NOT EXISTS audit (id INTEGER PRIMARY KEY)")
+```
+
+The adapter opens its native connection lazily on the first transaction and
+closes it with the owning `DataRuntime`. Direct `SqlDatabase` examples above
+are for standalone low-level use; application package code should resolve its
+port through `app.resource`.
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+For local development with the test and legacy dependencies:
+
+```bash
+pip install -e ".[dev]"
+pytest -q
+muscles-data-sql doctor --url sqlite:///./app.db
+```
+
+## Named SQL Connections
+
+`muscles-data-sql` can manage multiple SQL connections without becoming a generic
+storage registry. The registry is SQL-only: it owns SQL connection configs,
+lazy SQLAlchemy `EngineManager` instances, sessions, inspect and doctor reports.
+
+```python
+from muscles.data.sql import SqlConnectionConfig, SqlConnectionRegistry
+
+registry = SqlConnectionRegistry(
+    [
+        SqlConnectionConfig(name="default", url="sqlite:///./app.db"),
+        SqlConnectionConfig(name="analytics", url="sqlite:///./analytics.db", role="read"),
+    ]
+)
+
+session = registry.session("analytics")
+report = registry.inspect("analytics")
+```
+
+CLI diagnostics can read a JSON config:
+
+```json
+{
+  "connections": {
+    "default": "sqlite:///./app.db",
+    "analytics": {"url": "sqlite:///./analytics.db", "role": "read"}
+  }
+}
+```
+
+```bash
+muscles-data-sql inspect --config sql-connections.json --connection analytics
+muscles-data-sql doctor --config sql-connections.json --all
+```
+
+Diagnostic output uses safe URLs and does not print passwords from DSNs.
+
+## Advanced Query Example
+
+```python
+from sqlalchemy import func
+from muscles.data.sql import FilterClause, JoinClause, QuerySpec, SqlRepository
+
+spec = QuerySpec(
+    filters=[FilterClause("status", "eq", "active")],
+    joins=[JoinClause(table=orders, on=users.c.id == orders.c.user_id)],
+    select_columns=[users.c.id, func.count(orders.c.id).label("orders_total")],
+    group_by=[users.c.id],
+    order_by=[users.c.id.asc()],
+)
+rows = SqlRepository(session, users).aggregate(spec)
+```
+
+## Data operation policies
+
+`SqlRepository` accepts the policy pipeline and a request-scoped context from
+`muscles-data`. Policy predicates are combined with caller filters using AND;
+they apply to reads, counts, updates, deletes, creates, bulk creates and upserts.
+For update/delete, predicates are part of the SQL statement. Create and changed
+values are checked before execution. Under an active policy pipeline, upsert
+uses the constrained repository path instead of a native conflict statement.
+
+```python
+from muscles.data import DataOperationContext
+from muscles.data.sql import SqlRepository
+
+repo = SqlRepository(
+    session,
+    documents,
+    policies=data_runtime.policies,
+    context=DataOperationContext(
+        subject_type="account",
+        subject_uid=account_uid,
+        attributes={"workspace_uid": workspace_uid},
+    ),
+)
+rows = repo.find({"owner_uid": account_uid}, limit=50)
+```
+
+The policy contract is backend-neutral; each adapter is responsible for
+translating predicates and enforcing them before pagination or mutation. Direct
+SQLAlchemy session use is outside this enforcement boundary.
+
+Repository: https://github.com/butkoden/muscles-data-sql

muscles_data_sql-2.0.0/pyproject.toml

@@ -0,0 +1,52 @@
+[build-system]
+requires = ["setuptools>=69", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "muscles-data-sql"
+version = "2.0.0"
+description = "Muscles SQL data layer for repositories, migrations and AI-first workflow."
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.10"
+authors = [{ name = "Denis B.", email = "denis@butko.info" }]
+dependencies = []
+
+[project.optional-dependencies]
+postgres = ["psycopg[binary]>=3.1,<4.0"]
+data = ["muscles-data>=0.2.0,<1.0.0"]
+legacy = [
+  "muscles>=1.0.0rc1,<2.0.0",
+  "muscles-data>=0.2.0,<1.0.0",
+  "SQLAlchemy>=2.0,<3.0",
+  "click>=8.1,<9.0",
+  "pydantic>=2.7,<3.0",
+]
+dev = [
+  "pytest>=8.0,<9.0",
+  "muscles-data-sql[legacy]",
+]
+
+[project.urls]
+Homepage = "https://github.com/butkoden/muscles-data-sql"
+Repository = "https://github.com/butkoden/muscles-data-sql"
+Documentation = "https://github.com/butkoden/muscles-data-sql#readme"
+Releases = "https://github.com/butkoden/muscles-data-sql/releases"
+"muscles" = "https://pypi.org/project/muscles/"
+"muscles-documents" = "https://pypi.org/project/muscles-documents/"
+"muscles-ai" = "https://pypi.org/project/muscles-ai/"
+"muscles-otel" = "https://pypi.org/project/muscles-otel/"
+"muscles-benchmarks" = "https://pypi.org/project/muscles-benchmarks/"
+
+[project.scripts]
+muscles-data-sql = "muscles.data.sql.cli:main"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["muscles.data.sql*"]

muscles_data_sql-2.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

muscles_data_sql-2.0.0/src/muscles/data/sql/__init__.py

@@ -0,0 +1,114 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from .native_migrations import NativeSqlMigration, NativeSqlMigrationBackend, NativeSqlMigrationSourceBackend
+    from .drivers import SqlDriverCatalog
+    from .native import SqlDatabase, SqlTransaction
+    from .model_mapper import ModelMapper
+    from .config import DatabaseConfig
+    from .config import SqlConnectionConfig
+    from .connections import SqlConnectionRegistry, UnknownSqlConnection
+    from .engine import EngineManager
+    from .mapping import map_model
+    from .providers import SqlResourceGeneratorProvider
+    from .query import FilterClause, JoinClause, QuerySpec
+    from .repository import SqlRepository
+    from .uow import UnitOfWork
+    from .sqlalchemy_compat import text
+
+__all__ = [
+    "NativeSqlMigration",
+    "NativeSqlMigrationBackend",
+    "NativeSqlMigrationSourceBackend",
+    "SqlDriverCatalog",
+    "SqlDatabase",
+    "SqlTransaction",
+    "ModelMapper",
+    "DatabaseConfig",
+    "SqlConnectionConfig",
+    "SqlConnectionRegistry",
+    "UnknownSqlConnection",
+    "EngineManager",
+    "map_model",
+    "SqlRepository",
+    "UnitOfWork",
+    "text",
+    "QuerySpec",
+    "FilterClause",
+    "JoinClause",
+    "SqlResourceGeneratorProvider",
+]
+
+
+def __getattr__(name: str):
+    if name in {"NativeSqlMigration", "NativeSqlMigrationBackend", "NativeSqlMigrationSourceBackend"}:
+        from . import native_migrations
+
+        return getattr(native_migrations, name)
+    if name == "SqlDriverCatalog":
+        from .drivers import SqlDriverCatalog
+
+        return SqlDriverCatalog
+    if name == "ModelMapper":
+        from .model_mapper import ModelMapper
+
+        return ModelMapper
+    if name in {"SqlDatabase", "SqlTransaction"}:
+        from .native import SqlDatabase, SqlTransaction
+
+        return {"SqlDatabase": SqlDatabase, "SqlTransaction": SqlTransaction}[name]
+    if name == "DatabaseConfig":
+        from .config import DatabaseConfig
+
+        return DatabaseConfig
+    if name == "SqlConnectionConfig":
+        from .config import SqlConnectionConfig
+
+        return SqlConnectionConfig
+    if name == "SqlConnectionRegistry":
+        from .connections import SqlConnectionRegistry
+
+        return SqlConnectionRegistry
+    if name == "UnknownSqlConnection":
+        from .connections import UnknownSqlConnection
+
+        return UnknownSqlConnection
+    if name == "EngineManager":
+        from .engine import EngineManager
+
+        return EngineManager
+    if name == "map_model":
+        from .mapping import map_model
+
+        return map_model
+    if name == "SqlRepository":
+        from .repository import SqlRepository
+
+        return SqlRepository
+    if name == "UnitOfWork":
+        from .uow import UnitOfWork
+
+        return UnitOfWork
+    if name == "text":
+        from .sqlalchemy_compat import text
+
+        return text
+    if name == "QuerySpec":
+        from .query import QuerySpec
+
+        return QuerySpec
+    if name == "FilterClause":
+        from .query import FilterClause
+
+        return FilterClause
+    if name == "JoinClause":
+        from .query import JoinClause
+
+        return JoinClause
+    if name == "SqlResourceGeneratorProvider":
+        from .providers import SqlResourceGeneratorProvider
+
+        return SqlResourceGeneratorProvider
+    raise AttributeError(name)

muscles_data_sql-2.0.0/src/muscles/data/sql/cli.py

@@ -0,0 +1,76 @@
+import json
+from pathlib import Path
+
+import click
+
+from .config import DatabaseConfig
+from .connections import SqlConnectionRegistry, load_connection_registry
+from .providers import SqlGenerationRequest, SqlResourceGeneratorProvider
+
+
+@click.group()
+def main():
+    pass
+
+
+def _registry_from_options(url: str | None, config_path: str | None) -> SqlConnectionRegistry:
+    if config_path:
+        return load_connection_registry(config_path)
+    if url:
+        return SqlConnectionRegistry.from_database_config(DatabaseConfig(url=url))
+    raise click.UsageError("Use --url or --config.")
+
+
+@main.command("doctor")
+@click.option("--url", required=False)
+@click.option("--config", "config_path", type=click.Path(exists=True, dir_okay=False), required=False)
+@click.option("--connection", default="default", show_default=True)
+@click.option("--all", "all_connections", is_flag=True, default=False)
+def doctor(url: str | None, config_path: str | None, connection: str, all_connections: bool):
+    registry = _registry_from_options(url, config_path)
+    report = registry.inspect_all() if all_connections else registry.inspect(connection)
+    click.echo(json.dumps(report, ensure_ascii=False))
+
+
+@main.command("inspect")
+@click.option("--url", required=False)
+@click.option("--config", "config_path", type=click.Path(exists=True, dir_okay=False), required=False)
+@click.option("--connection", default="default", show_default=True)
+def inspect_cmd(url: str | None, config_path: str | None, connection: str):
+    registry = _registry_from_options(url, config_path)
+    click.echo(json.dumps(registry.inspect(connection), ensure_ascii=False))
+
+
+@main.group("generate")
+def generate():
+    pass
+
+
+@generate.command("resource")
+@click.argument("name")
+@click.option("--sql", is_flag=True, default=False)
+def generate_resource(name: str, sql: bool):
+    if sql:
+        provider = SqlResourceGeneratorProvider()
+        generated = provider.generate(
+            project_root=Path.cwd(),
+            request=SqlGenerationRequest(
+                generator_type="sql-resource",
+                name=name,
+                force=False,
+                with_tests=False,
+            ),
+        )
+        click.echo(generated[0])
+        return
+
+    root = Path("generated")
+    root.mkdir(exist_ok=True)
+    target = root / f"{name}.py"
+    content = f"class {name.title().replace('_', '')}Resource:\n    sql_enabled = {sql}\n"
+    target.write_text(content, encoding="utf-8")
+    click.echo(str(target))
+
+
+if __name__ == "__main__":
+    main()