datablade 0.0.0__py3-none-any.whl → 0.0.6__py3-none-any.whl

datablade/docs/ARCHITECTURE.md

@@ -0,0 +1,102 @@
+# datablade architecture overview
+
+This document explains how the `datablade` modules fit together and highlights the
+main decision points in the data-reading and SQL DDL workflows.
+
+## Package layout
+
+`datablade` is organized into a few focused namespaces:
+
+- `datablade.dataframes`: file readers, DataFrame cleanup, Parquet helpers
+- `datablade.sql`: dialect-aware quoting, DDL generation, bulk loading
+- `datablade.io`: HTTP JSON + ZIP helpers
+- `datablade.utils`: logging, string/list helpers
+- `datablade.core`: backward-compatible re-exports of the above modules
+
+::: mermaid
+flowchart TB
+    root[datablade] --> dataframes[datablade.dataframes]
+    root --> sql[datablade.sql]
+    root --> io[datablade.io]
+    root --> utils[datablade.utils]
+    root --> core[datablade.core - compat]
+
+    dataframes --> readers[readers.py<br/>read_file_smart/read_file_iter]
+    dataframes --> frames[frames.py<br/>cleaning + schema]
+
+    sql --> ddl[ddl.py<br/>DataFrame -> DDL]
+    sql --> ddl_arrow[ddl_pyarrow.py<br/>Parquet schema -> DDL]
+    sql --> bulk[bulk_load.py]
+    sql --> quoting[quoting.py]
+
+    io --> json_io[json.py]
+    io --> zip_io[zip.py]
+
+    utils --> logging_utils[logging.py]
+    utils --> strings_utils[strings.py]
+    utils --> lists_utils[lists.py]
+:::
+
+## Data reading pipeline (memory-aware)
+
+`read_file_smart` (and its chunked/streaming helpers) chooses the loading strategy
+based on file type, estimated memory, and optional Polars availability.
+
+::: mermaid
+flowchart TD
+    start([read_file_smart]) --> exists{file exists?}
+    exists -- no --> err_missing[error]
+    exists -- yes --> est[estimate memory]
+    est --> fits{fits in memory?}
+
+    fits -- yes --> direct[direct pandas read<br/>csv/xlsx/json/parquet]
+    fits -- no --> polars{use_polars?}
+
+    polars -- yes --> polars_ok{polars available?}
+    polars_ok -- yes --> polars_read[polars scan + collect]
+    polars_ok -- no --> chunked
+
+    polars -- no --> chunked[chunked pandas read]
+    chunked --> concat[concat chunks]
+
+    direct --> done([DataFrame])
+    polars_read --> done
+    concat --> done
+:::
+
+## Parquet partitioning pipeline
+
+Both `read_file_to_parquets` and `stream_to_parquets` chunk through a file, clean
+columns, optionally coerce numeric values, and write Parquet partitions.
+
+::: mermaid
+flowchart LR
+    input[(input file)] --> reader[read_file_chunked/read_file_iter]
+    reader --> clean[clean_dataframe_columns]
+    clean --> cast{convert_types?}
+    cast -- yes --> numeric[try_cast_string_columns_to_numeric]
+    cast -- no --> write
+    numeric --> write[write parquet partition]
+    write --> more{more chunks?}
+    more -- yes --> reader
+    more -- no --> done([parquet files])
+:::
+
+## SQL DDL generation
+
+DDL generation uses two entry points depending on source schema.
+
+::: mermaid
+flowchart TB
+    df[df: pandas DataFrame] --> ddl_df[ddl.generate_create_table]
+    parquet[parquet path] --> ddl_parquet[ddl_pyarrow.generate_create_table_from_parquet]
+
+    ddl_df --> qualify[_qualify_name + quote_identifier]
+    ddl_parquet --> qualify
+    qualify --> sql_stmt([CREATE TABLE statement])
+:::
+
+## Legacy `core` namespace
+
+`datablade.core` mirrors the organized modules and re-exports them for
+backward compatibility. New code should import from the newer modules.

datablade/docs/OBJECT_REGISTRY.md

@@ -0,0 +1,194 @@
+# Object Registry (draft)
+
+This document defines a draft specification for an in-memory object registry that
+provides SQL-like dot notation for database objects while keeping names and layout
+externalized in configuration.
+
+The registry is metadata-first. It does not connect to databases, perform I/O, or
+decide how content is produced. It simply gives a stable, structured namespace for
+object references and their associated content.
+
+## Goals
+
+- Provide dot-notation access to objects (catalog.schema.object.content).
+- Externalize names and layout in a config file to reduce hard-coded strings.
+- Allow host to be optional so connections can supply it at runtime.
+- Make content developer-managed (lazy by default, no implied loading).
+- Keep the model extensible across SQL dialects and non-SQL backends.
+
+## Non-goals
+
+- No database connections or live introspection.
+- No migrations, transactional DDL, or ORM behavior.
+- No enforcement of how content is loaded or computed.
+
+## Core concepts
+
+- ObjectRef: immutable reference to an object, with metadata and content.
+- ObjectNode: namespace container supporting dot and key access.
+- ObjectRegistry: root registry, config loading, and object iteration.
+- DialectAdapter: formatter for qualified names based on dialect rules.
+
+## Config schema (v1)
+
+Top-level keys:
+
+- version (int, required)
+- defaults (object, optional)
+- dialects (object, optional)
+- catalogs (object, optional)
+- hosts (object, optional)
+- metadata (object, optional)
+
+defaults:
+
+- dialect (string, optional; default "sqlserver")
+- host (string|null, optional)
+- catalog (string|null, optional)
+- schema (string|null, optional; default "dbo")
+- object_type (string, optional; default "table")
+- name_policy (string, optional; "preserve" | "lower" | "upper" | "normalize")
+
+dialects:
+
+- <dialect_name>:
+  - qualifier (string, required): dot-separated segments
+    - allowed: host, catalog, schema, object
+  - quote_style (string, optional):
+    - "sqlserver" | "postgres" | "mysql" | "duckdb" | "none"
+
+hosts:
+
+- <host_key>:
+  - host (string, optional)
+  - catalogs (map, required)
+
+catalogs (hostless or under host):
+
+- <catalog_key>:
+  - catalog (string, optional)
+  - schemas (map, optional)
+  - objects (map, optional)  # allow catalog-level objects
+
+schemas:
+
+- <schema_key>:
+  - schema (string, optional)
+  - objects (map, required)
+
+objects:
+
+- <object_key>:
+  - name (string, optional; defaults to key)
+  - object_type (string, optional; defaults to defaults.object_type)
+  - aliases (list[string], optional)
+  - content (any, optional)
+  - tags (map[string,string], optional)
+
+## Name policy and lookup
+
+Keys are the stable identifiers used for dot-notation. The name policy determines
+how keys and aliases are normalized for lookup.
+
+- preserve: do not transform keys; lookups are case-sensitive.
+- lower / upper: normalize keys and aliases to the chosen case.
+- normalize: transform keys into safe identifiers:
+  - lower-case
+  - replace invalid characters with "_"
+  - collapse multiple "_"
+  - prefix "_" if the key starts with a digit
+  - keep the original key as an implicit alias
+
+Lookup behavior:
+
+- Dot access uses normalized keys.
+- Bracket access checks raw key, then aliases, then normalized keys.
+- Aliases are only for lookup; they do not change the stored name.
+
+Collision rules:
+
+- Two siblings that normalize to the same key are errors.
+- An alias that collides with a sibling key or alias is an error in strict mode.
+
+## Host optionality
+
+Host is optional. If host is missing, qualified names use only the available
+segments. If a runtime connection provides host/catalog/schema, it can override
+missing fields.
+
+## Content semantics
+
+Content is developer-managed. The registry stores it but does not interpret it.
+Content can be a DataFrame, SQL text, a lazy loader, or a computed result.
+
+## Dialect qualification
+
+Dialect qualification uses `dialects.<name>.qualifier` to decide which segments
+to include and in what order. Missing segments are skipped.
+
+Example qualifiers:
+
+- sqlserver: catalog.schema.object
+- postgres: schema.object
+- nosql: collection
+
+## Validation rules (draft)
+
+Errors:
+
+- version missing or not an integer
+- neither catalogs nor hosts provided
+- unknown fields in strict mode
+- duplicate keys after normalization within a parent
+- alias collisions with sibling keys or aliases
+- dialect qualifier uses unknown segments
+
+Warnings:
+
+- name differs only by case from key under preserve
+- host provided under hostless catalogs root
+- objects without content (informational)
+
+## Example config
+
+```yaml
+version: 1
+
+defaults:
+  dialect: sqlserver
+  schema: dbo
+  name_policy: normalize
+
+dialects:
+  sqlserver:
+    qualifier: catalog.schema.object
+  postgres:
+    qualifier: schema.object
+
+catalogs:
+  sales:
+    catalog: SalesDW
+    schemas:
+      reporting:
+        schema: rpt
+        objects:
+          orders:
+            name: Orders
+            object_type: table
+            aliases: [orders_current]
+            content: null
+```
+
+## Usage sketch
+
+```python
+registry = ObjectRegistry.from_yaml("layout.yaml")
+
+df = registry.catalogs.sales.reporting.orders.content
+df2 = registry.catalogs.sales.reporting.orders_current.content
+
+df3 = registry.catalogs["sales"].schemas["rpt"].objects["Order Details"].content
+
+qualified = registry.catalogs.sales.reporting.orders.qualified(dialect="sqlserver")
+```
+

datablade/docs/README.md

@@ -0,0 +1,57 @@
+# datablade Documentation
+
+This folder contains the human-facing documentation for **datablade**.
+
+## Quick start
+
+Install:
+
+```bash
+pip install datablade
+```
+
+Access docs after installation:
+
+```bash
+python -m datablade.docs --list
+python -m datablade.docs --show USAGE
+python -m datablade.docs --write-dir .\\datablade-docs
+```
+
+Basic usage:
+
+```python
+import pandas as pd
+from datablade.dataframes import read_file_smart, clean_dataframe_columns
+from datablade.sql import Dialect, generate_create_table, generate_create_table_from_parquet
+
+df = read_file_smart("data.csv", verbose=True)
+df = clean_dataframe_columns(df)
+
+ddl = generate_create_table(df, table="my_table", dialect=Dialect.POSTGRES)
+print(ddl)
+
+# Or: generate DDL from a Parquet file schema without materializing rows
+# Note: nested Parquet types (struct/list/map/union) are dropped with a warning.
+ddl2 = generate_create_table_from_parquet(
+	"events.parquet",
+	table="events",
+	dialect=Dialect.POSTGRES,
+)
+print(ddl2)
+
+# Optional schema_spec overrides (type/nullability/string sizing)
+schema_spec = {
+	"columns": {"notes": {"sql_type": "text", "nullable": True}},
+}
+ddl3 = generate_create_table(df, table="my_table", dialect=Dialect.POSTGRES, schema_spec=schema_spec)
+print(ddl3)
+```
+
+Most file path parameters accept `str` or `pathlib.Path`.
+
+## Guides
+
+- See [docs/USAGE.md](USAGE.md) for the main usage guide (file reading, streaming, SQL, IO, logging).
+- See [docs/ARCHITECTURE.md](ARCHITECTURE.md) for an architecture overview with pipeline diagrams.
+- See [docs/TESTING.md](TESTING.md) for running tests locally.

datablade/docs/TESTING.md

@@ -0,0 +1,37 @@
+# Testing
+
+## Install test dependencies
+
+```bash
+pip install -e ".[test]"
+```
+
+## Run tests
+
+```bash
+pytest
+```
+
+## Coverage
+
+If you have `pytest-cov` available:
+
+```bash
+pytest --cov=datablade --cov-report=term-missing
+```
+
+If `pytest-cov` is problematic in your environment, you can use `coverage` directly:
+
+```bash
+coverage run -m pytest
+coverage report -m
+```
+
+## Lint (optional, for contributors)
+
+```bash
+pip install -e ".[dev]"
+black .
+isort .
+flake8
+```