datablade 0.0.5__py3-none-any.whl → 0.0.6__py3-none-any.whl

datablade/docs/USAGE.md

@@ -0,0 +1,409 @@
+# datablade Usage Guide
+
+## Installation
+
+```bash
+pip install datablade
+```
+
+Optional extras:
+
+```bash
+pip install "datablade[performance]"
+```
+
+## File reading
+
+### File format support matrix
+
+| File type | Streaming (iterator / Parquet streaming) | Materialized (read_file_smart) | Engine / SQL notes |
+| --- | --- | --- | --- |
+| `.csv`, `.tsv`, `.txt` | ✅ | ✅ | pandas by default; Polars `return_type="polars"` or `"polars_lazy"` (scan for large inputs) |
+| `.parquet` | ✅ | ✅ | pandas by default; Polars `return_type="polars"` or `"polars_lazy"`; SQL DDL from schema via `generate_create_table_from_parquet` |
+| JSON Lines (`.json` with `lines=True` or `.jsonl`) | ✅ | ✅ | pandas by default; Polars via `scan_ndjson` when `lines=True` |
+| JSON (non-lines) | ❌ | ✅ | pandas only (materialized) |
+| `.xlsx`, `.xls` | ✅ (read-only, `openpyxl`) | ✅ | pandas only |
+
+Notes:
+
+- **JSON Lines requirement:** streaming JSON requires JSON Lines (`lines=True`). Non-lines JSON only materializes.
+- **Excel limitations:** streaming Excel relies on `openpyxl` read-only mode; without it, Excel files fall back to full reads and must fit in `memory_fraction`.
+- **Polars return options:** use `return_type="polars"` for a Polars `DataFrame`, and `return_type="polars_lazy"` for a `LazyFrame` when scanning is supported.
+
+### Read into a DataFrame (convenience)
+
+Use this when you want a single in-memory DataFrame.
+
+```python
+from datablade.dataframes import read_file_smart
+
+df = read_file_smart("data.csv", verbose=True)
+```
+
+`read_file_smart()` may read in chunks internally, but it returns a single DataFrame by default.
+If you want a non-materializing return type, set `return_type`:
+
+```python
+from datablade.dataframes import read_file_smart
+
+# Iterator of pandas DataFrames (no concat)
+chunks = read_file_smart("huge.csv", return_type="iterator", memory_fraction=0.3)
+for chunk in chunks:
+    process(chunk)
+
+# Polars for large files (LazyFrame for large inputs, DataFrame for small ones)
+polars_obj = read_file_smart("huge.csv", return_type="polars")
+```
+
+#### Polars return types
+
+Use `return_type="polars"` (or `return_polars=True`) to get a Polars `DataFrame`. Use
+`return_type="polars_lazy"` for a lazy `LazyFrame` when the format supports Polars
+scanning.
+
+```python
+from datablade.dataframes import read_file_smart
+
+df_polars = read_file_smart("large.csv", return_type="polars", verbose=True)
+lf_polars = read_file_smart("large.csv", return_type="polars_lazy", verbose=True)
+```
+
+Notes:
+
+- For CSV/TSV/TXT and Parquet files, Polars uses `scan_*` for large inputs.
+- When a format is not supported by Polars scanning, datablade reads with pandas and
+  converts to Polars.
+- For very large files, the pandas chunk-concat fallback can temporarily spike memory
+  usage. Use `read_file_iter()` or `return_type="iterator"` to avoid concatenation.
+
+### Stream without materializing (recommended for very large files)
+
+Use this when you want to process arbitrarily large files without ever concatenating the full dataset.
+
+```python
+from datablade.dataframes import read_file_iter
+
+for chunk in read_file_iter("huge.csv", memory_fraction=0.3, verbose=True):
+    process(chunk)
+```
+
+If you already use `read_file_smart()`, you can request the same behavior:
+
+```python
+from datablade.dataframes import read_file_smart
+
+for chunk in read_file_smart("huge.csv", return_type="iterator"):
+    process(chunk)
+```
+
+Supported streaming formats:
+
+- `.csv`, `.tsv`, `.txt`
+- `.parquet`
+- `.json` (JSON Lines with `lines=True`, or standard JSON arrays with `ijson`)
+- `.xlsx`, `.xls` when `openpyxl` is installed (read-only streaming)
+
+JSON streaming notes:
+
+- Standard JSON streaming requires the optional `ijson` dependency.
+- If your JSON is nested, pass `record_path="item"` (or another path) to
+  `read_file_iter()` or `json_to_jsonl()`.
+- Use `json_to_jsonl()` to convert a standard JSON file into JSON Lines when
+  `lines=True` streaming is needed.
+
+Notes for Excel streaming:
+
+- Requires `openpyxl` and reads in read-only mode.
+- Use `chunksize` (or `rows_per_file` for `excel_to_parquets`) to control chunking.
+- If `openpyxl` is unavailable, Excel files fall back to full reads and must fit in
+  `memory_fraction` of available memory.
+
+### Stream to Parquet partitions
+
+If your goal is “partition to Parquet without ever materializing”, use `stream_to_parquets()`.
+
+```python
+from datablade.dataframes import excel_to_parquets, stream_to_parquets
+
+files = stream_to_parquets(
+    "huge.csv",
+    output_dir="partitioned/",
+    rows_per_file=200_000,
+    convert_types=True,
+    verbose=True,
+)
+print(len(files))
+```
+
+Excel to Parquet streaming:
+
+```python
+files = excel_to_parquets(
+    "large.xlsx",
+    output_dir="partitioned_excel/",
+    rows_per_file=100_000,
+    convert_types=True,
+    verbose=True,
+)
+```
+
+If you prefer the older helper that may choose chunk sizes automatically, you can also use `read_file_to_parquets()`.
+
+If you already have an iterator (for example from `read_file_iter()`), you can send it to a sink incrementally:
+
+```python
+from datablade.dataframes import read_file_iter, stream_to_sink
+
+chunks = read_file_iter("huge.csv", memory_fraction=0.3)
+files = stream_to_sink(chunks, output_dir="partitioned/")
+```
+
+## DataFrame helpers
+
+```python
+from datablade.dataframes import clean_dataframe_columns, try_cast_string_columns_to_numeric
+
+# clean_dataframe_columns:
+# - flattens MultiIndex columns
+# - coerces names to strings
+# - drops duplicate columns (keeps first)
+
+# try_cast_string_columns_to_numeric:
+# - converts object columns containing numeric strings
+```
+
+## SQL helpers
+
+```python
+import pandas as pd
+from datablade.sql import Dialect, generate_create_table, generate_create_table_from_parquet
+
+df = pd.DataFrame({"id": [1, 2], "name": ["a", "b"]})
+print(generate_create_table(df, table="t", dialect=Dialect.POSTGRES))
+
+# SQL Server batch separator (optional, only when catalog is supplied)
+print(
+    generate_create_table(
+        df,
+        catalog="db",
+        table="t",
+        dialect=Dialect.SQLSERVER,
+        use_go=True,
+    )
+)
+
+# Schema-only Parquet -> SQL (does not read/scan the data)
+ddl = generate_create_table_from_parquet(
+    "events.parquet",
+    table="events",
+    dialect=Dialect.POSTGRES,
+)
+print(ddl)
+
+# Opt-in fallback for complex types (struct/list/map/union) to JSON text columns.
+ddl, metadata = generate_create_table_from_parquet(
+    "events.parquet",
+    table="events",
+    dialect=Dialect.POSTGRES,
+    fallback_to_json=True,
+    return_metadata=True,
+)
+print(metadata.fallback_columns)
+
+# Column-level overrides (externalized schema controls)
+schema_spec = {
+    "defaults": {
+        "string": {
+            "prefer_length": "estimate",
+            "defined_pad": 2,
+            "min_length": 1,
+            "max_length": 200,
+            "empty_as_null": True,
+        }
+    },
+    "columns": {
+        "notes": {"sql_type": "nvarchar(max)"},
+        "code": {"string": {"max_length": 10}},
+        "legacy_id": {"sql_type": "varchar(32)", "nullable": True},
+    },
+}
+
+ddl = generate_create_table(
+    df,
+    table="t",
+    dialect=Dialect.SQLSERVER,
+    schema_spec=schema_spec,
+)
+print(ddl)
+
+# For Parquet DDL, schema_spec can override sql_type/nullable per column.
+ddl = generate_create_table_from_parquet(
+    "events.parquet",
+    table="events",
+    dialect=Dialect.SQLSERVER,
+    schema_spec={"columns": {"legacy_id": {"sql_type": "varchar(32)", "nullable": True}}},
+)
+print(ddl)
+```
+
+Notes:
+
+- Parquet DDL generation reads only the file schema via PyArrow (no DataFrame materialization).
+- By default, columns with no clean mapping (for example structs/lists/maps/unions) are dropped and a
+  warning is logged under logger name `datablade`.
+- Set `fallback_to_json=True` to keep complex types by mapping them to text columns
+  (NVARCHAR/TEXT/VARCHAR, depending on dialect) intended for JSON-encoded values.
+- Use `return_metadata=True` to receive a metadata object listing dropped columns and any
+  fallback-mapped columns.
+- `schema_spec` lets you override column types or string sizing. When omitted, the
+  existing inference behavior is unchanged.
+
+### SQL Server helpers
+
+Open a partitioned Parquet directory via OPENROWSET:
+
+```python
+from datablade.sql import sqlserver_openrowset_parquet
+
+sql = sqlserver_openrowset_parquet(
+    "C:/data/parts/*.parquet",
+    data_source="MyExternalDataSource",
+    table_alias="p",
+)
+print(sql)
+```
+
+Create a table from Parquet schema and generate BULK INSERT statements for
+memory-aware Parquet-to-CSV staging:
+
+```python
+from datablade.sql import sqlserver_create_and_insert_from_parquet
+
+sql, csv_files = sqlserver_create_and_insert_from_parquet(
+    "events.parquet",
+    output_dir="staging_csv",
+    table="events",
+    rows_per_file=200_000,
+)
+print(sql)
+```
+
+Stage multiple Parquet files and load via BCP:
+
+```python
+from datablade.sql import (
+    bulk_load_sqlserver_commands,
+    bulk_load_sqlserver_many,
+    sqlserver_create_and_stage_from_parquets,
+)
+
+ddl, csv_files = sqlserver_create_and_stage_from_parquets(
+    ["events_part1.parquet", "events_part2.parquet"],
+    output_dir="staging_csv",
+    table="events",
+    schema_strict=True,  # raise on schema drift across Parquet files
+)
+print(ddl)
+
+# Execute the DDL in SQL Server, then bulk load the staged CSVs via BCP.
+bulk_load_sqlserver_many(
+    csv_files,
+    table_name="events",
+    database="db",
+    server="localhost",
+    username="user",
+    password="pass",
+)
+
+# Or build the BCP command strings instead of executing:
+commands = bulk_load_sqlserver_commands(
+    csv_files,
+    table_name="events",
+    database="db",
+    server="localhost",
+    username="user",
+    password="pass",
+    redact_password=True,
+)
+print(commands)
+```
+
+Notes:
+
+- SQL Server bulk loading uses the `bcp` CLI. Ensure the SQL Server command line
+  utilities are installed and `bcp` is on your PATH.
+- When using `-U`/`-P`, the password is passed to the `bcp` process arguments
+  (even though logs are redacted). Prefer trusted connection (`-T`) or Azure AD
+  (`-G`) where possible.
+
+## IO helpers
+
+```python
+from datablade.io import get_json, get_zip
+
+data = get_json("https://api.example.com/data.json")
+get_zip("https://example.com/data.zip", path="./data")
+```
+
+## Logging
+
+datablade uses the standard Python logging system under the logger name `datablade`.
+
+```python
+import logging
+from datablade.utils import configure_logging
+
+configure_logging(level=logging.INFO)
+```
+
+Write logs to a file:
+
+```python
+import logging
+from datablade.utils import configure_logging
+
+configure_logging(level=logging.INFO, log_file="pipeline.log")
+```
+
+For rotation, pass a custom handler (e.g., `logging.handlers.RotatingFileHandler`).
+
+## Path handling
+
+Most APIs accept `str` or `pathlib.Path` for file paths.
+
+On case-insensitive filesystems, datablade can warn when a path's casing does not
+match the on-disk entry. To treat case mismatches as errors, enable strict mode:
+
+```python
+from datablade.utils import configure_paths
+
+configure_paths(path_strict=True)
+```
+
+## Backward compatibility
+
+Legacy imports under `datablade.core.*` remain available.
+New code should prefer `datablade.dataframes`, `datablade.sql`, `datablade.io`, and `datablade.utils`.
+
+## Optional facade (class-style)
+
+The primary API is module-level functions, but datablade also provides an optional convenience facade for users who prefer an object-style entrypoint with shared defaults.
+
+```python
+from datablade import Blade
+from datablade.sql import Dialect
+
+blade = Blade(memory_fraction=0.3, verbose=True, convert_types=True)
+
+for chunk in blade.iter("huge.csv"):
+    ...
+
+files = blade.stream_to_parquets("huge.csv", output_dir="partitioned/")
+
+# Generate DDL (CREATE TABLE) for a dialect
+ddl = blade.create_table_sql(
+    df,
+    table="my_table",
+    dialect=Dialect.POSTGRES,
+)
+```

datablade/docs/__init__.py

@@ -0,0 +1,87 @@
+from __future__ import annotations
+
+import argparse
+from importlib import resources
+from pathlib import Path
+
+
+def list_docs() -> list[str]:
+    files = []
+    for entry in resources.files(__package__).iterdir():
+        if entry.suffix == ".md":
+            files.append(entry.stem)
+    return sorted(files)
+
+
+def read_doc(name: str) -> str:
+    if not name:
+        raise ValueError("Document name is required.")
+    filename = name if name.lower().endswith(".md") else f"{name}.md"
+    doc_path = resources.files(__package__) / filename
+    if not doc_path.is_file():
+        available = ", ".join(list_docs()) or "none"
+        raise FileNotFoundError(f"Unknown document '{name}'. Available: {available}.")
+    return doc_path.read_text(encoding="utf-8")
+
+
+def write_docs(target_dir: str | Path) -> list[Path]:
+    target = Path(target_dir).expanduser().resolve()
+    target.mkdir(parents=True, exist_ok=True)
+    written: list[Path] = []
+    for entry in resources.files(__package__).iterdir():
+        if entry.suffix != ".md":
+            continue
+        content = entry.read_text(encoding="utf-8")
+        destination = target / entry.name
+        destination.write_text(content, encoding="utf-8")
+        written.append(destination)
+    return written
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Read datablade documentation bundled with the package."
+    )
+    parser.add_argument(
+        "--list",
+        action="store_true",
+        help="List available documentation files.",
+    )
+    parser.add_argument(
+        "--show",
+        metavar="NAME",
+        help="Print a documentation file to stdout (e.g. USAGE, ARCHITECTURE).",
+    )
+    parser.add_argument(
+        "--write-dir",
+        metavar="PATH",
+        help="Write all documentation files to a directory.",
+    )
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = _build_parser()
+    args = parser.parse_args(argv)
+
+    if args.list:
+        for name in list_docs():
+            print(name)
+        return 0
+
+    if args.show:
+        print(read_doc(args.show))
+        return 0
+
+    if args.write_dir:
+        written = write_docs(args.write_dir)
+        for path in written:
+            print(path)
+        return 0
+
+    parser.print_help()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

datablade/docs/__main__.py

@@ -0,0 +1,6 @@
+from __future__ import annotations
+
+from . import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

datablade/io/json.py

@@ -1,3 +1,6 @@
+"""HTTP helpers for retrieving JSON payloads."""
+
+import time
 from typing import Any, Dict
 
 import requests
@@ -5,13 +8,23 @@ import requests
 from ..utils.messages import print_verbose
 
 
-def get(url: str, verbose: bool = False, **kwargs: Any) -> Dict[str, Any]:
+def get(
+    url: str,
+    verbose: bool = False,
+    retries: int = 3,
+    backoff_factor: float = 0.5,
+    timeout: float | None = 10.0,
+    **kwargs: Any,
+) -> Dict[str, Any]:
     """
     Get JSON data from a URL using HTTP GET request.
 
     Args:
         url: The URL to fetch JSON data from (must be non-empty string).
         verbose: If True, prints error messages.
+        retries: Number of retry attempts after the initial request.
+        backoff_factor: Backoff factor for exponential retry delay.
+        timeout: Optional timeout (in seconds) passed to requests.get().
         **kwargs: Additional keyword arguments passed to requests.get().
 
     Returns:
@@ -24,10 +37,34 @@ def get(url: str, verbose: bool = False, **kwargs: Any) -> Dict[str, Any]:
     if not isinstance(url, str) or not url.strip():
         raise ValueError("url must be a non-empty string")
 
-    try:
-        response = requests.get(url, **kwargs)
-        response.raise_for_status()  # Raise exception for bad status codes
-        return response.json()
-    except requests.exceptions.RequestException as e:
-        print_verbose(f"Error fetching JSON from {url}: {e}", verbose=verbose)
-        raise
+    if retries < 0:
+        raise ValueError("retries must be >= 0")
+    if backoff_factor < 0:
+        raise ValueError("backoff_factor must be >= 0")
+
+    if "timeout" not in kwargs and timeout is not None:
+        kwargs["timeout"] = timeout
+
+    attempts = retries + 1
+    for attempt in range(1, attempts + 1):
+        try:
+            # Delegates HTTP details to requests; callers can pass headers/etc.
+            response = requests.get(url, **kwargs)
+            response.raise_for_status()  # Raise exception for bad status codes
+            return response.json()
+        except requests.exceptions.RequestException as e:
+            message = (
+                f"Error fetching JSON from {url} "
+                f"(attempt {attempt}/{attempts}): {e}"
+            )
+            print_verbose(message, verbose=verbose)
+            if attempt >= attempts:
+                raise requests.RequestException(
+                    f"Failed to fetch JSON from {url} after {attempts} attempts."
+                ) from e
+            sleep_time = backoff_factor * (2 ** (attempt - 1))
+            if sleep_time:
+                time.sleep(sleep_time)
+    raise requests.RequestException(
+        f"Failed to fetch JSON from {url} after {attempts} attempts."
+    )

datablade/io/zip.py

@@ -1,17 +1,24 @@
+"""HTTP helper for downloading and extracting ZIP archives."""
+
 import io
 import pathlib
+import time
 import zipfile
 from typing import Any, Optional
 
 import requests
 
 from ..utils.messages import print_verbose
+from ..utils.strings import ensure_directory
 
 
 def get(
     url: str,
     path: Optional[str | pathlib.Path] = None,
     verbose: bool = False,
+    retries: int = 3,
+    backoff_factor: float = 0.5,
+    timeout: float | None = 30.0,
     **kwargs: Any,
 ) -> Optional[io.BytesIO]:
     """
@@ -23,6 +30,9 @@ def get(
             If None, returns the ZIP data as a BytesIO object.
             If provided, extracts all files to the specified path.
         verbose: If True, prints progress messages.
+        retries: Number of retry attempts after the initial request.
+        backoff_factor: Backoff factor for exponential retry delay.
+        timeout: Optional timeout (in seconds) passed to requests.get().
         **kwargs: Additional keyword arguments passed to requests.get().
 
     Returns:
@@ -39,35 +49,63 @@ def get(
     if not isinstance(url, str) or not url.strip():
         raise ValueError("url must be a non-empty string")
 
-    try:
-        print_verbose(f"Downloading {url}", verbose=verbose)
-        response = requests.get(url, **kwargs)
-        response.raise_for_status()  # Raise exception for bad status codes
-        data = response.content
-        zip_buffer = io.BytesIO(data)
+    if retries < 0:
+        raise ValueError("retries must be >= 0")
+    if backoff_factor < 0:
+        raise ValueError("backoff_factor must be >= 0")
+
+    if "timeout" not in kwargs and timeout is not None:
+        kwargs["timeout"] = timeout
+
+    attempts = retries + 1
+    for attempt in range(1, attempts + 1):
+        try:
+            print_verbose(f"Downloading {url}", verbose=verbose)
+            response = requests.get(url, **kwargs)
+            response.raise_for_status()  # Raise exception for bad status codes
+            # Keep data in-memory to allow optional extraction or return BytesIO.
+            data = response.content
+            zip_buffer = io.BytesIO(data)
+
+            if path is None:
+                return zip_buffer
 
-        if path is None:
-            return zip_buffer
+            print_verbose(f"Saving data to {path}", verbose=verbose)
+            zip_buffer.seek(0)
+            with zipfile.ZipFile(zip_buffer, "r") as zip_ref:
+                # Unlike utils.strings.pathing(), extracting should work even if the
+                # destination directory doesn't exist yet.
+                extract_root = ensure_directory(path, verbose=verbose, label="path")
+                for zip_info in zip_ref.infolist():
+                    extract_path = extract_root / zip_info.filename
+                    ensure_directory(
+                        extract_path.parent,
+                        verbose=verbose,
+                        label="path",
+                    )
+                    with open(extract_path, "wb") as f:
+                        f.write(zip_ref.read(zip_info.filename))
+            return None
+        except requests.exceptions.RequestException as e:
+            message = (
+                f"Error downloading ZIP from {url} "
+                f"(attempt {attempt}/{attempts}): {e}"
+            )
+            print_verbose(message, verbose=verbose)
+            if attempt >= attempts:
+                raise requests.RequestException(
+                    f"Failed to download ZIP from {url} after {attempts} attempts."
+                ) from e
+            sleep_time = backoff_factor * (2 ** (attempt - 1))
+            if sleep_time:
+                time.sleep(sleep_time)
+        except zipfile.BadZipFile as e:
+            print_verbose(f"Error: Invalid ZIP file from {url}: {e}", verbose=verbose)
+            raise
+        except Exception as e:
+            print_verbose(f"Error processing ZIP file: {e}", verbose=verbose)
+            raise
 
-        print_verbose(f"Saving data to {path}", verbose=verbose)
-        zip_buffer.seek(0)
-        with zipfile.ZipFile(zip_buffer, "r") as zip_ref:
-            # Unlike utils.strings.pathing(), extracting should work even if the
-            # destination directory doesn't exist yet.
-            extract_root = pathlib.Path(path)
-            extract_root.mkdir(parents=True, exist_ok=True)
-            for zip_info in zip_ref.infolist():
-                extract_path = extract_root / zip_info.filename
-                extract_path.parent.mkdir(parents=True, exist_ok=True)
-                with open(extract_path, "wb") as f:
-                    f.write(zip_ref.read(zip_info.filename))
-        return None
-    except requests.exceptions.RequestException as e:
-        print_verbose(f"Error downloading ZIP from {url}: {e}", verbose=verbose)
-        raise
-    except zipfile.BadZipFile as e:
-        print_verbose(f"Error: Invalid ZIP file from {url}: {e}", verbose=verbose)
-        raise
-    except Exception as e:
-        print_verbose(f"Error processing ZIP file: {e}", verbose=verbose)
-        raise
+    raise requests.RequestException(
+        f"Failed to download ZIP from {url} after {attempts} attempts."
+    )