datablade 0.0.5__py3-none-any.whl → 0.0.6__py3-none-any.whl

{datablade-0.0.5.dist-info → datablade-0.0.6.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datablade
-Version: 0.0.5
+Version: 0.0.6
 Summary: datablade is a suite of functions to provide standard syntax across data engineering projects.
 Author-email: Brent Carpenetti <brentwc.git@pm.me>
 License: MIT License
@@ -63,7 +63,6 @@ Dynamic: license-file
 
 # datablade
 
-[![Tests](https://github.com/brentwc/data-prep/actions/workflows/test.yml/badge.svg)](https://github.com/brentwc/data-prep/actions/workflows/test.yml)
 [![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
@@ -115,20 +114,23 @@ datablade is ideal for:
 ## Installation
 
 ```bash
-pip install git+https://github.com/brentwc/data-prep.git
+pip install datablade
 ```
 
 **Optional dependencies:**
 
 ```bash
 # For high-performance file reading with Polars
-pip install git+https://github.com/brentwc/data-prep.git#egg=datablade[performance]
+pip install "datablade[performance]"
 
-# For development and testing
-pip install git+https://github.com/brentwc/data-prep.git#egg=datablade[dev]
+# For testing
+pip install "datablade[test]"
+
+# For development (includes testing + lint/format tooling)
+pip install "datablade[dev]"
 
 # All optional dependencies
-pip install git+https://github.com/brentwc/data-prep.git#egg=datablade[all]
+pip install "datablade[all]"
 ```
 
 ## Features
@@ -172,6 +174,7 @@ Multi-dialect SQL utilities:
 - Dialect-aware identifier quoting
 - CREATE TABLE generation for all dialects (from pandas DataFrames)
 - CREATE TABLE generation from Parquet schemas (schema-only, via PyArrow)
+- Optional `schema_spec` overrides for column types, nullability, and string sizing
 - Bulk loading helpers:
     - SQL Server: executes `bcp` via subprocess
     - PostgreSQL/MySQL/DuckDB: returns command strings you can run in your environment
@@ -215,10 +218,25 @@ ddl_from_parquet = generate_create_table_from_parquet(
 data = get_json('https://api.example.com/data.json')
 ```
 
+Most file path parameters accept `str` or `pathlib.Path`. To treat case mismatches
+as errors on case-insensitive filesystems, use `configure_paths(path_strict=True)`.
+
 ### Memory-Aware File Reading
 
+See the file format support matrix in the bundled USAGE doc:
+
+```bash
+python -m datablade.docs --show USAGE
+```
+
 ```python
-from datablade.dataframes import read_file_chunked, read_file_iter, read_file_to_parquets, stream_to_parquets
+from datablade.dataframes import (
+    excel_to_parquets,
+    read_file_chunked,
+    read_file_iter,
+    read_file_to_parquets,
+    stream_to_parquets,
+)
 
 # Read large files in chunks
 for chunk in read_file_chunked('huge_file.csv', memory_fraction=0.5):
@@ -232,6 +250,10 @@ for chunk in read_file_iter('huge_file.csv', memory_fraction=0.3, verbose=True):
 for chunk in read_file_iter('huge_file.parquet', memory_fraction=0.3, verbose=True):
     process(chunk)
 
+# Excel streaming is available with openpyxl installed (read-only mode)
+for chunk in read_file_iter('large.xlsx', chunksize=25_000, verbose=True):
+    process(chunk)
+
 # Partition large files to multiple Parquets
 files = read_file_to_parquets(
     'large_file.csv',
@@ -248,6 +270,15 @@ files = stream_to_parquets(
     convert_types=True,
     verbose=True,
 )
+
+# Excel streaming to Parquet partitions
+files = excel_to_parquets(
+    'large.xlsx',
+    output_dir='partitioned_excel/',
+    rows_per_file=200_000,
+    convert_types=True,
+    verbose=True,
+)
 ```
 
 ## Blade (Optional Facade)
@@ -284,10 +315,21 @@ ddl2 = blade.create_table_sql_from_parquet(
 
 ## Documentation
 
-- [Docs Home](docs/README.md) - Documentation landing page
-- [Usage Guide](docs/USAGE.md) - File reading (including streaming), SQL, IO, logging
-- [Testing Guide](docs/TESTING.md) - How to run tests locally
-- [Test Suite](tests/README.md) - Testing documentation and coverage
+Docs are bundled with the installed package:
+
+```bash
+python -m datablade.docs --list
+python -m datablade.docs --show USAGE
+python -m datablade.docs --write-dir .\datablade-docs
+```
+
+After writing docs to disk, open the markdown files locally:
+
+- README (docs landing page)
+- USAGE (file reading, streaming, SQL, IO, logging)
+- TESTING (how to run tests locally)
+- ARCHITECTURE (pipeline overview)
+- OBJECT_REGISTRY (registry reference)
 
 ## Testing
 
@@ -304,7 +346,11 @@ pytest
 pytest --cov=datablade --cov-report=html
 ```
 
-See [tests/README.md](tests/README.md) for detailed testing documentation.
+For detailed testing documentation, use the bundled TESTING doc:
+
+```bash
+python -m datablade.docs --show TESTING
+```
 
 ## Backward Compatibility
 
@@ -332,6 +378,12 @@ from datablade.core.json import get
 - **Streaming vs materializing**:
     - Use `read_file_iter()` to process arbitrarily large files chunk-by-chunk.
     - `read_file_smart()` returns a single DataFrame and may still be memory-intensive.
+- **Chunk concatenation**: the large-file pandas fallback in `read_file_smart()` can
+  temporarily spike memory usage during concat. Use `read_file_iter()` or
+  `return_type="iterator"` to avoid concatenation.
+- **Polars materialization**: when returning a pandas DataFrame, Polars still
+  collects into memory; use `return_type="polars"` or `"polars_lazy"` to keep
+  Polars frames.
 - **Parquet support**:
     - Streaming reads support single `.parquet` files.
     - Parquet “dataset directories” (Hive partitions / directory-of-parquets) are not a primary target API.
@@ -339,6 +391,9 @@ from datablade.core.json import get
     - Uses the Parquet schema (PyArrow) without scanning data.
     - Complex/nested columns (struct/list/map/union) are dropped and logged as warnings.
 - **DDL scope**: `CREATE TABLE` generation is column/type oriented (no indexes/constraints).
+- **SQL Server bulk load**: the SQL Server helpers use the `bcp` CLI and require it
+  to be installed and available on PATH. When using `-U`/`-P`, credentials are
+  passed via process args (logs are redacted); prefer `-T` or `-G` where possible.
 
 **Optional dependencies:**
 

datablade-0.0.6.dist-info/RECORD

@@ -0,0 +1,41 @@
+datablade/__init__.py,sha256=T1FBNEZfjdIFwyfRLNCQBroMXDfaYai1sQTaRzhA1Qg,1487
+datablade/blade.py,sha256=iKQtHhH-xs3EDH9ZNryheRGzE2OBkaxjZsA99iAG-p0,10330
+datablade/registry.py,sha256=zmfbt3-LthxV2lmwNAmt4KN-EnLRKAguMnYnk5xxzjg,20323
+datablade/core/__init__.py,sha256=VJ2lFumv-7CFKjIN_i4wPEaAGFEb-UhHPzsM2IyTUUM,871
+datablade/core/frames.py,sha256=iiTQPkWKQHslwkz_pu6bHVvbnisWSZF_b3Hgy1SvE-Q,662
+datablade/core/json.py,sha256=_0RoK_OXkOpwNG5Y7vdORQOCfs0ZVJAO0qeQqjxzpyE,121
+datablade/core/lists.py,sha256=OVj-KLyeOlbOxALyxvBLQpB0s78RO2QxmM4OxbI70Cc,132
+datablade/core/messages.py,sha256=_2CpeaDJIwmRlTm4UMrDTKnew1wZdyai6QeKFjN3rno,404
+datablade/core/strings.py,sha256=vwx0NW-m-2nOYW1aisRXUjrd4wERe2Y5Ka64wAyF0Mo,173
+datablade/core/zip.py,sha256=fmp6VTKH3JAaRWTKhWGHN-frO7Vr4EHlEYtUOlomNuc,119
+datablade/dataframes/__init__.py,sha256=blPM2EkNI6FA8unnsGpTDP4A12G23ChXy4djyhpgHMk,1316
+datablade/dataframes/frames.py,sha256=VaMgrY5F8uuWw6adx9g10sBlkW4TfZhH5AEo2TgD5N4,20993
+datablade/dataframes/readers.py,sha256=1RFW0J1iKGvmgrRjDMB8PoHcMhQhyfAlUh1h6_aF_2E,47469
+datablade/docs/ARCHITECTURE.md,sha256=ON65KUk3-EdwfnMglOT1CVZqyJNUXVbAUff803d7XDU,3290
+datablade/docs/OBJECT_REGISTRY.md,sha256=00wkTuo6Pnhl9kgRDu7mJFO3yxSBedQQS7r-ktnsqSs,5386
+datablade/docs/README.md,sha256=2uH6siz5ABEd6ZDjPYbXpH-zaOgyHzB3ik2e2eUj8Ks,1587
+datablade/docs/TESTING.md,sha256=zc1Qz3ZRGH9CZqWfzArDmv3wRfS8-346EiXhE_fO0co,463
+datablade/docs/USAGE.md,sha256=vJRmCF-dUeXAiWvvgUwc6fO3gpYNBZtwV-v2gmfoeeY,11866
+datablade/docs/__init__.py,sha256=t4KRQOU653CVdkFD00Aj0hdjrlcmx70D2W2QnybJDnc,2388
+datablade/docs/__main__.py,sha256=V-GK7vuR1aEvT_TgJPAZnapjLhwDXGOCSEseX4R6fts,112
+datablade/io/__init__.py,sha256=jDenqP4biI1gnRdoO9X5C489cBlHrLuR3xp_TIOljp8,295
+datablade/io/json.py,sha256=3Oao1lP8Dat3DT7MYyCes7feqPVFDHAPpFDemHuKVVw,2384
+datablade/io/zip.py,sha256=Ppl8CvGhWIYFcBelpSkinmT_oFDmQmqNiHuFHicSoQI,4268
+datablade/sql/__init__.py,sha256=Fs77XFT_hJcdxBZxLD9-6fgD2QEVtfp58LlZaEMFEJA,1466
+datablade/sql/bulk_load.py,sha256=7SniXGHMIvwg9P_r95Nt7Z72jkQnu-LuQtPvv_Wmq2I,19992
+datablade/sql/ddl.py,sha256=4xJ98LJmnD7PqloU8Ewa7J9iy2L6n9iPRsM5WtBKA0U,14958
+datablade/sql/ddl_pyarrow.py,sha256=yBfTbDAA7dE3BXrCSPqEFogOYohAb3p0v3AIgBRxNqc,15344
+datablade/sql/dialects.py,sha256=rM5qw6xdIuCSx81b6_TKYnsDhKeVpTdMDb_62HhtksA,266
+datablade/sql/quoting.py,sha256=Y8yn1GAsaWSIenQDidwU4j2YsZmYE9ppOifAPiTbStk,1345
+datablade/sql/schema_spec.py,sha256=HdEA5RS_yj8WNVH8wroWDyIK-8K4_fgMZC7rZBswkcs,2067
+datablade/sql/sqlserver.py,sha256=ASLhtHIslMePRi9pUY6vWTaebQ7TM4kO60N9QsXwhok,13726
+datablade/utils/__init__.py,sha256=KbvUZVWg2u53oUlfEIbDndwgz6f_j7N23Dl8Oz-vWXk,746
+datablade/utils/lists.py,sha256=h3AXo2SSE2iaR_rJfoCgfNRFV9AOWerL-hHFYo6S4n8,798
+datablade/utils/logging.py,sha256=kqM1389Wgzml69aja6g7yj8biNSi9AOPZlxr3YTnyp0,5491
+datablade/utils/messages.py,sha256=yZZTGTS_eD0PBZFCdzz51fqBFWgq9Sdq-GMR1a0FclY,537
+datablade/utils/strings.py,sha256=GLvYi7FPUHZ2nqNsZ0-mGbc3gu70UvVzrIy4bQWRHOo,7196
+datablade-0.0.6.dist-info/licenses/LICENSE,sha256=QyU-OkETSZ-L0Rltu-SAMQtQU_UXjwLnb_QlA_AXm0U,1072
+datablade-0.0.6.dist-info/METADATA,sha256=ou_j_HZwHrdNaZb26JhW_m3oRhILZ4tMFIPla9Sm8LM,13261
+datablade-0.0.6.dist-info/WHEEL,sha256=wUyA8OaulRlbfwMtmQsvNngGrxQHAvkKcvRmdizlJi0,92
+datablade-0.0.6.dist-info/top_level.txt,sha256=AwA5QxmfuaAs9XeXw1tCsboPsoffnMU-6CeLWMMUoUA,10
+datablade-0.0.6.dist-info/RECORD,,

{datablade-0.0.5.dist-info → datablade-0.0.6.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: setuptools (80.9.0)
+Generator: setuptools (80.10.2)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

datablade-0.0.5.dist-info/RECORD

@@ -1,31 +0,0 @@
-datablade/__init__.py,sha256=HGsVFJ4wSvSBXzKywpRpJweze_upiJbw-7yZ0JIlslY,1246
-datablade/blade.py,sha256=pT6NYKhrKal_0c2m1QKur2jZRWkJC1fE7sTO4ldiWcU,4442
-datablade/core/__init__.py,sha256=VJ2lFumv-7CFKjIN_i4wPEaAGFEb-UhHPzsM2IyTUUM,871
-datablade/core/frames.py,sha256=iiTQPkWKQHslwkz_pu6bHVvbnisWSZF_b3Hgy1SvE-Q,662
-datablade/core/json.py,sha256=_0RoK_OXkOpwNG5Y7vdORQOCfs0ZVJAO0qeQqjxzpyE,121
-datablade/core/lists.py,sha256=OVj-KLyeOlbOxALyxvBLQpB0s78RO2QxmM4OxbI70Cc,132
-datablade/core/messages.py,sha256=_2CpeaDJIwmRlTm4UMrDTKnew1wZdyai6QeKFjN3rno,404
-datablade/core/strings.py,sha256=vwx0NW-m-2nOYW1aisRXUjrd4wERe2Y5Ka64wAyF0Mo,173
-datablade/core/zip.py,sha256=fmp6VTKH3JAaRWTKhWGHN-frO7Vr4EHlEYtUOlomNuc,119
-datablade/dataframes/__init__.py,sha256=5nu_xgxIrL5GlX84mIkh66rCg5bUKor7bgP3EQwihBY,1122
-datablade/dataframes/frames.py,sha256=prbsOrq3rwLm-xPq2VrWVYyrdQLbjsuWsh_OQWq6Stw,16742
-datablade/dataframes/readers.py,sha256=_Ititzvk4hLRmITxDQIS-Ji6t9rc-wTBIEcjuPSNQ8U,19378
-datablade/io/__init__.py,sha256=jDenqP4biI1gnRdoO9X5C489cBlHrLuR3xp_TIOljp8,295
-datablade/io/json.py,sha256=3VSyNiDkLYA5YihlOdUyf2Z83iDxB5ztjc4vDtwvu9Q,1055
-datablade/io/zip.py,sha256=GaWCRNidUyeGJ4OBkImxIjtOV10mJaly9yLzyyBIE20,2716
-datablade/sql/__init__.py,sha256=5Cef9DA3ifgPD6WNcGQ_ncDloWLxESc7Cxd8PHybfnU,762
-datablade/sql/bulk_load.py,sha256=ugJMtzGc7Cb1FcuR7UhBaTPD76dqWj3oYOVwQkugeFw,12068
-datablade/sql/ddl.py,sha256=WmvjVLdMkPoMDZkM0CbsV5TJAaMihBWZRyDkURj5wAY,8373
-datablade/sql/ddl_pyarrow.py,sha256=QNdw4YN4CwUCvU2tsKHkCBCfFPFlYsUV5fUUC_MXpbs,10869
-datablade/sql/dialects.py,sha256=eishu7E5CaZXc6Jae7hn87qDbgyBuC_cSey-IR6SGKQ,207
-datablade/sql/quoting.py,sha256=l2pCcr55TyuPkpiUp3IFN8fmkp16fQ7ynvonTjd_OKQ,1291
-datablade/utils/__init__.py,sha256=HzIUWIBZpqNKvYVeVZfhqwBCNOE9b3-IdUoCwslCk1Y,706
-datablade/utils/lists.py,sha256=U2cX9bj8NfmL39sAdPPWis2xQ6CpC9mtc2I0LExQBOM,691
-datablade/utils/logging.py,sha256=SSzoRgBZiiKWrN_dsvHpqhqkgWiDqP0BGxfgr94QyyE,4227
-datablade/utils/messages.py,sha256=yZZTGTS_eD0PBZFCdzz51fqBFWgq9Sdq-GMR1a0FclY,537
-datablade/utils/strings.py,sha256=173z_TM6d5zojD3d3lGiM8tyU0z6IMmIqIn3E9zWdVw,2633
-datablade-0.0.5.dist-info/licenses/LICENSE,sha256=QyU-OkETSZ-L0Rltu-SAMQtQU_UXjwLnb_QlA_AXm0U,1072
-datablade-0.0.5.dist-info/METADATA,sha256=4u9PckSw3RVizI3uB2rGfKSR_2Vo2WmB4lk1CWuIP0k,11911
-datablade-0.0.5.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-datablade-0.0.5.dist-info/top_level.txt,sha256=AwA5QxmfuaAs9XeXw1tCsboPsoffnMU-6CeLWMMUoUA,10
-datablade-0.0.5.dist-info/RECORD,,