datagrunt 3.2.0__tar.gz → 3.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {datagrunt-3.2.0/src/datagrunt.egg-info → datagrunt-3.2.2}/PKG-INFO +5 -5
- {datagrunt-3.2.0 → datagrunt-3.2.2}/README.md +2 -2
- {datagrunt-3.2.0 → datagrunt-3.2.2}/pyproject.toml +4 -4
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/__init__.py +1 -1
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/databases/databases.py +13 -5
- {datagrunt-3.2.0 → datagrunt-3.2.2/src/datagrunt.egg-info}/PKG-INFO +5 -5
- {datagrunt-3.2.0 → datagrunt-3.2.2}/LICENSE +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/setup.cfg +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/__init__.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/ai/__init__.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/ai/engines.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/ai/factories.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/ai/prompts.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/csv_io/__init__.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/csv_io/csvcomponents.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/csv_io/engines.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/csv_io/factories.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/databases/__init__.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/file_io/__init__.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/file_io/fileproperties.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/__init__.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/engines.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/__init__.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/base.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/image_dedupe.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/layout_sorter.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/markdown_escape.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/ocr.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/pdfium_backend.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/pdfium_document.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/pdfium_native.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/pymupdf_backend.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/shapes.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/tables.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/text_block_builder.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/factories.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/pdfcomponents.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/csv_api/__init__.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/csv_api/_compat.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/csv_api/csvai.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/csv_api/csvreader.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/csv_api/csvwriter.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/pdf_api/__init__.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/pdf_api/pdfreader.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/pdf_api/pdfwriter.py +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt.egg-info/SOURCES.txt +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt.egg-info/dependency_links.txt +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt.egg-info/requires.txt +0 -0
- {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt.egg-info/top_level.txt +0 -0
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: datagrunt
|
|
3
|
-
Version: 3.2.
|
|
3
|
+
Version: 3.2.2
|
|
4
4
|
Summary: Read CSV files and convert to other file formats easily
|
|
5
5
|
Author-email: Martin Graham <datagrunt@datagrunt.io>
|
|
6
6
|
License: MIT License
|
|
7
|
-
Project-URL: Homepage, https://
|
|
7
|
+
Project-URL: Homepage, https://www.datagrunt.io/
|
|
8
8
|
Project-URL: Bug Tracker, https://github.com/pmgraham/datagrunt/issues
|
|
9
|
-
Project-URL: Documentation, https://
|
|
9
|
+
Project-URL: Documentation, https://www.datagrunt.io/
|
|
10
10
|
Project-URL: Source Code, https://github.com/pmgraham/datagrunt
|
|
11
11
|
Keywords: csv,data,duckdb,polars,pyarrow,xlsx,delimiter,ai,gemini,pdf
|
|
12
12
|
Classifier: License :: OSI Approved :: MIT License
|
|
@@ -55,7 +55,7 @@ Datagrunt is not an extension of or a replacement for DuckDB, Polars, or PyArrow
|
|
|
55
55
|
- **Path Object Support:** Full support for both string paths and `pathlib.Path` objects for modern, cross-platform file handling.
|
|
56
56
|
- **Multiple Processing Engines:** Choose from three powerful engines - [DuckDB](https://duckdb.org), [Polars](https://pola.rs), and [PyArrow](https://arrow.apache.org/docs/python/) - to handle your data processing needs.
|
|
57
57
|
- **Flexible Data Transformation:** Easily convert your processed CSV data into various formats including CSV, Excel, JSON, JSONL, and Parquet.
|
|
58
|
-
- **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on
|
|
58
|
+
- **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on all engines.
|
|
59
59
|
- **PDF Parsing & OCR:** Extract text, tables, and images from PDF files as dicts, DataFrames, or JSON, with optional [Tesseract](https://github.com/tesseract-ocr/tesseract) OCR for scanned pages. Powered by the permissively-licensed **PDFium** engine by default, with **PyMuPDF** available as an alternative.
|
|
60
60
|
- **AI-Powered Schema Analysis:** Use Google's Gemini models to automatically generate detailed schema reports for your CSV files, including data types, column classifications, and data quality checks.
|
|
61
61
|
- **Pythonic API:** Enjoy a clean and intuitive API that integrates seamlessly into your existing Python workflows.
|
|
@@ -319,7 +319,7 @@ tables with at least two rows and two columns. It is off by default.
|
|
|
319
319
|
| **Default for** | CSVReader | CSVWriter | - |
|
|
320
320
|
| **Export Quality** | Good | Excellent (especially JSON) | Native Parquet support |
|
|
321
321
|
|
|
322
|
-
_The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three.
|
|
322
|
+
_The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three. Mid-file lines starting with `#` are kept as data on all three engines. PDF parsing uses the **PDFium** engine by default (permissively licensed), with **PyMuPDF** available via `engine="pymupdf"` — see [PDF parsing](#pdf-parsing)._
|
|
323
323
|
|
|
324
324
|
## Primary Classes
|
|
325
325
|
|
|
@@ -15,7 +15,7 @@ Datagrunt is not an extension of or a replacement for DuckDB, Polars, or PyArrow
|
|
|
15
15
|
- **Path Object Support:** Full support for both string paths and `pathlib.Path` objects for modern, cross-platform file handling.
|
|
16
16
|
- **Multiple Processing Engines:** Choose from three powerful engines - [DuckDB](https://duckdb.org), [Polars](https://pola.rs), and [PyArrow](https://arrow.apache.org/docs/python/) - to handle your data processing needs.
|
|
17
17
|
- **Flexible Data Transformation:** Easily convert your processed CSV data into various formats including CSV, Excel, JSON, JSONL, and Parquet.
|
|
18
|
-
- **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on
|
|
18
|
+
- **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on all engines.
|
|
19
19
|
- **PDF Parsing & OCR:** Extract text, tables, and images from PDF files as dicts, DataFrames, or JSON, with optional [Tesseract](https://github.com/tesseract-ocr/tesseract) OCR for scanned pages. Powered by the permissively-licensed **PDFium** engine by default, with **PyMuPDF** available as an alternative.
|
|
20
20
|
- **AI-Powered Schema Analysis:** Use Google's Gemini models to automatically generate detailed schema reports for your CSV files, including data types, column classifications, and data quality checks.
|
|
21
21
|
- **Pythonic API:** Enjoy a clean and intuitive API that integrates seamlessly into your existing Python workflows.
|
|
@@ -279,7 +279,7 @@ tables with at least two rows and two columns. It is off by default.
|
|
|
279
279
|
| **Default for** | CSVReader | CSVWriter | - |
|
|
280
280
|
| **Export Quality** | Good | Excellent (especially JSON) | Native Parquet support |
|
|
281
281
|
|
|
282
|
-
_The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three.
|
|
282
|
+
_The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three. Mid-file lines starting with `#` are kept as data on all three engines. PDF parsing uses the **PDFium** engine by default (permissively licensed), with **PyMuPDF** available via `engine="pymupdf"` — see [PDF parsing](#pdf-parsing)._
|
|
283
283
|
|
|
284
284
|
## Primary Classes
|
|
285
285
|
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "datagrunt"
|
|
7
|
-
version = "3.2.
|
|
7
|
+
version = "3.2.2"
|
|
8
8
|
description = "Read CSV files and convert to other file formats easily"
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
authors = [{ name = "Martin Graham", email = "datagrunt@datagrunt.io" }]
|
|
@@ -45,9 +45,9 @@ dev = [
|
|
|
45
45
|
]
|
|
46
46
|
|
|
47
47
|
[project.urls]
|
|
48
|
-
Homepage = "https://
|
|
48
|
+
Homepage = "https://www.datagrunt.io/"
|
|
49
49
|
"Bug Tracker" = "https://github.com/pmgraham/datagrunt/issues"
|
|
50
|
-
Documentation = "https://
|
|
50
|
+
Documentation = "https://www.datagrunt.io/"
|
|
51
51
|
"Source Code" = "https://github.com/pmgraham/datagrunt"
|
|
52
52
|
|
|
53
53
|
[tool.setuptools.packages.find]
|
|
@@ -56,7 +56,7 @@ include = ["datagrunt*"]
|
|
|
56
56
|
exclude = ["tests*"]
|
|
57
57
|
|
|
58
58
|
[tool.bumpver]
|
|
59
|
-
current_version = "3.2.
|
|
59
|
+
current_version = "3.2.2"
|
|
60
60
|
version_pattern = "MAJOR.MINOR.PATCH"
|
|
61
61
|
commit_message = "build: bump version {old_version} -> {new_version} [skip ci]"
|
|
62
62
|
commit = true
|
|
@@ -226,12 +226,18 @@ class DuckDBQueries:
|
|
|
226
226
|
strict_mode=false,
|
|
227
227
|
skip={self.skip_rows});
|
|
228
228
|
"""
|
|
229
|
+
# comment='' pins comment handling OFF despite auto_detect=true. The
|
|
230
|
+
# sniffer otherwise infers comment='#' for files mixing a leading '#'
|
|
231
|
+
# block with '#'-prefixed data rows and swallows the data rows
|
|
232
|
+
# (issue #141). Leading comment blocks are already handled via skip,
|
|
233
|
+
# and mid-file '#' lines are data on the polars/pyarrow engines.
|
|
229
234
|
else:
|
|
230
235
|
return f"""
|
|
231
236
|
CREATE OR REPLACE TABLE {self.database_table_name} AS
|
|
232
237
|
SELECT *
|
|
233
238
|
FROM read_csv('{self._escaped_filepath_literal}',
|
|
234
239
|
auto_detect=true,
|
|
240
|
+
comment='',
|
|
235
241
|
delim='{self._escape_sql_literal(self.delimiter)}',
|
|
236
242
|
header=true,
|
|
237
243
|
quote='{self.quotechar.replace("'", "''")}',
|
|
@@ -275,6 +281,7 @@ class DuckDBQueries:
|
|
|
275
281
|
SELECT *
|
|
276
282
|
FROM read_csv('{self._escaped_filepath_literal}',
|
|
277
283
|
auto_detect=true,
|
|
284
|
+
comment='',
|
|
278
285
|
delim='{self._escape_sql_literal(self.delimiter)}',
|
|
279
286
|
header=true,
|
|
280
287
|
quote='{self.quotechar.replace("'", "''")}',
|
|
@@ -308,9 +315,9 @@ class DuckDBQueries:
|
|
|
308
315
|
from datagrunt.core.csv_io import CSVColumns
|
|
309
316
|
|
|
310
317
|
cols = CSVColumns(self.filepath, delimiter=self.delimiter).columns
|
|
311
|
-
cols_param =
|
|
312
|
-
read_csv = f"""read_csv('{self.
|
|
313
|
-
delim='{self.delimiter}',
|
|
318
|
+
cols_param = self._build_lenient_columns_param(cols)
|
|
319
|
+
read_csv = f"""read_csv('{self._escaped_filepath_literal}',
|
|
320
|
+
delim='{self._escape_sql_literal(self.delimiter)}',
|
|
314
321
|
header=true,
|
|
315
322
|
columns={cols_param},
|
|
316
323
|
quote='{self.quotechar.replace("'", "''")}',
|
|
@@ -320,9 +327,10 @@ class DuckDBQueries:
|
|
|
320
327
|
strict_mode=false,
|
|
321
328
|
skip={self.skip_rows})"""
|
|
322
329
|
else:
|
|
323
|
-
read_csv = f"""read_csv('{self.
|
|
330
|
+
read_csv = f"""read_csv('{self._escaped_filepath_literal}',
|
|
324
331
|
auto_detect=true,
|
|
325
|
-
|
|
332
|
+
comment='',
|
|
333
|
+
delim='{self._escape_sql_literal(self.delimiter)}',
|
|
326
334
|
header=true,
|
|
327
335
|
quote='{self.quotechar.replace("'", "''")}',
|
|
328
336
|
null_padding=true,
|
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: datagrunt
|
|
3
|
-
Version: 3.2.
|
|
3
|
+
Version: 3.2.2
|
|
4
4
|
Summary: Read CSV files and convert to other file formats easily
|
|
5
5
|
Author-email: Martin Graham <datagrunt@datagrunt.io>
|
|
6
6
|
License: MIT License
|
|
7
|
-
Project-URL: Homepage, https://
|
|
7
|
+
Project-URL: Homepage, https://www.datagrunt.io/
|
|
8
8
|
Project-URL: Bug Tracker, https://github.com/pmgraham/datagrunt/issues
|
|
9
|
-
Project-URL: Documentation, https://
|
|
9
|
+
Project-URL: Documentation, https://www.datagrunt.io/
|
|
10
10
|
Project-URL: Source Code, https://github.com/pmgraham/datagrunt
|
|
11
11
|
Keywords: csv,data,duckdb,polars,pyarrow,xlsx,delimiter,ai,gemini,pdf
|
|
12
12
|
Classifier: License :: OSI Approved :: MIT License
|
|
@@ -55,7 +55,7 @@ Datagrunt is not an extension of or a replacement for DuckDB, Polars, or PyArrow
|
|
|
55
55
|
- **Path Object Support:** Full support for both string paths and `pathlib.Path` objects for modern, cross-platform file handling.
|
|
56
56
|
- **Multiple Processing Engines:** Choose from three powerful engines - [DuckDB](https://duckdb.org), [Polars](https://pola.rs), and [PyArrow](https://arrow.apache.org/docs/python/) - to handle your data processing needs.
|
|
57
57
|
- **Flexible Data Transformation:** Easily convert your processed CSV data into various formats including CSV, Excel, JSON, JSONL, and Parquet.
|
|
58
|
-
- **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on
|
|
58
|
+
- **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on all engines.
|
|
59
59
|
- **PDF Parsing & OCR:** Extract text, tables, and images from PDF files as dicts, DataFrames, or JSON, with optional [Tesseract](https://github.com/tesseract-ocr/tesseract) OCR for scanned pages. Powered by the permissively-licensed **PDFium** engine by default, with **PyMuPDF** available as an alternative.
|
|
60
60
|
- **AI-Powered Schema Analysis:** Use Google's Gemini models to automatically generate detailed schema reports for your CSV files, including data types, column classifications, and data quality checks.
|
|
61
61
|
- **Pythonic API:** Enjoy a clean and intuitive API that integrates seamlessly into your existing Python workflows.
|
|
@@ -319,7 +319,7 @@ tables with at least two rows and two columns. It is off by default.
|
|
|
319
319
|
| **Default for** | CSVReader | CSVWriter | - |
|
|
320
320
|
| **Export Quality** | Good | Excellent (especially JSON) | Native Parquet support |
|
|
321
321
|
|
|
322
|
-
_The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three.
|
|
322
|
+
_The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three. Mid-file lines starting with `#` are kept as data on all three engines. PDF parsing uses the **PDFium** engine by default (permissively licensed), with **PyMuPDF** available via `engine="pymupdf"` — see [PDF parsing](#pdf-parsing)._
|
|
323
323
|
|
|
324
324
|
## Primary Classes
|
|
325
325
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/text_block_builder.py
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|