datagrunt 3.2.0__tar.gz → 3.2.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. {datagrunt-3.2.0/src/datagrunt.egg-info → datagrunt-3.2.2}/PKG-INFO +5 -5
  2. {datagrunt-3.2.0 → datagrunt-3.2.2}/README.md +2 -2
  3. {datagrunt-3.2.0 → datagrunt-3.2.2}/pyproject.toml +4 -4
  4. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/__init__.py +1 -1
  5. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/databases/databases.py +13 -5
  6. {datagrunt-3.2.0 → datagrunt-3.2.2/src/datagrunt.egg-info}/PKG-INFO +5 -5
  7. {datagrunt-3.2.0 → datagrunt-3.2.2}/LICENSE +0 -0
  8. {datagrunt-3.2.0 → datagrunt-3.2.2}/setup.cfg +0 -0
  9. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/__init__.py +0 -0
  10. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/ai/__init__.py +0 -0
  11. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/ai/engines.py +0 -0
  12. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/ai/factories.py +0 -0
  13. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/ai/prompts.py +0 -0
  14. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/csv_io/__init__.py +0 -0
  15. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/csv_io/csvcomponents.py +0 -0
  16. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/csv_io/engines.py +0 -0
  17. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/csv_io/factories.py +0 -0
  18. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/databases/__init__.py +0 -0
  19. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/file_io/__init__.py +0 -0
  20. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/file_io/fileproperties.py +0 -0
  21. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/__init__.py +0 -0
  22. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/engines.py +0 -0
  23. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/__init__.py +0 -0
  24. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/base.py +0 -0
  25. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/image_dedupe.py +0 -0
  26. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/layout_sorter.py +0 -0
  27. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/markdown_escape.py +0 -0
  28. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/ocr.py +0 -0
  29. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/pdfium_backend.py +0 -0
  30. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/pdfium_document.py +0 -0
  31. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/pdfium_native.py +0 -0
  32. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/pymupdf_backend.py +0 -0
  33. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/shapes.py +0 -0
  34. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/tables.py +0 -0
  35. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/extraction/text_block_builder.py +0 -0
  36. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/factories.py +0 -0
  37. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/core/pdf_io/pdfcomponents.py +0 -0
  38. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/csv_api/__init__.py +0 -0
  39. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/csv_api/_compat.py +0 -0
  40. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/csv_api/csvai.py +0 -0
  41. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/csv_api/csvreader.py +0 -0
  42. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/csv_api/csvwriter.py +0 -0
  43. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/pdf_api/__init__.py +0 -0
  44. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/pdf_api/pdfreader.py +0 -0
  45. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt/pdf_api/pdfwriter.py +0 -0
  46. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt.egg-info/SOURCES.txt +0 -0
  47. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt.egg-info/dependency_links.txt +0 -0
  48. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt.egg-info/requires.txt +0 -0
  49. {datagrunt-3.2.0 → datagrunt-3.2.2}/src/datagrunt.egg-info/top_level.txt +0 -0
@@ -1,12 +1,12 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: datagrunt
3
- Version: 3.2.0
3
+ Version: 3.2.2
4
4
  Summary: Read CSV files and convert to other file formats easily
5
5
  Author-email: Martin Graham <datagrunt@datagrunt.io>
6
6
  License: MIT License
7
- Project-URL: Homepage, https://pmgraham.github.io/datagrunt
7
+ Project-URL: Homepage, https://www.datagrunt.io/
8
8
  Project-URL: Bug Tracker, https://github.com/pmgraham/datagrunt/issues
9
- Project-URL: Documentation, https://pmgraham.github.io/datagrunt
9
+ Project-URL: Documentation, https://www.datagrunt.io/
10
10
  Project-URL: Source Code, https://github.com/pmgraham/datagrunt
11
11
  Keywords: csv,data,duckdb,polars,pyarrow,xlsx,delimiter,ai,gemini,pdf
12
12
  Classifier: License :: OSI Approved :: MIT License
@@ -55,7 +55,7 @@ Datagrunt is not an extension of or a replacement for DuckDB, Polars, or PyArrow
55
55
  - **Path Object Support:** Full support for both string paths and `pathlib.Path` objects for modern, cross-platform file handling.
56
56
  - **Multiple Processing Engines:** Choose from three powerful engines - [DuckDB](https://duckdb.org), [Polars](https://pola.rs), and [PyArrow](https://arrow.apache.org/docs/python/) - to handle your data processing needs.
57
57
  - **Flexible Data Transformation:** Easily convert your processed CSV data into various formats including CSV, Excel, JSON, JSONL, and Parquet.
58
- - **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on the Polars and PyArrow engines.
58
+ - **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on all engines.
59
59
  - **PDF Parsing & OCR:** Extract text, tables, and images from PDF files as dicts, DataFrames, or JSON, with optional [Tesseract](https://github.com/tesseract-ocr/tesseract) OCR for scanned pages. Powered by the permissively-licensed **PDFium** engine by default, with **PyMuPDF** available as an alternative.
60
60
  - **AI-Powered Schema Analysis:** Use Google's Gemini models to automatically generate detailed schema reports for your CSV files, including data types, column classifications, and data quality checks.
61
61
  - **Pythonic API:** Enjoy a clean and intuitive API that integrates seamlessly into your existing Python workflows.
@@ -319,7 +319,7 @@ tables with at least two rows and two columns. It is off by default.
319
319
  | **Default for** | CSVReader | CSVWriter | - |
320
320
  | **Export Quality** | Good | Excellent (especially JSON) | Native Parquet support |
321
321
 
322
- _The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three. One known divergence: mid-file lines starting with `#` are kept as data on Polars and PyArrow, but the DuckDB engine may still drop them. PDF parsing uses the **PDFium** engine by default (permissively licensed), with **PyMuPDF** available via `engine="pymupdf"` — see [PDF parsing](#pdf-parsing)._
322
+ _The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three. Mid-file lines starting with `#` are kept as data on all three engines. PDF parsing uses the **PDFium** engine by default (permissively licensed), with **PyMuPDF** available via `engine="pymupdf"` — see [PDF parsing](#pdf-parsing)._
323
323
 
324
324
  ## Primary Classes
325
325
 
@@ -15,7 +15,7 @@ Datagrunt is not an extension of or a replacement for DuckDB, Polars, or PyArrow
15
15
  - **Path Object Support:** Full support for both string paths and `pathlib.Path` objects for modern, cross-platform file handling.
16
16
  - **Multiple Processing Engines:** Choose from three powerful engines - [DuckDB](https://duckdb.org), [Polars](https://pola.rs), and [PyArrow](https://arrow.apache.org/docs/python/) - to handle your data processing needs.
17
17
  - **Flexible Data Transformation:** Easily convert your processed CSV data into various formats including CSV, Excel, JSON, JSONL, and Parquet.
18
- - **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on the Polars and PyArrow engines.
18
+ - **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on all engines.
19
19
  - **PDF Parsing & OCR:** Extract text, tables, and images from PDF files as dicts, DataFrames, or JSON, with optional [Tesseract](https://github.com/tesseract-ocr/tesseract) OCR for scanned pages. Powered by the permissively-licensed **PDFium** engine by default, with **PyMuPDF** available as an alternative.
20
20
  - **AI-Powered Schema Analysis:** Use Google's Gemini models to automatically generate detailed schema reports for your CSV files, including data types, column classifications, and data quality checks.
21
21
  - **Pythonic API:** Enjoy a clean and intuitive API that integrates seamlessly into your existing Python workflows.
@@ -279,7 +279,7 @@ tables with at least two rows and two columns. It is off by default.
279
279
  | **Default for** | CSVReader | CSVWriter | - |
280
280
  | **Export Quality** | Good | Excellent (especially JSON) | Native Parquet support |
281
281
 
282
- _The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three. One known divergence: mid-file lines starting with `#` are kept as data on Polars and PyArrow, but the DuckDB engine may still drop them. PDF parsing uses the **PDFium** engine by default (permissively licensed), with **PyMuPDF** available via `engine="pymupdf"` — see [PDF parsing](#pdf-parsing)._
282
+ _The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three. Mid-file lines starting with `#` are kept as data on all three engines. PDF parsing uses the **PDFium** engine by default (permissively licensed), with **PyMuPDF** available via `engine="pymupdf"` — see [PDF parsing](#pdf-parsing)._
283
283
 
284
284
  ## Primary Classes
285
285
 
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "datagrunt"
7
- version = "3.2.0"
7
+ version = "3.2.2"
8
8
  description = "Read CSV files and convert to other file formats easily"
9
9
  readme = "README.md"
10
10
  authors = [{ name = "Martin Graham", email = "datagrunt@datagrunt.io" }]
@@ -45,9 +45,9 @@ dev = [
45
45
  ]
46
46
 
47
47
  [project.urls]
48
- Homepage = "https://pmgraham.github.io/datagrunt"
48
+ Homepage = "https://www.datagrunt.io/"
49
49
  "Bug Tracker" = "https://github.com/pmgraham/datagrunt/issues"
50
- Documentation = "https://pmgraham.github.io/datagrunt"
50
+ Documentation = "https://www.datagrunt.io/"
51
51
  "Source Code" = "https://github.com/pmgraham/datagrunt"
52
52
 
53
53
  [tool.setuptools.packages.find]
@@ -56,7 +56,7 @@ include = ["datagrunt*"]
56
56
  exclude = ["tests*"]
57
57
 
58
58
  [tool.bumpver]
59
- current_version = "3.2.0"
59
+ current_version = "3.2.2"
60
60
  version_pattern = "MAJOR.MINOR.PATCH"
61
61
  commit_message = "build: bump version {old_version} -> {new_version} [skip ci]"
62
62
  commit = true
@@ -23,7 +23,7 @@ Attributes:
23
23
  __license__: The license under which the package is released.
24
24
  """
25
25
 
26
- __version__ = "3.2.0"
26
+ __version__ = "3.2.2"
27
27
  __author__ = "Martin Graham"
28
28
  __license__ = "MIT"
29
29
 
@@ -226,12 +226,18 @@ class DuckDBQueries:
226
226
  strict_mode=false,
227
227
  skip={self.skip_rows});
228
228
  """
229
+ # comment='' pins comment handling OFF despite auto_detect=true. The
230
+ # sniffer otherwise infers comment='#' for files mixing a leading '#'
231
+ # block with '#'-prefixed data rows and swallows the data rows
232
+ # (issue #141). Leading comment blocks are already handled via skip,
233
+ # and mid-file '#' lines are data on the polars/pyarrow engines.
229
234
  else:
230
235
  return f"""
231
236
  CREATE OR REPLACE TABLE {self.database_table_name} AS
232
237
  SELECT *
233
238
  FROM read_csv('{self._escaped_filepath_literal}',
234
239
  auto_detect=true,
240
+ comment='',
235
241
  delim='{self._escape_sql_literal(self.delimiter)}',
236
242
  header=true,
237
243
  quote='{self.quotechar.replace("'", "''")}',
@@ -275,6 +281,7 @@ class DuckDBQueries:
275
281
  SELECT *
276
282
  FROM read_csv('{self._escaped_filepath_literal}',
277
283
  auto_detect=true,
284
+ comment='',
278
285
  delim='{self._escape_sql_literal(self.delimiter)}',
279
286
  header=true,
280
287
  quote='{self.quotechar.replace("'", "''")}',
@@ -308,9 +315,9 @@ class DuckDBQueries:
308
315
  from datagrunt.core.csv_io import CSVColumns
309
316
 
310
317
  cols = CSVColumns(self.filepath, delimiter=self.delimiter).columns
311
- cols_param = "{" + ", ".join(f"'{c}': 'VARCHAR'" for c in cols) + "}"
312
- read_csv = f"""read_csv('{self.filepath}',
313
- delim='{self.delimiter}',
318
+ cols_param = self._build_lenient_columns_param(cols)
319
+ read_csv = f"""read_csv('{self._escaped_filepath_literal}',
320
+ delim='{self._escape_sql_literal(self.delimiter)}',
314
321
  header=true,
315
322
  columns={cols_param},
316
323
  quote='{self.quotechar.replace("'", "''")}',
@@ -320,9 +327,10 @@ class DuckDBQueries:
320
327
  strict_mode=false,
321
328
  skip={self.skip_rows})"""
322
329
  else:
323
- read_csv = f"""read_csv('{self.filepath}',
330
+ read_csv = f"""read_csv('{self._escaped_filepath_literal}',
324
331
  auto_detect=true,
325
- delim='{self.delimiter}',
332
+ comment='',
333
+ delim='{self._escape_sql_literal(self.delimiter)}',
326
334
  header=true,
327
335
  quote='{self.quotechar.replace("'", "''")}',
328
336
  null_padding=true,
@@ -1,12 +1,12 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: datagrunt
3
- Version: 3.2.0
3
+ Version: 3.2.2
4
4
  Summary: Read CSV files and convert to other file formats easily
5
5
  Author-email: Martin Graham <datagrunt@datagrunt.io>
6
6
  License: MIT License
7
- Project-URL: Homepage, https://pmgraham.github.io/datagrunt
7
+ Project-URL: Homepage, https://www.datagrunt.io/
8
8
  Project-URL: Bug Tracker, https://github.com/pmgraham/datagrunt/issues
9
- Project-URL: Documentation, https://pmgraham.github.io/datagrunt
9
+ Project-URL: Documentation, https://www.datagrunt.io/
10
10
  Project-URL: Source Code, https://github.com/pmgraham/datagrunt
11
11
  Keywords: csv,data,duckdb,polars,pyarrow,xlsx,delimiter,ai,gemini,pdf
12
12
  Classifier: License :: OSI Approved :: MIT License
@@ -55,7 +55,7 @@ Datagrunt is not an extension of or a replacement for DuckDB, Polars, or PyArrow
55
55
  - **Path Object Support:** Full support for both string paths and `pathlib.Path` objects for modern, cross-platform file handling.
56
56
  - **Multiple Processing Engines:** Choose from three powerful engines - [DuckDB](https://duckdb.org), [Polars](https://pola.rs), and [PyArrow](https://arrow.apache.org/docs/python/) - to handle your data processing needs.
57
57
  - **Flexible Data Transformation:** Easily convert your processed CSV data into various formats including CSV, Excel, JSON, JSONL, and Parquet.
58
- - **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on the Polars and PyArrow engines.
58
+ - **Robust by Default:** Fail-fast validation with clear errors (invalid engine names, missing paths, directories, encrypted PDFs), graceful handling of empty files, no `UnicodeDecodeError` when constructing a reader over a non-UTF-8 file, and sane comment semantics — only leading `#` lines are treated as comments, so `#`-prefixed data rows such as hex colors are preserved on all engines.
59
59
  - **PDF Parsing & OCR:** Extract text, tables, and images from PDF files as dicts, DataFrames, or JSON, with optional [Tesseract](https://github.com/tesseract-ocr/tesseract) OCR for scanned pages. Powered by the permissively-licensed **PDFium** engine by default, with **PyMuPDF** available as an alternative.
60
60
  - **AI-Powered Schema Analysis:** Use Google's Gemini models to automatically generate detailed schema reports for your CSV files, including data types, column classifications, and data quality checks.
61
61
  - **Pythonic API:** Enjoy a clean and intuitive API that integrates seamlessly into your existing Python workflows.
@@ -319,7 +319,7 @@ tables with at least two rows and two columns. It is off by default.
319
319
  | **Default for** | CSVReader | CSVWriter | - |
320
320
  | **Export Quality** | Good | Excellent (especially JSON) | Native Parquet support |
321
321
 
322
- _The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three. One known divergence: mid-file lines starting with `#` are kept as data on Polars and PyArrow, but the DuckDB engine may still drop them. PDF parsing uses the **PDFium** engine by default (permissively licensed), with **PyMuPDF** available via `engine="pymupdf"` — see [PDF parsing](#pdf-parsing)._
322
+ _The engines above apply to CSV processing. Whichever you pick, results are consistent: leading `#` comment lines, leading blank lines, logical record counts (quoted embedded newlines count as one record), and column-name normalization — including collision handling like `Col A,col_a` → `col_a, col_a_1` — behave identically across all three. Mid-file lines starting with `#` are kept as data on all three engines. PDF parsing uses the **PDFium** engine by default (permissively licensed), with **PyMuPDF** available via `engine="pymupdf"` — see [PDF parsing](#pdf-parsing)._
323
323
 
324
324
  ## Primary Classes
325
325
 
File without changes
File without changes