csv-to-db-py 0.1.0__tar.gz

csv_to_db_py-0.1.0/.github/workflows/publish.yaml

@@ -0,0 +1,19 @@
+name: publish
+
+on:
+  push:
+    tags:
+      - '[0-9]*'
+
+permissions:
+  contents: read
+  id-token: write
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1

csv_to_db_py-0.1.0/.github/workflows/workflow.yaml

@@ -0,0 +1,43 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  ruff-format:
+    runs-on: ubuntu-latest
+    container:
+      image: python:3.13-slim
+    steps:
+      - uses: actions/checkout@v4
+      - name: install ruff
+        run: pip install ruff
+      - name: lint
+        run: ruff format --check
+
+  ruff-check:
+    runs-on: ubuntu-latest
+    container:
+      image: python:3.13-slim
+    steps:
+      - uses: actions/checkout@v4
+      - name: install ruff
+        run: pip install ruff
+      - name: lint
+        run: ruff check
+
+  test:
+    runs-on: ubuntu-latest
+    container:
+      image: python:3.13-slim
+    steps:
+      - uses: actions/checkout@v4
+      - name: install uv
+        run: pip install uv
+      - name: install dependencies
+        run: uv sync --dev
+      - name: run tests
+        run: uv run pytest

csv_to_db_py-0.1.0/.gitignore

@@ -0,0 +1,25 @@
+mainConfig.json
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+
+# Distribution / packaging
+build/
+dist/
+*.egg-info/
+
+# Environments
+.env
+.venv
+venv/
+
+# Ruff
+.ruff_cache/
+
+# Logs
+*.log
+
+.serena/

csv_to_db_py-0.1.0/.superpowers/sdd/.gitignore

@@ -0,0 +1 @@
+*

csv_to_db_py-0.1.0/.superpowers/sdd/2026-10-08-csv-to-db-hardening/final-review-fix-report.md

@@ -0,0 +1,91 @@
+# Final Review — Fix Report
+
+Date: 2026-10-08
+Branch: `fixes` (fix wave on top of fc851e2)
+Scope: whole-branch final review findings (2 Important + minor batch)
+
+## Important 1 — Schema drift destroys old data before insert
+
+**File:** `csv_to_db_py/utils/csv_loader.py`
+
+`_truncate_if_exists` now takes `expected_columns`. When the table exists it
+compares the DB column names (`inspect(engine).get_columns(table_name)` →
+`.name`) against the expected columns **before** any `TRUNCATE`. On mismatch it
+raises a `ValueError` explaining the schema drift and suggesting
+`DROP TABLE ... CASCADE` + recreating dependent views; nothing is truncated.
+Only when the names match is the table truncated. Return contract unchanged:
+`True` if truncated, `False` if the table did not exist.
+
+Call site updated in `overwrite_table_with_csv_data` to pass `csv_data.columns`.
+
+### Column-name comparison behavior (checked)
+
+`create_table_from_csv` builds its `CREATE TABLE IF NOT EXISTS` with
+`quote_identifier(column)` (double-quoted identifiers) over the already
+normalized `dataframe.columns`. Quoted identifiers preserve the exact case on
+Postgres, so on the normal overwrite path `get_columns(...)` returns the very
+same names as `df.columns`. I therefore compare **exact** sorted name lists
+without normalization. Rationale: the guard exists to avoid destroying data
+when layouts differ; a normalized (fuzzy) comparison could mask a genuine
+rename (e.g. `col name` → `colname` normalizing to the same key) and wrongly
+truncate. Exact comparison is the correct, stricter choice. No normalization
+applied.
+
+A one-line `ponytail:` comment documents that TYPE drift is still not guarded
+(it still fails loudly after truncate), which is acceptable since the name
+guard covers the documented case.
+
+**Tests** (`tests/test_errors.py`, +2 new):
+- `test_truncate_raises_on_schema_drift_without_truncating`: stub inspector
+  with `has_table` → True and `get_columns` returning a column different from
+  the dataframe's → `_truncate_if_exists` raises `ValueError`; the stub engine
+  uses the existing `BoomConn` (raises `RuntimeError` on execute), so a
+  `ValueError` proves TRUNCATE never ran.
+- `test_truncate_executes_when_columns_match`: stub inspector whose columns
+  match → TRUNCATE executes (asserted via a counting connection) and returns
+  `True`.
+- Existing `test_truncate_if_exists_propagates_execute_error` updated: the
+  `FakeInspector` now also returns `get_columns` matching `["col"]`.
+
+## Important 2 — README Testing section stale
+
+`README.md` Testing section now runs `uv run pytest` instead of
+`uv run python tests/test_datetime_detection.py` (that file is a pytest module
+without `__main__`), and the prose reflects the pytest reality.
+
+## Minor batch
+
+1. `csv_to_db_py/config.py`: `validate_config` guards `DATABASE_CONFIG` with
+   `isinstance(cfg.get("DATABASE_CONFIG"), dict)` → `ValueError` (was
+   AttributeError on `.get` of a non-dict). Error message concatenation merged
+   into a single f-string.
+2. `tests/test_datetime_detection.py`: `TEST_FILE` now
+   `Path(__file__).parent / "date_formats_test.csv"` (CWD-independent).
+3. `csv_to_db_py/main.py`: the two
+   `from csv_to_db_py.utils.cleaned_csv import ...` lines merged into one.
+4. `csv_to_db_py/main.py`: `parse_args` annotated
+   `-> argparse.Namespace`.
+5. `csv_to_db_py/utils/cleaned_csv.py`: `apply_filter` annotated
+   `(df: pl.DataFrame, filter_config: dict) -> pl.DataFrame`.
+6. `README.md`: Table Overwrite Strategy reworded — table is always
+   pre-created by `create_table_from_csv` (no longer "created automatically by
+   `to_sql`"); column-name drift now blocks before truncating, type drift
+   still fails loudly after truncate.
+7. `csv_to_db_py/utils/csv_loader.py`: stale "au moins 90%" comments replaced
+   with the configurable `datetime_detection_threshold` wording; one-line
+   docstring added to `convert_datetime_columns`.
+
+## Verification
+
+```
+$ UV_PROJECT_ENVIRONMENT=/tmp/uv/csv-to-db-py uv run pytest -q
+47 passed, 80 warnings in 0.40s
+
+$ UV_PROJECT_ENVIRONMENT=/tmp/uv/csv-to-db-py uv run ruff check .
+All checks passed!  (Ruff: No issues found)
+
+$ UV_PROJECT_ENVIRONMENT=/tmp/uv/csv-to-db-py uv run ruff format .
+24 files left unchanged
+```
+
+Baseline before the fix wave: 45 passed → 47 after (2 new drift tests).

csv_to_db_py-0.1.0/.superpowers/sdd/2026-10-08-csv-to-db-hardening/plan-path

@@ -0,0 +1 @@
+docs/superpowers/plans/2026-10-08-csv-to-db-hardening.md

csv_to_db_py-0.1.0/.superpowers/sdd/2026-10-08-csv-to-db-hardening/progress.md

@@ -0,0 +1,70 @@
+# SDD ledger — plan: docs/superpowers/plans/2026-10-08-csv-to-db-hardening.md
+
+## Pre-flight scan
+
+| Tasks | Shared file | What one produces / other consumes | Found |
+|---|---|---|---|
+| T1 → T7 | `csv_to_db_py/main.py` | imports (T1) then argparse (T7) | sequential, fine |
+| T1 → T5 | `table_creator.py`, `csv_loader.py` | import fixes, then error handling | sequential, fine |
+| T5 → T6 | `csv_loader.py` | error propagation, then datetime refactor (different functions) | sequential, fine |
+| T6 → T7 | `csv_loader.py` | `convert_datetime_columns`, then delete `polars_type_to_sql` | sequential, fine |
+| T4 → T11 | `cleaned_csv.py` | `quote_identifier`, then `apply_filter` (different functions) | sequential, fine |
+| T2 → T3 | `config.py` + tests | validation, then tests import prod modules | **CONFLICT: config.py reads `../mainConfig.json` at import; file is gitignored → absent in CI → tests cannot import** |
+| T3 → T8 | `pyproject.toml` | pytest (dev), then drivers (deps) | sequential, fine |
+| T6 test | config threshold | test needs threshold 0.7; local config has no key → default 1.0 | **NOTE: T6 test must monkeypatch `config["datetime_detection_threshold"] = 0.7`** |
+| T9 | `compose.yml` | file is untracked (`?? compose.yml`) | commit will `git add` it whole; expected |
+
+## Pre-flight rulings
+
+- `Ruling: config import resilience` — config.py must fall back to `mainConfig.json.example-strict` when `mainConfig.json` is absent, then validate. Without it, tests and CI crash on `import csv_to_db_py.config` (gitignored config missing in CI, and installed package can't find `../mainConfig.json`). Amend Task 2. — Cost if wrong: small rework in T2.
+- `Ruling: T6 test threshold` — T6 test monkeypatches `config["datetime_detection_threshold"] = 0.7` before calling `convert_datetime_columns`; carried in T6 dispatch. — Cost if wrong: test silently uses default 1.0 and fails.
+
+## Tasks
+
+Task 1: pending
+
+Task 1: complete (commits 7ffe98c..f7751b0, review clean)
+Task 1: minor (deferred): main.py:101-102 two csv_to_db_py.utils.cleaned_csv import lines could merge into one
+
+Task 2: complete (commits f7751b0..02798e6, review clean)
+Task 2: minor (deferred): config.py:28 DATABASE_CONFIG non-dict would raise AttributeError not ValueError; f-string for error message; fallback load path not covered by test
+
+Task 3: complete (commits 02798e6..bd3e9d5, review clean; 22 tests passing confirmed by controller)
+Task 3: minor (deferred): TEST_FILE path is CWD-relative (use Path(__file__).parent); batch test replicates prod formula (accepted, no importable helper)
+
+Task 4: complete (commits bd3e9d5..ab9ee61, review clean)
+Task 4: minor (deferred): long f-string in table_creator.py:86 readability
+
+Task 5: complete (commits ab9ee61..55479cd, review clean)
+Task 5: minor (deferred): table_creator.py has pre-existing duplicate `columns = dataframe.columns` assignment (out of scope)
+
+Task 6: complete (commits 55479cd..27c87bb, review clean; fix round 1/5: test data 4 values at threshold 0.7 per ruling)
+Task 6: minor (deferred): stale "90%" comments in csv_loader.py:110,172 (pre-existing); convert_datetime_columns lacks docstring
+
+Task 7: complete (commits 27c87bb..11102ec, review clean)
+Task 7: minor (deferred): parse_args lacks return type annotation (cosmetic)
+
+Task 8: fix round 1/5 (oracle driver) — in progress
+Task 8: Ruling: cx_Oracle cannot build on Python 3.13 (deprecated C extension, no cp313 wheel; sandbox build failed on missing pkg_resources). Its official successor python-oracledb ships cp313 wheels and SQLAlchemy supports dialect `oracle+oracledb`. Replace cx_oracle with oracledb so the connector installs everywhere (including CI uv sync) and the oracle test can run. — Cost if wrong: dialect name differs from some users' expectations; small.
+
+Task 8: complete (commits 11102ec..f08c415, review clean; fix round 1/5: cx_oracle → oracledb per ruling)
+Task 8: minor (deferred): make_url identity vs string round-trip in connector test (matches brief, adequate)
+
+Task 9: complete (commits f08c415..8593195, review clean)
+
+Task 10: complete (commits 8593195..49aa2f3, review clean)
+
+Task 11: complete (commits 49aa2f3..ff80dc8, review clean)
+Task 11: minor (deferred): apply_filter lacks type hints; FILTER empty-string value now runs filter (was truthy-skipped) — harmless
+
+Task 12: complete (commits ff80dc8..dbb469e, review clean)
+
+Task 13: complete (commits dbb469e..fc851e2, review clean)
+Task 13: minor (deferred): README "Good for most cases" 100000 framing vs new 10000 default; launch.json windows python path (pre-existing)
+ALL TASKS COMPLETE. Launching final whole-branch review (base 7ffe98c, head fc851e2).
+
+Final review: "With fixes" — 2 Important + minor batch. Dispatching ONE fix subagent (fix base fc851e2).
+
+Final review fix wave: complete (fc851e2..9ee3e7a, re-review clean — all 9 findings addressed, no new Critical/Important breakage)
+Out-of-scope note for user: quote_identifier uses postgres double-quote syntax on all backends (MySQL expects backticks) — pre-existing, not introduced by this branch.
+Final review: clean. Workspace cleanup next.