dumpling-cli 0.3.0__tar.gz → 0.4.1__tar.gz

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/CHANGELOG.md

@@ -7,6 +7,24 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
 
 ## [Unreleased]
 
+## [0.4.1] - 2026-05-03
+
+### Fixed
+
+- **INSERT row parsing with JSON casts**: Values such as `'{"k":1}'::jsonb` are parsed so the cell’s unescaped payload is valid JSON for JSON path rules and anonymization; trailing casts like `::jsonb` / `::text` are preserved on output.
+
+## [0.4.0] - 2026-05-02
+
+### Added
+
+- **`--dump-decode` CLI**: Decode PostgreSQL **custom-format** (`pg_dump -Fc`) or **directory-format** archives by running **`pg_restore -f -`** (plain SQL to stdout, no database), then anonymize—built for workflows such as **`heroku pg:backups:download`**. Requires PostgreSQL client tools (`pg_restore` on `PATH`, or **`--pg-restore-path`**).
+- **`--dump-decode-arg`** (repeatable): Extra arguments forwarded to `pg_restore`.
+- **`--dump-decode-keep-input`**: Keep the archive after a successful run. **By default** the `--input` path is **removed** after success so only anonymized output remains. **`--check`** with **`--dump-decode`** requires **`--dump-decode-keep-input`** (otherwise the dump would be deleted before config iteration).
+
+### Changed
+
+- README and mdBook documentation for PostgreSQL archive decoding and Heroku-style examples.
+
 ## [0.3.0] - 2026-05-02
 
 ### Added
@@ -43,5 +61,7 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
 - Configurable output scan severities and per-category thresholds via `[output_scan]`.
 - JSON report section for output scan findings including category, count, threshold, severity, and sample locations.
 
+[0.4.1]: https://github.com/ababic/dumpling/compare/v0.4.0...v0.4.1
+[0.4.0]: https://github.com/ababic/dumpling/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/ababic/dumpling/compare/v0.2.0...v0.3.0
 [0.2.0]: https://github.com/ababic/dumpling/compare/v0.1.0...v0.2.0

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/Cargo.lock

@@ -262,7 +262,7 @@ dependencies = [
 
 [[package]]
 name = "dumpling"
-version = "0.3.0"
+version = "0.4.1"
 dependencies = [
  "anyhow",
  "chrono",

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "dumpling"
-version = "0.3.0"
+version = "0.4.1"
 edition = "2021"
 readme = "README.md"
 

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dumpling-cli
-Version: 0.3.0
+Version: 0.4.1
 Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
@@ -19,18 +19,48 @@ Keywords: postgres,sqlite,mssql,sql,anonymization,cli,rust
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
 
-# Dumpling
+<p align="center">
+  <img src="assets/logo.svg" width="140" height="140" alt="Dumpling logo: a dumpling with steam" />
+</p>
 
-**Dumpling** is a static anonymizer for plain SQL dumps. It supports PostgreSQL (`pg_dump` plain format), SQLite (`.dump`), and SQL Server / MSSQL (SSMS / mssql-scripter output). It lets you safely share, test with, or store database snapshots by replacing sensitive column data according to configurable rules — without ever touching a live database.
+<h1 align="center">Dumpling</h1>
+
+<p align="center">
+  <strong>Sanitize SQL dumps before they go anywhere.</strong><br />
+  Turn huge <code>pg_dump</code> / SQLite / SQL Server exports into shareable, test-friendly snapshots — no DB connection, no secrets left by accident.
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/dumpling-cli/"><img src="https://img.shields.io/pypi/v/dumpling-cli.svg" alt="PyPI version" /></a>
+  <a href="https://pypi.org/project/dumpling-cli/"><img src="https://img.shields.io/pypi/pyversions/dumpling-cli.svg" alt="Python versions" /></a>
+  <a href="https://pypi.org/project/dumpling-cli/"><img src="https://img.shields.io/pypi/l/dumpling-cli.svg" alt="PyPI license" /></a>
+  <a href="https://github.com/ababic/dumpling/actions/workflows/tests.yml"><img src="https://github.com/ababic/dumpling/actions/workflows/tests.yml/badge.svg" alt="Tests" /></a>
+  <a href="https://github.com/ababic/dumpling/actions/workflows/ci.yml"><img src="https://github.com/ababic/dumpling/actions/workflows/ci.yml/badge.svg" alt="Lint" /></a>
+  <img src="https://img.shields.io/badge/rust-stable-orange?logo=rust" alt="Rust stable" />
+</p>
+
+<p align="center">
+  <a href="https://ababic.github.io/dumpling/"><strong>Documentation</strong></a>
+  &nbsp;·&nbsp;
+  <a href="https://github.com/ababic/dumpling"><strong>GitHub</strong></a>
+</p>
+
+<p align="center">
+  <sub><em>Disclaimer: This project is entirely vibe-coded, but with strong human guidance, review, and attention to quality and safety.</em></sub>
+</p>
+
+---
+
+**Dumpling** reads plain-text SQL dumps (PostgreSQL `pg_dump`, SQLite `.dump`, SQL Server / MSSQL scripts) and rewrites sensitive columns using rules you define in TOML. Everything runs offline on files — ideal for CI, staging share-outs, and compliance-minded workflows.
 
 ## Why Dumpling?
 
-- **No live database required.** Works entirely on dump files; nothing connects to your database.
-- **Streaming and memory-efficient.** Processes dumps line by line, so even multi-gigabyte files stay manageable.
-- **Fail-safe by default.** If no configuration is found, Dumpling exits non-zero and tells you exactly where it looked. Silence is never mistaken for success.
-- **Deterministic anonymization.** Domain mappings ensure the same source value always produces the same pseudonym, keeping foreign-key relationships intact across tables.
-- **CI/CD ready.** `--check` mode, strict-coverage enforcement, JSON reports, and residual-PII scan gates plug cleanly into any pipeline.
-- **Flexible configuration.** Rules live in a `.dumplingconf` file or directly in `pyproject.toml` — no extra tooling needed.
+- **Offline by design** — works on dump files only; nothing connects to your database.
+- **Streams giant files** — line-by-line processing keeps multi‑GB dumps reasonable on modest hardware.
+- **Fails loud, not silent** — missing config exits non‑zero and lists where Dumpling looked; use `--allow-noop` only when you mean it.
+- **Stable pseudonyms** — optional domain mappings keep the same source value as the same fake value across tables (foreign keys stay consistent).
+- **Pipeline-ready** — `--check`, strict coverage, JSON reports, and residual PII scans fit pre-merge gates and release automation.
+- **Configure once** — `.dumplingconf` or `[tool.dumpling]` in `pyproject.toml`; install via **Rust** (`cargo`) or **`pip install dumpling-cli`**.
 
 ---
 
@@ -289,7 +319,16 @@ Produced by `pg_dump --format=plain`. Handles:
 - `"double-quoted"` identifiers
 - `''`-escaped string literals
 
-Binary, custom, and directory formats from `pg_dump` are not supported — use `--format=plain` when running `pg_dump`.
+Binary, custom, and directory formats from `pg_dump` are not parsed directly — Dumpling’s SQL pipeline expects plain text. Use either:
+
+- **`pg_dump --format=plain`** when you control capture, or
+- **`dumpling --dump-decode`** with `--input` set to a **custom-format** (`.dump`) or **directory-format** folder: Dumpling runs `pg_restore -f -` and streams the resulting SQL (same as a manual `pg_restore` “script” output, no database required). Requires PostgreSQL client tools on `PATH` (`pg_restore`), or set `--pg-restore-path`. Use `--dump-decode-arg` to pass extra flags (e.g. `--no-owner --no-acl`). **By default** the archive is removed after a fully successful run; pass **`--dump-decode-keep-input`** to retain it. **`--check`** requires **`--dump-decode-keep-input`** so the archive still exists if changes would be detected.
+
+Example (e.g. after `heroku pg:backups:download`):
+
+```bash
+dumpling --dump-decode -i latest.dump -c .dumplingconf -o anonymized.sql
+```
 
 ### SQLite (`--format sqlite`)
 
@@ -335,6 +374,22 @@ Supported predicate operators:
 
 Predicates can target nested JSON values using dot notation (`payload.profile.tier`) or Django-style notation (`payload__profile__tier`). For JSON arrays, path segments are evaluated against each element, so list-of-dicts structures can be matched naturally.
 
+### JSON path list targeting
+
+JSON list/array traversal is automatic once a path segment resolves to an array.
+
+- **All elements in an array**: use the next field name directly.
+  - `payload.items.kind` or `payload__items__kind`
+  - Matches/rewrites `kind` for every object in `items`.
+- **Specific array index**: use a numeric segment.
+  - `payload.items.0.kind` or `payload__items__0__kind`
+  - Targets only the first element.
+- **Nested arrays**: combine field and index segments as needed.
+  - `payload.groups.members.email`
+  - `payload.groups.1.members.0.email`
+
+This path behavior is shared by both `row_filters` predicates and JSON-path anonymization rules in `[rules]`.
+
 ```toml
 [row_filters."public.users"]
 retain = [
@@ -477,5 +532,5 @@ See the [CI guardrails documentation](docs/src/ci-guardrails.md) for full pipeli
 
 ## Full documentation
 
-Detailed docs, including the configuration reference and release process, are available at the project's [GitHub Pages site](https://github.com) (built from `docs/src/`).
+Detailed docs, including the configuration reference and release process, are available at the project's [GitHub Pages site](https://ababic.github.io/dumpling/) (built from `docs/src/`).
 

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/README.md

@@ -1,15 +1,45 @@
-# Dumpling
+<p align="center">
+  <img src="assets/logo.svg" width="140" height="140" alt="Dumpling logo: a dumpling with steam" />
+</p>
+
+<h1 align="center">Dumpling</h1>
+
+<p align="center">
+  <strong>Sanitize SQL dumps before they go anywhere.</strong><br />
+  Turn huge <code>pg_dump</code> / SQLite / SQL Server exports into shareable, test-friendly snapshots — no DB connection, no secrets left by accident.
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/dumpling-cli/"><img src="https://img.shields.io/pypi/v/dumpling-cli.svg" alt="PyPI version" /></a>
+  <a href="https://pypi.org/project/dumpling-cli/"><img src="https://img.shields.io/pypi/pyversions/dumpling-cli.svg" alt="Python versions" /></a>
+  <a href="https://pypi.org/project/dumpling-cli/"><img src="https://img.shields.io/pypi/l/dumpling-cli.svg" alt="PyPI license" /></a>
+  <a href="https://github.com/ababic/dumpling/actions/workflows/tests.yml"><img src="https://github.com/ababic/dumpling/actions/workflows/tests.yml/badge.svg" alt="Tests" /></a>
+  <a href="https://github.com/ababic/dumpling/actions/workflows/ci.yml"><img src="https://github.com/ababic/dumpling/actions/workflows/ci.yml/badge.svg" alt="Lint" /></a>
+  <img src="https://img.shields.io/badge/rust-stable-orange?logo=rust" alt="Rust stable" />
+</p>
+
+<p align="center">
+  <a href="https://ababic.github.io/dumpling/"><strong>Documentation</strong></a>
+  &nbsp;·&nbsp;
+  <a href="https://github.com/ababic/dumpling"><strong>GitHub</strong></a>
+</p>
+
+<p align="center">
+  <sub><em>Disclaimer: This project is entirely vibe-coded, but with strong human guidance, review, and attention to quality and safety.</em></sub>
+</p>
 
-**Dumpling** is a static anonymizer for plain SQL dumps. It supports PostgreSQL (`pg_dump` plain format), SQLite (`.dump`), and SQL Server / MSSQL (SSMS / mssql-scripter output). It lets you safely share, test with, or store database snapshots by replacing sensitive column data according to configurable rules — without ever touching a live database.
+---
+
+**Dumpling** reads plain-text SQL dumps (PostgreSQL `pg_dump`, SQLite `.dump`, SQL Server / MSSQL scripts) and rewrites sensitive columns using rules you define in TOML. Everything runs offline on files — ideal for CI, staging share-outs, and compliance-minded workflows.
 
 ## Why Dumpling?
 
-- **No live database required.** Works entirely on dump files; nothing connects to your database.
-- **Streaming and memory-efficient.** Processes dumps line by line, so even multi-gigabyte files stay manageable.
-- **Fail-safe by default.** If no configuration is found, Dumpling exits non-zero and tells you exactly where it looked. Silence is never mistaken for success.
-- **Deterministic anonymization.** Domain mappings ensure the same source value always produces the same pseudonym, keeping foreign-key relationships intact across tables.
-- **CI/CD ready.** `--check` mode, strict-coverage enforcement, JSON reports, and residual-PII scan gates plug cleanly into any pipeline.
-- **Flexible configuration.** Rules live in a `.dumplingconf` file or directly in `pyproject.toml` — no extra tooling needed.
+- **Offline by design** — works on dump files only; nothing connects to your database.
+- **Streams giant files** — line-by-line processing keeps multi‑GB dumps reasonable on modest hardware.
+- **Fails loud, not silent** — missing config exits non‑zero and lists where Dumpling looked; use `--allow-noop` only when you mean it.
+- **Stable pseudonyms** — optional domain mappings keep the same source value as the same fake value across tables (foreign keys stay consistent).
+- **Pipeline-ready** — `--check`, strict coverage, JSON reports, and residual PII scans fit pre-merge gates and release automation.
+- **Configure once** — `.dumplingconf` or `[tool.dumpling]` in `pyproject.toml`; install via **Rust** (`cargo`) or **`pip install dumpling-cli`**.
 
 ---
 
@@ -268,7 +298,16 @@ Produced by `pg_dump --format=plain`. Handles:
 - `"double-quoted"` identifiers
 - `''`-escaped string literals
 
-Binary, custom, and directory formats from `pg_dump` are not supported — use `--format=plain` when running `pg_dump`.
+Binary, custom, and directory formats from `pg_dump` are not parsed directly — Dumpling’s SQL pipeline expects plain text. Use either:
+
+- **`pg_dump --format=plain`** when you control capture, or
+- **`dumpling --dump-decode`** with `--input` set to a **custom-format** (`.dump`) or **directory-format** folder: Dumpling runs `pg_restore -f -` and streams the resulting SQL (same as a manual `pg_restore` “script” output, no database required). Requires PostgreSQL client tools on `PATH` (`pg_restore`), or set `--pg-restore-path`. Use `--dump-decode-arg` to pass extra flags (e.g. `--no-owner --no-acl`). **By default** the archive is removed after a fully successful run; pass **`--dump-decode-keep-input`** to retain it. **`--check`** requires **`--dump-decode-keep-input`** so the archive still exists if changes would be detected.
+
+Example (e.g. after `heroku pg:backups:download`):
+
+```bash
+dumpling --dump-decode -i latest.dump -c .dumplingconf -o anonymized.sql
+```
 
 ### SQLite (`--format sqlite`)
 
@@ -314,6 +353,22 @@ Supported predicate operators:
 
 Predicates can target nested JSON values using dot notation (`payload.profile.tier`) or Django-style notation (`payload__profile__tier`). For JSON arrays, path segments are evaluated against each element, so list-of-dicts structures can be matched naturally.
 
+### JSON path list targeting
+
+JSON list/array traversal is automatic once a path segment resolves to an array.
+
+- **All elements in an array**: use the next field name directly.
+  - `payload.items.kind` or `payload__items__kind`
+  - Matches/rewrites `kind` for every object in `items`.
+- **Specific array index**: use a numeric segment.
+  - `payload.items.0.kind` or `payload__items__0__kind`
+  - Targets only the first element.
+- **Nested arrays**: combine field and index segments as needed.
+  - `payload.groups.members.email`
+  - `payload.groups.1.members.0.email`
+
+This path behavior is shared by both `row_filters` predicates and JSON-path anonymization rules in `[rules]`.
+
 ```toml
 [row_filters."public.users"]
 retain = [
@@ -456,4 +511,4 @@ See the [CI guardrails documentation](docs/src/ci-guardrails.md) for full pipeli
 
 ## Full documentation
 
-Detailed docs, including the configuration reference and release process, are available at the project's [GitHub Pages site](https://github.com) (built from `docs/src/`).
+Detailed docs, including the configuration reference and release process, are available at the project's [GitHub Pages site](https://ababic.github.io/dumpling/) (built from `docs/src/`).

dumpling_cli-0.4.1/assets/logo.svg

@@ -0,0 +1,33 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128" role="img" aria-label="Dumpling logo">
+  <defs>
+    <linearGradient id="steam" x1="0%" y1="100%" x2="0%" y2="0%">
+      <stop offset="0%" stop-color="#e8f4fc" stop-opacity="0"/>
+      <stop offset="50%" stop-color="#cfe9fb" stop-opacity="0.85"/>
+      <stop offset="100%" stop-color="#b8dff8" stop-opacity="0"/>
+    </linearGradient>
+    <linearGradient id="dough" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" stop-color="#fff8ef"/>
+      <stop offset="45%" stop-color="#f4dcc4"/>
+      <stop offset="100%" stop-color="#e8b896"/>
+    </linearGradient>
+    <linearGradient id="shadow" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" stop-color="#c4865a" stop-opacity="0.35"/>
+      <stop offset="100%" stop-color="#8b5a3c" stop-opacity="0.15"/>
+    </linearGradient>
+  </defs>
+  <!-- Steam -->
+  <path fill="url(#steam)" d="M44 18c2-6 8-10 14-8s8 10 4 15c-3 4-2 9 2 12 5 4 5 12 0 16-6 5-16 4-20-3-2-4 0-9 4-11 3-2 3-6 0-9-4-4-4-10 0-12z"/>
+  <path fill="url(#steam)" opacity="0.75" d="M64 14c3-5 9-7 14-4 5 3 6 10 2 14-4 4-3 10 2 13 6 4 7 13 1 18-7 6-19 4-24-5-2-5 1-11 6-13 4-2 4-7 1-10-5-4-5-11 0-13z"/>
+  <path fill="url(#steam)" opacity="0.6" d="M82 20c2-5 8-8 13-5 5 3 7 10 3 15-3 4-2 9 3 12 5 3 7 11 2 16-6 6-17 5-22-3-2-4 0-9 5-11 3-2 4-6 1-9-4-4-4-10 0-13z"/>
+  <!-- Plate -->
+  <ellipse cx="64" cy="108" rx="52" ry="10" fill="#dfe8ef"/>
+  <ellipse cx="64" cy="106" rx="48" ry="8" fill="#eef4f8"/>
+  <!-- Dumpling body -->
+  <ellipse cx="64" cy="82" rx="42" ry="28" fill="url(#dough)" stroke="#d4a574" stroke-width="2"/>
+  <ellipse cx="64" cy="96" rx="38" ry="12" fill="url(#shadow)"/>
+  <!-- Pleats -->
+  <path fill="none" stroke="#c9956a" stroke-width="1.8" stroke-linecap="round" d="M34 58c6 10 14 16 30 16s24-6 30-16"/>
+  <path fill="none" stroke="#d9b08a" stroke-width="1.2" stroke-linecap="round" opacity="0.9" d="M42 54c5 8 13 13 22 13s17-5 22-13"/>
+  <!-- Highlight -->
+  <ellipse cx="48" cy="76" rx="10" ry="6" fill="#ffffff" opacity="0.35"/>
+</svg>

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/docs/src/configuration.md

@@ -6,7 +6,7 @@ Use `--format` to declare the SQL dialect of your input file:
 
 | Value | Description |
 |---|---|
-| `postgres` (default) | PostgreSQL `pg_dump` plain-text format. Supports `COPY … FROM stdin` blocks, `"double-quoted"` identifiers, `''`-escaped strings. |
+| `postgres` (default) | PostgreSQL `pg_dump` plain-text format. Supports `COPY … FROM stdin` blocks, `"double-quoted"` identifiers, `''`-escaped strings. Custom-format (`-Fc`) or directory dumps can be decoded on the fly with `dumpling --dump-decode` (wraps `pg_restore -f -`; requires client tools). By default the archive is deleted after success; use `--dump-decode-keep-input` to retain it. |
 | `sqlite` | SQLite `.dump` format. Adds `INSERT OR REPLACE INTO` / `INSERT OR IGNORE INTO` support. No COPY blocks. |
 | `mssql` | SQL Server / MSSQL plain SQL. Adds `[bracket]` identifier quoting, `N'…'` Unicode string literals, and `nvarchar(n)` / `nchar(n)` length extraction. No COPY blocks. |
 
@@ -17,6 +17,28 @@ dumpling --format sqlite -i data.db.sql -o anonymized.sql
 dumpling --format mssql  -i backup.sql  -o anonymized.sql
 ```
 
+### PostgreSQL custom-format archives (`--dump-decode`)
+
+Heroku PGBackups and many pipelines ship **`pg_dump` custom format** (`-Fc`) or **directory-format** dumps to save bandwidth. Dumpling’s SQL engine still expects **plain text**; use **`--dump-decode`** so Dumpling runs **`pg_restore -f -`** (script to stdout, no database) and pipes the result through the same anonymizer as a normal plain-SQL file.
+
+**Requirements:** PostgreSQL client tools on `PATH` (`pg_restore`), or set **`--pg-restore-path`**. Use **`--dump-decode-arg`** (repeatable) for extra `pg_restore` flags, e.g. `--dump-decode-arg=--no-owner --dump-decode-arg=--no-acl`.
+
+**Input deletion:** After a **fully successful** run, Dumpling **removes** the `--input` path (single file or directory-format folder) by default so only the anonymized output remains. Pass **`--dump-decode-keep-input`** to retain the archive.
+
+**Check mode:** **`--check`** with **`--dump-decode`** requires **`--dump-decode-keep-input`**. Otherwise the default would delete the dump before you can iterate on config.
+
+Example (e.g. after `heroku pg:backups:download`):
+
+```bash
+dumpling --dump-decode -i latest.dump -c .dumplingconf -o anonymized.sql
+```
+
+Dry run while keeping the downloaded file:
+
+```bash
+dumpling --dump-decode --dump-decode-keep-input --check -i latest.dump -c .dumplingconf
+```
+
 ---
 
 ## Configuration sources

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/docs/src/getting-started.md

@@ -28,4 +28,6 @@ cargo test --all-targets --all-features
 dumpling -i dump.sql -o sanitized.sql
 ```
 
+If your input is a PostgreSQL **custom-format** file (not plain SQL), decode and anonymize in one step with **`--dump-decode`** (needs `pg_restore` from PostgreSQL client tools). See [PostgreSQL custom-format archives](configuration.md#postgresql-custom-format-archives---dump-decode) in the configuration guide.
+
 For full command examples and strategy options, see the repository `README.md`.

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/docs/src/index.md

@@ -1,6 +1,6 @@
 # Dumpling documentation
 
-Dumpling is a streaming anonymizer for plain SQL dumps. It supports PostgreSQL (`pg_dump` plain format), SQLite (`.dump`), and SQL Server / MSSQL (SSMS / mssql-scripter plain SQL output).
+Dumpling is a streaming anonymizer for plain SQL dumps. It supports PostgreSQL (`pg_dump` plain format), SQLite (`.dump`), and SQL Server / MSSQL (SSMS / mssql-scripter plain SQL output). For PostgreSQL **custom-format** archives (e.g. Heroku `pg:backups:download`), use **`--dump-decode`** so Dumpling invokes `pg_restore` and streams plain SQL—see [Dump format](configuration.html#postgresql-custom-format-archives---dump-decode) in the configuration guide.
 
 This documentation covers the operating model for day-to-day use:
 

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "dumpling-cli"
-version = "0.3.0"
+version = "0.4.1"
 description = "Static anonymizer for plain SQL dumps (PostgreSQL, SQLite, SQL Server)."
 readme = "README.md"
 requires-python = ">=3.8"

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/src/main.rs

@@ -1,6 +1,7 @@
 use std::fs::File;
 use std::io::{self, BufRead, BufReader, BufWriter, Write};
 use std::path::{Path, PathBuf};
+use std::process::{Command, Stdio};
 
 use clap::{ArgAction, Parser, Subcommand};
 
@@ -13,6 +14,7 @@ mod settings;
 mod sql;
 mod transform;
 
+use anyhow::Context;
 use regex::Regex;
 use report::Reporter;
 use scan::{OutputScanner, ScanningWriter};
@@ -105,6 +107,26 @@ struct Cli {
     #[arg(long = "security-profile", default_value = "standard")]
     security_profile: String,
 
+    /// Decode PostgreSQL custom-format or directory-format dumps via `pg_restore -f -` before anonymizing.
+    /// Requires `--input` pointing at the archive file or directory and `--format postgres`. Requires a
+    /// PostgreSQL client install (`pg_restore` on PATH unless overridden by `--pg-restore-path`).
+    #[arg(long = "dump-decode", action = ArgAction::SetTrue)]
+    dump_decode: bool,
+
+    /// Keep the input archive after `--dump-decode` (default: delete file or directory after a fully
+    /// successful run). Cannot retain the archive with `--check` (would delete before verifying changes).
+    #[arg(long = "dump-decode-keep-input", action = ArgAction::SetTrue)]
+    dump_decode_keep_input: bool,
+
+    /// `pg_restore` executable to use with `--dump-decode` (default: `pg_restore` on PATH).
+    #[arg(long = "pg-restore-path", default_value = "pg_restore")]
+    pg_restore_path: PathBuf,
+
+    /// Extra arguments forwarded to `pg_restore` before the archive path (repeatable). Example:
+    /// `--dump-decode-arg=--no-owner` `--dump-decode-arg=--no-acl`
+    #[arg(long = "dump-decode-arg")]
+    dump_decode_arg: Vec<String>,
+
     #[command(subcommand)]
     command: Option<Commands>,
 }
@@ -184,6 +206,14 @@ fn run_anonymize(cli: Cli) -> anyhow::Result<()> {
     if cli.check && (cli.in_place || cli.output.is_some()) {
         anyhow::bail!("--check cannot be used together with --output or --in-place");
     }
+    if cli.dump_decode && !cli.dump_decode_keep_input && cli.check {
+        anyhow::bail!(
+            "--dump-decode removes the input archive on success by default; use --dump-decode-keep-input with --check"
+        );
+    }
+    if cli.dump_decode && cli.in_place {
+        anyhow::bail!("--dump-decode cannot be used with --in-place");
+    }
 
     // Resolve config from provided path or discover in CWD
     let resolved_config: ResolvedConfig =
@@ -247,36 +277,97 @@ fn run_anonymize(cli: Cli) -> anyhow::Result<()> {
             other
         ),
     };
+    if cli.dump_decode && dump_format != DumpFormat::Postgres {
+        anyhow::bail!(
+            "--dump-decode only applies to PostgreSQL dumps; use --format postgres (default)"
+        );
+    }
 
     // Compile table include/exclude regex patterns
     let include_res = compile_patterns(&cli.include_table)?;
     let exclude_res = compile_patterns(&cli.exclude_table)?;
 
-    // Determine IO
-    let (mut reader, input_path_for_inplace): (Box<dyn BufRead>, Option<PathBuf>) = match &cli.input
+    // Determine IO (optional pg_restore child when --dump-decode)
+    let mut pg_restore_child: Option<std::process::Child> = None;
+    let (mut reader, input_path_for_inplace): (Box<dyn BufRead>, Option<PathBuf>) = if cli
+        .dump_decode
     {
-        Some(path) => {
-            // Enforce extension allowlist if provided
-            if !cli.allow_ext.is_empty() && !has_allowed_extension(path, &cli.allow_ext) {
-                let actual = path
-                    .extension()
-                    .and_then(|s| s.to_str())
-                    .unwrap_or("<none>")
-                    .to_string();
-                anyhow::bail!(
-                    "input file extension '{}' is not in allowed set {:?}",
-                    actual,
-                    cli.allow_ext
-                );
-            }
-            let f = File::open(path)?;
-            (Box::new(BufReader::new(f)), Some(path.clone()))
+        let archive_path = cli
+                .input
+                .as_ref()
+                .ok_or_else(|| {
+                    anyhow::anyhow!(
+                        "--dump-decode requires --input pointing at a pg_dump custom-format file or directory-format directory"
+                    )
+                })?;
+        if !cli.allow_ext.is_empty() && !has_allowed_extension(archive_path, &cli.allow_ext) {
+            let actual = archive_path
+                .extension()
+                .and_then(|s| s.to_str())
+                .unwrap_or("<none>")
+                .to_string();
+            anyhow::bail!(
+                "input file extension '{}' is not in allowed set {:?}",
+                actual,
+                cli.allow_ext
+            );
+        }
+        if !archive_path.exists() {
+            anyhow::bail!(
+                "--dump-decode input path does not exist: {}",
+                archive_path.display()
+            );
+        }
+        eprintln!(
+            "dumpling: decoding PostgreSQL archive via {} -f - {}",
+            cli.pg_restore_path.display(),
+            archive_path.display()
+        );
+        let mut cmd = Command::new(&cli.pg_restore_path);
+        for a in &cli.dump_decode_arg {
+            cmd.arg(a);
         }
-        None => {
-            if !cli.allow_ext.is_empty() {
-                eprintln!("dumpling: --allow-ext provided but no --input file; extension check is ignored for stdin");
+        cmd.arg("-f")
+            .arg("-")
+            .arg(archive_path)
+            .stdout(Stdio::piped())
+            .stderr(Stdio::inherit());
+        let mut child = cmd.spawn().with_context(|| {
+            format!(
+                "failed to spawn `{}`; install PostgreSQL client tools or set --pg-restore-path",
+                cli.pg_restore_path.display()
+            )
+        })?;
+        let stdout = child
+            .stdout
+            .take()
+            .ok_or_else(|| anyhow::anyhow!("pg_restore stdout missing"))?;
+        pg_restore_child = Some(child);
+        (Box::new(BufReader::new(stdout)), Some(archive_path.clone()))
+    } else {
+        match &cli.input {
+            Some(path) => {
+                if !cli.allow_ext.is_empty() && !has_allowed_extension(path, &cli.allow_ext) {
+                    let actual = path
+                        .extension()
+                        .and_then(|s| s.to_str())
+                        .unwrap_or("<none>")
+                        .to_string();
+                    anyhow::bail!(
+                        "input file extension '{}' is not in allowed set {:?}",
+                        actual,
+                        cli.allow_ext
+                    );
+                }
+                let f = File::open(path)?;
+                (Box::new(BufReader::new(f)), Some(path.clone()))
+            }
+            None => {
+                if !cli.allow_ext.is_empty() {
+                    eprintln!("dumpling: --allow-ext provided but no --input file; extension check is ignored for stdin");
+                }
+                (Box::new(BufReader::new(io::stdin())), None)
             }
-            (Box::new(BufReader::new(io::stdin())), None)
         }
     };
 
@@ -330,12 +421,30 @@ fn run_anonymize(cli: Cli) -> anyhow::Result<()> {
         dump_format,
     );
     let mut writer = output;
-    if let Some(scanner) = output_scanner.as_mut() {
+    let proc_res = if let Some(scanner) = output_scanner.as_mut() {
         let mut scanning_writer = ScanningWriter::new(&mut writer, scanner);
-        processor.process(&mut reader, &mut scanning_writer)?;
+        processor.process(&mut reader, &mut scanning_writer)
     } else {
-        processor.process(&mut reader, &mut writer)?;
+        processor.process(&mut reader, &mut writer)
+    };
+
+    if let Some(mut child) = pg_restore_child {
+        if proc_res.is_err() {
+            let _ = child.kill();
+        }
+        let status = child
+            .wait()
+            .with_context(|| format!("waiting for `{}`", cli.pg_restore_path.display()))?;
+        if proc_res.is_ok() && !status.success() {
+            anyhow::bail!(
+                "`{}` exited with status {}",
+                cli.pg_restore_path.display(),
+                status
+            );
+        }
     }
+
+    proc_res?;
     let coverage = processor.sensitive_coverage_summary();
     reporter.report.sensitive_columns_detected = coverage.detected.clone();
     reporter.report.sensitive_columns_covered = coverage.covered.clone();
@@ -363,7 +472,10 @@ fn run_anonymize(cli: Cli) -> anyhow::Result<()> {
 
     // If in-place, do the swap now
     if cli.in_place {
-        let input_path = input_path_for_inplace.unwrap();
+        let input_path = input_path_for_inplace
+            .as_ref()
+            .ok_or_else(|| anyhow::anyhow!("--in-place requires an --input path"))?
+            .clone();
         let mut tmp = input_path.clone();
         tmp.set_extension("sql.dumpling.tmp");
         writer.flush()?;
@@ -405,9 +517,30 @@ fn run_anonymize(cli: Cli) -> anyhow::Result<()> {
         std::process::exit(1);
     }
 
+    if cli.dump_decode && !cli.dump_decode_keep_input {
+        if let Some(ref p) = input_path_for_inplace {
+            match remove_pg_archive(p) {
+                Ok(()) => eprintln!("dumpling: removed input archive {}", p.display()),
+                Err(e) => eprintln!(
+                    "dumpling: warning: could not remove input archive {}: {}",
+                    p.display(),
+                    e
+                ),
+            }
+        }
+    }
+
     Ok(())
 }
 
+fn remove_pg_archive(path: &Path) -> std::io::Result<()> {
+    if path.is_dir() {
+        std::fs::remove_dir_all(path)
+    } else {
+        std::fs::remove_file(path)
+    }
+}
+
 fn compile_patterns(patterns: &[String]) -> anyhow::Result<Vec<Regex>> {
     let mut out = Vec::new();
     for p in patterns {
@@ -494,6 +627,24 @@ mod tests_main {
         }
     }
 
+    #[test]
+    fn test_dump_decode_flags_parse() {
+        let cli = Cli::parse_from([
+            "dumpling",
+            "--dump-decode",
+            "--dump-decode-keep-input",
+            "--pg-restore-path",
+            "/usr/bin/pg_restore",
+            "--dump-decode-arg=--no-owner",
+            "-i",
+            "/tmp/latest.dump",
+        ]);
+        assert!(cli.dump_decode);
+        assert!(cli.dump_decode_keep_input);
+        assert_eq!(cli.pg_restore_path, PathBuf::from("/usr/bin/pg_restore"));
+        assert_eq!(cli.dump_decode_arg, vec!["--no-owner"]);
+    }
+
     #[test]
     fn test_lint_policy_allow_noop_flag() {
         let cli = Cli::parse_from(["dumpling", "lint-policy", "--allow-noop"]);

{dumpling_cli-0.3.0 → dumpling_cli-0.4.1}/src/sql.rs

@@ -1155,20 +1155,22 @@ struct Cell {
     original: Option<String>, // None for NULL
     was_quoted: bool,
     was_default: bool,
+    trailing_expr: Option<String>,
 }
 
 impl Cell {
     fn render_original(&self) -> String {
+        let trailing = self.trailing_expr.as_deref().unwrap_or("");
         if self.was_default {
-            return "DEFAULT".to_string();
+            return format!("DEFAULT{trailing}");
         }
         match &self.original {
-            None => "NULL".to_string(),
+            None => format!("NULL{trailing}"),
             Some(s) => {
                 if self.was_quoted {
-                    format!("'{}'", s.replace('\'', "''"))
+                    format!("'{}'{trailing}", s.replace('\'', "''"))
                 } else {
-                    s.clone()
+                    format!("{s}{trailing}")
                 }
             }
         }
@@ -1176,14 +1178,15 @@ impl Cell {
 }
 
 fn render_cell(repl: &Replacement, original: &Cell) -> String {
+    let trailing = original.trailing_expr.as_deref().unwrap_or("");
     if repl.is_null {
-        return "NULL".to_string();
+        return format!("NULL{trailing}");
     }
     let should_quote = repl.force_quoted || original.was_quoted;
     if should_quote {
-        format!("'{}'", repl.value.replace('\'', "''"))
+        format!("'{}'{trailing}", repl.value.replace('\'', "''"))
     } else {
-        repl.value.clone()
+        format!("{}{trailing}", repl.value)
     }
 }
 
@@ -1243,7 +1246,9 @@ fn parse_parenthesized_values(s: &str) -> anyhow::Result<(Vec<Cell>, usize)> {
     let mut cells: Vec<Cell> = Vec::new();
     let mut in_single = false;
     let mut buf = String::new();
+    let mut trailing_expr = String::new();
     let mut was_quoted = false;
+    let mut closed_quoted_literal = false;
     while i < chs.len() {
         let c = chs[i];
         if in_single {
@@ -1255,6 +1260,7 @@ fn parse_parenthesized_values(s: &str) -> anyhow::Result<(Vec<Cell>, usize)> {
                     continue;
                 } else {
                     in_single = false;
+                    closed_quoted_literal = true;
                     i += 1;
                     continue;
                 }
@@ -1282,17 +1288,19 @@ fn parse_parenthesized_values(s: &str) -> anyhow::Result<(Vec<Cell>, usize)> {
                 }
                 ')' => {
                     // end cell, end row
-                    let cell = finalize_cell(&buf, was_quoted);
+                    let cell = finalize_cell(&buf, was_quoted, &trailing_expr);
                     cells.push(cell);
                     i += 1;
                     return Ok((cells, i));
                 }
                 ',' => {
                     // end cell
-                    let cell = finalize_cell(&buf, was_quoted);
+                    let cell = finalize_cell(&buf, was_quoted, &trailing_expr);
                     cells.push(cell);
                     buf.clear();
+                    trailing_expr.clear();
                     was_quoted = false;
+                    closed_quoted_literal = false;
                     i += 1;
                     // consume following spaces
                     while i < chs.len() && chs[i].is_whitespace() {
@@ -1300,11 +1308,19 @@ fn parse_parenthesized_values(s: &str) -> anyhow::Result<(Vec<Cell>, usize)> {
                     }
                 }
                 c if c.is_whitespace() => {
-                    // skip insignificant whitespace between tokens when unquoted
+                    // Preserve whitespace after a quoted literal so explicit SQL casts stay intact.
+                    if was_quoted && closed_quoted_literal {
+                        trailing_expr.push(c);
+                    }
+                    // Skip insignificant whitespace between tokens when unquoted.
                     i += 1;
                 }
                 other => {
-                    buf.push(other);
+                    if was_quoted && closed_quoted_literal {
+                        trailing_expr.push(other);
+                    } else {
+                        buf.push(other);
+                    }
                     i += 1;
                 }
             }
@@ -1313,12 +1329,21 @@ fn parse_parenthesized_values(s: &str) -> anyhow::Result<(Vec<Cell>, usize)> {
     anyhow::bail!("unterminated values row")
 }
 
-fn finalize_cell(buf: &str, was_quoted: bool) -> Cell {
+fn finalize_cell(buf: &str, was_quoted: bool, trailing_expr: &str) -> Cell {
+    let trailing = {
+        let t = trailing_expr.trim();
+        if t.is_empty() {
+            None
+        } else {
+            Some(t.to_string())
+        }
+    };
     if was_quoted {
         Cell {
             original: Some(buf.to_string()),
             was_quoted: true,
             was_default: false,
+            trailing_expr: trailing,
         }
     } else {
         let t = buf.trim();
@@ -1327,18 +1352,21 @@ fn finalize_cell(buf: &str, was_quoted: bool) -> Cell {
                 original: None,
                 was_quoted: false,
                 was_default: false,
+                trailing_expr: None,
             }
         } else if t.eq_ignore_ascii_case("default") {
             Cell {
                 original: None,
                 was_quoted: false,
                 was_default: true,
+                trailing_expr: None,
             }
         } else {
             Cell {
                 original: Some(t.to_string()),
                 was_quoted: false,
                 was_default: false,
+                trailing_expr: None,
             }
         }
     }
@@ -2370,6 +2398,90 @@ COPY public.events (id, payload) FROM stdin;
         );
     }
 
+    #[test]
+    fn parse_values_rows_tracks_trailing_cast_for_quoted_literals() {
+        let rows =
+            parse_values_rows("(1, '{\"profile\":{\"secret\":\"alpha\"}}'::jsonb, 'note'::text)")
+                .unwrap();
+        assert_eq!(rows.len(), 1);
+        assert_eq!(rows[0].len(), 3);
+        assert_eq!(
+            rows[0][1].original.as_deref(),
+            Some("{\"profile\":{\"secret\":\"alpha\"}}")
+        );
+        assert_eq!(rows[0][1].trailing_expr.as_deref(), Some("::jsonb"));
+        assert_eq!(rows[0][2].original.as_deref(), Some("note"));
+        assert_eq!(rows[0][2].trailing_expr.as_deref(), Some("::text"));
+    }
+
+    #[test]
+    fn pipeline_anonymizes_nested_json_paths_for_jsonb_cast_insert_rows() {
+        let mut rules: HashMap<String, HashMap<String, AnonymizerSpec>> = HashMap::new();
+        let mut cols: HashMap<String, AnonymizerSpec> = HashMap::new();
+        cols.insert(
+            "payload.profile.secret".to_string(),
+            AnonymizerSpec {
+                strategy: "string".to_string(),
+                salt: None,
+                min: None,
+                max: None,
+                length: Some(8),
+                min_days: None,
+                max_days: None,
+                min_seconds: None,
+                max_seconds: None,
+                domain: Some("secrets".to_string()),
+                unique_within_domain: None,
+                as_string: Some(true),
+                locale: None,
+                faker: None,
+                format: None,
+            },
+        );
+        rules.insert("public.events".to_string(), cols);
+        let cfg = ResolvedConfig {
+            salt: None,
+            rules,
+            row_filters: HashMap::new(),
+            column_cases: HashMap::new(),
+            sensitive_columns: HashMap::new(),
+            output_scan: crate::settings::OutputScanConfig::default(),
+            source_path: None,
+        };
+        let reg = AnonymizerRegistry::from_config(&cfg);
+        let mut proc =
+            SqlStreamProcessor::new(reg, cfg, Vec::new(), Vec::new(), None, DumpFormat::Postgres);
+        let input = r#"
+CREATE TABLE public.events (id int, payload jsonb);
+INSERT INTO public.events (id, payload) VALUES
+  (1, '{"profile":{"tier":"gold","secret":"alpha"}}'::jsonb),
+  (2, '{"profile":{"tier":"gold","secret":"alpha"}}'::jsonb);
+"#;
+        let mut reader = std::io::BufReader::new(input.as_bytes());
+        let mut out = Vec::new();
+        proc.process(&mut reader, &mut out).unwrap();
+        let s = String::from_utf8(out).unwrap();
+        assert!(!s.contains("alpha"), "nested secret should be anonymized");
+        assert!(s.contains("::jsonb"), "jsonb cast should be preserved");
+
+        let insert_pos = s.find("INSERT INTO public.events").unwrap();
+        let insert_tail = &s[insert_pos..];
+        let insert_end = insert_tail.find(";\n").unwrap() + insert_pos;
+        let ins_stmt = &s[insert_pos..=insert_end];
+        let vals_idx = ins_stmt.to_uppercase().find("VALUES").unwrap();
+        let ins_block = strip_trailing_semicolon(ins_stmt[vals_idx + "VALUES".len()..].trim());
+        let ins_rows = parse_values_rows(ins_block).unwrap();
+        assert_eq!(ins_rows[0][1].trailing_expr.as_deref(), Some("::jsonb"));
+        assert_eq!(ins_rows[1][1].trailing_expr.as_deref(), Some("::jsonb"));
+        let v0 =
+            serde_json::from_str::<serde_json::Value>(ins_rows[0][1].original.as_ref().unwrap())
+                .unwrap();
+        let v1 =
+            serde_json::from_str::<serde_json::Value>(ins_rows[1][1].original.as_ref().unwrap())
+                .unwrap();
+        assert_eq!(v0["profile"]["secret"], v1["profile"]["secret"]);
+    }
+
     #[test]
     fn generated_values_fit_length_restricted_columns_from_create_table() {
         let mut rules: HashMap<String, HashMap<String, AnonymizerSpec>> = HashMap::new();