dumpling-cli 0.3.0__tar.gz → 0.4.0__tar.gz

{dumpling_cli-0.3.0 → dumpling_cli-0.4.0}/CHANGELOG.md

@@ -7,6 +7,18 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-05-02
+
+### Added
+
+- **`--dump-decode` CLI**: Decode PostgreSQL **custom-format** (`pg_dump -Fc`) or **directory-format** archives by running **`pg_restore -f -`** (plain SQL to stdout, no database), then anonymize—built for workflows such as **`heroku pg:backups:download`**. Requires PostgreSQL client tools (`pg_restore` on `PATH`, or **`--pg-restore-path`**).
+- **`--dump-decode-arg`** (repeatable): Extra arguments forwarded to `pg_restore`.
+- **`--dump-decode-keep-input`**: Keep the archive after a successful run. **By default** the `--input` path is **removed** after success so only anonymized output remains. **`--check`** with **`--dump-decode`** requires **`--dump-decode-keep-input`** (otherwise the dump would be deleted before config iteration).
+
+### Changed
+
+- README and mdBook documentation for PostgreSQL archive decoding and Heroku-style examples.
+
 ## [0.3.0] - 2026-05-02
 
 ### Added
@@ -43,5 +55,6 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
 - Configurable output scan severities and per-category thresholds via `[output_scan]`.
 - JSON report section for output scan findings including category, count, threshold, severity, and sample locations.
 
+[0.4.0]: https://github.com/ababic/dumpling/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/ababic/dumpling/compare/v0.2.0...v0.3.0
 [0.2.0]: https://github.com/ababic/dumpling/compare/v0.1.0...v0.2.0

{dumpling_cli-0.3.0 → dumpling_cli-0.4.0}/Cargo.lock

@@ -262,7 +262,7 @@ dependencies = [
 
 [[package]]
 name = "dumpling"
-version = "0.3.0"
+version = "0.4.0"
 dependencies = [
  "anyhow",
  "chrono",

{dumpling_cli-0.3.0 → dumpling_cli-0.4.0}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "dumpling"
-version = "0.3.0"
+version = "0.4.0"
 edition = "2021"
 readme = "README.md"
 

{dumpling_cli-0.3.0 → dumpling_cli-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dumpling-cli
-Version: 0.3.0
+Version: 0.4.0
 Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
@@ -289,7 +289,16 @@ Produced by `pg_dump --format=plain`. Handles:
 - `"double-quoted"` identifiers
 - `''`-escaped string literals
 
-Binary, custom, and directory formats from `pg_dump` are not supported — use `--format=plain` when running `pg_dump`.
+Binary, custom, and directory formats from `pg_dump` are not parsed directly — Dumpling’s SQL pipeline expects plain text. Use either:
+
+- **`pg_dump --format=plain`** when you control capture, or
+- **`dumpling --dump-decode`** with `--input` set to a **custom-format** (`.dump`) or **directory-format** folder: Dumpling runs `pg_restore -f -` and streams the resulting SQL (same as a manual `pg_restore` “script” output, no database required). Requires PostgreSQL client tools on `PATH` (`pg_restore`), or set `--pg-restore-path`. Use `--dump-decode-arg` to pass extra flags (e.g. `--no-owner --no-acl`). **By default** the archive is removed after a fully successful run; pass **`--dump-decode-keep-input`** to retain it. **`--check`** requires **`--dump-decode-keep-input`** so the archive still exists if changes would be detected.
+
+Example (e.g. after `heroku pg:backups:download`):
+
+```bash
+dumpling --dump-decode -i latest.dump -c .dumplingconf -o anonymized.sql
+```
 
 ### SQLite (`--format sqlite`)
 
@@ -477,5 +486,5 @@ See the [CI guardrails documentation](docs/src/ci-guardrails.md) for full pipeli
 
 ## Full documentation
 
-Detailed docs, including the configuration reference and release process, are available at the project's [GitHub Pages site](https://github.com) (built from `docs/src/`).
+Detailed docs, including the configuration reference and release process, are available at the project's [GitHub Pages site](https://ababic.github.io/dumpling/) (built from `docs/src/`).
 

{dumpling_cli-0.3.0 → dumpling_cli-0.4.0}/README.md

@@ -268,7 +268,16 @@ Produced by `pg_dump --format=plain`. Handles:
 - `"double-quoted"` identifiers
 - `''`-escaped string literals
 
-Binary, custom, and directory formats from `pg_dump` are not supported — use `--format=plain` when running `pg_dump`.
+Binary, custom, and directory formats from `pg_dump` are not parsed directly — Dumpling’s SQL pipeline expects plain text. Use either:
+
+- **`pg_dump --format=plain`** when you control capture, or
+- **`dumpling --dump-decode`** with `--input` set to a **custom-format** (`.dump`) or **directory-format** folder: Dumpling runs `pg_restore -f -` and streams the resulting SQL (same as a manual `pg_restore` “script” output, no database required). Requires PostgreSQL client tools on `PATH` (`pg_restore`), or set `--pg-restore-path`. Use `--dump-decode-arg` to pass extra flags (e.g. `--no-owner --no-acl`). **By default** the archive is removed after a fully successful run; pass **`--dump-decode-keep-input`** to retain it. **`--check`** requires **`--dump-decode-keep-input`** so the archive still exists if changes would be detected.
+
+Example (e.g. after `heroku pg:backups:download`):
+
+```bash
+dumpling --dump-decode -i latest.dump -c .dumplingconf -o anonymized.sql
+```
 
 ### SQLite (`--format sqlite`)
 
@@ -456,4 +465,4 @@ See the [CI guardrails documentation](docs/src/ci-guardrails.md) for full pipeli
 
 ## Full documentation
 
-Detailed docs, including the configuration reference and release process, are available at the project's [GitHub Pages site](https://github.com) (built from `docs/src/`).
+Detailed docs, including the configuration reference and release process, are available at the project's [GitHub Pages site](https://ababic.github.io/dumpling/) (built from `docs/src/`).

{dumpling_cli-0.3.0 → dumpling_cli-0.4.0}/docs/src/configuration.md

@@ -6,7 +6,7 @@ Use `--format` to declare the SQL dialect of your input file:
 
 | Value | Description |
 |---|---|
-| `postgres` (default) | PostgreSQL `pg_dump` plain-text format. Supports `COPY … FROM stdin` blocks, `"double-quoted"` identifiers, `''`-escaped strings. |
+| `postgres` (default) | PostgreSQL `pg_dump` plain-text format. Supports `COPY … FROM stdin` blocks, `"double-quoted"` identifiers, `''`-escaped strings. Custom-format (`-Fc`) or directory dumps can be decoded on the fly with `dumpling --dump-decode` (wraps `pg_restore -f -`; requires client tools). By default the archive is deleted after success; use `--dump-decode-keep-input` to retain it. |
 | `sqlite` | SQLite `.dump` format. Adds `INSERT OR REPLACE INTO` / `INSERT OR IGNORE INTO` support. No COPY blocks. |
 | `mssql` | SQL Server / MSSQL plain SQL. Adds `[bracket]` identifier quoting, `N'…'` Unicode string literals, and `nvarchar(n)` / `nchar(n)` length extraction. No COPY blocks. |
 
@@ -17,6 +17,28 @@ dumpling --format sqlite -i data.db.sql -o anonymized.sql
 dumpling --format mssql  -i backup.sql  -o anonymized.sql
 ```
 
+### PostgreSQL custom-format archives (`--dump-decode`)
+
+Heroku PGBackups and many pipelines ship **`pg_dump` custom format** (`-Fc`) or **directory-format** dumps to save bandwidth. Dumpling’s SQL engine still expects **plain text**; use **`--dump-decode`** so Dumpling runs **`pg_restore -f -`** (script to stdout, no database) and pipes the result through the same anonymizer as a normal plain-SQL file.
+
+**Requirements:** PostgreSQL client tools on `PATH` (`pg_restore`), or set **`--pg-restore-path`**. Use **`--dump-decode-arg`** (repeatable) for extra `pg_restore` flags, e.g. `--dump-decode-arg=--no-owner --dump-decode-arg=--no-acl`.
+
+**Input deletion:** After a **fully successful** run, Dumpling **removes** the `--input` path (single file or directory-format folder) by default so only the anonymized output remains. Pass **`--dump-decode-keep-input`** to retain the archive.
+
+**Check mode:** **`--check`** with **`--dump-decode`** requires **`--dump-decode-keep-input`**. Otherwise the default would delete the dump before you can iterate on config.
+
+Example (e.g. after `heroku pg:backups:download`):
+
+```bash
+dumpling --dump-decode -i latest.dump -c .dumplingconf -o anonymized.sql
+```
+
+Dry run while keeping the downloaded file:
+
+```bash
+dumpling --dump-decode --dump-decode-keep-input --check -i latest.dump -c .dumplingconf
+```
+
 ---
 
 ## Configuration sources

{dumpling_cli-0.3.0 → dumpling_cli-0.4.0}/docs/src/getting-started.md

@@ -28,4 +28,6 @@ cargo test --all-targets --all-features
 dumpling -i dump.sql -o sanitized.sql
 ```
 
+If your input is a PostgreSQL **custom-format** file (not plain SQL), decode and anonymize in one step with **`--dump-decode`** (needs `pg_restore` from PostgreSQL client tools). See [PostgreSQL custom-format archives](configuration.md#postgresql-custom-format-archives---dump-decode) in the configuration guide.
+
 For full command examples and strategy options, see the repository `README.md`.

{dumpling_cli-0.3.0 → dumpling_cli-0.4.0}/docs/src/index.md

@@ -1,6 +1,6 @@
 # Dumpling documentation
 
-Dumpling is a streaming anonymizer for plain SQL dumps. It supports PostgreSQL (`pg_dump` plain format), SQLite (`.dump`), and SQL Server / MSSQL (SSMS / mssql-scripter plain SQL output).
+Dumpling is a streaming anonymizer for plain SQL dumps. It supports PostgreSQL (`pg_dump` plain format), SQLite (`.dump`), and SQL Server / MSSQL (SSMS / mssql-scripter plain SQL output). For PostgreSQL **custom-format** archives (e.g. Heroku `pg:backups:download`), use **`--dump-decode`** so Dumpling invokes `pg_restore` and streams plain SQL—see [Dump format](configuration.html#postgresql-custom-format-archives---dump-decode) in the configuration guide.
 
 This documentation covers the operating model for day-to-day use:
 

{dumpling_cli-0.3.0 → dumpling_cli-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "dumpling-cli"
-version = "0.3.0"
+version = "0.4.0"
 description = "Static anonymizer for plain SQL dumps (PostgreSQL, SQLite, SQL Server)."
 readme = "README.md"
 requires-python = ">=3.8"

{dumpling_cli-0.3.0 → dumpling_cli-0.4.0}/src/main.rs

@@ -1,6 +1,7 @@
 use std::fs::File;
 use std::io::{self, BufRead, BufReader, BufWriter, Write};
 use std::path::{Path, PathBuf};
+use std::process::{Command, Stdio};
 
 use clap::{ArgAction, Parser, Subcommand};
 
@@ -13,6 +14,7 @@ mod settings;
 mod sql;
 mod transform;
 
+use anyhow::Context;
 use regex::Regex;
 use report::Reporter;
 use scan::{OutputScanner, ScanningWriter};
@@ -105,6 +107,26 @@ struct Cli {
     #[arg(long = "security-profile", default_value = "standard")]
     security_profile: String,
 
+    /// Decode PostgreSQL custom-format or directory-format dumps via `pg_restore -f -` before anonymizing.
+    /// Requires `--input` pointing at the archive file or directory and `--format postgres`. Requires a
+    /// PostgreSQL client install (`pg_restore` on PATH unless overridden by `--pg-restore-path`).
+    #[arg(long = "dump-decode", action = ArgAction::SetTrue)]
+    dump_decode: bool,
+
+    /// Keep the input archive after `--dump-decode` (default: delete file or directory after a fully
+    /// successful run). Cannot retain the archive with `--check` (would delete before verifying changes).
+    #[arg(long = "dump-decode-keep-input", action = ArgAction::SetTrue)]
+    dump_decode_keep_input: bool,
+
+    /// `pg_restore` executable to use with `--dump-decode` (default: `pg_restore` on PATH).
+    #[arg(long = "pg-restore-path", default_value = "pg_restore")]
+    pg_restore_path: PathBuf,
+
+    /// Extra arguments forwarded to `pg_restore` before the archive path (repeatable). Example:
+    /// `--dump-decode-arg=--no-owner` `--dump-decode-arg=--no-acl`
+    #[arg(long = "dump-decode-arg")]
+    dump_decode_arg: Vec<String>,
+
     #[command(subcommand)]
     command: Option<Commands>,
 }
@@ -184,6 +206,14 @@ fn run_anonymize(cli: Cli) -> anyhow::Result<()> {
     if cli.check && (cli.in_place || cli.output.is_some()) {
         anyhow::bail!("--check cannot be used together with --output or --in-place");
     }
+    if cli.dump_decode && !cli.dump_decode_keep_input && cli.check {
+        anyhow::bail!(
+            "--dump-decode removes the input archive on success by default; use --dump-decode-keep-input with --check"
+        );
+    }
+    if cli.dump_decode && cli.in_place {
+        anyhow::bail!("--dump-decode cannot be used with --in-place");
+    }
 
     // Resolve config from provided path or discover in CWD
     let resolved_config: ResolvedConfig =
@@ -247,36 +277,97 @@ fn run_anonymize(cli: Cli) -> anyhow::Result<()> {
             other
         ),
     };
+    if cli.dump_decode && dump_format != DumpFormat::Postgres {
+        anyhow::bail!(
+            "--dump-decode only applies to PostgreSQL dumps; use --format postgres (default)"
+        );
+    }
 
     // Compile table include/exclude regex patterns
     let include_res = compile_patterns(&cli.include_table)?;
     let exclude_res = compile_patterns(&cli.exclude_table)?;
 
-    // Determine IO
-    let (mut reader, input_path_for_inplace): (Box<dyn BufRead>, Option<PathBuf>) = match &cli.input
+    // Determine IO (optional pg_restore child when --dump-decode)
+    let mut pg_restore_child: Option<std::process::Child> = None;
+    let (mut reader, input_path_for_inplace): (Box<dyn BufRead>, Option<PathBuf>) = if cli
+        .dump_decode
     {
-        Some(path) => {
-            // Enforce extension allowlist if provided
-            if !cli.allow_ext.is_empty() && !has_allowed_extension(path, &cli.allow_ext) {
-                let actual = path
-                    .extension()
-                    .and_then(|s| s.to_str())
-                    .unwrap_or("<none>")
-                    .to_string();
-                anyhow::bail!(
-                    "input file extension '{}' is not in allowed set {:?}",
-                    actual,
-                    cli.allow_ext
-                );
-            }
-            let f = File::open(path)?;
-            (Box::new(BufReader::new(f)), Some(path.clone()))
+        let archive_path = cli
+                .input
+                .as_ref()
+                .ok_or_else(|| {
+                    anyhow::anyhow!(
+                        "--dump-decode requires --input pointing at a pg_dump custom-format file or directory-format directory"
+                    )
+                })?;
+        if !cli.allow_ext.is_empty() && !has_allowed_extension(archive_path, &cli.allow_ext) {
+            let actual = archive_path
+                .extension()
+                .and_then(|s| s.to_str())
+                .unwrap_or("<none>")
+                .to_string();
+            anyhow::bail!(
+                "input file extension '{}' is not in allowed set {:?}",
+                actual,
+                cli.allow_ext
+            );
+        }
+        if !archive_path.exists() {
+            anyhow::bail!(
+                "--dump-decode input path does not exist: {}",
+                archive_path.display()
+            );
+        }
+        eprintln!(
+            "dumpling: decoding PostgreSQL archive via {} -f - {}",
+            cli.pg_restore_path.display(),
+            archive_path.display()
+        );
+        let mut cmd = Command::new(&cli.pg_restore_path);
+        for a in &cli.dump_decode_arg {
+            cmd.arg(a);
         }
-        None => {
-            if !cli.allow_ext.is_empty() {
-                eprintln!("dumpling: --allow-ext provided but no --input file; extension check is ignored for stdin");
+        cmd.arg("-f")
+            .arg("-")
+            .arg(archive_path)
+            .stdout(Stdio::piped())
+            .stderr(Stdio::inherit());
+        let mut child = cmd.spawn().with_context(|| {
+            format!(
+                "failed to spawn `{}`; install PostgreSQL client tools or set --pg-restore-path",
+                cli.pg_restore_path.display()
+            )
+        })?;
+        let stdout = child
+            .stdout
+            .take()
+            .ok_or_else(|| anyhow::anyhow!("pg_restore stdout missing"))?;
+        pg_restore_child = Some(child);
+        (Box::new(BufReader::new(stdout)), Some(archive_path.clone()))
+    } else {
+        match &cli.input {
+            Some(path) => {
+                if !cli.allow_ext.is_empty() && !has_allowed_extension(path, &cli.allow_ext) {
+                    let actual = path
+                        .extension()
+                        .and_then(|s| s.to_str())
+                        .unwrap_or("<none>")
+                        .to_string();
+                    anyhow::bail!(
+                        "input file extension '{}' is not in allowed set {:?}",
+                        actual,
+                        cli.allow_ext
+                    );
+                }
+                let f = File::open(path)?;
+                (Box::new(BufReader::new(f)), Some(path.clone()))
+            }
+            None => {
+                if !cli.allow_ext.is_empty() {
+                    eprintln!("dumpling: --allow-ext provided but no --input file; extension check is ignored for stdin");
+                }
+                (Box::new(BufReader::new(io::stdin())), None)
             }
-            (Box::new(BufReader::new(io::stdin())), None)
         }
     };
 
@@ -330,12 +421,30 @@ fn run_anonymize(cli: Cli) -> anyhow::Result<()> {
         dump_format,
     );
     let mut writer = output;
-    if let Some(scanner) = output_scanner.as_mut() {
+    let proc_res = if let Some(scanner) = output_scanner.as_mut() {
         let mut scanning_writer = ScanningWriter::new(&mut writer, scanner);
-        processor.process(&mut reader, &mut scanning_writer)?;
+        processor.process(&mut reader, &mut scanning_writer)
     } else {
-        processor.process(&mut reader, &mut writer)?;
+        processor.process(&mut reader, &mut writer)
+    };
+
+    if let Some(mut child) = pg_restore_child {
+        if proc_res.is_err() {
+            let _ = child.kill();
+        }
+        let status = child
+            .wait()
+            .with_context(|| format!("waiting for `{}`", cli.pg_restore_path.display()))?;
+        if proc_res.is_ok() && !status.success() {
+            anyhow::bail!(
+                "`{}` exited with status {}",
+                cli.pg_restore_path.display(),
+                status
+            );
+        }
     }
+
+    proc_res?;
     let coverage = processor.sensitive_coverage_summary();
     reporter.report.sensitive_columns_detected = coverage.detected.clone();
     reporter.report.sensitive_columns_covered = coverage.covered.clone();
@@ -363,7 +472,10 @@ fn run_anonymize(cli: Cli) -> anyhow::Result<()> {
 
     // If in-place, do the swap now
     if cli.in_place {
-        let input_path = input_path_for_inplace.unwrap();
+        let input_path = input_path_for_inplace
+            .as_ref()
+            .ok_or_else(|| anyhow::anyhow!("--in-place requires an --input path"))?
+            .clone();
         let mut tmp = input_path.clone();
         tmp.set_extension("sql.dumpling.tmp");
         writer.flush()?;
@@ -405,9 +517,30 @@ fn run_anonymize(cli: Cli) -> anyhow::Result<()> {
         std::process::exit(1);
     }
 
+    if cli.dump_decode && !cli.dump_decode_keep_input {
+        if let Some(ref p) = input_path_for_inplace {
+            match remove_pg_archive(p) {
+                Ok(()) => eprintln!("dumpling: removed input archive {}", p.display()),
+                Err(e) => eprintln!(
+                    "dumpling: warning: could not remove input archive {}: {}",
+                    p.display(),
+                    e
+                ),
+            }
+        }
+    }
+
     Ok(())
 }
 
+fn remove_pg_archive(path: &Path) -> std::io::Result<()> {
+    if path.is_dir() {
+        std::fs::remove_dir_all(path)
+    } else {
+        std::fs::remove_file(path)
+    }
+}
+
 fn compile_patterns(patterns: &[String]) -> anyhow::Result<Vec<Regex>> {
     let mut out = Vec::new();
     for p in patterns {
@@ -494,6 +627,24 @@ mod tests_main {
         }
     }
 
+    #[test]
+    fn test_dump_decode_flags_parse() {
+        let cli = Cli::parse_from([
+            "dumpling",
+            "--dump-decode",
+            "--dump-decode-keep-input",
+            "--pg-restore-path",
+            "/usr/bin/pg_restore",
+            "--dump-decode-arg=--no-owner",
+            "-i",
+            "/tmp/latest.dump",
+        ]);
+        assert!(cli.dump_decode);
+        assert!(cli.dump_decode_keep_input);
+        assert_eq!(cli.pg_restore_path, PathBuf::from("/usr/bin/pg_restore"));
+        assert_eq!(cli.dump_decode_arg, vec!["--no-owner"]);
+    }
+
     #[test]
     fn test_lint_policy_allow_noop_flag() {
         let cli = Cli::parse_from(["dumpling", "lint-policy", "--allow-noop"]);