@danielarndt0/cnpj-db-loader 2.3.1 → 2.4.0-beta.1

package/docs/architecture.md

@@ -95,3 +95,11 @@ planner -> source-reader -> parser -> normalizer -> staging-writer -> materializ
 ```
 
 - Materialization now stores lightweight staging validation markers (row count and max staging id) in the materialization checkpoint table so reruns can verify the live staging state quickly and reuse lookup reconciliation when the staging snapshot is unchanged. The runtime validates that the required import tables already exist but no longer creates or alters them automatically.
+
+## PostgreSQL direct import workflow
+
+The PostgreSQL direct import workflow is a hybrid execution path. It keeps file detection, Receita parsing, validation and sanitization in the loader, then exports normalized CSV files and a generated `psql` script for direct PostgreSQL loading.
+
+This keeps the parsing rules centralized in the TypeScript layouts while allowing PostgreSQL to run the heaviest bulk load and materialization work with set-based SQL.
+
+The generated script resets staging tables, loads CSV files with `\copy`, upserts domain and final tables, materializes `establishment_secondary_cnaes`, and refreshes planner statistics with `ANALYZE`.

package/docs/commands.md

@@ -22,6 +22,8 @@
 | `database cleanup materialized` | Truncate simplified final relational tables populated by materialization, including establishment secondary CNAEs when available, in safe order.                  |
 | `database cleanup checkpoints`  | Clear load checkpoints, materialization checkpoints, or both without truncating staging or final tables.                                                          |
 | `database cleanup plans`        | Delete saved import plans. Related plan files and materialization checkpoints are removed by database cascade.                                                    |
+| `postgres generate-script`      | Generate a direct `psql` import script that loads sanitized Receita files without rewriting them into new CSV files.                                              |
+| `postgres export-csv`           | Convert sanitized Receita files into normalized PostgreSQL-ready CSV files and generate a direct `psql` import script for audit/debug workflows.                  |
 | `import <input>`                | Run the full pipeline: plan, load validated files into staging/direct final targets, materialize staged datasets into final tables, and finalize the import plan. |
 | `import load <input>`           | Prepare the plan and run only the load phase. Heavy datasets stop in `staging_*`; domain datasets still upsert directly into the final schema.                    |
 | `import materialize <input>`    | Resume from the saved import plan and materialize staged datasets into the final relational tables with resumable chunks.                                         |
@@ -50,6 +52,8 @@ cnpj-db-loader database cleanup staging --validated-path ./downloads/sanitized -
 cnpj-db-loader database cleanup materialized --dataset companies --force
 cnpj-db-loader database cleanup checkpoints --phase materialization --validated-path ./downloads/sanitized --force
 cnpj-db-loader database cleanup plans --validated-path ./downloads/sanitized --force
+cnpj-db-loader postgres generate-script ./downloads/sanitized --output ./downloads/postgres-direct --force
+psql "postgres://user:password@localhost:5432/cnpj" -f ./downloads/postgres-direct/import-postgres-direct.sql
 cnpj-db-loader import ./downloads/sanitized
 cnpj-db-loader import ./downloads/sanitized --db-url "postgresql://user:password@localhost:5432/cnpj"
 cnpj-db-loader import ./downloads/sanitized --dataset companies --load-batch-size 500
@@ -63,3 +67,22 @@ cnpj-db-loader quarantine list --dataset establishments --limit 10
 cnpj-db-loader quarantine list --terminal --after-id 500
 cnpj-db-loader quarantine show 42
 ```
+
+## PostgreSQL direct import helper
+
+```bash
+cnpj-db-loader postgres generate-script <input> [--output <path>] [--dataset <dataset>] [--script-name <name>] [--source-encoding <encoding>] [-f]
+cnpj-db-loader postgres export-csv <input> [--output <path>] [--dataset <dataset>] [--script-name <name>] [-f]
+```
+
+`postgres generate-script` is the recommended hybrid workflow. The loader performs extraction, validation and sanitization, then generates a `psql` script that loads the sanitized Receita files directly through `\copy`.
+
+`postgres export-csv` remains available when you explicitly want a normalized CSV output tree for audit/debug purposes.
+
+Options:
+
+- `--output <path>`: directory where manifest and SQL script are generated.
+- `--dataset <dataset>`: generate only one dataset block.
+- `--script-name <name>`: custom generated SQL script name.
+- `--source-encoding <encoding>`: source file encoding for `psql` copy operations. Defaults to `WIN1252`.
+- `-f, --force`: skip confirmation.

package/docs/postgres-direct.md

@@ -0,0 +1,138 @@
+# PostgreSQL direct import workflow
+
+The PostgreSQL direct import workflow is a hybrid path for environments where the standard resumable importer is too expensive for a full monthly load.
+
+It keeps the safe preparation steps inside CNPJ DB Loader and moves the heaviest database load/materialization work into a generated `psql` script.
+
+## Intended flow
+
+```bash
+cnpj-db-loader federal-revenue download --output ./downloads
+cnpj-db-loader extract ./downloads/<reference>
+cnpj-db-loader validate ./downloads/<reference>/extracted
+cnpj-db-loader sanitize ./downloads/<reference>/extracted
+cnpj-db-loader postgres generate-script ./downloads/<reference>/sanitized --output ./downloads/<reference>/postgres-direct --force
+psql "postgres://postgres:postgres@localhost:5432/cnpj" -f ./downloads/<reference>/postgres-direct/import-postgres-direct.sql
+```
+
+The loader remains responsible for:
+
+- Federal Revenue download and local manifest control
+- extraction
+- validation
+- sanitization
+- preserving the sanitized Receita files without rewriting the whole dataset
+- generating the final `psql` import script
+- optionally exporting PostgreSQL-ready CSV files through `postgres export-csv` when an audit/debug CSV tree is useful
+
+PostgreSQL is then responsible for:
+
+- `\copy` loading sanitized Receita files into temporary raw tables
+- SQL-side conversion of dates, numeric values and nullable fields
+- staging table population
+- set-based final table upserts
+- `establishment_secondary_cnaes` materialization
+- planner statistics refresh through `ANALYZE`
+
+## Why this exists
+
+The standard `import` command is safer and resumable, but it keeps more orchestration inside the Node.js process. That is useful for production safety, checkpoints, quarantine and incremental recovery.
+
+The direct PostgreSQL path is optimized for bulk loading after the input files have already been sanitized. It avoids per-batch Node.js database inserts and avoids rewriting the full dataset into a second CSV tree. Instead, `psql` streams the sanitized Receita files into temporary text tables and PostgreSQL performs the value conversion and materialization with set-based SQL.
+
+Use this when you want to benchmark or run a faster controlled load on a local machine.
+
+## Command
+
+```bash
+cnpj-db-loader postgres generate-script <input> [--output <path>] [--dataset <dataset>] [--script-name <name>] [--source-encoding <encoding>] [-f]
+```
+
+### Arguments
+
+| Argument  | Description                              |
+| --------- | ---------------------------------------- |
+| `<input>` | Path to the sanitized dataset directory. |
+
+### Options
+
+| Option                         | Description                                                                                                                                           |
+| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--output <path>`              | Custom output directory for the generated SQL script and manifest.                                                                                    |
+| `--dataset <dataset>`          | Generate a script only for one dataset block. Useful for debugging.                                                                                   |
+| `--script-name <name>`         | Name of the generated SQL script. Defaults to `import-postgres-direct.sql`.                                                                           |
+| `--source-encoding <encoding>` | Source file encoding used by `psql` while reading the sanitized Receita files. Defaults to `WIN1252`. Use `UTF8` only if the files are already UTF-8. |
+| `-f, --force`                  | Skip the confirmation prompt.                                                                                                                         |
+
+## Output structure
+
+The command creates a small PostgreSQL direct output directory:
+
+```text
+postgres-direct/
+  manifest.json
+  import-postgres-direct.sql
+```
+
+Unlike `postgres export-csv`, this command does not create a second tree of converted CSV files. The generated SQL script points directly to the sanitized Receita files.
+
+This is faster for large monthly loads because it avoids reading and writing the entire dataset again just to add headers or change delimiters.
+
+## Generated script behavior
+
+The generated `import-postgres-direct.sql` script:
+
+1. enables `ON_ERROR_STOP` for `psql`;
+2. starts a transaction;
+3. truncates the `staging_*` tables and restarts their identities;
+4. sets the configured client encoding for `psql` copy operations;
+5. loads domain datasets from sanitized Receita files into temporary raw text tables;
+6. upserts final domain tables;
+7. loads large datasets from sanitized Receita files into temporary raw text tables;
+8. converts values inside PostgreSQL and inserts them into `staging_companies`, `staging_establishments`, `staging_partners` and `staging_simples_options`;
+9. materializes final `companies`, `establishments`, `partners` and `simples_options` tables using set-based SQL;
+10. populates `establishment_secondary_cnaes` from `secondary_cnaes_raw`;
+11. runs `ANALYZE` on the main final tables;
+12. commits the transaction.
+
+The script does not recreate the schema. Run the normal schema first:
+
+```bash
+cnpj-db-loader schema generate --profile full --output ./sql/schema.sql
+psql "postgres://postgres:postgres@localhost:5432/cnpj" -f ./sql/schema.sql
+```
+
+## Important notes
+
+The generated script is designed for full controlled loads and benchmarks. It is not a replacement for the standard resumable `import` command when you need checkpoint-based recovery, row quarantine or long-running incremental resume behavior.
+
+The generated script resets staging tables, but it does not truncate final tables. Final tables are updated through `ON CONFLICT` upserts.
+
+For a fully clean rebuild, reset the database or run the appropriate database cleanup command before executing the generated script.
+
+## Windows usage
+
+On Windows, the script uses `\copy`, not server-side `COPY`.
+
+This is intentional. With `\copy`, the `psql` client reads local files and streams them to PostgreSQL. This avoids common Windows service permission issues where the PostgreSQL service user cannot read files from your working directory.
+
+Example:
+
+```powershell
+psql "postgres://postgres:postgres@localhost:5432/cnpj" -f "D:/cnpj-data/2026-05/postgres-direct/import-postgres-direct.sql"
+```
+
+## Recommended comparison benchmark
+
+To compare the standard and hybrid paths:
+
+```bash
+# Standard path
+cnpj-db-loader import ./downloads/<reference>/sanitized --load-batch-size 500 --materialize-batch-size 50000 --verbose-progress
+
+# Hybrid path
+cnpj-db-loader postgres generate-script ./downloads/<reference>/sanitized --output ./downloads/<reference>/postgres-direct --force
+psql "postgres://postgres:postgres@localhost:5432/cnpj" -f ./downloads/<reference>/postgres-direct/import-postgres-direct.sql
+```
+
+Compare total duration, disk usage, PostgreSQL CPU usage, WAL growth and final row counts.

package/docs/releases/v2.4.0.md

@@ -0,0 +1,40 @@
+# v2.4.0 — PostgreSQL Direct Import Workflow
+
+This release adds a hybrid PostgreSQL direct import workflow.
+
+The loader can now generate a ready-to-run `psql` script that loads sanitized Receita Federal files directly through `\copy`, converts values inside PostgreSQL and materializes the final schema using set-based SQL.
+
+The previous CSV export path remains available for audit/debug workflows, but the recommended fast path no longer rewrites the entire dataset into a second CSV tree.
+
+## Added
+
+- Added `postgres generate-script` command.
+- Added direct `psql` script generation from sanitized Receita files.
+- Added SQL-side raw temporary tables and value conversion for dates, numerics and nullable fields.
+- Kept `postgres export-csv` for optional PostgreSQL-ready CSV export with headers, UTF-8 output and normalized values.
+- Added generated `import-postgres-direct.sql` script.
+- Added generated `manifest.json` for exported files and row counts.
+- Added set-based SQL materialization for:
+  - `companies`
+  - `establishments`
+  - `establishment_secondary_cnaes`
+  - `partners`
+  - `simples_options`
+- Added domain table loading through temporary tables and final upserts.
+- Added documentation for the hybrid PostgreSQL workflow.
+
+## Purpose
+
+This workflow is designed for controlled bulk-load scenarios where the standard resumable importer is too slow for local full monthly loads.
+
+The recommended flow is:
+
+1. use the loader for download, extraction, validation and sanitization;
+2. generate a direct `psql` script from the sanitized files;
+3. run the generated `psql` script to load and materialize the database.
+
+## Notes
+
+The standard `import` command remains the safest option when checkpoint-based resume, row quarantine and detailed recovery behavior are required.
+
+The new PostgreSQL direct workflow is intended for faster controlled imports and benchmarking while keeping extraction, validation and sanitization inside the loader. Value conversion for this path happens inside PostgreSQL to avoid unnecessary full-dataset rewriting.

package/docs/usage.md

@@ -117,6 +117,20 @@ The final import summary also includes baseline metrics for preparatory scan tim
 
 The exact preparatory scan runs only when no saved import plan exists for the same validated source files and batch size. On resume, the importer reuses the saved plan and then reuses the checkpoint table to continue from the last committed byte offset instead of restarting the data load itself. Rows that fail after retries are written to `import_quarantine`, so a few bad rows do not stop the entire dataset. Running `sanitize` first reduces how often the importer has to fall back to those slower recovery paths.
 
+## Hybrid PostgreSQL direct import
+
+After sanitization, you can generate a direct `psql` script and let PostgreSQL load the sanitized Receita files without rewriting the full dataset into another CSV tree:
+
+```bash
+cnpj-db-loader sanitize ./downloads/<reference>/extracted
+cnpj-db-loader postgres generate-script ./downloads/<reference>/sanitized --output ./downloads/<reference>/postgres-direct --force
+psql "postgres://postgres:postgres@localhost:5432/cnpj" -f ./downloads/<reference>/postgres-direct/import-postgres-direct.sql
+```
+
+Use this flow when you want PostgreSQL to perform the heavy bulk load and set-based materialization directly. Use `postgres export-csv` only when you need an intermediate normalized CSV tree for audit/debug purposes. Use the standard `import` command when you need checkpoint-based resume and row quarantine recovery.
+
+See [PostgreSQL Direct Import](./postgres-direct.md) for details.
+
 ## Quarantine analysis
 
 Use the `quarantine` service after a long-running import when you want to inspect the rows that could not be inserted.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@danielarndt0/cnpj-db-loader",
-  "version": "2.3.1",
+  "version": "2.4.0-beta.1",
   "publishConfig": {
     "access": "public"
   },