@danielarndt0/cnpj-db-loader 2.3.1 → 2.4.0-beta.2

package/docs/architecture.md

@@ -21,7 +21,7 @@ The import pipeline now uses:
 - deterministic dataset order to respect foreign keys
 - an exact preparatory scan that counts total source rows and planned batches before the first write
 - streaming file reads to avoid loading the full dataset into RAM
-- an optional sanitize step that removes known low-level byte issues before import starts
+- an optional sanitize step that writes clean UTF-8 files and removes known low-level byte issues before import starts
 - COPY-based staged writes for the large datasets followed by staged-to-final materialization
 - conflict-safe upserts for the smaller domain datasets
 - `import_plans` and `import_plan_files` to persist exact import plans and avoid recounting the same source files on resume
@@ -95,3 +95,11 @@ planner -> source-reader -> parser -> normalizer -> staging-writer -> materializ
 ```
 
 - Materialization now stores lightweight staging validation markers (row count and max staging id) in the materialization checkpoint table so reruns can verify the live staging state quickly and reuse lookup reconciliation when the staging snapshot is unchanged. The runtime validates that the required import tables already exist but no longer creates or alters them automatically.
+
+## PostgreSQL direct import workflow
+
+The PostgreSQL direct import workflow is a hybrid execution path. It keeps file detection, Receita parsing, validation and sanitization in the loader, then exports normalized CSV files and a generated `psql` script for direct PostgreSQL loading.
+
+This keeps the parsing rules centralized in the TypeScript layouts while allowing PostgreSQL to run the heaviest bulk load and materialization work with set-based SQL.
+
+The generated script resets staging tables, loads CSV files with `\copy`, upserts domain and final tables, materializes `establishment_secondary_cnaes`, and refreshes planner statistics with `ANALYZE`.

package/docs/cli.md

@@ -12,7 +12,7 @@ cnpj-db-loader federal-revenue sync [reference] [--reference <yyyy-mm>] [--curre
 cnpj-db-loader inspect <input>
 cnpj-db-loader extract <input> [--output <path>]
 cnpj-db-loader validate <input>
-cnpj-db-loader sanitize <input> [--output <path>] [--dataset <name>] [-f]
+cnpj-db-loader sanitize <input> [--output <path>] [--dataset <name>] [--source-encoding <encoding>] [-f]
 cnpj-db-loader schema print [--profile <profile>]
 cnpj-db-loader schema generate [--name <name>] [--output <path>] [--profile <profile>]
 cnpj-db-loader database config set <url>

package/docs/commands.md

@@ -22,6 +22,8 @@
 | `database cleanup materialized` | Truncate simplified final relational tables populated by materialization, including establishment secondary CNAEs when available, in safe order.                  |
 | `database cleanup checkpoints`  | Clear load checkpoints, materialization checkpoints, or both without truncating staging or final tables.                                                          |
 | `database cleanup plans`        | Delete saved import plans. Related plan files and materialization checkpoints are removed by database cascade.                                                    |
+| `postgres generate-script`      | Generate a direct `psql` import script that loads sanitized Receita files without rewriting them into new CSV files.                                              |
+| `postgres export-csv`           | Convert sanitized Receita files into normalized PostgreSQL-ready CSV files and generate a direct `psql` import script for audit/debug workflows.                  |
 | `import <input>`                | Run the full pipeline: plan, load validated files into staging/direct final targets, materialize staged datasets into final tables, and finalize the import plan. |
 | `import load <input>`           | Prepare the plan and run only the load phase. Heavy datasets stop in `staging_*`; domain datasets still upsert directly into the final schema.                    |
 | `import materialize <input>`    | Resume from the saved import plan and materialize staged datasets into the final relational tables with resumable chunks.                                         |
@@ -50,6 +52,8 @@ cnpj-db-loader database cleanup staging --validated-path ./downloads/sanitized -
 cnpj-db-loader database cleanup materialized --dataset companies --force
 cnpj-db-loader database cleanup checkpoints --phase materialization --validated-path ./downloads/sanitized --force
 cnpj-db-loader database cleanup plans --validated-path ./downloads/sanitized --force
+cnpj-db-loader postgres generate-script ./downloads/sanitized --output ./downloads/postgres-direct --force
+psql "postgres://user:password@localhost:5432/cnpj" -f ./downloads/postgres-direct/import-postgres-direct.sql
 cnpj-db-loader import ./downloads/sanitized
 cnpj-db-loader import ./downloads/sanitized --db-url "postgresql://user:password@localhost:5432/cnpj"
 cnpj-db-loader import ./downloads/sanitized --dataset companies --load-batch-size 500
@@ -63,3 +67,22 @@ cnpj-db-loader quarantine list --dataset establishments --limit 10
 cnpj-db-loader quarantine list --terminal --after-id 500
 cnpj-db-loader quarantine show 42
 ```
+
+## PostgreSQL direct import helper
+
+```bash
+cnpj-db-loader postgres generate-script <input> [--output <path>] [--dataset <dataset>] [--script-name <name>] [--source-encoding <encoding>] [-f]
+cnpj-db-loader postgres export-csv <input> [--output <path>] [--dataset <dataset>] [--script-name <name>] [-f]
+```
+
+`postgres generate-script` is the recommended hybrid workflow. The loader performs extraction, validation and sanitization, then generates a `psql` script that loads the sanitized Receita files directly through `\copy`.
+
+`postgres export-csv` remains available when you explicitly want a normalized CSV output tree for audit/debug purposes.
+
+Options:
+
+- `--output <path>`: directory where manifest and SQL script are generated.
+- `--dataset <dataset>`: generate only one dataset block.
+- `--script-name <name>`: custom generated SQL script name.
+- `--source-encoding <encoding>`: source file encoding for `psql` copy operations. Defaults to `UTF8`.
+- `-f, --force`: skip confirmation.

package/docs/postgres-direct.md

@@ -0,0 +1,138 @@
+# PostgreSQL direct import workflow
+
+The PostgreSQL direct import workflow is a hybrid path for environments where the standard resumable importer is too expensive for a full monthly load.
+
+It keeps the safe preparation steps inside CNPJ DB Loader and moves the heaviest database load/materialization work into a generated `psql` script.
+
+## Intended flow
+
+```bash
+cnpj-db-loader federal-revenue download --output ./downloads
+cnpj-db-loader extract ./downloads/<reference>
+cnpj-db-loader validate ./downloads/<reference>/extracted
+cnpj-db-loader sanitize ./downloads/<reference>/extracted
+cnpj-db-loader postgres generate-script ./downloads/<reference>/sanitized --output ./downloads/<reference>/postgres-direct --source-encoding UTF8 --force
+psql "postgres://postgres:postgres@localhost:5432/cnpj" -f ./downloads/<reference>/postgres-direct/import-postgres-direct.sql
+```
+
+The loader remains responsible for:
+
+- Federal Revenue download and local manifest control
+- extraction
+- validation
+- sanitization
+- preserving the sanitized Receita files without rewriting the whole dataset
+- generating the final `psql` import script
+- optionally exporting PostgreSQL-ready CSV files through `postgres export-csv` when an audit/debug CSV tree is useful
+
+PostgreSQL is then responsible for:
+
+- `\copy` loading sanitized Receita files into temporary raw tables
+- SQL-side conversion of dates, numeric values and nullable fields
+- staging table population
+- set-based final table upserts
+- `establishment_secondary_cnaes` materialization
+- planner statistics refresh through `ANALYZE`
+
+## Why this exists
+
+The standard `import` command is safer and resumable, but it keeps more orchestration inside the Node.js process. That is useful for production safety, checkpoints, quarantine and incremental recovery.
+
+The direct PostgreSQL path is optimized for bulk loading after the input files have already been sanitized. It avoids per-batch Node.js database inserts and avoids rewriting the full dataset into a second CSV tree. Instead, `psql` streams the sanitized Receita files into temporary text tables and PostgreSQL performs the value conversion and materialization with set-based SQL.
+
+Use this when you want to benchmark or run a faster controlled load on a local machine.
+
+## Command
+
+```bash
+cnpj-db-loader postgres generate-script <input> [--output <path>] [--dataset <dataset>] [--script-name <name>] [--source-encoding <encoding>] [-f]
+```
+
+### Arguments
+
+| Argument  | Description                              |
+| --------- | ---------------------------------------- |
+| `<input>` | Path to the sanitized dataset directory. |
+
+### Options
+
+| Option                         | Description                                                                                                                                                                                                                                          |
+| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--output <path>`              | Custom output directory for the generated SQL script and manifest.                                                                                                                                                                                   |
+| `--dataset <dataset>`          | Generate a script only for one dataset block. Useful for debugging.                                                                                                                                                                                  |
+| `--script-name <name>`         | Name of the generated SQL script. Defaults to `import-postgres-direct.sql`.                                                                                                                                                                          |
+| `--source-encoding <encoding>` | Source file encoding used by `psql` while reading the sanitized Receita files. Defaults to `UTF8` because the current `sanitize` command writes UTF-8 output. Use `WIN1252` or `LATIN1` only for legacy sanitized files generated by older versions. |
+| `-f, --force`                  | Skip the confirmation prompt.                                                                                                                                                                                                                        |
+
+## Output structure
+
+The command creates a small PostgreSQL direct output directory:
+
+```text
+postgres-direct/
+  manifest.json
+  import-postgres-direct.sql
+```
+
+Unlike `postgres export-csv`, this command does not create a second tree of converted CSV files. The generated SQL script points directly to the sanitized Receita files. Current sanitized files are expected to be clean UTF-8 by default.
+
+This is faster for large monthly loads because it avoids reading and writing the entire dataset again just to add headers or change delimiters.
+
+## Generated script behavior
+
+The generated `import-postgres-direct.sql` script:
+
+1. enables `ON_ERROR_STOP` for `psql`;
+2. starts a transaction;
+3. truncates the `staging_*` tables and restarts their identities;
+4. sets the configured client encoding for `psql` copy operations;
+5. loads domain datasets from sanitized Receita files into temporary raw text tables;
+6. upserts final domain tables;
+7. loads large datasets from sanitized Receita files into temporary raw text tables;
+8. converts values inside PostgreSQL and inserts them into `staging_companies`, `staging_establishments`, `staging_partners` and `staging_simples_options`;
+9. materializes final `companies`, `establishments`, `partners` and `simples_options` tables using set-based SQL;
+10. populates `establishment_secondary_cnaes` from `secondary_cnaes_raw`;
+11. runs `ANALYZE` on the main final tables;
+12. commits the transaction.
+
+The script does not recreate the schema. Run the normal schema first:
+
+```bash
+cnpj-db-loader schema generate --profile full --output ./sql/schema.sql
+psql "postgres://postgres:postgres@localhost:5432/cnpj" -f ./sql/schema.sql
+```
+
+## Important notes
+
+The generated script is designed for full controlled loads and benchmarks. It is not a replacement for the standard resumable `import` command when you need checkpoint-based recovery, row quarantine or long-running incremental resume behavior.
+
+The generated script resets staging tables, but it does not truncate final tables. Final tables are updated through `ON CONFLICT` upserts.
+
+For a fully clean rebuild, reset the database or run the appropriate database cleanup command before executing the generated script.
+
+## Windows usage
+
+On Windows, the script uses `\copy`, not server-side `COPY`.
+
+This is intentional. With `\copy`, the `psql` client reads local files and streams them to PostgreSQL. This avoids common Windows service permission issues where the PostgreSQL service user cannot read files from your working directory.
+
+Example:
+
+```powershell
+psql "postgres://postgres:postgres@localhost:5432/cnpj" -f "D:/cnpj-data/2026-05/postgres-direct/import-postgres-direct.sql"
+```
+
+## Recommended comparison benchmark
+
+To compare the standard and hybrid paths:
+
+```bash
+# Standard path
+cnpj-db-loader import ./downloads/<reference>/sanitized --load-batch-size 500 --materialize-batch-size 50000 --verbose-progress
+
+# Hybrid path
+cnpj-db-loader postgres generate-script ./downloads/<reference>/sanitized --output ./downloads/<reference>/postgres-direct --source-encoding UTF8 --force
+psql "postgres://postgres:postgres@localhost:5432/cnpj" -f ./downloads/<reference>/postgres-direct/import-postgres-direct.sql
+```
+
+Compare total duration, disk usage, PostgreSQL CPU usage, WAL growth and final row counts.

package/docs/sanitize.md

@@ -4,7 +4,16 @@
 
 `sanitize` prepares a clean dataset tree before PostgreSQL import.
 
-It removes known low-level byte issues, especially `0x00` / NUL bytes, from validated dataset files and writes the result to a new output directory. The goal is to reduce slow fallback work during import so PostgreSQL receives cleaner files from the start.
+The command now performs robust text sanitization for Receita Federal files:
+
+- reads legacy Receita files using a configurable source encoding;
+- writes sanitized output as clean UTF-8;
+- removes NUL bytes;
+- removes invalid bytes that cannot be safely decoded;
+- removes problematic control characters while preserving line breaks;
+- keeps the original dataset file names and directory structure.
+
+This makes the sanitized tree safer for both the standard loader import flow and the hybrid PostgreSQL direct import flow.
 
 ## Command
 
@@ -14,18 +23,19 @@ cnpj-db-loader sanitize <input>
 
 ## Options
 
-| Option             | Description                                                               |
-| ------------------ | ------------------------------------------------------------------------- |
-| `--output <path>`  | Custom output directory for the sanitized dataset tree.                   |
-| `--dataset <name>` | Sanitize only one dataset block, such as `establishments` or `companies`. |
-| `-f, --force`      | Skip the confirmation prompt.                                             |
+| Option                         | Description                                                                                                           |
+| ------------------------------ | --------------------------------------------------------------------------------------------------------------------- |
+| `--output <path>`              | Custom output directory for the sanitized dataset tree.                                                               |
+| `--dataset <name>`             | Sanitize only one dataset block, such as `establishments` or `companies`.                                             |
+| `--source-encoding <encoding>` | Source file encoding used while reading Receita files. Defaults to `WIN1252`. Supported: `WIN1252`, `LATIN1`, `UTF8`. |
+| `-f, --force`                  | Skip the confirmation prompt.                                                                                         |
 
 ## Default output behavior
 
-- when the validated path is `.../extracted`, the default sanitized output is `.../sanitized`
-- otherwise the default output is `<validated-path>-sanitized`
+- when the validated path is `.../extracted`, the default sanitized output is `.../sanitized`;
+- otherwise the default output is `<validated-path>-sanitized`.
 
-## Recommended flow
+## Recommended standard flow
 
 ```bash
 cnpj-db-loader inspect ./downloads
@@ -35,15 +45,41 @@ cnpj-db-loader sanitize ./downloads/extracted
 cnpj-db-loader import ./downloads/sanitized --load-batch-size 500 --materialize-batch-size 50000 --verbose-progress
 ```
 
+## Recommended hybrid PostgreSQL flow
+
+Because sanitized files are now written as UTF-8, the direct PostgreSQL script can use `UTF8` as the source encoding.
+
+```bash
+cnpj-db-loader sanitize ./downloads/extracted --output ./downloads/sanitized --force
+cnpj-db-loader postgres generate-script ./downloads/sanitized --output ./downloads/postgres-direct --source-encoding UTF8 --force
+psql -d "postgres://postgres:postgres@localhost:5432/cnpj" -f ./downloads/postgres-direct/import-postgres-direct.sql
+```
+
 ## What it improves
 
-- fewer UTF-8 / NUL-byte related insert failures
-- less row-by-row fallback during import
-- better import throughput for large datasets
-- cleaner quarantine data because known low-level issues are removed earlier
+- fewer encoding-related `COPY` failures;
+- fewer UTF-8 / NUL-byte related insert failures;
+- no invalid bytes in sanitized output;
+- fewer problematic control characters in PostgreSQL input files;
+- less row-by-row fallback during standard import;
+- better throughput for large datasets;
+- cleaner quarantine data because known low-level issues are removed earlier.
+
+## Encoding notes
+
+The default source encoding is `WIN1252`, which matches the common legacy encoding used by Receita files.
+
+If a source dataset still fails because of undefined Windows-1252 bytes, `LATIN1` can be used as a more permissive decoder:
+
+```bash
+cnpj-db-loader sanitize ./downloads/extracted --source-encoding LATIN1 --output ./downloads/sanitized --force
+```
+
+The output is still UTF-8 in both cases.
 
 ## Notes
 
-- `sanitize` does not replace validation; it assumes the dataset tree is already valid
-- `import` still keeps quarantine and retry logic for unexpected issues that survive sanitization
-- no database schema changes are required to use `sanitize`
+- `sanitize` does not replace validation; it assumes the dataset tree is already valid.
+- `sanitize` preserves file names and relative paths so existing import logic can keep detecting datasets by name.
+- `import` still keeps quarantine and retry logic for unexpected issues that survive sanitization.
+- no database schema changes are required to use `sanitize`.

package/docs/usage.md

@@ -117,6 +117,20 @@ The final import summary also includes baseline metrics for preparatory scan tim
 
 The exact preparatory scan runs only when no saved import plan exists for the same validated source files and batch size. On resume, the importer reuses the saved plan and then reuses the checkpoint table to continue from the last committed byte offset instead of restarting the data load itself. Rows that fail after retries are written to `import_quarantine`, so a few bad rows do not stop the entire dataset. Running `sanitize` first reduces how often the importer has to fall back to those slower recovery paths.
 
+## Hybrid PostgreSQL direct import
+
+After sanitization, you can generate a direct `psql` script and let PostgreSQL load the sanitized Receita files without rewriting the full dataset into another CSV tree:
+
+```bash
+cnpj-db-loader sanitize ./downloads/<reference>/extracted
+cnpj-db-loader postgres generate-script ./downloads/<reference>/sanitized --output ./downloads/<reference>/postgres-direct --force
+psql "postgres://postgres:postgres@localhost:5432/cnpj" -f ./downloads/<reference>/postgres-direct/import-postgres-direct.sql
+```
+
+Use this flow when you want PostgreSQL to perform the heavy bulk load and set-based materialization directly. Use `postgres export-csv` only when you need an intermediate normalized CSV tree for audit/debug purposes. Use the standard `import` command when you need checkpoint-based resume and row quarantine recovery.
+
+See [PostgreSQL Direct Import](./postgres-direct.md) for details.
+
 ## Quarantine analysis
 
 Use the `quarantine` service after a long-running import when you want to inspect the rows that could not be inserted.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@danielarndt0/cnpj-db-loader",
-  "version": "2.3.1",
+  "version": "2.4.0-beta.2",
   "publishConfig": {
     "access": "public"
   },
@@ -46,10 +46,10 @@
     "cli": "node --no-deprecation --import tsx src/cli.ts",
     "test": "vitest run",
     "lint": "eslint src",
+    "check": "npm run lint && npm run typecheck && npm run build",
     "format": "prettier . --write",
     "format:check": "prettier . --check",
-    "typecheck": "tsc --noEmit",
-    "prepublishOnly": "npm run lint && npm run typecheck && npm run build"
+    "typecheck": "tsc --noEmit"
   },
   "dependencies": {
     "commander": "^12.1.0",