ciphergate-cli 0.15.0__tar.gz → 0.16.0__tar.gz

{ciphergate_cli-0.15.0 → ciphergate_cli-0.16.0}/CHANGELOG.md

@@ -6,6 +6,27 @@ versioning: [PEP 440](https://peps.python.org/pep-0440/) with SemVer meaning (pr
 
 ## [Unreleased]
 
+## [0.16.0] - 2026-11-02
+
+`ciphergate review` writes a local HTML page that shows, as a trace, what sanitize would change
+and how the run went. The CLI's other commands and the report format are unchanged.
+
+### Added
+
+- `ciphergate review INPUT --policy P -o page.html` writes one local HTML page that shows
+  what sanitize would change. It covers findings, how many are transformed, kept, blocked
+  or waiting for review, a per-category table, rescan results, possible partial finds,
+  and each record with its changes highlighted and labelled with category, action, rule
+  and reason code. No sanitized file is written. The page holds the would-be output; the
+  raw input is added only with `--include-input`. The page has no scripts, loads nothing,
+  is written atomically, and is not overwritten without `--force`. The exit code matches
+  `plan`. The page is laid out as a trace: a tree of the run, its steps with timings, and the
+  records; each record opens as Input and Output panels.
+- `scripts/benchmark/ner_names.py` measures the optional NER detector on generated log lines.
+  With `en_core_web_sm` it found any part of 57-59% of names and the whole name in 41%, missed
+  most lowercase and upper-case names, and flagged 3 of 16 name-free lines. Details and limits
+  are in `docs/detector-coverage.md`. The detector itself is unchanged.
+
 ## [0.15.0] - 2026-11-02
 
 JSON reports now include the next-step hints, so CI pipelines can act on them. The field is new;
@@ -395,7 +416,8 @@ Pre-release (alpha). Licensed under Apache-2.0. Not a final release: the securit
 See `docs/limitations.md`. Deferred: optional NER/Presidio detectors, stdin/stdout streaming,
 directory processing, structured log parsers, key files, reversible token vault, lossy decoding.
 
-[Unreleased]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.15.0...HEAD
+[Unreleased]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.16.0...HEAD
+[0.16.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.15.0...v0.16.0
 [0.15.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.14.0...v0.15.0
 [0.14.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.13.0...v0.14.0
 [0.13.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.12.0...v0.13.0

{ciphergate_cli-0.15.0 → ciphergate_cli-0.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ciphergate-cli
-Version: 0.15.0
+Version: 0.16.0
 Summary: Local-first, environment- and destination-aware inspection and sanitization of logs and test datasets.
 Project-URL: Homepage, https://github.com/aashishkumar-tech/ciphergate
 Project-URL: Source, https://github.com/aashishkumar-tech/ciphergate
@@ -50,7 +50,7 @@ vendor ticket. CipherGate replaces that with a local, explainable, scriptable ga
 inspect  ->  plan  ->  sanitize --apply  ->  rescan
 ```
 
-> **Status: 0.15.0 (alpha).** No detector finds every sensitive value. A passing run means the
+> **Status: 0.16.0 (alpha).** No detector finds every sensitive value. A passing run means the
 > configured processing completed under the selected policy; it is **not** proof that the output
 > contains no sensitive information, and it is not a compliance (GDPR, HIPAA, PCI DSS, ...) or
 > anonymization guarantee. Policy templates must be reviewed before use.
@@ -91,6 +91,8 @@ inspect  ->  plan  ->  sanitize --apply  ->  rescan
   and its source, gates, key *status* (never key material), no raw values or fingerprints.
   A failed run's report lists `next_steps` as `{"code", "message"}` pairs, the same hints the
   terminal prints as `next:` lines, so CI can act on them.
+- **Review page:** `ciphergate review` writes one local HTML file that shows what sanitize
+  would change, record by record, with run metrics. See [Review page](#review-page).
 
 ## Install
 
@@ -236,8 +238,9 @@ healthcare or finance are starting points, **not** HIPAA/PCI controls.
 
 ## Commands and exit codes
 
-`init`, `validate-policy`, `inspect`, `plan`, `sanitize` (needs `--apply`), `rescan`, `explain`
-(decision for a hypothetical finding — no data needed). All support `--help`.
+`init`, `validate-policy`, `inspect`, `plan`, `sanitize` (needs `--apply`), `rescan`, `review`
+(local HTML page, see below), `explain` (decision for a hypothetical finding — no data
+needed). All support `--help`.
 
 Common options: `--policy`, `--format csv|json|jsonl|text|logfmt|jsonlog|accesslog|syslog|stacktrace`, `--report PATH`,
 `--report-format json|text` (JSON without `--report` prints only JSON on stdout),
@@ -485,6 +488,50 @@ kubectl logs my-pod | ciphergate sanitize - --format text --policy policies/prd_
 | 8 | Unexpected internal error (safe message) |
 | 130 | Interrupted (no output published) |
 
+### Review page
+
+`ciphergate review` runs the same steps as `sanitize --apply`, including the rescan, but writes
+no sanitized file. It writes one HTML page instead:
+
+```bash
+ciphergate review app.log --policy policies/developer_logs.yml   --environment dev --destination local_dev -o build/review.html
+```
+
+Open the file in a browser. It is laid out like a trace: a tree on the left, details of the
+selected item on the right.
+
+- **review run:** a verdict, then metrics. These include findings and how many are
+  transformed, kept by an `allow` rule, need review or are blocked (with percentages), records
+  with findings, values changed, records dropped, rescan results and possible partial finds.
+- **Steps**, with measured timings: the plan pass (read input, detect, decide), transform,
+  rescan and output.
+  - *detect* shows counts by category, detector and confidence.
+  - *decide* shows the policy and context, what happened to each category, counts by action,
+    rule and reason code, plus gates and key status.
+- **Records with findings:** each one opens as Input and Output panels with a decisions table.
+  Changed parts are highlighted, and hovering over one shows its category, action, rule and
+  reason code. Values kept by an `allow` rule are underlined. A dashed outline marks a
+  *possible partial find*: a replaced name or address next to a capitalised word that stayed.
+  This is a hint, not a check.
+
+What the page contains, and how to handle it:
+
+- By default it contains the would-be **output**, not the input. Treat it like the sanitized
+  file: detectors can miss values.
+- `--include-input` also adds the **raw input** next to the output, with a warning banner. A
+  page made with it must be handled like the original file. Do not share, upload or commit it.
+- If the run is blocked, needs review or a key is missing, there is no output preview. The page
+  shows the plan and the decisions for each record. Planned transforms are labelled as not run.
+- There are no default locations: `-o` is required. `build/` is in `.gitignore`. The page is not
+  overwritten without `--force`, can't be the input or the policy file, and is written
+  atomically (0600 on POSIX). Directories and stdin/stdout are not supported.
+- The page has no scripts and a Content-Security-Policy that blocks all loads, so opening it
+  makes no network request. All text is escaped.
+- The exit code is the plan's: 0 ready, 1 blocked/review, 6 key/config, 7 rescan found gating
+  findings. Use `sanitize` and `rescan` as the CI gate. Up to 500 records are shown, each cut
+  to 4,000 characters; the counts include all records. It takes a little longer than
+  `sanitize` (about 15% on a 40,000-line synthetic log).
+
 ### Troubleshooting
 
 - `destination_required_but_missing` — pass `--destination` (and `--environment`).

{ciphergate_cli-0.15.0 → ciphergate_cli-0.16.0}/README.md

@@ -11,7 +11,7 @@ vendor ticket. CipherGate replaces that with a local, explainable, scriptable ga
 inspect  ->  plan  ->  sanitize --apply  ->  rescan
 ```
 
-> **Status: 0.15.0 (alpha).** No detector finds every sensitive value. A passing run means the
+> **Status: 0.16.0 (alpha).** No detector finds every sensitive value. A passing run means the
 > configured processing completed under the selected policy; it is **not** proof that the output
 > contains no sensitive information, and it is not a compliance (GDPR, HIPAA, PCI DSS, ...) or
 > anonymization guarantee. Policy templates must be reviewed before use.
@@ -52,6 +52,8 @@ inspect  ->  plan  ->  sanitize --apply  ->  rescan
   and its source, gates, key *status* (never key material), no raw values or fingerprints.
   A failed run's report lists `next_steps` as `{"code", "message"}` pairs, the same hints the
   terminal prints as `next:` lines, so CI can act on them.
+- **Review page:** `ciphergate review` writes one local HTML file that shows what sanitize
+  would change, record by record, with run metrics. See [Review page](#review-page).
 
 ## Install
 
@@ -197,8 +199,9 @@ healthcare or finance are starting points, **not** HIPAA/PCI controls.
 
 ## Commands and exit codes
 
-`init`, `validate-policy`, `inspect`, `plan`, `sanitize` (needs `--apply`), `rescan`, `explain`
-(decision for a hypothetical finding — no data needed). All support `--help`.
+`init`, `validate-policy`, `inspect`, `plan`, `sanitize` (needs `--apply`), `rescan`, `review`
+(local HTML page, see below), `explain` (decision for a hypothetical finding — no data
+needed). All support `--help`.
 
 Common options: `--policy`, `--format csv|json|jsonl|text|logfmt|jsonlog|accesslog|syslog|stacktrace`, `--report PATH`,
 `--report-format json|text` (JSON without `--report` prints only JSON on stdout),
@@ -446,6 +449,50 @@ kubectl logs my-pod | ciphergate sanitize - --format text --policy policies/prd_
 | 8 | Unexpected internal error (safe message) |
 | 130 | Interrupted (no output published) |
 
+### Review page
+
+`ciphergate review` runs the same steps as `sanitize --apply`, including the rescan, but writes
+no sanitized file. It writes one HTML page instead:
+
+```bash
+ciphergate review app.log --policy policies/developer_logs.yml   --environment dev --destination local_dev -o build/review.html
+```
+
+Open the file in a browser. It is laid out like a trace: a tree on the left, details of the
+selected item on the right.
+
+- **review run:** a verdict, then metrics. These include findings and how many are
+  transformed, kept by an `allow` rule, need review or are blocked (with percentages), records
+  with findings, values changed, records dropped, rescan results and possible partial finds.
+- **Steps**, with measured timings: the plan pass (read input, detect, decide), transform,
+  rescan and output.
+  - *detect* shows counts by category, detector and confidence.
+  - *decide* shows the policy and context, what happened to each category, counts by action,
+    rule and reason code, plus gates and key status.
+- **Records with findings:** each one opens as Input and Output panels with a decisions table.
+  Changed parts are highlighted, and hovering over one shows its category, action, rule and
+  reason code. Values kept by an `allow` rule are underlined. A dashed outline marks a
+  *possible partial find*: a replaced name or address next to a capitalised word that stayed.
+  This is a hint, not a check.
+
+What the page contains, and how to handle it:
+
+- By default it contains the would-be **output**, not the input. Treat it like the sanitized
+  file: detectors can miss values.
+- `--include-input` also adds the **raw input** next to the output, with a warning banner. A
+  page made with it must be handled like the original file. Do not share, upload or commit it.
+- If the run is blocked, needs review or a key is missing, there is no output preview. The page
+  shows the plan and the decisions for each record. Planned transforms are labelled as not run.
+- There are no default locations: `-o` is required. `build/` is in `.gitignore`. The page is not
+  overwritten without `--force`, can't be the input or the policy file, and is written
+  atomically (0600 on POSIX). Directories and stdin/stdout are not supported.
+- The page has no scripts and a Content-Security-Policy that blocks all loads, so opening it
+  makes no network request. All text is escaped.
+- The exit code is the plan's: 0 ready, 1 blocked/review, 6 key/config, 7 rescan found gating
+  findings. Use `sanitize` and `rescan` as the CI gate. Up to 500 records are shown, each cut
+  to 4,000 characters; the counts include all records. It takes a little longer than
+  `sanitize` (about 15% on a 40,000-line synthetic log).
+
 ### Troubleshooting
 
 - `destination_required_but_missing` — pass `--destination` (and `--environment`).

{ciphergate_cli-0.15.0 → ciphergate_cli-0.16.0}/docs/architecture.md

@@ -40,6 +40,7 @@ Report (reporting/)            privacy-safe JSON / text summary
 | `transforms/` | Execute approved actions; HMAC pseudonyms; key lookup | log values or keys |
 | `pipeline/` | Orchestration, two passes, rescan, atomic output, path safety | publish on any failure |
 | `reporting/` | Report building, control-char escaping, field-path masking, rendering | include values/fingerprints |
+| `pipeline/review.py`, `reporting/review_page.py` | `review`: sanitize steps without publishing, per-record changes, local HTML page | publish output, add input without `--include-input`, load anything |
 | `cli.py` | Typer CLI, exit codes, safe error rendering, SIGTERM -> interrupt | enable rich tracebacks (they print locals) |
 
 ## Why two passes

{ciphergate_cli-0.15.0 → ciphergate_cli-0.16.0}/docs/detector-coverage.md

@@ -32,8 +32,31 @@ in `tests/unit/test_presidio_ner.py` use a fake engine. `tests/integration/test_
 runs the real Presidio with `en_core_web_sm`: it finds a common synthetic name and place, ignores
 text with no entities, removes them in a `sanitize` run with network access disabled, and fails
 closed when the model is missing. CI runs it in the `ner` job (not a required check). These are
-smoke tests on a few sentences: accuracy on real text has not been measured by this project,
-and the small model misses names that the larger `en_core_web_lg` may find.
+smoke tests on a few sentences.
+
+`scripts/benchmark/ner_names.py` measures name detection on 320 generated log lines (8 regions x
+5 ways of writing a name x 8 lines) plus 16 name-free lines. With `en_core_web_sm` (spaCy 3.8,
+Presidio 2.2.366), seeds 7 and 11:
+
+| | seed 7 | seed 11 |
+|---|---|---|
+| names with any part found | 184/320 (57%) | 190/320 (59%) |
+| names found in full | 130/320 (41%) | 132/320 (41%) |
+| name-free lines flagged | 3/16 | 3/16 |
+
+- When only part of a name is found, the rest stays in the output (for example the family name).
+- `First Last` and `Last, First` are found most often (50-61 of 64). Lowercase (27/64) and
+  upper-case (15/64) names are mostly missed.
+- English, Chinese and German names were found more often (27-35 of 40) than Indian and West
+  African names (12-17 of 40).
+- The false alarms were product names (`Kafka`, `Jenkins`, `Grafana`).
+- `min_score` from 0.4 to 0.85 changed nothing: Presidio gives these spaCy entities one fixed
+  score, so the threshold cannot trade misses against false alarms with this model.
+- The default `en_core_web_lg` was not measured. Run the script with `--model en_core_web_lg`
+  after installing it.
+
+These are synthetic lines written by the project, not a public accuracy claim. Treat the NER
+detector as an extra layer: do not rely on it alone to remove names, especially from free text.
 
 ## Text/log specifics
 

{ciphergate_cli-0.15.0 → ciphergate_cli-0.16.0}/docs/security-and-privacy.md

@@ -75,6 +75,21 @@ values are replaced with `<redacted-key>`; control and bidi characters are escap
 Logging uses the standard library and never receives values. `--verbose` lists more items;
 `--debug` adds traceback frame locations for internal errors without exception messages.
 
+## Review page
+
+`ciphergate review` is the only command that puts data into a file other than the output. By
+default the page holds the would-be sanitized output, so it is as sensitive as that output.
+With `--include-input` it also holds the raw input and must be handled like the input; the
+page and the command's stderr both say so. The JSON report never contains either.
+
+The would-be output is written to a `mkstemp` file in the page's directory, rescanned, and
+always removed; it is never published. The page itself uses the same atomic,
+no-overwrite-by-default writer as outputs and refuses the input and policy paths. It contains
+no scripts. A Content-Security-Policy (`default-src 'none'`) and `no-referrer` stop any load
+or request, and every text is HTML-escaped with control and bidi characters shown as escapes.
+Field paths in the decision tables use the same masking as reports. The possible-partial-find
+mark is a heuristic and can miss partial finds or flag whole ones.
+
 ## Files
 
 Inputs must be regular files (FIFOs/devices refused). A directory INPUT starts a batch run:

{ciphergate_cli-0.15.0 → ciphergate_cli-0.16.0}/scripts/benchmark/README.md

@@ -30,6 +30,14 @@ Only CipherGate is required. Any other tool that is not installed is skipped and
 credentials that must not be committed. The tool refuses a path inside the repository. Results
 are repeatable for the same `--seed` and tool versions.
 
+## Name detection (`ner_names.py`)
+
+`python scripts/benchmark/ner_names.py --model en_core_web_sm` measures the optional NER detector
+on generated log lines: names from eight regions written five ways (`First Last`, `Last, First`,
+lowercase, first name only, upper case) and name-free lines. It needs `presidio-analyzer` and the
+spaCy model installed in the same environment, and prints only counts. Results for
+`en_core_web_sm` are in [docs/detector-coverage.md](../../docs/detector-coverage.md).
+
 ## Reading the output
 
 Each row is a data class, and each cell is caught/total. A positive counts only if the finding

{ciphergate_cli-0.15.0 → ciphergate_cli-0.16.0}/src/ciphergate/__init__.py

@@ -24,7 +24,7 @@ Supported public API (everything else is internal and may change without notice)
     rescan_directory(root, policy, context=None, *, recursive=False, strict=None) -> BatchReport
 """
 
-__version__ = "0.15.0"
+__version__ = "0.16.0"
 
 from ciphergate.config import RuntimeSettings
 from ciphergate.constants import (

{ciphergate_cli-0.15.0 → ciphergate_cli-0.16.0}/src/ciphergate/cli.py

@@ -56,6 +56,7 @@ from ciphergate.pipeline.batch import (
     sanitize_directory,
 )
 from ciphergate.pipeline.paths import validate_output_path
+from ciphergate.pipeline.review import ReviewOptions, review_file
 from ciphergate.pipeline.sanitizer import (
     SanitizeOptions,
     inspect_file,
@@ -823,6 +824,86 @@ def rescan_command(
     )
 
 
+@app.command("review")
+def review_command(
+    input_path: Annotated[Path, typer.Argument(metavar="INPUT", help="File to review.")],
+    policy: PolicyOpt,
+    output: Annotated[
+        Path,
+        typer.Option("--output", "-o", help="HTML page to write, e.g. build/review.html."),
+    ],
+    include_input: Annotated[
+        bool,
+        typer.Option(
+            "--include-input",
+            help="Also put the RAW input on the page; handle the page like the input.",
+        ),
+    ] = False,
+    fmt: FormatOpt = None,
+    extension_format: ExtFormatOpt = None,
+    environment: EnvOpt = None,
+    target_environment: TargetOpt = None,
+    destination: DestOpt = None,
+    use_case: UseCaseOpt = None,
+    profile: ProfileOpt = None,
+    max_bytes: MaxBytesOpt = None,
+    encoding: EncodingOpt = None,
+    path_display: PathDisplayOpt = None,
+    force: ForceOpt = False,
+    quiet: QuietOpt = False,
+    verbose: VerboseOpt = False,
+    no_color: NoColorOpt = False,
+    debug: DebugOpt = False,
+) -> None:
+    """Write a local HTML page showing what sanitize would change, with run metrics.
+
+    Writes no sanitized file. The page holds the would-be output; with --include-input it also
+    holds the raw input. The exit code is the plan's (1 blocked/review, 6 key, 7 rescan).
+    """
+    _configure_logging(verbose, debug)
+
+    def body() -> int:
+        if is_stdio(input_path) or is_stdio(output):
+            raise UsageError(
+                "review needs a file INPUT and an --output file", code="stdio_unsupported"
+            )
+        if _is_directory_input(input_path):
+            raise UsageError("review takes a single file, not a directory", code="review_directory")
+        runtime = _runtime(
+            max_bytes, encoding, path_display, extension_formats=extension_format, fmt=fmt
+        )
+        options = ReviewOptions(
+            force=force,
+            include_input=include_input,
+            format=fmt,
+            runtime=runtime,
+            protected=(policy,),
+        )
+        data = review_file(
+            input_path,
+            output,
+            load_policy(policy),
+            _context(environment, target_environment, destination, use_case, profile),
+            options,
+        )
+        code = _emit(
+            data.report,
+            report_path=None,
+            report_format=None,
+            quiet=quiet,
+            verbose=verbose,
+            force=force,
+        )
+        held = "the would-be output" if data.has_output else "the plan only"
+        if include_input:
+            held += " and the RAW INPUT; handle it like the input file"
+        sys.stdout.flush()
+        sys.stderr.write(f"review page written: {escape_control(str(output))} ({held})\n")
+        return code
+
+    _guard(debug, body)
+
+
 @app.command("explain")
 def explain_command(
     policy: PolicyOpt,