mainframe-migration-toolkit 0.2.0__tar.gz

mainframe_migration_toolkit-0.2.0/.claude/skills/analyze-mainframe-similarity/SKILL.md

@@ -0,0 +1,30 @@
+---
+name: analyze-mainframe-similarity
+description: Cluster COBOL and JCL source by deterministic textual similarity, inspect every proposed cluster, and add an AI verdict and concise justification to JSON and CSV outputs. Use for duplicate or near-duplicate codebase analysis.
+---
+
+# Analyze mainframe similarity
+
+1. Run the deterministic scan:
+
+   ```text
+   python -m mainframe_toolkit similarity scan <source-root> --threshold <threshold> --json similarity.json --csv similarity.csv --review-request similarity-review-request.json
+   ```
+
+   Use the requested threshold, or 80 when absent. Never edit deterministic scores or cluster membership manually.
+
+2. For each non-singleton cluster in the review request, read every original file and its normalized evidence. Check all pair relationships; connected-component chaining alone is not proof of equivalence. Ignore comments, sequence columns, copyright blocks, whitespace, generated names, and environment-only identifiers. Compare JCL execution flow/DD semantics or COBOL inputs, outputs, branches, calculations, side effects, error behavior, and record layouts.
+
+3. Assign exactly one verdict per requested cluster:
+
+   - `identico`: same executable semantics; differences are non-semantic.
+   - `similar`: materially shared flow/business implementation but at least one real semantic variation.
+   - `diferente`: superficial boilerplate or chaining grouped programs with different behavior.
+
+4. Write the requested review JSON with a short evidence-based Portuguese justification, then apply it:
+
+   ```text
+   python -m mainframe_toolkit similarity apply-review similarity.json similarity-review.json --json similarity-reviewed.json --csv similarity-reviewed.csv
+   ```
+
+5. Verify both reviewed files contain `parecer_ia` and `justificativa_ia` for every row. Keep singletons as `diferente` unless direct inspection establishes a duplicate omitted by the threshold.

mainframe_migration_toolkit-0.2.0/.claude/skills/migrate-mainframe-job/SKILL.md

@@ -0,0 +1,63 @@
+---
+name: migrate-mainframe-job
+description: Convert one specification-led JCL job and its COBOL programs to an incrementally validated PySpark pipeline with audited in-memory handoffs and final golden-dataset equality. Use when asked to migrate, continue, repair, or finish a mainframe job in this repository.
+---
+
+# Migrate mainframe job
+
+Migrate `$ARGUMENTS`, or the single job requested by the user. Do not widen the job scope.
+
+## Preflight
+
+1. Install the repo package if its import fails:
+
+   ```text
+   python -m pip install -e ".[all]"
+   ```
+
+2. Run `python -m mainframe_toolkit inspect . --output .migration-work/project-inspection.json`. Resolve structural errors before conversion; unresolved dependencies may proceed only through the synthetic/external rules below.
+3. Read the selected `spec/jcl/<JOB>-jcl.json` and source JCL completely. Reconcile their ordered steps, conditions, DD concatenation, SORT/IDCAMS control cards, input/output datasets, GDGs, and return-code behavior. The spec defines intended business behavior; source resolves mechanics the spec leaves open.
+4. Read every referenced `spec/cobol/<PROGRAM>-cbl.json`, COBOL source, used copybook, and relevant entry in `external-programs/*.json`. Do not infer a record layout from narrative text when a copybook exists. Distinguish physical COBOL width from any expanded textual transport width.
+5. Inventory every supplied input and golden file before generating anything. Freeze clock-dependent behavior behind an injected processing date.
+
+## Target shape
+
+Create only the files needed under `converted/<JOB>/`:
+
+```text
+<job_lower>.py
+programs/__init__.py
+programs/<program_lower>.py
+tests/...
+RELATORIO_FINAL.md                 # only after acceptance
+```
+
+Create one module per COBOL program. Represent utility steps separately when they implement a transformation. Every module exposes `run(context)` and returns a mapping of logical dataset names to PySpark DataFrames, or a `ProgramResult` when a return code matters.
+
+The JCL-named entrypoint contains no transformation, validation, lookup, calculation, filtering, joining, or formatting rules. It only initializes Spark and `PipelineContext`, imports modules, declares `PipelineProgram` order/conditions, loads runtime arguments, and calls `PipelineRunner`. It must accept `--through <PROGRAM>` so prefixes are executable.
+
+Use `mainframe_toolkit.io` for local/S3 CSV and Parquet, `sequential`/`copybook` for fixed records, `sorting` for stable multi-key sort, `external` for subprogram adapters, and COBOL decimal helpers instead of cloning generic logic into modules.
+
+## Ordered conversion gate
+
+For each executable step in JCL order:
+
+1. Implement only that program's behavior from its spec and source. Preserve COBOL decimal/truncation, key, mutation, EOF, return-code, and error semantics unless the specification explicitly overrides them.
+2. Add its import and `PipelineProgram` entry to the JCL-named file immediately.
+3. Run fast module tests, then execute the entire pipeline prefix with `--through <PROGRAM>` using supplied inputs.
+4. If a required input, dataset, clock value, or external implementation is absent, create the smallest deterministic substitute that makes semantic sense. Keep relationships valid, use a stable seed, place it under `converted/<JOB>/synthetic/`, and write a machine-readable provenance manifest. Never overwrite or shadow supplied data.
+5. Confirm the prefix has no import, analysis, Spark, schema, conversion, or runtime errors. Confirm each program output exists both in the in-memory registry and its audit path. Fix the current or earlier responsible module before moving to the next step.
+
+Do not use the final golden comparison for a partial prefix unless that prefix is itself the documented final producer.
+
+## Final equivalence loop
+
+1. Execute the full pipeline with a clean run ID. Make final row ordering explicit using contract keys before export; Spark partition order is not a business order.
+2. Build the Java validator if needed, then invoke it through `python -m mainframe_toolkit golden validate ...` with the final output, matching golden file, format/delimiter, and ordering keys.
+3. Acceptance is `status=MATCH`, zero differing cells, equal row/column counts, and exit code 0.
+4. On mismatch, start from the first reported row/key/column. Trace that field backward through the per-program audit outputs and JCL data flow; identify the earliest program that creates the wrong value, row, multiplicity, or ordering. Correct that module, rerun its prefix, then rerun the full pipeline and validator.
+5. Repeat until 100% equal. Do not weaken comparison options, discard differing rows, alter the golden dataset, or encode golden values as business rules.
+
+## Final artifact
+
+Only after acceptance, create `converted/<JOB>/RELATORIO_FINAL.md` with: programs converted in order; material spec/source decisions; supplied versus synthetic inputs and external adapters; tests/checkpoints executed; exact final comparison result; and the shortest commands needed to run the job and validator. Do not add deployment guidance.

mainframe_migration_toolkit-0.2.0/.claude/skills/validate-golden-dataset/SKILL.md

@@ -0,0 +1,12 @@
+---
+name: validate-golden-dataset
+description: Compare a converted pipeline's final CSV or Parquet output cell by cell against its golden dataset and localize mismatches using audit lineage. Use for final equivalence checks or divergence repair.
+---
+
+# Validate golden dataset
+
+Build `validator-java` when its executable JAR is absent, then run `python -m mainframe_toolkit golden validate` for `$ARGUMENTS`. Supply the real delimiter/header/null/decimal options and explicit ordering keys whenever record order is not contractual.
+
+Require equal schema, column order, row count, and normalized cell values. Exit code 0 and zero differences are the only passing result.
+
+For a mismatch, use the report's first row/key/column and the audit manifest to trace the field backward through program outputs. Fix the earliest converted program that introduces the divergence, rerun its full prefix, then the full job and validator. Repeat without changing the golden dataset or relaxing comparison semantics.

mainframe_migration_toolkit-0.2.0/.gitignore

@@ -0,0 +1,9 @@
+.venv/
+.tmp/
+.pytest_tmp/
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+target/

mainframe_migration_toolkit-0.2.0/AGENTS.md

@@ -0,0 +1,10 @@
+# Mainframe migration toolkit
+
+- Treat `spec/` as the intended behavior, the JCL/COBOL/copybook sources as the mechanical detail, and `golden-dataset/` as the final observable acceptance criterion. Record material conflicts in the final report.
+- Convert a job strictly in JCL execution order. After each program, import it from the JCL-named entrypoint and execute the complete prefix through that program. Fix every syntax or runtime failure before continuing.
+- Keep intermediate datasets as PySpark DataFrames in `PipelineContext`; never use an audit file as the handoff between programs. Publish every program output so the runtime also materializes an audit copy.
+- Put business rules only in program modules. The JCL-named entrypoint may initialize Spark/context, import programs, declare their order/conditions, parse runtime arguments, and run the pipeline.
+- Generate deterministic synthetic data or an external-program adapter only for an unavailable dependency. Never replace supplied data. Preserve referential integrity and identify every synthetic resource in the final report.
+- Completion requires a full run and 100% cell-level equality with the golden dataset. Use explicit ordering keys when row order is not part of the contract. Trace mismatches to the responsible program, fix it, and rerun until equal.
+- Do not add deployment, infrastructure, production hardening, architecture prose, README files, or per-step reports. Generate only functional core code, tests, audit artifacts, and the final `RELATORIO_FINAL.md` for each converted job.
+

mainframe_migration_toolkit-0.2.0/CLAUDE.md

@@ -0,0 +1,2 @@
+@AGENTS.md
+

mainframe_migration_toolkit-0.2.0/PKG-INFO

@@ -0,0 +1,16 @@
+Metadata-Version: 2.5
+Name: mainframe-migration-toolkit
+Version: 0.2.0
+Summary: Runtime and deterministic tools for COBOL/JCL to PySpark migrations
+Author: Mainframe Migration Toolkit
+Requires-Python: >=3.10
+Provides-Extra: all
+Requires-Dist: pyarrow>=15; extra == 'all'
+Requires-Dist: pyspark<5,>=3.5; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=5; extra == 'dev'
+Requires-Dist: pytest>=8.2; extra == 'dev'
+Provides-Extra: parquet
+Requires-Dist: pyarrow>=15; extra == 'parquet'
+Provides-Extra: spark
+Requires-Dist: pyspark<5,>=3.5; extra == 'spark'

mainframe_migration_toolkit-0.2.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+[project]
+name = "mainframe-migration-toolkit"
+version = "0.2.0"
+description = "Runtime and deterministic tools for COBOL/JCL to PySpark migrations"
+requires-python = ">=3.10"
+authors = [{ name = "Mainframe Migration Toolkit" }]
+dependencies = []
+
+[project.optional-dependencies]
+spark = ["pyspark>=3.5,<5"]
+parquet = ["pyarrow>=15"]
+all = [
+  "pyspark>=3.5,<5",
+  "pyarrow>=15",
+]
+dev = [
+  "pytest>=8.2",
+  "pytest-cov>=5",
+]
+
+[project.scripts]
+mainframe-toolkit = "mainframe_toolkit.cli:main"
+mft = "mainframe_toolkit.cli:main"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/mainframe_toolkit"]
+
+[tool.hatch.build.targets.wheel.force-include]
+"AGENTS.md" = "mainframe_toolkit/_workspace/AGENTS.md"
+"CLAUDE.md" = "mainframe_toolkit/_workspace/CLAUDE.md"
+".claude/skills" = "mainframe_toolkit/_workspace/.claude/skills"
+"validator-java" = "mainframe_toolkit/_workspace/validator-java"
+
+[tool.hatch.build.targets.sdist]
+include = ["/pyproject.toml", "/src", "/AGENTS.md", "/CLAUDE.md", "/.claude/skills", "/validator-java"]
+
+[tool.pytest.ini_options]
+addopts = "-q --basetemp=.pytest_tmp"
+testpaths = ["tests"]

mainframe_migration_toolkit-0.2.0/src/mainframe_toolkit/__init__.py

@@ -0,0 +1,92 @@
+"""Reusable runtime for specification-led COBOL/JCL migrations."""
+
+from .cobol import (
+    cobol_decimal,
+    decode_overpunch,
+    decode_packed_decimal,
+    encode_packed_decimal,
+    is_numeric,
+    quantize_cobol,
+)
+from .copybook import CopybookParseError, layout_to_dict, parse_copybook, parse_copybook_text
+from .external import ExternalProgramRegistry, external_program
+from .io import (
+    configure_s3,
+    read_csv,
+    read_dataset,
+    read_delimited_rows,
+    read_parquet,
+    read_s3_csv,
+    read_s3_parquet,
+    s3_uri,
+    write_csv,
+    write_dataset,
+    write_parquet,
+    write_s3_csv,
+    write_s3_parquet,
+)
+from .jcl import JCLJob, JCLStep, parse_jcl
+from .pipeline import (
+    DatasetRegistry,
+    PipelineContext,
+    PipelineProgram,
+    ProgramResult,
+    PipelineRunner,
+    PipelineRunError,
+    ProgramReturnCodeError,
+)
+from .sequential import FieldSpec, RecordLayout, convert_sequential, iter_sequential
+from .sorting import SortKey, sort_dataframe, sort_records
+from .synthetic import synthetic_dataframe, synthetic_rows
+from .workspace import WorkspaceInitResult, init_workspace
+
+__all__ = [
+    "CopybookParseError",
+    "DatasetRegistry",
+    "ExternalProgramRegistry",
+    "FieldSpec",
+    "JCLJob",
+    "JCLStep",
+    "PipelineContext",
+    "PipelineProgram",
+    "ProgramResult",
+    "ProgramReturnCodeError",
+    "PipelineRunError",
+    "PipelineRunner",
+    "RecordLayout",
+    "SortKey",
+    "WorkspaceInitResult",
+    "cobol_decimal",
+    "configure_s3",
+    "convert_sequential",
+    "decode_overpunch",
+    "decode_packed_decimal",
+    "encode_packed_decimal",
+    "external_program",
+    "is_numeric",
+    "init_workspace",
+    "iter_sequential",
+    "layout_to_dict",
+    "parse_copybook",
+    "parse_copybook_text",
+    "parse_jcl",
+    "quantize_cobol",
+    "read_csv",
+    "read_dataset",
+    "read_delimited_rows",
+    "read_parquet",
+    "read_s3_csv",
+    "read_s3_parquet",
+    "s3_uri",
+    "sort_dataframe",
+    "sort_records",
+    "synthetic_dataframe",
+    "synthetic_rows",
+    "write_csv",
+    "write_dataset",
+    "write_parquet",
+    "write_s3_csv",
+    "write_s3_parquet",
+]
+
+__version__ = "0.2.0"

mainframe_migration_toolkit-0.2.0/src/mainframe_toolkit/__main__.py

@@ -0,0 +1,5 @@
+from .cli import main
+
+
+raise SystemExit(main())
+

mainframe_migration_toolkit-0.2.0/src/mainframe_toolkit/cli.py

@@ -0,0 +1,361 @@
+from __future__ import annotations
+
+import argparse
+import csv
+import hashlib
+import json
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any, Sequence
+
+from .jcl import parse_jcl
+from .sequential import RecordLayout, convert_sequential
+from .similarity import (
+    analyze_codebase,
+    apply_ai_reviews,
+    write_ai_review_payload,
+    write_results,
+)
+from .specs import inspect_project, write_inspection
+from .synthetic import synthetic_rows
+from .workspace import init_workspace
+
+
+def _json_dump(value: Any, path: str | Path | None = None) -> None:
+    content = json.dumps(value, ensure_ascii=False, indent=2, default=str) + "\n"
+    if path is None or str(path) == "-":
+        sys.stdout.write(content)
+        return
+    output = Path(path)
+    output.parent.mkdir(parents=True, exist_ok=True)
+    output.write_text(content, encoding="utf-8")
+
+
+def _inspect(args: argparse.Namespace) -> int:
+    result = inspect_project(args.project, strict=args.strict)
+    if args.output:
+        write_inspection(result, args.output)
+    else:
+        _json_dump(result.to_dict())
+    return 0 if result.valid else 2
+
+
+def _init_workspace(args: argparse.Namespace) -> int:
+    result = init_workspace(args.destination, force=args.force)
+    _json_dump(result.to_dict())
+    return 0
+
+
+def _jcl_parse(args: argparse.Namespace) -> int:
+    _json_dump(parse_jcl(args.source).to_dict(), args.output)
+    return 0
+
+
+def _copybook_parse(args: argparse.Namespace) -> int:
+    from .copybook import layout_to_dict, parse_copybook
+
+    parsed = parse_copybook(args.source)
+    _json_dump(layout_to_dict(parsed), args.output)
+    return 0
+
+
+def _sequential_convert(args: argparse.Namespace) -> int:
+    layout = RecordLayout.from_json(args.layout)
+    result = convert_sequential(
+        args.source,
+        args.destination,
+        layout,
+        output_format=args.format,
+        mode=args.mode,
+        delimiter=args.delimiter,
+        on_error=args.on_error,
+        batch_size=args.batch_size,
+    )
+    _json_dump(result)
+    return 0
+
+
+def _similarity_scan(args: argparse.Namespace) -> int:
+    report = analyze_codebase(
+        args.root,
+        threshold=args.threshold,
+        shingle_size=args.shingle_size,
+        exhaustive_limit=args.exhaustive_limit,
+    )
+    write_results(report, args.json, args.csv)
+    write_ai_review_payload(report, args.review_request)
+    _json_dump(
+        {
+            **report.statistics,
+            "json": str(Path(args.json)),
+            "csv": str(Path(args.csv)),
+            "review_request": str(Path(args.review_request)),
+        }
+    )
+    return 0
+
+
+def _similarity_apply(args: argparse.Namespace) -> int:
+    json_output = args.json or args.report
+    csv_output = args.csv or str(Path(json_output).with_suffix(".csv"))
+    result = apply_ai_reviews(
+        args.report,
+        args.reviews,
+        json_path=json_output,
+        csv_path=csv_output,
+    )
+    records = result.records if hasattr(result, "records") else result.get("arquivos", [])
+    missing = sum(not row.get("parecer_ia") or not row.get("justificativa_ia") for row in records)
+    _json_dump({"json": json_output, "csv": csv_output, "rows": len(records), "reviews_missing": missing})
+    return 0 if missing == 0 else 1
+
+
+def _synthetic(args: argparse.Namespace) -> int:
+    payload = json.loads(Path(args.schema).read_text(encoding="utf-8"))
+    schema = payload.get("schema", payload) if isinstance(payload, dict) else payload
+    if not isinstance(schema, dict) or not all(isinstance(value, str) for value in schema.values()):
+        raise ValueError("schema must be a JSON object mapping field names to types")
+    rows = synthetic_rows(schema, count=args.rows, seed=args.seed)
+    output = Path(args.output)
+    output.parent.mkdir(parents=True, exist_ok=True)
+    if args.format == "jsonl":
+        with output.open("w", encoding="utf-8") as stream:
+            for row in rows:
+                stream.write(json.dumps(row, ensure_ascii=False, default=str) + "\n")
+    else:
+        with output.open("w", encoding="utf-8", newline="") as stream:
+            writer = csv.DictWriter(stream, fieldnames=list(schema), delimiter=args.delimiter)
+            writer.writeheader()
+            writer.writerows(rows)
+    digest = hashlib.sha256(output.read_bytes()).hexdigest()
+    manifest = Path(args.manifest) if args.manifest else output.with_suffix(output.suffix + ".manifest.json")
+    _json_dump(
+        {
+            "synthetic": True,
+            "output": str(output),
+            "format": args.format,
+            "rows": len(rows),
+            "seed": args.seed,
+            "schema": schema,
+            "sha256": digest,
+        },
+        manifest,
+    )
+    _json_dump({"output": str(output), "manifest": str(manifest), "rows": len(rows), "seed": args.seed})
+    return 0
+
+
+def _toolkit_root() -> Path:
+    override = os.environ.get("MAINFRAME_TOOLKIT_ROOT")
+    if override:
+        return Path(override).resolve()
+    candidate = Path(__file__).resolve().parents[2]
+    return candidate if (candidate / "validator-java" / "pom.xml").is_file() else Path.cwd()
+
+
+def _validator_dir(value: str | None) -> Path:
+    return Path(value).resolve() if value else _toolkit_root() / "validator-java"
+
+
+def _maven_executable(directory: Path) -> str:
+    wrapper = directory / ("mvnw.cmd" if os.name == "nt" else "mvnw")
+    if wrapper.is_file():
+        return str(wrapper)
+    executable = shutil.which("mvn") or shutil.which("mvn.cmd")
+    if not executable:
+        raise RuntimeError("Maven 3.9+ or the validator-java Maven wrapper is required")
+    return executable
+
+
+def _golden_build(args: argparse.Namespace) -> int:
+    directory = _validator_dir(args.validator_dir)
+    pom = directory / "pom.xml"
+    if not pom.is_file():
+        raise FileNotFoundError(pom)
+    command = [_maven_executable(directory), "-q", "-f", str(pom), "package"]
+    if args.skip_tests:
+        command.insert(-1, "-DskipTests")
+    process_environment = None
+    java_home = getattr(args, "java_home", None)
+    if java_home:
+        process_environment = os.environ.copy()
+        process_environment["JAVA_HOME"] = str(Path(java_home).resolve())
+    return subprocess.run(command, check=False, env=process_environment).returncode
+
+
+def _resolve_validator_jar(args: argparse.Namespace) -> Path:
+    if args.jar:
+        jar = Path(args.jar).resolve()
+    elif os.environ.get("MAINFRAME_VALIDATOR_JAR"):
+        jar = Path(os.environ["MAINFRAME_VALIDATOR_JAR"]).resolve()
+    else:
+        jar = _validator_dir(args.validator_dir) / "target" / "dataset-validator.jar"
+    if not jar.is_file() and args.build:
+        code = _golden_build(
+            argparse.Namespace(
+                validator_dir=args.validator_dir,
+                skip_tests=False,
+                java_home=(str(Path(args.java).resolve().parent.parent) if args.java else None),
+            )
+        )
+        if code != 0:
+            raise RuntimeError(f"validator-java build failed with exit code {code}")
+    if not jar.is_file():
+        raise FileNotFoundError(jar)
+    return jar
+
+
+def _golden_validate(args: argparse.Namespace) -> int:
+    requested_java = args.java or os.environ.get("MAINFRAME_JAVA")
+    java = str(Path(requested_java).resolve()) if requested_java else shutil.which("java")
+    if not java:
+        raise RuntimeError("Java 17+ is required to run the dataset validator")
+    command = [java, "-jar", str(_resolve_validator_jar(args)), args.expected, args.actual]
+    scalar_options = {
+        "--format": args.format,
+        "--expected-format": args.expected_format,
+        "--actual-format": args.actual_format,
+        "--delimiter": args.delimiter,
+        "--expected-delimiter": args.expected_delimiter,
+        "--actual-delimiter": args.actual_delimiter,
+        "--header": args.header,
+        "--expected-header": args.expected_header,
+        "--actual-header": args.actual_header,
+        "--charset": args.charset,
+        "--max-differences": args.max_differences,
+        "--report": args.report,
+    }
+    for option, value in scalar_options.items():
+        if value is not None:
+            command.extend((option, str(value)))
+    if args.no_header:
+        command.append("--no-header")
+    if args.trim:
+        command.append("--trim")
+    if args.normalize_decimal:
+        command.append("--normalize-decimal")
+    for value in args.normalize:
+        command.extend(("--normalize", value))
+    for value in args.key:
+        command.extend(("--key", value))
+    return subprocess.run(command, check=False).returncode
+
+
+def _parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(prog="mainframe-toolkit")
+    subcommands = parser.add_subparsers(dest="command", required=True)
+
+    initialize = subcommands.add_parser("init-workspace", help="materialize a migration workspace")
+    initialize.add_argument("destination", nargs="?", default=".")
+    initialize.add_argument(
+        "--force",
+        action="store_true",
+        help="update conflicting toolkit-managed files without replacing project inputs",
+    )
+    initialize.set_defaults(handler=_init_workspace)
+
+    inspect = subcommands.add_parser("inspect", help="validate the migration project contract")
+    inspect.add_argument("project", nargs="?", default=".")
+    inspect.add_argument("--output", "-o")
+    inspect.add_argument("--strict", action="store_true")
+    inspect.set_defaults(handler=_inspect)
+
+    jcl = subcommands.add_parser("jcl", help="JCL source tools").add_subparsers(dest="jcl_command", required=True)
+    jcl_parse = jcl.add_parser("parse")
+    jcl_parse.add_argument("source")
+    jcl_parse.add_argument("--output", "-o")
+    jcl_parse.set_defaults(handler=_jcl_parse)
+
+    copybook = subcommands.add_parser("copybook", help="copybook layout tools").add_subparsers(dest="copybook_command", required=True)
+    copybook_parse = copybook.add_parser("parse")
+    copybook_parse.add_argument("source")
+    copybook_parse.add_argument("--output", "-o")
+    copybook_parse.set_defaults(handler=_copybook_parse)
+
+    sequential = subcommands.add_parser("sequential", help="sequential dataset tools").add_subparsers(dest="sequential_command", required=True)
+    convert = sequential.add_parser("convert")
+    convert.add_argument("source")
+    convert.add_argument("destination")
+    convert.add_argument("--layout", required=True)
+    convert.add_argument("--format", choices=("csv", "parquet"))
+    convert.add_argument("--mode", choices=("line", "fixed"), default="line")
+    convert.add_argument("--delimiter", default=",")
+    convert.add_argument("--on-error", choices=("raise", "skip", "null"), default="raise")
+    convert.add_argument("--batch-size", type=int, default=10_000)
+    convert.set_defaults(handler=_sequential_convert)
+
+    similarity = subcommands.add_parser("similarity", help="deterministic source clustering").add_subparsers(dest="similarity_command", required=True)
+    scan = similarity.add_parser("scan")
+    scan.add_argument("root")
+    scan.add_argument("--threshold", type=float, default=80.0)
+    scan.add_argument("--shingle-size", type=int, default=3)
+    scan.add_argument("--exhaustive-limit", type=int, default=80)
+    scan.add_argument("--json", default="similarity.json")
+    scan.add_argument("--csv", default="similarity.csv")
+    scan.add_argument("--review-request", default="similarity-review-request.json")
+    scan.set_defaults(handler=_similarity_scan)
+    review = similarity.add_parser("apply-review")
+    review.add_argument("report")
+    review.add_argument("reviews")
+    review.add_argument("--json")
+    review.add_argument("--csv")
+    review.set_defaults(handler=_similarity_apply)
+
+    synthetic = subcommands.add_parser("synthetic", help="create deterministic fallback rows")
+    synthetic.add_argument("--schema", required=True)
+    synthetic.add_argument("--output", required=True)
+    synthetic.add_argument("--manifest")
+    synthetic.add_argument("--rows", type=int, default=10)
+    synthetic.add_argument("--seed", default="0")
+    synthetic.add_argument("--format", choices=("jsonl", "csv"), default="jsonl")
+    synthetic.add_argument("--delimiter", default=",")
+    synthetic.set_defaults(handler=_synthetic)
+
+    golden = subcommands.add_parser("golden", help="Java cell-level dataset validator").add_subparsers(dest="golden_command", required=True)
+    build = golden.add_parser("build")
+    build.add_argument("--validator-dir")
+    build.add_argument("--java-home")
+    build.add_argument("--skip-tests", action="store_true")
+    build.set_defaults(handler=_golden_build)
+    validate = golden.add_parser("validate")
+    validate.add_argument("expected")
+    validate.add_argument("actual")
+    validate.add_argument("--jar")
+    validate.add_argument("--java")
+    validate.add_argument("--validator-dir")
+    validate.add_argument("--build", action=argparse.BooleanOptionalAction, default=True)
+    validate.add_argument("--format", choices=("auto", "csv", "parquet"), default="auto")
+    validate.add_argument("--expected-format", choices=("auto", "csv", "parquet"))
+    validate.add_argument("--actual-format", choices=("auto", "csv", "parquet"))
+    validate.add_argument("--delimiter", default=",")
+    validate.add_argument("--expected-delimiter")
+    validate.add_argument("--actual-delimiter")
+    validate.add_argument("--no-header", action="store_true")
+    validate.add_argument("--header", choices=("true", "false"))
+    validate.add_argument("--expected-header", choices=("true", "false"))
+    validate.add_argument("--actual-header", choices=("true", "false"))
+    validate.add_argument("--charset", default="UTF-8")
+    validate.add_argument("--normalize", action="append", default=[])
+    validate.add_argument("--trim", action="store_true")
+    validate.add_argument("--normalize-decimal", action="store_true")
+    validate.add_argument("--key", action="append", default=[])
+    validate.add_argument("--max-differences", type=int, default=100)
+    validate.add_argument("--report", "-r", default="-")
+    validate.set_defaults(handler=_golden_validate)
+    return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    try:
+        args = _parser().parse_args(argv)
+        return int(args.handler(args))
+    except (FileNotFoundError, ValueError, RuntimeError) as exc:
+        sys.stderr.write(f"error: {exc}\n")
+        return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())