scanner-cli 0.1.0__py3-none-any.whl

scanner_cli-0.1.0.dist-info/METADATA

@@ -0,0 +1,199 @@
+Metadata-Version: 2.4
+Name: scanner-cli
+Version: 0.1.0
+Summary: Python command-line interface for Scanner API
+Author: Scanner, Inc.
+Author-email: support@scanner.dev
+Requires-Python: >=3.10,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: PyYAML (>=6.0.2,<7.0.0)
+Requires-Dist: click (>=8.1.7,<9.0.0)
+Requires-Dist: pydantic (>=2.0,<3.0)
+Requires-Dist: scanner-client (>=0.1.0,<0.2.0)
+Description-Content-Type: text/markdown
+
+# scanner-cli
+
+This is a Python CLI for the Scanner API.
+
+## Usage
+
+To install the CLI, run
+
+```
+pip install scanner-cli
+```
+
+## Environment Variables
+
+For various commands, you will need to supply some Scanner configuration values.
+
+For `run-tests`, `validate`, and `sync`, you will need these values:
+- Scanner API URL
+- Scanner API key
+
+For `sync`, you will also need this value:
+- Scanner Team ID
+
+You can find these values in **Settings > General** and **Settings > API Keys**
+in your Scanner account.
+
+You can either set these values as environment variables:
+
+```
+# required for run-tests, validate, sync commands:
+export SCANNER_API_URL=<Scanner API URL>
+export SCANNER_API_KEY=<Scanner API key>
+
+# required for sync command:
+export SCANNER_TEAM_ID=<Scanner Team ID>
+```
+
+or provide them as arguments to the CLI:
+
+```
+scanner-cli <command> \
+    --api-url=<Scanner API URL> \
+    --api-key=<Scanner API key> \
+    --team-id=<Scanner Team ID> \
+    ...
+```
+
+## Commands
+
+Available commands are
+- `run-tests` - run tests on detection rules as code
+- `validate` - validate detection rules as code
+- `sync` - sync detection rules to Scanner
+- `migrate-elastic-rules` - migrate Elastic SIEM detection rules to Scanner YAML rules
+
+### `run-tests` and `validate`
+
+To validate or run tests on files:
+
+```
+scanner-cli validate -f detections/errors.yaml -f detections/unauthorized_logins.yaml
+scanner-cli run-tests -f detections/errors.yaml -f detections/unauthorized_logins.yaml
+```
+
+To validate or run tests on directories:
+
+```
+scanner-cli validate -d detections
+scanner-cli run-tests -d detections
+```
+
+To recursively validate or run-tests on a directory, use `-r` or `--recursive`:
+
+```
+scanner-cli validate -r -d detections
+scanner-cli run-tests -r -d detections
+
+scanner-cli validate --recursive -d detections
+scanner-cli run-tests --recursive -d detections
+```
+
+This will only validate or run tests on YAML files with the correct schema header.
+
+A file or directory must be provided. Multiple files and/or directories can be provided.
+
+### `sync`
+
+This command syncs detection rules to your Scanner account.
+
+You can sync individual files or full directories.
+
+```
+scanner-cli sync -f detections/errors.yaml -f detections/unauthorized_logins.yaml
+scanner-cli sync -r -d detections
+```
+
+#### `sync_key`
+
+Each detection rule must have the `sync_key` field defined. This is a unique
+identifier for the rule and can be any string you wish, as long as it is
+unique.
+
+#### `event_sink_keys`
+
+If your detection rules have `event_sink_keys` defined, you must provide a sync
+configuration file in YAML format using the `--sync-config-file` flag.
+
+This file allows you to map `event_sink_keys` in your detection rules (eg.
+`low_severity_alerts`, `high_severity_alerts`, etc.) to specific Scanner event
+sinks (eg. "Custom webhook", "Slack alerts channel", etc.)
+
+```
+scanner-cli sync --sync-config-file sync_config.yaml -r -d detections/
+```
+
+Where is what the sync configuration file looks like:
+```
+event_sink_keys:
+  low_severity_alerts:
+    - sink_id: "5098b2bd-065c-4c0d-9f11-685e88808fc2"
+  medium_severity_alerts:
+    - sink_id: "5098b2bd-065c-4c0d-9f11-685e88808fc2"
+    - sink_id: "62741ace-ea22-4255-8a36-b921a282d61e"
+  high_severity_alerts:
+    - sink_id: "5098b2bd-065c-4c0d-9f11-685e88808fc2"
+    - sink_id: "62741ace-ea22-4255-8a36-b921a282d61e"
+    - sink_id: "2313eb73-0020-4814-9b35-864e3d3439e0"
+```
+
+In this example, the `low_severity_alerts` event sink key is mapped to a single
+event sink, while the `medium_severity_alerts` and `high_severity_alerts` event
+sink keys are mapped to multiple event sinks.
+
+For instance, you might send low severity alerts to a SOAR webhook, but send
+medium and high severity alerts to a high-priority Slack channel and multiple
+webhooks.
+
+To find the Sink ID for a specific event sink, visit **Settings > Event Sinks**
+and click on the event sink you want to use.
+
+### `migrate-elastic-rules`
+
+This command migrates an `ndjson` file containing Elastic SIEM detection rules to Scanner YAML rules.
+
+For each rule in the `ndjson` file, a YAML file will be created in the output directory.
+
+```
+scanner-cli migrate-elastic-rules --elastic-files-rule elastic_rules.ndjson --output-dir scanner_rules/
+```
+
+Optionally, you can provide a migrate config file in YAML format.
+
+The migrate config file allows you to map Elastic Data View IDs to Scanner
+query terms. This way, the queries in your Scanner detection rules can be more
+selective about which logs they check against.
+
+```
+scanner-cli migrate-elastic-rules \
+    --migrate-config-file elastic_to_scanner_config.yaml \
+    --elastic-files-rule elastic_rules.ndjson \
+    --output-dir scanner_rules/
+```
+
+Here is an example migrate config file with a few mappings:
+- Map from an Elastic Data View ID to a Scanner index.
+- Map from an Elastic Data View wildcard name to a Scanner query for a specific
+  source type.
+- Map from an Elastic Data View wildcard name to a Scanner query with multiple
+  terms.
+
+```
+data_view_id_to_query_term:
+  e9874b58-5cee-40e0-8b49-5c3739572ea2: |-
+    @index={ 0f75b7fd-ea1e-421a-b4ee-007ac8570a20 | "application_logs" }
+  log-sources:aws:cloudtrail:*: |-
+    %ingest.source_type="aws:cloudtrail"
+  log-sources:non-prod-app-logs:*: |-
+    my_env=("staging" or "dev") and my_type="app_logs"
+```
+

scanner_cli-0.1.0.dist-info/RECORD

@@ -0,0 +1,13 @@
+src/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+src/cli.py,sha256=rShrleZukvvGZGP_d0BCzLu4LD_Ri0NHa6Eu0mxNBbM,13231
+src/detection_rule_yaml.py,sha256=z_GDtBLcfteib5N5XvRRY7Enk8xOFEE2G_FLNgRsnqI,1177
+src/migrate/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+src/migrate/elastic.py,sha256=y_cn_3Y_t8NHwYcpAQQLfnCkEBSULoU_vxxGwk47QAc,8681
+src/sync.py,sha256=ch5pAIOpRADleaVnxNql2dRwzYiwiEvJxCRbZY1oaJ0,4070
+src/sync_git_repo.py,sha256=o5tWV_LDpvetuleSIaue6xdjhUCWc2Stfhutf5wSf1k,8297
+src/sync_git_repo_result.py,sha256=N1id87RjdzNZSQvWwpbaaq_DBVexF0Td2-tdHVq-HUo,3637
+src/utils.py,sha256=E5tJmAJrIQdYhAX60ZYqtqvHCh3lu4G32mzpEip4r4g,859
+scanner_cli-0.1.0.dist-info/METADATA,sha256=YRfnM2xZJri81eLWU2LtqMizgF_NkvtJSAzrKU8zd1o,6023
+scanner_cli-0.1.0.dist-info/WHEEL,sha256=kJCRJT_g0adfAJzTx2GUMmS80rTJIVHRCfG0DQgLq3o,88
+scanner_cli-0.1.0.dist-info/entry_points.txt,sha256=vqXMrIG6N6pY66bNf0y-gbUxbU8v5dXvuL3mV832Fh8,43
+scanner_cli-0.1.0.dist-info/RECORD,,

scanner_cli-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: poetry-core 2.3.1
+Root-Is-Purelib: true
+Tag: py3-none-any

scanner_cli-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+scanner-cli=src.cli:cli
+

src/cli.py

@@ -0,0 +1,402 @@
+"""Contains code for Python CLI"""
+
+import os
+from typing import Any, Callable, Optional
+
+import click
+
+from scanner_client.scanner import Scanner
+
+import src.migrate.elastic as elastic_cmd
+import src.sync as sync_cmd
+import src.sync_git_repo as sync_git_repo_cmd
+from src.detection_rule_yaml import validate_and_read_file
+from src.utils import format_exception_message
+
+_DEFAULT_CLICK_OPTIONS = [
+    click.option(
+        "--api-url",
+        envvar="SCANNER_API_URL",
+        help="The API URL of your Scanner instance. Go to Settings > API Keys in Scanner to find your API URL.",
+    ),
+    click.option(
+        "--api-key",
+        envvar="SCANNER_API_KEY",
+        help="Scanner API key. Go to Settings > API Keys in Scanner to find your API keys or to create a new API key.",
+    ),
+    click.option(
+        "-f",
+        "--file",
+        "file_paths",
+        help="Detection rule file. This must be .yml or .yaml file with the correct schema header.",
+        multiple=True,
+    ),
+    click.option(
+        "-d",
+        "--dir",
+        "directories",
+        help="Directory of detection rule files. Only .yml or .yaml files with the correct schema header will be processed.",
+        multiple=True,
+    ),
+    click.option(
+        "-r",
+        "recursive",
+        is_flag=True,
+        show_default=True,
+        default=False,
+        help="Recursively search directory for valid YAML files.",
+    ),
+]
+
+
+def _default_click_options(func) -> Callable[..., Any]:
+    for option in reversed(_DEFAULT_CLICK_OPTIONS):
+        func = option(func)
+
+    return func
+
+
+def _is_valid_file(file_path: str) -> bool:
+    try:
+        validate_and_read_file(file_path)
+        return True
+    except:
+        return False
+
+
+def _get_valid_files_in_directory(directory: str, recursive: bool) -> list[str]:
+    if not os.path.exists(directory):
+        raise click.exceptions.ClickException(
+            message=(f"Directory {directory} not found.")
+        )
+
+    # Skip symlinks under directory recursion. The user pointed at a dir,
+    # not specific files — a symlink in there (e.g. via an attacker PR)
+    # could read sensitive content outside the dir. `followlinks=False`
+    # blocks symlinked sub-dirs; the per-file `islink` check blocks
+    # symlinked files. Explicit `-f` paths bypass this gate because the
+    # user vouched for the file by name.
+    files: list[str] = []
+    for root, _dirs, filenames in os.walk(directory, followlinks=False):
+        for name in filenames:
+            full = os.path.join(root, name)
+            if not os.path.islink(full) and _is_valid_file(full):
+                files.append(full)
+        if not recursive:
+            break
+    return files
+
+
+def _get_valid_files(
+    file_paths: tuple[str, ...], directories: tuple[str, ...], recursive: bool
+) -> list[str]:
+    files = [f for f in file_paths if _is_valid_file(f)]
+
+    for d in directories:
+        files.extend(_get_valid_files_in_directory(d, recursive))
+
+    return files
+
+
+def _validate_default_options(
+    api_url: Optional[str],
+    api_key: Optional[str],
+    file_paths: tuple[str, ...],
+    directories: tuple[str, ...],
+) -> tuple[str, str, tuple[str, ...], tuple[str, ...]]:
+    if api_url is None:
+        raise click.exceptions.UsageError(
+            message=(
+                "Pass --api-url option or set `SCANNER_API_URL` environment variable."
+            )
+        )
+
+    if api_key is None:
+        raise click.exceptions.UsageError(
+            message=(
+                "Pass --api-key option or set `SCANNER_API_KEY` environment variable."
+            )
+        )
+
+    if not file_paths and not directories:
+        raise click.exceptions.UsageError(
+            message=("Either --file or --dir must be provided.")
+        )
+
+    return (api_url, api_key, file_paths, directories)
+
+
+@click.group()
+def cli():
+    """Python CLI for Scanner API"""
+
+
+@cli.command()
+@_default_click_options
+def validate(
+    api_url: Optional[str],
+    api_key: Optional[str],
+    file_paths: tuple[str, ...],
+    directories: tuple[str, ...],
+    recursive: bool,
+):
+    """Validate detection rule files"""
+    api_url, api_key, file_paths, directories = _validate_default_options(
+        api_url, api_key, file_paths, directories
+    )
+
+    scanner_client = Scanner(api_url, api_key)
+
+    files = _get_valid_files(file_paths, directories, recursive)
+    click.echo(f'Validating {len(files)} {"file" if len(files) == 1 else "files"}')
+
+    any_failures: bool = False
+
+    for file in files:
+        try:
+            result = scanner_client.detection_rule_yaml.validate(
+                validate_and_read_file(file)
+            )
+
+            if result.is_valid:
+                if result.warning:
+                    click.echo(
+                        f"{file}: "
+                        + click.style("OK. ", fg="green")
+                        + click.style(f"Warning: {result.warning}", fg="yellow")
+                    )
+                else:
+                    click.echo(f"{file}: " + click.style("OK", fg="green"))
+            else:
+                any_failures = True
+                click.echo(f"{file}: " + click.style(f"{result.error}", fg="red"))
+        except Exception as e:
+            any_failures = True
+            error_msg = format_exception_message(
+                e, "An exception occurred when attempting to validate file"
+            )
+            click.echo(f"{file}: " + click.style(error_msg, fg="red"))
+
+    if any_failures:
+        # To make it so the CLI exits with a non-zero exit code
+        raise click.ClickException(
+            "`validate` failed for one or more files. See https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/detections-and-alerting/detection-rules/detection-rules-as-code/writing-detection-rules for requirements."
+        )
+    else:
+        click.secho(
+            "All specified detection rule files are valid. Use `run-tests` to run the detection rule tests.",
+            bold=True,
+        )
+
+
+@cli.command()
+@_default_click_options
+def run_tests(
+    api_url: Optional[str],
+    api_key: Optional[str],
+    file_paths: tuple[str, ...],
+    directories: tuple[str, ...],
+    recursive: bool,
+):
+    """Run detection rule tests"""
+    api_url, api_key, file_paths, directories = _validate_default_options(
+        api_url, api_key, file_paths, directories
+    )
+
+    scanner_client = Scanner(api_url, api_key)
+
+    files = _get_valid_files(file_paths, directories, recursive)
+    click.echo(
+        f'Running tests on {len(files)} {"file" if len(files) == 1 else "files"}'
+    )
+
+    any_validation_errors: bool = False
+    any_test_failures: bool = False
+
+    for file in files:
+        click.secho(f"{file}", bold=True)
+        try:
+            rule_yaml = validate_and_read_file(file)
+
+            # Check for validation errors
+            validation_result = scanner_client.detection_rule_yaml.validate(rule_yaml)
+            if not validation_result.is_valid:
+                any_validation_errors = True
+                click.secho(f"Validation error: {validation_result.error}", fg="red")
+                click.echo("")
+                continue
+
+            if validation_result.warning:
+                click.secho(f"Warning: {validation_result.warning}", fg="yellow")
+
+            # Run detection rule tests
+            run_tests_response = scanner_client.detection_rule_yaml.run_tests(rule_yaml)
+            results = run_tests_response.results.to_dict()
+
+            if len(results) == 0:
+                click.secho("No tests found", fg="yellow")
+            else:
+                for name, status in results.items():
+                    if status == "Passed":
+                        click.echo(f"{name}: " + click.style("OK", fg="green"))
+                    else:
+                        any_test_failures = True
+                        click.echo(f"{name}: " + click.style("Test failed", fg="red"))
+        except Exception as e:
+            any_test_failures = True
+            error_msg = format_exception_message(
+                e, "An exception occurred when attempting to run tests"
+            )
+            click.secho(error_msg, fg="red")
+
+        click.echo("")
+
+    error_messages = []
+    if any_validation_errors:
+        error_messages.append(
+            "Validation failed for one or more files. See https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/detections-and-alerting/detection-rules/detection-rules-as-code/writing-detection-rules for requirements."
+        )
+
+    if any_test_failures:
+        error_messages.append(
+            "`run-tests` failed for one or more files. See https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/detections-and-alerting/detection-rules/detection-rules-as-code/cli#failing-tests for more information."
+        )
+
+    if error_messages:
+        # To make it so the CLI exits with a non-zero exit code
+        raise click.ClickException("\n".join(error_messages))
+
+
+@cli.command()
+@_default_click_options
+@click.option(
+    "--team-id",
+    envvar="SCANNER_TEAM_ID",
+    help="The team ID to which you want to sync the Scanner rules. Go to Settings > General to find the Team ID.",
+    required=True,
+)
+@click.option(
+    "--sync-config-file",
+    help=(
+        "Optional. The path to the sync configuration file the CLI will use to "
+        "sync detection rules to Scanner (eg. contains event_sink_keys mappings, etc)."
+    ),
+    required=False,
+)
+def sync(
+    api_url: Optional[str],
+    api_key: Optional[str],
+    file_paths: tuple[str, ...],
+    directories: tuple[str, ...],
+    recursive: bool,
+    team_id: str,
+    sync_config_file: Optional[str],
+):
+    """Sync detection rules to Scanner"""
+    api_url, api_key, file_paths, directories = _validate_default_options(
+        api_url, api_key, file_paths, directories
+    )
+    if team_id is None:
+        raise click.exceptions.UsageError(
+            message=(
+                "Pass --team-id option or set `SCANNER_TEAM_ID` environment variable."
+            )
+        )
+
+    scanner_client: Scanner = Scanner(api_url, api_key)
+    files: list[str] = _get_valid_files(file_paths, directories, recursive)
+    # Note: In the Scanner UI, the tenant_id is called Team ID.
+    sync_cmd.sync(scanner_client, files, team_id, sync_config_file)
+
+
+@cli.command(
+    name="sync-git-repo",
+    short_help="Sync an entire git repo to Scanner.",
+)
+@click.option(
+    "--api-url",
+    envvar="SCANNER_API_URL",
+    help="The API URL of your Scanner instance. Go to Settings > API Keys in Scanner to find your API URL.",
+)
+@click.option(
+    "--api-key",
+    envvar="SCANNER_API_KEY",
+    help="Scanner API key. Go to Settings > API Keys in Scanner to find your API keys or to create a new API key.",
+)
+@click.option(
+    "--push-key",
+    required=True,
+    help="The push key identifying the target GithubRepoSyncSource on Scanner. Configure one under Settings > Detection Rule Sync.",
+)
+@click.option(
+    "--json",
+    "json_output",
+    is_flag=True,
+    default=False,
+    help="Emit a single JSON object on stdout instead of human-readable output. Exit code is non-zero iff sync failed.",
+)
+@click.argument(
+    "path",
+    type=click.Path(exists=True, file_okay=False, dir_okay=True),
+    default=".",
+)
+def sync_git_repo(
+    api_url: Optional[str],
+    api_key: Optional[str],
+    push_key: str,
+    json_output: bool,
+    path: str,
+):
+    """Sync an entire git repo to Scanner via the push-sync zipball endpoint.
+
+    PATH is a directory inside a git repo (defaults to the current directory).
+    The current branch (must not be detached) and commit SHA (with `+dirty`
+    suffix if the working tree is dirty) are uploaded alongside the zip; the
+    server reconciles detection rules against the `--push-key` sync source.
+    """
+    if api_url is None:
+        raise click.exceptions.UsageError(
+            message=(
+                "Pass --api-url option or set `SCANNER_API_URL` environment variable."
+            )
+        )
+    if api_key is None:
+        raise click.exceptions.UsageError(
+            message=(
+                "Pass --api-key option or set `SCANNER_API_KEY` environment variable."
+            )
+        )
+
+    scanner_client: Scanner = Scanner(api_url, api_key)
+    sync_git_repo_cmd.sync_git_repo(scanner_client, path, push_key, json_output)
+
+
+@cli.command()
+@click.option(
+    "-c",
+    "--migrate-config-file",
+    help="Optional. The path to the migration configuration file the CLI will use to migrate Elastic rules (eg. contains data_view_id_to_query_term mappings, etc).",
+)
+@click.option(
+    "-f",
+    "--elastic-rules-file",
+    help="The path to the Elastic detection rules ndjson file the CLI will migrate.",
+    required=True,
+)
+@click.option(
+    "-o",
+    "--output-dir",
+    help="The directory where the migrated rules will be saved. Will create one YAML file per rule.",
+    required=True,
+)
+def migrate_elastic_rules(
+    migrate_config_file: Optional[str], elastic_rules_file: str, output_dir: str
+):
+    """Migrate Elastic SIEM rules to Scanner rules"""
+    elastic_cmd.migrate_elastic_rules(
+        migrate_config_file, elastic_rules_file, output_dir
+    )
+
+
+if __name__ == "__main__":
+    cli()

src/detection_rule_yaml.py

@@ -0,0 +1,42 @@
+"""Schema-header validation for detection rule YAML files.
+
+These helpers used to live in `scanner_client.detection_rule_yaml`, but
+the upstream library dropped them — schema validation is a CLI concern
+(we want a friendly local error before any network round-trip), so the
+canonical home is here.
+"""
+
+import os
+
+
+DETECTION_RULE_SCHEMA_HEADER = (
+    "# schema: https://scanner.dev/schema/scanner-detection-rule.v1.json"
+)
+
+
+def has_yaml_extension(file_path: str) -> bool:
+    return file_path.endswith(".yml") or file_path.endswith(".yaml")
+
+
+def contains_schema(contents: str) -> bool:
+    return DETECTION_RULE_SCHEMA_HEADER in contents
+
+
+def validate_and_read_file(file_path: str) -> str:
+    if not os.path.exists(file_path):
+        raise Exception(f"File {file_path} not found.")
+
+    if not has_yaml_extension(file_path):
+        raise Exception(
+            f"File {file_path} does not have a .yml or .yaml extension."
+        )
+
+    with open(file_path, "r") as f:
+        contents = f.read()
+
+    if not contains_schema(contents):
+        raise Exception(
+            f"File {file_path} does not contain the correct schema header."
+        )
+
+    return contents

src/migrate/elastic.py

@@ -0,0 +1,235 @@
+import json
+import os
+from typing import Any, Optional
+
+import click
+import yaml
+from yaml import Dumper
+
+
+class YamlStrLiteral(str):
+    """
+    A class to represent a YAML literal string, which may have multiple lines.
+    Helps us to represent a string as a literal block scalar in YAML.
+    """
+    pass
+
+
+def _represent_literal(dumper: Dumper, data: YamlStrLiteral) -> Any:
+    return dumper.represent_scalar(
+            yaml.resolver.BaseResolver.DEFAULT_SCALAR_TAG,
+            data,
+            style="|"
+    )
+
+
+yaml.add_representer(YamlStrLiteral, _represent_literal)
+
+
+def _validate_options(migrate_config_file: Optional[str], elastic_rules_file: str, output_dir: str):
+    if migrate_config_file and not os.path.exists(migrate_config_file):
+        raise click.exceptions.UsageError(
+            message=(
+                "Config file not found."
+            )
+        )
+    if not os.path.exists(elastic_rules_file):
+        raise click.exceptions.UsageError(
+            message=(
+                "Elastic rules file not found."
+            )
+        )
+    if not os.path.isdir(output_dir):
+        raise click.exceptions.UsageError(
+            message=(
+                "Output directory not found."
+            )
+        )
+
+
+def _migrate_to_scanner_yml(elastic_detection_rule: dict[str, Any], config: dict[str, Any]) -> dict[str, Any]:
+    name: str = elastic_detection_rule.get('name', '')
+    description: YamlStrLiteral = YamlStrLiteral(elastic_detection_rule.get('note', ''))
+    enabled: bool = elastic_detection_rule.get('enabled', True)
+    severity: str = _capitalize_first_letter(elastic_detection_rule.get('severity', 'unknown'))
+    query_text: YamlStrLiteral = YamlStrLiteral(_get_query_text_for_detection_rule(elastic_detection_rule, config))
+    time_range_s: int = 300
+    run_frequency_s: int = 60
+    event_sink_key: str = _migrate_severity_to_event_sink_key(severity)
+    file_name: str = _generate_rule_file_name(name)
+    scanner_rule: dict[str, Any] = {
+        'sync_key': file_name,
+        'name': name,
+        'description': description,
+        'enabled': enabled,
+        'severity': severity,
+        'query_text': query_text,
+        'time_range_s': time_range_s,
+        'run_frequency_s': run_frequency_s,
+        'event_sink_keys': [event_sink_key],
+    }
+    yaml_content: str = yaml.dump(
+        scanner_rule,
+        default_flow_style=False,
+        sort_keys=False
+    )
+    yaml_content = yaml_content.replace('\\', '\\\\')
+    yaml_content = "# schema: https://scanner.dev/schema/scanner-detection-rule.v1.json\n" + yaml_content
+    return {
+        'file_name': file_name,
+        'yaml_content': yaml_content,
+    }
+
+
+def _generate_rule_file_name(name: str) -> str:
+    chars = []
+    for c in name:
+        if c.isalnum():
+            chars.append(c.lower())
+        elif chars and chars[-1] != ' ':
+            chars.append(' ')
+    file_name = ''.join(chars)
+    file_name = file_name.strip()
+    file_name = file_name.replace(' ', '_')
+    return f"{file_name}.yml"
+
+
+def _migrate_severity_to_event_sink_key(severity: str) -> str:
+    return f"{severity.lower()}_severity_alerts"
+
+
+def _capitalize_first_letter(text: str) -> str:
+    if text:
+        return text[0].upper() + text[1:]
+    return text
+
+
+def _get_query_text_for_detection_rule(elastic_detection_rule: dict[str, Any], config: dict[str, Any]) -> str:
+    query_parts: list[str] = []
+    data_view_id: Optional[str] = elastic_detection_rule.get('data_view_id')
+    data_view_id_query_term: Optional[str] = config.get('data_view_id_to_query_term', {}).get(data_view_id)
+    if data_view_id_query_term:
+        query_parts.append(data_view_id_query_term)
+    main_query: Optional[str] = elastic_detection_rule.get('query')
+    if main_query:
+        query_parts.append(main_query)
+    filters: list[dict[str, Any]] = elastic_detection_rule.get('filters', [])
+    for filter in filters:
+        query_text = _get_query_text_for_filter(filter)
+        if query_text:
+            query_parts.append(query_text)
+    return "\n".join(query_parts)
+
+
+def _get_query_text_for_filter(f: dict[str, Any]) -> Optional[str]:
+    should_negate: bool = f.get('meta', {}).get('negate', False)
+    filter_query: dict[str, Any] = f.get('query', {})
+    query_text: Optional[str] = _get_query_text_for_filter_query(filter_query)
+    if query_text is None:
+        return None
+    if should_negate:
+        query_text = f"not {query_text}"
+    return query_text
+
+
+def _get_query_text_for_filter_query(filter_query: dict[str, Any]) -> Optional[str]:
+    match_phrase_query: Optional[dict[str, Any]] = filter_query.get('match_phrase')
+    bool_query: Optional[dict[str, Any]] = filter_query.get('bool')
+    exists_query: Optional[dict[str, Any]] = filter_query.get('exists')
+    if match_phrase_query:
+        return _get_query_text_for_match_phrase_query(match_phrase_query)
+    elif bool_query:
+        return _get_query_text_for_bool_query(bool_query)
+    elif exists_query:
+        return _get_query_text_for_exists_query(exists_query)
+    return None
+
+
+def _get_query_text_for_match_phrase_query(match_phrase_query: dict[str, Any]) -> str:
+    query_parts: list[str] = []
+    for field, value in match_phrase_query.items():
+        query_parts.append(f"{field}: \"{value}\"")
+    has_multiple_parts: bool = len(query_parts) > 1
+    query_text: str = "\n".join(query_parts)
+    if has_multiple_parts:
+        return f"({query_text})"
+    else:
+        return query_text
+
+
+def _get_query_text_for_bool_query(bool_query: dict[str, Any]) -> Optional[str]:
+    minimum_should_match: Optional[int] = bool_query.get('minimum_should_match')
+    if minimum_should_match != 1:
+        # Print error message that this is unsupported
+        click.secho("Unsupported filter: Only support bool minimum_should_match = 1", fg="red")
+        return None
+    query_parts: list[str] = []
+    should_clauses: list[dict[str, Any]] = bool_query.get('should', [])
+    for should_clause in should_clauses:
+        query_text: Optional[str] = _get_query_text_for_filter_query(should_clause)
+        if query_text is None:
+            return None
+        query_parts.append(query_text)
+    has_multiple_parts: bool = len(query_parts) > 1
+    returned_query_text: str = " or ".join(query_parts)
+    if has_multiple_parts:
+        return f"({returned_query_text})"
+    else:
+        return returned_query_text
+
+
+def _get_query_text_for_exists_query(exists_query: dict[str, Any]) -> Optional[str]:
+    field: Optional[str] = exists_query.get('field')
+    if field:
+        return f"{field}: *"
+    else:
+        return None
+
+
+def migrate_elastic_rules(migrate_config_file: Optional[str], elastic_rules_file: str, output_dir: str):
+    _validate_options(migrate_config_file, elastic_rules_file, output_dir)
+
+    any_failures: bool = False
+
+    count: int = 0
+    try:
+        config: dict[str, Any] = {}
+        if migrate_config_file:
+            with open(migrate_config_file, 'r') as file:
+                config = yaml.safe_load(file)
+        with open(elastic_rules_file, 'r') as file:
+            for raw_line in file:
+                line: str = raw_line.strip()
+                if not line:  # Skip empty lines
+                    continue
+                try:
+                    elastic_detection_rule: dict[str, Any] = json.loads(line)
+                    migrated: dict[str, Any] = _migrate_to_scanner_yml(elastic_detection_rule, config)
+                    output_file_path: str = f"{output_dir}/{migrated['file_name']}"
+                    with open(output_file_path, 'w') as output_file:
+                        output_file.write(migrated['yaml_content'])
+                    count += 1
+                    click.echo(click.style("Migrated", fg="green") + f": {output_file_path}")
+                except json.JSONDecodeError as e:
+                    click.secho(f"Error parsing JSON on line: {line}", fg="red")
+                    click.secho(f"Error details: {e}", fg="red")
+                    click.echo("")
+                    any_failures = True
+                    continue
+    except yaml.YAMLError as e:
+        any_failures = True
+        click.secho(f"YAML Error: {e}", fg="red")
+        click.echo("")
+    except BaseException as e:
+        any_failures = True
+        click.secho(f"Error: {e}", fg="red")
+        click.echo("")
+
+    click.secho(f"Successfully migrated {count} rules", fg="green")
+    click.echo(click.style("Output directory", fg="green") + f": {output_dir}")
+
+    if any_failures:
+        # To make it so the CLI exits with a non-zero exit code
+        raise click.ClickException(
+            "migrate-elastic-rules failed for one or more rules"
+        )

src/sync.py

@@ -0,0 +1,114 @@
+from typing import Any, Optional
+
+import click
+import yaml
+
+from scanner_client.scanner import Scanner
+
+from src.detection_rule_yaml import validate_and_read_file
+
+
+def _extract_sync_key(rule_yaml: str) -> Optional[str]:
+    """Pull the rule's `push_key` (preferred) or legacy `sync_key` out of
+    the YAML so we can print it in per-file status lines. The server is
+    the authoritative validator — this is just for cosmetic output."""
+    try:
+        doc: Any = yaml.safe_load(rule_yaml)
+    except yaml.YAMLError:
+        return None
+    if not isinstance(doc, dict):
+        return None
+    key = doc.get("push_key") or doc.get("sync_key")
+    return key if isinstance(key, str) else None
+
+
+def sync(
+    scanner_client: Scanner,
+    files: list[str],
+    tenant_id: str,
+    sync_config_file: Optional[str],
+):
+    click.echo(
+        f'Syncing {len(files)} detection rule {"file" if len(files) == 1 else "files"} to Scanner Team {tenant_id}'
+    )
+
+    # We don't parse the sync config ourselves — the server does, via the
+    # `/v1/detection_rule_yaml/upsert_by_push_key` endpoint. Keeping the
+    # CLI thin here means a new field doesn't require a CLI release.
+    sync_config_yaml: str = ""
+    if sync_config_file:
+        with open(sync_config_file, "r") as f:
+            sync_config_yaml = f.read()
+
+    # Server-side validate every file first so a syntax error in one rule
+    # aborts before we apply any of them. We slurp each rule's YAML once
+    # here and reuse it for the upsert pass below.
+    click.echo("Validating rules before syncing...")
+    any_failures = False
+    rule_yamls: dict[str, str] = {}
+    sync_keys: dict[str, str] = {}
+    for file in files:
+        try:
+            rule_yaml = validate_and_read_file(file)
+            result = scanner_client.detection_rule_yaml.validate(rule_yaml)
+            if not result.is_valid:
+                any_failures = True
+                click.echo(f"{file}: " + click.style(f"{result.error}", fg="red"))
+                continue
+
+            # Missing key is a hard error before we hit the server. We
+            # accept either `push_key` (preferred) or `sync_key` (legacy)
+            # — same field on the wire.
+            sync_key = _extract_sync_key(rule_yaml)
+            if not sync_key:
+                any_failures = True
+                click.secho(
+                    f"Error: push_key (or sync_key) not found in {file}",
+                    fg="red",
+                )
+                continue
+
+            click.echo(f"{file}: " + click.style("Valid", fg="green"))
+            rule_yamls[file] = rule_yaml
+            sync_keys[file] = sync_key
+        except Exception as e:
+            any_failures = True
+            click.echo(f"{file}: " + click.style(e, fg="red"))
+
+    if any_failures:
+        raise click.ClickException(
+            "validate failed for one or more files. Sync aborted."
+        )
+
+    click.secho("All rules are valid", fg="green")
+    click.echo("")
+
+    click.echo("Syncing rules to Scanner...")
+    num_created = 0
+    num_updated = 0
+    for file in files:
+        try:
+            resp = scanner_client.detection_rule_yaml.upsert_by_push_key(
+                rule_yaml=rule_yamls[file],
+                sync_config_yaml=sync_config_yaml,
+            )
+            label = "Created" if resp.created else "Updated"
+            click.echo(f"{sync_keys[file]}: " + click.style(label, fg="green"))
+            if resp.created:
+                num_created += 1
+            else:
+                num_updated += 1
+        except Exception as e:
+            any_failures = True
+            click.echo(click.style("Failed to sync file", fg="red") + f": {file}")
+            click.echo(click.style("Error", fg="red") + f": {e}")
+            click.echo("")
+            break
+
+    if any_failures:
+        raise click.ClickException("sync failed for one or more files")
+
+    if num_created > 0:
+        click.secho(f"Created {num_created} rule(s)", fg="green")
+    if num_updated > 0:
+        click.secho(f"Updated {num_updated} rule(s)", fg="green")

src/sync_git_repo.py

@@ -0,0 +1,226 @@
+"""sync-git-repo command implementation.
+
+Packages a git working tree into a zip archive and uploads it to Scanner's
+push-sync endpoint. The caller supplies the `push_key` that identifies the
+target `GithubRepoSyncSource` on the server side — this lets the CLI stay
+oblivious to per-repo Scanner config; we just ship bytes.
+"""
+
+import io
+import os
+import subprocess
+import zipfile
+
+import click
+
+from scanner_client.scanner import Scanner
+
+from src.sync_git_repo_result import (
+    SyncGitRepoFailure,
+    SyncGitRepoFailureStatus,
+    SyncGitRepoResult,
+    SyncGitRepoWarning,
+)
+
+
+def _run_git(repo_path: str, *args: str) -> str:
+    """Run `git -C repo_path <args>` and return stdout, raising
+    `UsageError` if git is missing or the command fails."""
+    try:
+        result = subprocess.run(
+            ["git", "-C", repo_path, *args],
+            capture_output=True,
+            text=True,
+        )
+    except FileNotFoundError:
+        raise click.exceptions.UsageError("`git` not found in PATH.")
+    if result.returncode != 0:
+        raise click.exceptions.UsageError(
+            f"git {' '.join(args)} failed: {result.stderr.strip() or result.stdout.strip()}"
+        )
+    return result.stdout
+
+
+def _resolve_git_root(path: str) -> str:
+    return _run_git(path, "rev-parse", "--show-toplevel").strip()
+
+
+def _get_branch(repo_root: str) -> str:
+    branch = _run_git(repo_root, "rev-parse", "--abbrev-ref", "HEAD").strip()
+    if branch == "HEAD":
+        raise click.exceptions.UsageError(
+            "Repository is in a detached HEAD state. Check out a branch before "
+            "running `sync-git-repo`."
+        )
+    return branch
+
+
+def _get_commit_sha(repo_root: str) -> str:
+    sha = _run_git(repo_root, "rev-parse", "HEAD").strip()
+    # `git status --porcelain` is empty iff the working tree (incl. index) is
+    # clean. Untracked files count as dirty too — matches what a user
+    # eyeballing `git status` would call "dirty".
+    porcelain = _run_git(repo_root, "status", "--porcelain")
+    if porcelain.strip():
+        sha = f"{sha}+dirty"
+    return sha
+
+
+def _list_trackable_files(repo_root: str) -> list[str]:
+    """Return the repo-relative paths of every file `git status` would
+    consider part of the project: tracked + untracked, minus anything
+    gitignored. `-z` keeps us safe against filenames with spaces or
+    newlines."""
+    try:
+        result = subprocess.run(
+            [
+                "git",
+                "-C",
+                repo_root,
+                "ls-files",
+                "--cached",
+                "--others",
+                "--exclude-standard",
+                "-z",
+            ],
+            capture_output=True,
+        )
+    except FileNotFoundError:
+        raise click.exceptions.UsageError("`git` not found in PATH.")
+    if result.returncode != 0:
+        raise click.exceptions.UsageError(
+            f"git ls-files failed: {result.stderr.decode(errors='replace').strip()}"
+        )
+    raw = result.stdout
+    if not raw:
+        return []
+    # `-z` emits NUL-terminated entries with a trailing NUL.
+    # `surrogateescape` round-trips non-UTF-8 bytes through Python's
+    # filesystem-string convention (PEP 383), so paths flow back into
+    # `os.path` / `open` / `zipfile.write` cleanly — and a repo with
+    # exotic filename encodings doesn't abort the whole sync at the
+    # listing step.
+    return [p.decode(errors="surrogateescape") for p in raw.split(b"\0") if p]
+
+
+def _build_zipball_bytes(repo_root: str) -> bytes:
+    """Return the bytes of a zip containing every file `git ls-files`
+    considers part of the project (tracked + untracked, minus gitignored).
+
+    Uses the working tree — not `git archive HEAD` — so uncommitted and
+    untracked files are included. We set "+dirty" on the commit SHA to indicate
+    that the working tree is dirty.
+
+    The single top-level directory matches GitHub's `/zipball/<ref>`
+    layout that the upload endpoint expects. The short-sha slot is
+    cosmetic for us (commit SHA travels out-of-band as a query param),
+    so we pin it to a sentinel rather than computing one.
+    """
+    prefix = "repo-0000000/"
+    buf = io.BytesIO()
+    with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
+        for rel in _list_trackable_files(repo_root):
+            full = os.path.join(repo_root, rel)
+            # Skip symlinks. `zipfile.write` follows them and embeds the
+            # target's contents, which would silently exfiltrate files
+            # outside the repo (e.g. an attacker-controlled PR adding a
+            # symlink to /etc/passwd or ~/.aws/credentials and a CI run
+            # that pipes the upload to Scanner).
+            if os.path.islink(full):
+                continue
+            # `ls-files` can list deleted-but-still-indexed paths; skip
+            # them so zipfile doesn't raise on a missing file.
+            if not os.path.isfile(full):
+                continue
+            zf.write(full, arcname=prefix + rel)
+    return buf.getvalue()
+
+
+def sync_git_repo(
+    scanner_client: Scanner,
+    path: str,
+    push_key: str,
+    json_output: bool,
+) -> None:
+    repo_root = _resolve_git_root(path)
+    branch = _get_branch(repo_root)
+    commit_sha = _get_commit_sha(repo_root)
+
+    if not json_output:
+        click.echo(f"Repo root:  {repo_root}")
+        click.echo(f"Push key:   {push_key}")
+        click.echo(f"Branch:     {branch}")
+        click.echo(f"Commit SHA: {commit_sha}")
+
+    zipball_bytes = _build_zipball_bytes(repo_root)
+
+    if not json_output:
+        click.echo(f"Uploading zipball ({len(zipball_bytes)} bytes)...")
+    resp = scanner_client.github_sync.upload_zipball(
+        zipball_bytes=zipball_bytes,
+        push_key=push_key,
+        branch=branch,
+        commit_sha=commit_sha,
+    )
+
+    if json_output:
+        result = SyncGitRepoResult(
+            push_key=push_key,
+            branch=branch,
+            commit_sha=commit_sha,
+            detection_rules_synced=resp.detection_rules_synced,
+            detection_rules_deleted=resp.detection_rules_deleted,
+            failures=[
+                SyncGitRepoFailure(
+                    file_path=f.file_path,
+                    check_status=SyncGitRepoFailureStatus(f.check_status.value),
+                    # `reason` is `None | str | Unset`; only str is real content.
+                    reason=f.reason if isinstance(f.reason, str) else None,
+                )
+                for f in resp.failures
+            ],
+            warnings=[
+                SyncGitRepoWarning(
+                    file_path=w.file_path,
+                    detection_rule_id=str(w.detection_rule_id),
+                    messages=w.messages,
+                )
+                for w in resp.warnings
+            ],
+        )
+        click.echo(result.model_dump_json(indent=2))
+        if resp.failures:
+            # `Exit` is Click's mechanism for setting an exit code without
+            # printing an error message — stdout stays a clean JSON blob
+            # for automation, but `$?` still signals failure.
+            raise click.exceptions.Exit(code=1)
+        return
+
+    if resp.warnings:
+        click.secho("Warnings:", fg="yellow")
+        for warning in resp.warnings:
+            click.secho(f"  {warning.file_path}:", fg="yellow")
+            for message in warning.messages:
+                click.secho(f"    - {message}", fg="yellow")
+
+    # `failures` non-empty means *nothing* synced (whole-zip transaction),
+    # so reporting it as a hard error matches reality.
+    if resp.failures:
+        n = len(resp.failures)
+        click.secho(
+            f"Failed to sync {n} rule file{'' if n == 1 else 's'}:",
+            fg="red",
+        )
+        for failure in resp.failures:
+            line = f"  {failure.file_path} [{failure.check_status.value}]"
+            # `reason` is `None | str | Unset`; only str is real content.
+            if isinstance(failure.reason, str):
+                line += f": {failure.reason}"
+            click.secho(line, fg="red")
+        raise click.ClickException("Sync aborted server-side; no rules were applied.")
+
+    click.secho(
+        f"Synced {resp.detection_rules_synced} rule(s), "
+        f"deleted {resp.detection_rules_deleted} rule(s)",
+        fg="green",
+    )

src/sync_git_repo_result.py

@@ -0,0 +1,107 @@
+"""User-facing schema for the JSON emitted by `scanner-cli sync-git-repo --json`.
+
+This is a *first-class CLI contract* — independent of the underlying
+`scanner-client-python` wire types. Downstream consumers (TS, Python, etc.)
+should treat the committed `schemas/sync-git-repo-result.schema.json` as
+the canonical definition; this module is just where we generate it from.
+
+When you change a field here, run `scripts/dump_schema.py` and commit the
+regenerated schema in the same PR.
+"""
+
+from enum import Enum
+
+from pydantic import BaseModel, Field
+
+
+class SyncGitRepoFailureStatus(str, Enum):
+    """Why a detection-rule file in the uploaded zip didn't sync.
+
+    Mirrors the server-side `DetectionRuleSyncStatus` set, but is owned by
+    the CLI — we may extend it independently if the CLI grows pre-upload
+    checks of its own. New values may be added in a non-breaking way; old
+    values will not be removed without a CLI major-version bump.
+    """
+
+    ERROR_IN_QUERY = "ErrorInQuery"
+    ERROR_IN_RULE = "ErrorInRule"
+    FAILED_TESTS = "FailedTests"
+
+
+class SyncGitRepoFailure(BaseModel):
+    """A single rule file that prevented the sync."""
+
+    file_path: str = Field(
+        description=(
+            "Repo-relative path of the failing detection-rule YAML."
+        ),
+    )
+    check_status: SyncGitRepoFailureStatus = Field(
+        description="Categorical reason this file failed.",
+    )
+    reason: str | None = Field(
+        default=None,
+        description=(
+            "Free-form detail (e.g. parse error message, failing test "
+            "names). May be absent when `check_status` is self-explanatory."
+        ),
+    )
+
+
+class SyncGitRepoWarning(BaseModel):
+    """A non-fatal advisory attached to one synced rule."""
+
+    file_path: str = Field(
+        description="Repo-relative path of the rule the warning pertains to.",
+    )
+    detection_rule_id: str = Field(
+        description=(
+            "UUID of the resulting Scanner detection rule, as a string."
+        ),
+    )
+    messages: list[str] = Field(
+        description="Human-readable warning messages.",
+    )
+
+
+class SyncGitRepoResult(BaseModel):
+    """Top-level shape of `sync-git-repo --json` output.
+
+    `failures` non-empty implies the sync was rejected as a whole — i.e.
+    `detection_rules_synced` and `detection_rules_deleted` are both 0 — so
+    consumers can treat the failure list as the authoritative signal.
+    """
+
+    push_key: str = Field(
+        description="The `--push-key` value the CLI uploaded under.",
+    )
+    branch: str = Field(
+        description="Git branch the upload was associated with.",
+    )
+    commit_sha: str = Field(
+        description=(
+            "Commit SHA of HEAD at upload time. Carries a `+dirty` suffix "
+            "iff the working tree had uncommitted or untracked changes "
+            "(which were nevertheless included in the uploaded zip)."
+        ),
+    )
+    detection_rules_synced: int = Field(
+        description="Number of rules created or updated by the upload.",
+    )
+    detection_rules_deleted: int = Field(
+        description=(
+            "Number of rules removed because their file no longer exists "
+            "in the uploaded repo."
+        ),
+    )
+    failures: list[SyncGitRepoFailure] = Field(
+        default_factory=list,
+        description=(
+            "Per-file failures. Non-empty means the entire sync was "
+            "aborted server-side and no rules were applied."
+        ),
+    )
+    warnings: list[SyncGitRepoWarning] = Field(
+        default_factory=list,
+        description="Non-fatal advisories from a successful sync.",
+    )

src/utils.py

@@ -0,0 +1,21 @@
+"""Utility functions for the Scanner CLI."""
+
+
+def format_exception_message(e: Exception, base_msg: str) -> str:
+    """Format exception message with better error details for API responses."""
+    if hasattr(e, 'args') and len(e.args) > 0:
+        response = e.args[0]
+        status_code = getattr(response, 'status_code', None)
+        status_text = f" (HTTP {status_code})" if status_code else ""
+        
+        if hasattr(response, 'content'):
+            if response.content:
+                return f"{base_msg}{status_text}: {response.content!r}"
+            else:
+                return f"{base_msg}: Empty response{status_text or ' (status code: unknown)'}"
+        elif status_code:
+            return f"{base_msg}: HTTP {status_code}"
+        else:
+            return f"{base_msg}: {str(response)}"
+    else:
+        return f"{base_msg}: {str(e)}"