scanner-cli 0.1.0rc6__py3-none-any.whl → 0.1.0rc8__py3-none-any.whl

scanner_cli-0.1.0rc8.dist-info/METADATA

@@ -0,0 +1,196 @@
+Metadata-Version: 2.1
+Name: scanner-cli
+Version: 0.1.0rc8
+Summary: Python command-line interface for Scanner API
+Author: Scanner, Inc.
+Author-email: support@scanner.dev
+Requires-Python: >=3.10,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: PyYAML (>=6.0.2,<7.0.0)
+Requires-Dist: click (>=8.1.7,<9.0.0)
+Requires-Dist: scanner-client (>=0.1.0rc8,<0.2.0)
+Description-Content-Type: text/markdown
+
+# scanner-cli
+
+This is a Python CLI for the Scanner API.
+
+## Usage
+
+To install the CLI, run
+
+```
+pip install scanner-cli
+```
+
+## Environment Variables
+
+For various commands, you will need to supply some Scanner configuration values.
+
+For `run-tests`, `validate`, and `sync`, you will need these values:
+- Scanner API URL
+- Scanner API key
+
+For `sync`, you will also need this value:
+- Scanner Team ID
+
+You can find these values in **Settings > General** and **Settings > API Keys**
+in your Scanner account.
+
+You can either set these values as environment variables:
+
+```
+# required for run-tests, validate, sync commands:
+export SCANNER_API_URL=<Scanner API URL>
+export SCANNER_API_KEY=<Scanner API key>
+
+# required for sync command:
+export SCANNER_TEAM_ID=<Scanner Team ID>
+```
+
+or provide them as arguments to the CLI:
+
+```
+scanner-cli <command> \
+    --api-url=<Scanner API URL> \
+    --api-key=<Scanner API key> \
+    --team-id=<Scanner Team ID> \
+    ...
+```
+
+## Commands
+
+Available commands are
+- `run-tests` - run tests on detection rules as code
+- `validate` - validate detection rules as code
+- `sync` - sync detection rules to Scanner
+- `migrate-elastic-rules` - migrate Elastic SIEM detection rules to Scanner YAML rules
+
+### `run-tests` and `validate`
+
+To validate or run tests on files:
+
+```
+scanner-cli validate -f detections/errors.yaml -f detections/unauthorized_logins.yaml
+scanner-cli run-tests -f detections/errors.yaml -f detections/unauthorized_logins.yaml
+```
+
+To validate or run tests on directories:
+
+```
+scanner-cli validate -d detections
+scanner-cli run-tests -d detections
+```
+
+To recursively validate or run-tests on a directory, use `-r` or `--recursive`:
+
+```
+scanner-cli validate -r -d detections
+scanner-cli run-tests -r -d detections
+
+scanner-cli validate --recursive -d detections
+scanner-cli run-tests --recursive -d detections
+```
+
+This will only validate or run tests on YAML files with the correct schema header.
+
+A file or directory must be provided. Multiple files and/or directories can be provided.
+
+### `sync`
+
+This command syncs detection rules to your Scanner account.
+
+You can sync individual files or full directories.
+
+```
+scanner-cli sync -f detections/errors.yaml -f detections/unauthorized_logins.yaml
+scanner-cli sync -r -d detections
+```
+
+#### `sync_key`
+
+Each detection rule must have the `sync_key` field defined. This is a unique
+identifier for the rule and can be any string you wish, as long as it is
+unique.
+
+#### `event_sink_keys`
+
+If your detection rules have `event_sink_keys` defined, you must provide a sync
+configuration file in YAML format using the `--sync-config-file` flag.
+
+This file allows you to map `event_sink_keys` in your detection rules (eg.
+`low_severity_alerts`, `high_severity_alerts`, etc.) to specific Scanner event
+sinks (eg. "Custom webhook", "Slack alerts channel", etc.)
+
+```
+scanner-cli sync --sync-config-file sync_config.yaml -r -d detections/
+```
+
+Where is what the sync configuration file looks like:
+```
+event_sink_keys:
+  low_severity_alerts:
+    - sink_id: "5098b2bd-065c-4c0d-9f11-685e88808fc2"
+  medium_severity_alerts:
+    - sink_id: "5098b2bd-065c-4c0d-9f11-685e88808fc2"
+    - sink_id: "62741ace-ea22-4255-8a36-b921a282d61e"
+  high_severity_alerts:
+    - sink_id: "5098b2bd-065c-4c0d-9f11-685e88808fc2"
+    - sink_id: "62741ace-ea22-4255-8a36-b921a282d61e"
+    - sink_id: "2313eb73-0020-4814-9b35-864e3d3439e0"
+```
+
+In this example, the `low_severity_alerts` event sink key is mapped to a single
+event sink, while the `medium_severity_alerts` and `high_severity_alerts` event
+sink keys are mapped to multiple event sinks.
+
+For instance, you might send low severity alerts to a SOAR webhook, but send
+medium and high severity alerts to a high-priority Slack channel and multiple
+webhooks.
+
+To find the Sink ID for a specific event sink, visit **Settings > Event Sinks**
+and click on the event sink you want to use.
+
+### `migrate-elastic-rules`
+
+This command migrates an `ndjson` file containing Elastic SIEM detection rules to Scanner YAML rules.
+
+For each rule in the `ndjson` file, a YAML file will be created in the output directory.
+
+```
+scanner-cli migrate-elastic-rules --elastic-files-rule elastic_rules.ndjson --output-dir scanner_rules/
+```
+
+Optionally, you can provide a migrate config file in YAML format.
+
+The migrate config file allows you to map Elastic Data View IDs to Scanner
+query terms. This way, the queries in your Scanner detection rules can be more
+selective about which logs they check against.
+
+```
+scanner-cli migrate-elastic-rules \
+    --migrate-config-file elastic_to_scanner_config.yaml \
+    --elastic-files-rule elastic_rules.ndjson \
+    --output-dir scanner_rules/
+```
+
+Here is an example migrate config file with a few mappings:
+- Map from an Elastic Data View ID to a Scanner index.
+- Map from an Elastic Data View wildcard name to a Scanner query for a specific
+  source type.
+- Map from an Elastic Data View wildcard name to a Scanner query with multiple
+  terms.
+
+```
+data_view_id_to_query_term:
+  e9874b58-5cee-40e0-8b49-5c3739572ea2: |-
+    @index={ 0f75b7fd-ea1e-421a-b4ee-007ac8570a20 | "application_logs" }
+  log-sources:aws:cloudtrail:*: |-
+    %ingest.source_type="aws:cloudtrail"
+  log-sources:non-prod-app-logs:*: |-
+    my_env=("staging" or "dev") and my_type="app_logs"
+```
+

scanner_cli-0.1.0rc8.dist-info/RECORD

@@ -0,0 +1,9 @@
+src/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+src/cli.py,sha256=1in8DDTlhYNXaaxvLwdYWiNHBr4VMnTBIzq59fLkhXc,9100
+src/migrate/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+src/migrate/elastic.py,sha256=y_cn_3Y_t8NHwYcpAQQLfnCkEBSULoU_vxxGwk47QAc,8681
+src/sync.py,sha256=WABCtvI6fpwXNJKo9lzafw3HkGYljpGjre5ywKnO0yA,7738
+scanner_cli-0.1.0rc8.dist-info/METADATA,sha256=Dch4WTSbmhU152rnEBOAFDh1eYRbNlMXBi-vK3Qgd10,5890
+scanner_cli-0.1.0rc8.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
+scanner_cli-0.1.0rc8.dist-info/entry_points.txt,sha256=vqXMrIG6N6pY66bNf0y-gbUxbU8v5dXvuL3mV832Fh8,43
+scanner_cli-0.1.0rc8.dist-info/RECORD,,

src/cli.py

@@ -2,14 +2,17 @@
 
 import glob
 import os
-from typing import Any, Callable
+from typing import Any, Callable, Optional
 
 import click
 
 from scanner_client import Scanner
 from scanner_client.detection_rule_yaml import validate_and_read_file
 
-_CLICK_OPTIONS = [
+import src.migrate.elastic as elastic_cmd
+import src.sync as sync_cmd
+
+_DEFAULT_CLICK_OPTIONS = [
     click.option(
         "--api-url",
         envvar="SCANNER_API_URL",
@@ -24,14 +27,14 @@ _CLICK_OPTIONS = [
         "-f",
         "--file",
         "file_paths",
-        help="File to validate. This must be .yml or .yaml file with the correct schema header.",
+        help="Detection rule file. This must be .yml or .yaml file with the correct schema header.",
         multiple=True,
     ),
     click.option(
         "-d",
         "--dir",
         "directories",
-        help="Directory to validate. Only .yml or .yaml files with the correct schema header will be validated.",
+        help="Directory of detection rule files. Only .yml or .yaml files with the correct schema header will be processed.",
         multiple=True,
     ),
     click.option(
@@ -45,8 +48,8 @@ _CLICK_OPTIONS = [
 ]
 
 
-def _click_options(func) -> Callable[..., Any]:
-    for option in reversed(_CLICK_OPTIONS):
+def _default_click_options(func) -> Callable[..., Any]:
+    for option in reversed(_DEFAULT_CLICK_OPTIONS):
         func = option(func)
 
     return func
@@ -80,7 +83,7 @@ def _get_valid_files(file_paths: str, directories: str, recursive: bool) -> list
     return files
 
 
-def _validate_shared_options(api_url: str, api_key: str, file_paths: str, directories: str) -> None:
+def _validate_default_options(api_url: str, api_key: str, file_paths: str, directories: str) -> None:
     if api_url is None:
         raise click.exceptions.UsageError(
             message=(
@@ -109,60 +112,156 @@ def cli():
 
 
 @cli.command()
-@_click_options
+@_default_click_options
 def validate(api_url: str, api_key: str, file_paths: str, directories: str, recursive: bool):
-    """ Validate detection rules """
-    _validate_shared_options(api_url, api_key, file_paths, directories)
+    """ Validate detection rule files """
+    _validate_default_options(api_url, api_key, file_paths, directories)
 
     scanner_client = Scanner(api_url, api_key)
 
     files = _get_valid_files(file_paths, directories, recursive)
     click.echo(f'Validating {len(files)} {"file" if len(files) == 1 else "files"}')
 
+    any_failures: bool = False
+
     for file in files:
         try:
             result = scanner_client.detection_rule_yaml.validate(file)
 
             if result.is_valid:
-                click.echo(f"{file}: " + click.style("Valid", fg="green"))
+                click.echo(f"{file}: " + click.style("OK", fg="green"))
             else:
+                any_failures = True
                 click.echo(f"{file}: " + click.style(f"{result.error}", fg="red"))
         except Exception as e:
-            click.echo(f"{file}: " + click.style(e, fg="red"))
-
+            any_failures = True
+            response = e.args[0]
+            click.echo(f"{file}: " + click.style(f"An exception occurred when attempting to validate file: {response.content}", fg="red"))
+
+    if any_failures:
+        # To make it so the CLI exits with a non-zero exit code
+        raise click.ClickException(
+            "`validate` failed for one or more files. See https://docs.scanner.dev/scanner/using-scanner/beta-features/detection-rules-as-code/writing-detection-rules for requirements."
+        )
+    else:
+        click.secho("All specified detection rule files are valid. Use `run-tests` to run the detection rule tests.", bold=True)
 
 
 @cli.command()
-@_click_options
+@_default_click_options
 def run_tests(api_url: str, api_key: str, file_paths: str, directories: str, recursive: bool):
     """ Run detection rule tests """
-    _validate_shared_options(api_url, api_key, file_paths, directories)
+    _validate_default_options(api_url, api_key, file_paths, directories)
 
     scanner_client = Scanner(api_url, api_key)
 
     files = _get_valid_files(file_paths, directories, recursive)
     click.echo(f'Running tests on {len(files)} {"file" if len(files) == 1 else "files"}')
 
+    any_validation_errors: bool = False
+    any_test_failures: bool = False
+
     for file in files:
+        click.secho(f"{file}", bold=True)
         try:
-            response = scanner_client.detection_rule_yaml.run_tests(file)
-            results = response.results.to_dict()
+            # Check for validation errors
+            validation_result = scanner_client.detection_rule_yaml.validate(file)
+            if not validation_result.is_valid:
+                any_validation_errors = True
+                click.secho(f"Validation error: {validation_result.error}", fg="red")
+                click.echo("")
+                continue
+
+            # Run detection rule tests
+            run_tests_response = scanner_client.detection_rule_yaml.run_tests(file)
+            results = run_tests_response.results.to_dict()
 
-            click.secho(f"{file}", bold=True)
             if len(results) == 0:
                 click.secho("No tests found", fg="yellow")
             else:
-                for name, status in response.results.to_dict().items():
+                for name, status in results.items():
                     if status == "Passed":
-                        click.echo(f"{name}: " + click.style("Passed", fg="green"))
+                        click.echo(f"{name}: " + click.style("OK", fg="green"))
                     else:
-                        click.echo(f"{name}: " + click.style("Failed", fg="red"))
-
-            click.echo("")
+                        any_test_failures = True
+                        click.echo(f"{name}: " + click.style("Test failed", fg="red"))
         except Exception as e:
-            click.secho(f"{file}", bold=True)
-            click.secho(e, fg="red")
-            click.echo("")
+            any_test_failures = True
+            response = e.args[0]
+            click.secho(f"An exception occurred when attempting to run tests: {response.content}", fg="red")
+
+        click.echo("")
+
+    error_messages = []
+    if any_validation_errors:
+        error_messages.append("Validation failed for one or more files. See https://docs.scanner.dev/scanner/using-scanner/beta-features/detection-rules-as-code/writing-detection-rules for requirements.")
+
+    if any_test_failures:
+        error_messages.append("`run-tests` failed for one or more files. See https://docs.scanner.dev/scanner/using-scanner/beta-features/detection-rules-as-code/cli#failing-tests for more information.")
+
+    if error_messages:
+        # To make it so the CLI exits with a non-zero exit code
+        raise click.ClickException("\n".join(error_messages))
+
+
+@cli.command()
+@_default_click_options
+@click.option(
+    "--team-id",
+    envvar="SCANNER_TEAM_ID",
+    help="The team ID to which you want to sync the Scanner rules. Go to Settings > General to find the Team ID.",
+    required=True,
+)
+@click.option(
+    "--sync-config-file",
+    help="Optional. The path to the sync configuration file the CLI will use to sync detection rules to Scanner. (eg. contains event_sink_keys mappings, etc).",
+    required=False,
+)
+def sync(
+    api_url: str,
+    api_key: str,
+    file_paths: str,
+    directories: str,
+    recursive: bool,
+    team_id: str,
+    sync_config_file: Optional[str]
+):
+    """ Sync detection rules to Scanner """
+    _validate_default_options(api_url, api_key, file_paths, directories)
+    if team_id is None:
+        raise click.exceptions.UsageError(
+            message=(
+                "Pass --team-id option or set `SCANNER_TEAM_ID` environment variable."
+            )
+        )
+
+    scanner_client: Scanner = Scanner(api_url, api_key)
+    files: list[str] = _get_valid_files(file_paths, directories, recursive)
+    # Note: In the Scanner UI, the tenant_id is called Team ID.
+    sync_cmd.sync(scanner_client, files, team_id, sync_config_file)
+
+
+@cli.command()
+@click.option(
+    "-c",
+    "--migrate-config-file",
+    help="Optional. The path to the migration configuration file the CLI will use to migrate Elastic rules (eg. contains data_view_id_to_query_term mappings, etc).",
+)
+@click.option(
+    "-f",
+    "--elastic-rules-file",
+    help="The path to the Elastic detection rules ndjson file the CLI will migrate.",
+    required=True,
+)
+@click.option(
+    "-o",
+    "--output-dir",
+    help="The directory where the migrated rules will be saved. Will create one YAML file per rule.",
+    required=True,
+)
+def migrate_elastic_rules(migrate_config_file: Optional[str], elastic_rules_file: str, output_dir: str):
+    """ Migrate Elastic SIEM rules to Scanner rules """
+    elastic_cmd.migrate_elastic_rules(migrate_config_file, elastic_rules_file, output_dir)
 
 
 if __name__ == "__main__":

src/migrate/elastic.py

@@ -0,0 +1,235 @@
+import json
+import os
+from typing import Any, Optional
+
+import click
+import yaml
+from yaml import Dumper
+
+
+class YamlStrLiteral(str):
+    """
+    A class to represent a YAML literal string, which may have multiple lines.
+    Helps us to represent a string as a literal block scalar in YAML.
+    """
+    pass
+
+
+def _represent_literal(dumper: Dumper, data: YamlStrLiteral) -> Any:
+    return dumper.represent_scalar(
+            yaml.resolver.BaseResolver.DEFAULT_SCALAR_TAG,
+            data,
+            style="|"
+    )
+
+
+yaml.add_representer(YamlStrLiteral, _represent_literal)
+
+
+def _validate_options(migrate_config_file: Optional[str], elastic_rules_file: str, output_dir: str):
+    if migrate_config_file and not os.path.exists(migrate_config_file):
+        raise click.exceptions.UsageError(
+            message=(
+                "Config file not found."
+            )
+        )
+    if not os.path.exists(elastic_rules_file):
+        raise click.exceptions.UsageError(
+            message=(
+                "Elastic rules file not found."
+            )
+        )
+    if not os.path.isdir(output_dir):
+        raise click.exceptions.UsageError(
+            message=(
+                "Output directory not found."
+            )
+        )
+
+
+def _migrate_to_scanner_yml(elastic_detection_rule: dict[str, Any], config: dict[str, Any]) -> dict[str, Any]:
+    name: str = elastic_detection_rule.get('name', '')
+    description: YamlStrLiteral = YamlStrLiteral(elastic_detection_rule.get('note', ''))
+    enabled: bool = elastic_detection_rule.get('enabled', True)
+    severity: str = _capitalize_first_letter(elastic_detection_rule.get('severity', 'unknown'))
+    query_text: YamlStrLiteral = YamlStrLiteral(_get_query_text_for_detection_rule(elastic_detection_rule, config))
+    time_range_s: int = 300
+    run_frequency_s: int = 60
+    event_sink_key: str = _migrate_severity_to_event_sink_key(severity)
+    file_name: str = _generate_rule_file_name(name)
+    scanner_rule: dict[str, Any] = {
+        'sync_key': file_name,
+        'name': name,
+        'description': description,
+        'enabled': enabled,
+        'severity': severity,
+        'query_text': query_text,
+        'time_range_s': time_range_s,
+        'run_frequency_s': run_frequency_s,
+        'event_sink_keys': [event_sink_key],
+    }
+    yaml_content: str = yaml.dump(
+        scanner_rule,
+        default_flow_style=False,
+        sort_keys=False
+    )
+    yaml_content = yaml_content.replace('\\', '\\\\')
+    yaml_content = "# schema: https://scanner.dev/schema/scanner-detection-rule.v1.json\n" + yaml_content
+    return {
+        'file_name': file_name,
+        'yaml_content': yaml_content,
+    }
+
+
+def _generate_rule_file_name(name: str) -> str:
+    chars = []
+    for c in name:
+        if c.isalnum():
+            chars.append(c.lower())
+        elif chars and chars[-1] != ' ':
+            chars.append(' ')
+    file_name = ''.join(chars)
+    file_name = file_name.strip()
+    file_name = file_name.replace(' ', '_')
+    return f"{file_name}.yml"
+
+
+def _migrate_severity_to_event_sink_key(severity: str) -> str:
+    return f"{severity.lower()}_severity_alerts"
+
+
+def _capitalize_first_letter(text: str) -> str:
+    if text:
+        return text[0].upper() + text[1:]
+    return text
+
+
+def _get_query_text_for_detection_rule(elastic_detection_rule: dict[str, Any], config: dict[str, Any]) -> str:
+    query_parts: list[str] = []
+    data_view_id: Optional[str] = elastic_detection_rule.get('data_view_id')
+    data_view_id_query_term: Optional[str] = config.get('data_view_id_to_query_term', {}).get(data_view_id)
+    if data_view_id_query_term:
+        query_parts.append(data_view_id_query_term)
+    main_query: Optional[str] = elastic_detection_rule.get('query')
+    if main_query:
+        query_parts.append(main_query)
+    filters: list[dict[str, Any]] = elastic_detection_rule.get('filters', [])
+    for filter in filters:
+        query_text = _get_query_text_for_filter(filter)
+        if query_text:
+            query_parts.append(query_text)
+    return "\n".join(query_parts)
+
+
+def _get_query_text_for_filter(f: dict[str, Any]) -> Optional[str]:
+    should_negate: bool = f.get('meta', {}).get('negate', False)
+    filter_query: dict[str, Any] = f.get('query', {})
+    query_text: Optional[str] = _get_query_text_for_filter_query(filter_query)
+    if query_text is None:
+        return None
+    if should_negate:
+        query_text = f"not {query_text}"
+    return query_text
+
+
+def _get_query_text_for_filter_query(filter_query: dict[str, Any]) -> Optional[str]:
+    match_phrase_query: Optional[dict[str, Any]] = filter_query.get('match_phrase')
+    bool_query: Optional[dict[str, Any]] = filter_query.get('bool')
+    exists_query: Optional[dict[str, Any]] = filter_query.get('exists')
+    if match_phrase_query:
+        return _get_query_text_for_match_phrase_query(match_phrase_query)
+    elif bool_query:
+        return _get_query_text_for_bool_query(bool_query)
+    elif exists_query:
+        return _get_query_text_for_exists_query(exists_query)
+    return None
+
+
+def _get_query_text_for_match_phrase_query(match_phrase_query: dict[str, Any]) -> str:
+    query_parts: list[str] = []
+    for field, value in match_phrase_query.items():
+        query_parts.append(f"{field}: \"{value}\"")
+    has_multiple_parts: bool = len(query_parts) > 1
+    query_text: str = "\n".join(query_parts)
+    if has_multiple_parts:
+        return f"({query_text})"
+    else:
+        return query_text
+
+
+def _get_query_text_for_bool_query(bool_query: dict[str, Any]) -> Optional[str]:
+    minimum_should_match: Optional[int] = bool_query.get('minimum_should_match')
+    if minimum_should_match != 1:
+        # Print error message that this is unsupported
+        click.secho("Unsupported filter: Only support bool minimum_should_match = 1", fg="red")
+        return None
+    query_parts: list[str] = []
+    should_clauses: list[dict[str, Any]] = bool_query.get('should', [])
+    for should_clause in should_clauses:
+        query_text: Optional[str] = _get_query_text_for_filter_query(should_clause)
+        if query_text is None:
+            return None
+        query_parts.append(query_text)
+    has_multiple_parts: bool = len(query_parts) > 1
+    returned_query_text: str = " or ".join(query_parts)
+    if has_multiple_parts:
+        return f"({returned_query_text})"
+    else:
+        return returned_query_text
+
+
+def _get_query_text_for_exists_query(exists_query: dict[str, Any]) -> Optional[str]:
+    field: Optional[str] = exists_query.get('field')
+    if field:
+        return f"{field}: *"
+    else:
+        return None
+
+
+def migrate_elastic_rules(migrate_config_file: Optional[str], elastic_rules_file: str, output_dir: str):
+    _validate_options(migrate_config_file, elastic_rules_file, output_dir)
+
+    any_failures: bool = False
+
+    count: int = 0
+    try:
+        config: dict[str, Any] = {}
+        if migrate_config_file:
+            with open(migrate_config_file, 'r') as file:
+                config = yaml.safe_load(file)
+        with open(elastic_rules_file, 'r') as file:
+            for raw_line in file:
+                line: str = raw_line.strip()
+                if not line:  # Skip empty lines
+                    continue
+                try:
+                    elastic_detection_rule: dict[str, Any] = json.loads(line)
+                    migrated: dict[str, Any] = _migrate_to_scanner_yml(elastic_detection_rule, config)
+                    output_file_path: str = f"{output_dir}/{migrated['file_name']}"
+                    with open(output_file_path, 'w') as output_file:
+                        output_file.write(migrated['yaml_content'])
+                    count += 1
+                    click.echo(click.style("Migrated", fg="green") + f": {output_file_path}")
+                except json.JSONDecodeError as e:
+                    click.secho(f"Error parsing JSON on line: {line}", fg="red")
+                    click.secho(f"Error details: {e}", fg="red")
+                    click.echo("")
+                    any_failures = True
+                    continue
+    except yaml.YAMLError as e:
+        any_failures = True
+        click.secho(f"YAML Error: {e}", fg="red")
+        click.echo("")
+    except BaseException as e:
+        any_failures = True
+        click.secho(f"Error: {e}", fg="red")
+        click.echo("")
+
+    click.secho(f"Successfully migrated {count} rules", fg="green")
+    click.echo(click.style("Output directory", fg="green") + f": {output_dir}")
+
+    if any_failures:
+        # To make it so the CLI exits with a non-zero exit code
+        raise click.ClickException(
+            "migrate-elastic-rules failed for one or more rules"
+        )

src/sync.py

@@ -0,0 +1,199 @@
+from dataclasses import dataclass
+from typing import Any, Optional
+import typing
+
+import click
+import yaml
+import sys
+
+from scanner_client import Scanner, NotFound
+from scanner_client.detection_rule import string_to_detection_severity, DetectionSeverity
+from scanner_client.detection_rule_yaml import validate_and_read_file
+
+
+@dataclass
+class DetectionRuleConfig:
+   file: str
+   detection_rule: dict[str, Any]
+   event_sink_ids: list[str]
+
+
+def _get_detection_rule_id_for_sync_key(
+    scanner_client: Scanner,
+    sync_key: str,
+) -> Optional[str]:
+    try:
+        detection_rule = scanner_client.detection_rule.get_by_sync_key(sync_key)
+        return detection_rule.id
+    except NotFound:
+        return None
+
+
+def _read_sync_config_from_file(sync_config_file: str) -> dict[str, Any]:
+    with open(sync_config_file, 'r') as f:
+        contents = f.read()
+        any_sync_config: Any = yaml.safe_load(contents)
+        if not isinstance(any_sync_config, dict):
+            raise click.exceptions.UsageError(
+                message=(
+                    "sync_config_file must be a YAML file with a dictionary at the root"
+                )
+            )
+        sync_config: dict[str, Any] = any_sync_config
+        return sync_config
+
+
+def _get_event_sink_ids(detection_rule: dict[str, Any], sync_config: dict[str, Any]) -> list[str]:
+    event_sink_ids: list[str] = []
+    detection_rule_event_sink_keys = detection_rule.get('event_sink_keys', [])
+    sync_config_event_sink_keys = sync_config.get('event_sink_keys', {})
+    if not isinstance(sync_config_event_sink_keys, dict):
+        raise click.exceptions.UsageError(
+            message=(
+                "Please supply a sync-config-file where event_sink_keys is present"
+            )
+        )
+    for event_sink_key in detection_rule_event_sink_keys:
+        mapped_event_sinks = sync_config_event_sink_keys.get(event_sink_key)
+        if not isinstance(mapped_event_sinks, list):
+            raise click.exceptions.UsageError(
+                message=(
+                    f"Please supply a sync-config-file where {event_sink_key} is present under event_sink_keys."
+                )
+            )
+        for mapped_event_sink in mapped_event_sinks:
+            if not isinstance(mapped_event_sink, dict):
+                raise click.exceptions.UsageError(
+                    message=(
+                        f"Event sink key {event_sink_key} must be a list of dictionaries in sync config file."
+                    )
+                )
+            event_sink_id = mapped_event_sink.get('sink_id')
+            if not event_sink_id:
+                raise click.exceptions.UsageError(
+                    message=(
+                        f"Event sink key {event_sink_key} must have a sink_id in sync config file."
+                    )
+                )
+            event_sink_ids.append(event_sink_id)
+    return event_sink_ids
+
+
+def sync(scanner_client: Scanner, files: list[str], tenant_id: str, sync_config_file: Optional[str]):
+    click.echo(f'Syncing {len(files)} detection rule {"file" if len(files) == 1 else "files"} to Scanner Team {tenant_id}')
+
+    any_failures: bool = False
+
+    sync_config: dict[str, Any] = {}
+    if sync_config_file:
+        sync_config = _read_sync_config_from_file(sync_config_file)
+
+    click.echo("Validating rules before syncing...")
+    detection_rule_configs: list[DetectionRuleConfig] = []
+    for file in files:
+        try:
+            result = scanner_client.detection_rule_yaml.validate(file)
+
+            if result.is_valid:
+                click.echo(f"{file}: " + click.style("Valid", fg="green"))
+            else:
+                any_failures = True
+                click.echo(f"{file}: " + click.style(f"{result.error}", fg="red"))
+                continue
+
+            contents: str = validate_and_read_file(file)
+            detection_rule: dict[str, Any] = yaml.safe_load(contents)
+
+            if not detection_rule.get('sync_key'):
+                any_failures = True
+                click.secho(f"Error: sync_key not found in {file}", fg="red")
+                continue
+
+            # Will raise an exception if the event sink keys are not valid
+            event_sink_ids: list[str] = _get_event_sink_ids(detection_rule, sync_config)
+
+            detection_rule_configs.append(DetectionRuleConfig(
+                file=file,
+                detection_rule=detection_rule,
+                event_sink_ids=event_sink_ids,
+            ))
+        except Exception as e:
+            any_failures = True
+            click.echo(f"{file}: " + click.style(e, fg="red"))
+
+    if any_failures:
+        # To make it so the CLI exits with a non-zero exit code
+        raise click.ClickException(
+            "validate failed for one or more files. Sync aborted."
+        )
+
+    click.secho("All rules are valid", fg="green")
+    click.echo("")
+
+    click.echo("Syncing rules to Scanner...")
+    num_created: int = 0
+    num_updated: int = 0
+    for detection_rule_config in detection_rule_configs:
+        file = detection_rule_config.file
+        detection_rule = detection_rule_config.detection_rule
+        event_sink_ids = detection_rule_config.event_sink_ids
+
+        try:
+            # At this point, we know sync_key is present
+            sync_key: str = detection_rule['sync_key']
+            detection_rule_id: Optional[str] = _get_detection_rule_id_for_sync_key(
+                scanner_client,
+                sync_key,
+            )
+
+            severity: DetectionSeverity = string_to_detection_severity(
+                detection_rule['severity']
+            )
+
+            if detection_rule_id:
+                scanner_client.detection_rule.update(
+                    sync_key=sync_key,
+                    detection_rule_id=detection_rule_id,
+                    name=detection_rule['name'],
+                    description=detection_rule['description'],
+                    time_range_s=detection_rule['time_range_s'],
+                    run_frequency_s=detection_rule['run_frequency_s'],
+                    enabled=detection_rule['enabled'],
+                    severity=severity,
+                    query_text=detection_rule['query_text'],
+                    event_sink_ids=event_sink_ids,
+                )
+                click.echo(f"{sync_key}: " + click.style("Updated", fg="green"))
+                num_updated += 1
+            else:
+                scanner_client.detection_rule.create(
+                    sync_key=detection_rule['sync_key'],
+                    tenant_id=tenant_id,
+                    name=detection_rule['name'],
+                    description=detection_rule['description'],
+                    time_range_s=detection_rule['time_range_s'],
+                    run_frequency_s=detection_rule['run_frequency_s'],
+                    enabled=detection_rule['enabled'],
+                    severity=severity,
+                    query_text=detection_rule['query_text'],
+                    event_sink_ids=event_sink_ids,
+                )
+                click.echo(f"{sync_key}: " + click.style("Created", fg="green"))
+                num_created += 1
+        except Exception as e:
+            any_failures = True
+            click.echo(click.style("Failed to sync file", fg="red") + f": {file}")
+            click.echo(click.style("Error", fg="red") + f": {e}")
+            click.echo("")
+            break
+
+    if any_failures:
+        # To make it so the CLI exits with a non-zero exit code
+        raise click.ClickException(
+            "sync failed for one or more files"
+        )
+
+    if num_created > 0:
+        click.secho(f"Created {num_created} rule(s)", fg="green")
+    if num_updated > 0:
+        click.secho(f"Updated {num_updated} rule(s)", fg="green")

scanner_cli-0.1.0rc6.dist-info/METADATA

@@ -1,65 +0,0 @@
-Metadata-Version: 2.1
-Name: scanner-cli
-Version: 0.1.0rc6
-Summary: Python command-line interface for Scanner API
-Author: Scanner, Inc.
-Author-email: support@scanner.dev
-Requires-Python: >=3.10,<4.0
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Requires-Dist: click (>=8.1.7,<9.0.0)
-Requires-Dist: scanner-client (>=0.1.0rc5,<0.2.0)
-Description-Content-Type: text/markdown
-
-# scanner-cli
-
-This is a Python CLI for the Scanner API.
-
-## Usage
-
-To install the CLI, run
-
-```
-pip install scanner-cli
-```
-
-You will need to provide the API URL of your Scanner instance and an API key. Go
-to **Settings > API Keys** to find your API URL and API key.
-
-You can either set these values as environment variables:
-
-```
-export SCANNER_API_URL=<Scanner API URL>
-export SCANNER_API_KEY=<Scanner API key>
-```
-
-or provide them as arguments to the CLI:
-
-```
-scanner-cli <command> --api-url=<Scanner API URL> --api-key=<Scanner API key>
-```
-
-## Commands
-
-Available commands are
-- `run-tests` - run tests on detection rules as code
-- `validate` - validate detection rules as code
-
-To validate or run tests on files:
-
-```
-scanner-cli <command> -f detections/errors.yaml -f detections/unauthorized_logins.yaml
-```
-
-To validate or run tests on directories:
-
-```
-scanner-cli <command> -d detections
-```
-
-This will only validate or run tests on YAML files with the correct schema header.
-
-A file or directory must be provided. Multiple files and/or directories can be provided.
-

scanner_cli-0.1.0rc6.dist-info/RECORD

@@ -1,6 +0,0 @@
-src/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-src/cli.py,sha256=EPfmwoCBiSUKGBReXF5PENDFTWKb-eIAWZkiSNwmogY,5035
-scanner_cli-0.1.0rc6.dist-info/METADATA,sha256=sGfEtgjuEFr833V-j6lVAn7LRDCmYKDHi8_uY6932ak,1607
-scanner_cli-0.1.0rc6.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
-scanner_cli-0.1.0rc6.dist-info/entry_points.txt,sha256=vqXMrIG6N6pY66bNf0y-gbUxbU8v5dXvuL3mV832Fh8,43
-scanner_cli-0.1.0rc6.dist-info/RECORD,,