contentctl 5.0.5__tar.gz → 5.2.0__tar.gz

contentctl-5.2.0/PKG-INFO

@@ -0,0 +1,74 @@
+Metadata-Version: 2.3
+Name: contentctl
+Version: 5.2.0
+Summary: Splunk Content Control Tool
+License: Apache 2.0
+Author: STRT
+Author-email: research@splunk.com
+Requires-Python: >=3.11,<3.14
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: Jinja2 (>=3.1.4,<4.0.0)
+Requires-Dist: PyYAML (>=6.0.2,<7.0.0)
+Requires-Dist: attackcti (>=0.4.0,<0.5.0)
+Requires-Dist: bottle (>=0.12.25,<0.14.0)
+Requires-Dist: docker (>=7.1.0,<8.0.0)
+Requires-Dist: gitpython (>=3.1.43,<4.0.0)
+Requires-Dist: pycvesearch (>=1.2,<2.0)
+Requires-Dist: pydantic (>=2.9.2,<2.10.0)
+Requires-Dist: pygit2 (>=1.15.1,<2.0.0)
+Requires-Dist: questionary (>=2.0.1,<3.0.0)
+Requires-Dist: requests (>=2.32.3,<2.33.0)
+Requires-Dist: semantic-version (>=2.10.0,<3.0.0)
+Requires-Dist: setuptools (>=69.5.1,<76.0.0)
+Requires-Dist: splunk-sdk (>=2.0.2,<3.0.0)
+Requires-Dist: tqdm (>=4.66.5,<5.0.0)
+Requires-Dist: tyro (>=0.9.2,<0.10.0)
+Requires-Dist: xmltodict (>=0.13,<0.15)
+Description-Content-Type: text/markdown
+
+# contentctl
+<p align="center">
+<img src="https://raw.githubusercontent.com/splunk/contentctl/refs/heads/main/docs/contentctl_logo_white.png" title="In case you're wondering, it's a capybara" alt="the logo for the contentctl project, which depicts a doodled 4 legged animal that is supposed to represent a capybara, with the name of the project below it" width="250" height="250"></p>
+
+> [!NOTE]
+> Looking to migrate from an earlier release to the new contentctl v5+ ? Check out our migration guide [here](docs/contentctl_v5_migration_guide.md). 
+
+## What is contentctl?
+`contentctl` is a tool developed by the Splunk Threat Research Team to help with managing the content living in [splunk/security_content](https://github.com/splunk/security_content) and producing the Enterprise Security Content Update app for Splunk. While its development is largely driven by STRT's needs, it has been somewhat genericized and can be used by customers and partners to package their own content. Simply put, `contentctl` is the workhorse that packages detections, macros, lookups, dashboards into a Splunk app that you can use, and that understands the YAML structure and project layout we've selected to keep development clean.
+
+## Quick Start Guide
+Check out our [User Guide](docs/UserGuide.md) to get started!
+
+## Content Testing
+Read more about how `contentctl` can help test and validate your content in a real Splunk instance [here](docs/ContentTestingGuide.md).
+
+## Sample CICD Workflows
+Already using `contentctl`, or looking to get started with it already configured in GitHub Actions? [Our guide](docs/Sample_CICD_Templates.md) includes workflows to help you build and test your app.
+
+## Contribution Guide
+Read [the Contribution Guidelines](CONTRIBUTING.md) for this project before opening a Pull Request.
+
+## Ecosystem
+| Project               | Description                                             |
+| --------------------- | ------------------------------------------------------- |
+| [Splunk Security Content](https://github.com/splunk/security_content)          | Splunk Threat Research Team's Content included in the [Enterprise Security Content Update App (ESCU)](https://splunkbase.splunk.com/app/3449)|
+| [Splunk Attack Range](https://github.com/splunk/attack_range)          | Easily deploy a preconfigured Splunk Environment locally or on AWS containing a Splunk Instance, Windows and Linux Machines, and Attacker Tools like Kali Linux.  Automatically simulate attacks or run your own|
+| [Splunk Attack Data](https://github.com/splunk/attack_data)          | Repository of Attack Simulation Data for writing and Testing Detections|                         |
+| [Splunk contentctl](https://github.com/splunk/contentctl)          | Generate, validate, build, test, and deploy custom Security Content|
+| [SigmaHQ Sigma Rules](https://github.com/SigmaHQ/sigma) | Official Repository for Sigma Rules. These rules are an excellent starting point for new content. |
+| [PurpleSharp Attack Simulation](https://github.com/mvelazc0/PurpleSharp) | Open source adversary simulation tool for Windows Active Directory environments (integrated into Attack Range)|
+| [Red Canary Atomic Red Team](https://github.com/redcanaryco/atomic-red-team)          | Library of attack simulations mapped to the MITRE ATT&CK® framework (integrated into Attack Range)|
+
+## License
+Copyright 2023 Splunk Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+

contentctl-5.2.0/README.md

@@ -0,0 +1,41 @@
+# contentctl
+<p align="center">
+<img src="https://raw.githubusercontent.com/splunk/contentctl/refs/heads/main/docs/contentctl_logo_white.png" title="In case you're wondering, it's a capybara" alt="the logo for the contentctl project, which depicts a doodled 4 legged animal that is supposed to represent a capybara, with the name of the project below it" width="250" height="250"></p>
+
+> [!NOTE]
+> Looking to migrate from an earlier release to the new contentctl v5+ ? Check out our migration guide [here](docs/contentctl_v5_migration_guide.md). 
+
+## What is contentctl?
+`contentctl` is a tool developed by the Splunk Threat Research Team to help with managing the content living in [splunk/security_content](https://github.com/splunk/security_content) and producing the Enterprise Security Content Update app for Splunk. While its development is largely driven by STRT's needs, it has been somewhat genericized and can be used by customers and partners to package their own content. Simply put, `contentctl` is the workhorse that packages detections, macros, lookups, dashboards into a Splunk app that you can use, and that understands the YAML structure and project layout we've selected to keep development clean.
+
+## Quick Start Guide
+Check out our [User Guide](docs/UserGuide.md) to get started!
+
+## Content Testing
+Read more about how `contentctl` can help test and validate your content in a real Splunk instance [here](docs/ContentTestingGuide.md).
+
+## Sample CICD Workflows
+Already using `contentctl`, or looking to get started with it already configured in GitHub Actions? [Our guide](docs/Sample_CICD_Templates.md) includes workflows to help you build and test your app.
+
+## Contribution Guide
+Read [the Contribution Guidelines](CONTRIBUTING.md) for this project before opening a Pull Request.
+
+## Ecosystem
+| Project               | Description                                             |
+| --------------------- | ------------------------------------------------------- |
+| [Splunk Security Content](https://github.com/splunk/security_content)          | Splunk Threat Research Team's Content included in the [Enterprise Security Content Update App (ESCU)](https://splunkbase.splunk.com/app/3449)|
+| [Splunk Attack Range](https://github.com/splunk/attack_range)          | Easily deploy a preconfigured Splunk Environment locally or on AWS containing a Splunk Instance, Windows and Linux Machines, and Attacker Tools like Kali Linux.  Automatically simulate attacks or run your own|
+| [Splunk Attack Data](https://github.com/splunk/attack_data)          | Repository of Attack Simulation Data for writing and Testing Detections|                         |
+| [Splunk contentctl](https://github.com/splunk/contentctl)          | Generate, validate, build, test, and deploy custom Security Content|
+| [SigmaHQ Sigma Rules](https://github.com/SigmaHQ/sigma) | Official Repository for Sigma Rules. These rules are an excellent starting point for new content. |
+| [PurpleSharp Attack Simulation](https://github.com/mvelazc0/PurpleSharp) | Open source adversary simulation tool for Windows Active Directory environments (integrated into Attack Range)|
+| [Red Canary Atomic Red Team](https://github.com/redcanaryco/atomic-red-team)          | Library of attack simulations mapped to the MITRE ATT&CK® framework (integrated into Attack Range)|
+
+## License
+Copyright 2023 Splunk Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

{contentctl-5.0.5 → contentctl-5.2.0}/contentctl/actions/detection_testing/infrastructures/DetectionTestingInfrastructure.py

@@ -89,7 +89,7 @@ class DetectionTestingManagerOutputDto:
     start_time: Union[datetime.datetime, None] = None
     replay_index: str = "contentctl_testing_index"
     replay_host: str = "CONTENTCTL_HOST"
-    timeout_seconds: int = 60
+    timeout_seconds: int = 120
     terminate: bool = False
 
 

{contentctl-5.0.5 → contentctl-5.2.0}/contentctl/contentctl.py

@@ -143,20 +143,6 @@ def test_common_func(config: test_common):
     raise Exception("There was at least one unsuccessful test")
 
 
-CONTENTCTL_5_WARNING = """
-*****************************************************************************
-WARNING - THIS IS AN ALPHA BUILD OF CONTENTCTL 5.
-THERE HAVE BEEN NUMEROUS CHANGES IN CONTENTCTL (ESPECIALLY TO YML FORMATS). 
-YOU ALMOST CERTAINLY DO NOT WANT TO USE THIS BUILD.
-IF YOU ENCOUNTER ERRORS, PLEASE USE THE LATEST CURRENTYLY SUPPORTED RELEASE: 
-
-CONTENTCTL==4.4.7 
-
-YOU HAVE BEEN WARNED!
-*****************************************************************************
-"""
-
-
 def get_random_compliment():
     compliments = [
         "Your detection rules are like a zero-day shield! 🛡️",
@@ -187,7 +173,6 @@ class RecognizeCommand:
 
 
 def main():
-    print(CONTENTCTL_5_WARNING)
     try:
         configFile = pathlib.Path("contentctl.yml")
 
@@ -299,7 +284,6 @@ def main():
             )
 
         print(e)
-        print(CONTENTCTL_5_WARNING)
         sys.exit(1)
 
 

{contentctl-5.0.5 → contentctl-5.2.0}/contentctl/input/director.py

@@ -1,30 +1,29 @@
 import os
 import sys
-from pathlib import Path
 from dataclasses import dataclass, field
-from pydantic import ValidationError
+from pathlib import Path
 from uuid import UUID
-from contentctl.input.yml_reader import YmlReader
 
-from contentctl.objects.detection import Detection
-from contentctl.objects.story import Story
+from pydantic import ValidationError
 
-from contentctl.objects.baseline import Baseline
-from contentctl.objects.investigation import Investigation
-from contentctl.objects.playbook import Playbook
-from contentctl.objects.deployment import Deployment
-from contentctl.objects.macro import Macro
-from contentctl.objects.lookup import LookupAdapter, Lookup
-from contentctl.objects.atomic import AtomicEnrichment
-from contentctl.objects.security_content_object import SecurityContentObject
-from contentctl.objects.data_source import DataSource
-from contentctl.objects.dashboard import Dashboard
 from contentctl.enrichments.attack_enrichment import AttackEnrichment
 from contentctl.enrichments.cve_enrichment import CveEnrichment
-
+from contentctl.helper.utils import Utils
+from contentctl.input.yml_reader import YmlReader
+from contentctl.objects.atomic import AtomicEnrichment
+from contentctl.objects.baseline import Baseline
 from contentctl.objects.config import validate
+from contentctl.objects.dashboard import Dashboard
+from contentctl.objects.data_source import DataSource
+from contentctl.objects.deployment import Deployment
+from contentctl.objects.detection import Detection
 from contentctl.objects.enums import SecurityContentType
-from contentctl.helper.utils import Utils
+from contentctl.objects.investigation import Investigation
+from contentctl.objects.lookup import Lookup, LookupAdapter
+from contentctl.objects.macro import Macro
+from contentctl.objects.playbook import Playbook
+from contentctl.objects.security_content_object import SecurityContentObject
+from contentctl.objects.story import Story
 
 
 @dataclass
@@ -113,20 +112,6 @@ class Director:
         self.createSecurityContent(SecurityContentType.detections)
         self.createSecurityContent(SecurityContentType.dashboards)
 
-        from contentctl.objects.abstract_security_content_objects.detection_abstract import (
-            MISSING_SOURCES,
-        )
-
-        if len(MISSING_SOURCES) > 0:
-            missing_sources_string = "\n 🟡 ".join(sorted(list(MISSING_SOURCES)))
-            print(
-                "WARNING: The following data_sources have been used in detections, but are not yet defined.\n"
-                "This is not yet an error since not all data_sources have been defined, but will be convered to an error soon:\n 🟡 "
-                f"{missing_sources_string}"
-            )
-        else:
-            print("No missing data_sources!")
-
     def createSecurityContent(self, contentType: SecurityContentType) -> None:
         if contentType in [
             SecurityContentType.deployments,

{contentctl-5.0.5 → contentctl-5.2.0}/contentctl/objects/abstract_security_content_objects/detection_abstract.py

@@ -51,8 +51,6 @@ from contentctl.objects.security_content_object import SecurityContentObject
 from contentctl.objects.test_group import TestGroup
 from contentctl.objects.unit_test import UnitTest
 
-MISSING_SOURCES: set[str] = set()
-
 # Those AnalyticsTypes that we do not test via contentctl
 SKIPPED_ANALYTICS_TYPES: set[str] = {AnalyticsType.Correlation}
 
@@ -476,7 +474,7 @@ class Detection_Abstract(SecurityContentObject):
                         "name": lookup.name,
                         "description": lookup.description,
                         "filename": lookup.filename.name,
-                        "default_match": "true" if lookup.default_match else "false",
+                        "default_match": lookup.default_match,
                         "case_sensitive_match": "true"
                         if lookup.case_sensitive_match
                         else "false",
@@ -514,7 +512,7 @@ class Detection_Abstract(SecurityContentObject):
             baseline.tags.detections = new_detections
 
         # Data source may be defined 1 on each line, OR they may be defined as
-        # SOUCE_1 AND ANOTHERSOURCE AND A_THIRD_SOURCE
+        # SOURCE_1 AND ANOTHERSOURCE AND A_THIRD_SOURCE
         # if more than 1 data source is required for a detection (for example, because it includes a join)
         # Parse and update the list to resolve individual names and remove potential duplicates
         updated_data_source_names: set[str] = set()
@@ -524,27 +522,9 @@ class Detection_Abstract(SecurityContentObject):
             updated_data_source_names.update(split_data_sources)
 
         sources = sorted(list(updated_data_source_names))
-
-        matched_data_sources: list[DataSource] = []
-        missing_sources: list[str] = []
-        for source in sources:
-            try:
-                matched_data_sources += DataSource.mapNamesToSecurityContentObjects(
-                    [source], director
-                )
-            except Exception:
-                # We gobble this up and add it to a global set so that we
-                # can print it ONCE at the end of the build of datasources.
-                # This will be removed later as per the note below
-                MISSING_SOURCES.add(source)
-
-        if len(missing_sources) > 0:
-            # This will be changed to ValueError when we have a complete list of data sources
-            print(
-                "WARNING: The following exception occurred when mapping the data_source field to "
-                f"DataSource objects:{missing_sources}"
-            )
-
+        matched_data_sources = DataSource.mapNamesToSecurityContentObjects(
+            sources, director
+        )
         self.data_source_objects = matched_data_sources
 
         for story in self.tags.analytic_story:
@@ -1075,3 +1055,30 @@ class Detection_Abstract(SecurityContentObject):
         # Return the summary
 
         return summary_dict
+
+    @model_validator(mode="after")
+    def validate_data_source_output_fields(self):
+        # Skip validation for Hunting and Correlation types, or non-production detections
+        if self.status != DetectionStatus.production or self.type in {
+            AnalyticsType.Hunting,
+            AnalyticsType.Correlation,
+        }:
+            return self
+
+        # Validate that all required output fields are present in the search
+        for data_source in self.data_source_objects:
+            if not data_source.output_fields:
+                continue
+
+            missing_fields = [
+                field for field in data_source.output_fields if field not in self.search
+            ]
+
+            if missing_fields:
+                raise ValueError(
+                    f"Data source '{data_source.name}' has output fields "
+                    f"{missing_fields} that are not present in the search "
+                    f"for detection '{self.name}'"
+                )
+
+        return self

{contentctl-5.0.5 → contentctl-5.2.0}/contentctl/objects/data_source.py

@@ -17,10 +17,12 @@ class DataSource(SecurityContentObject):
     source: str = Field(...)
     sourcetype: str = Field(...)
     separator: Optional[str] = None
+    separator_value: None | str = None
     configuration: Optional[str] = None
     supported_TA: list[TA] = []
     fields: None | list = None
     field_mappings: None | list = None
+    mitre_components: list[str] = []
     convert_to_log_source: None | list = None
     example_log: None | str = None
     output_fields: list[str] = []

{contentctl-5.0.5 → contentctl-5.2.0}/contentctl/objects/lookup.py

@@ -6,9 +6,10 @@ import pathlib
 import re
 from enum import StrEnum, auto
 from functools import cached_property
-from typing import TYPE_CHECKING, Annotated, Any, Literal, Optional, Self
+from typing import TYPE_CHECKING, Annotated, Any, Literal, Self
 
 from pydantic import (
+    BeforeValidator,
     Field,
     FilePath,
     NonNegativeInt,
@@ -69,7 +70,19 @@ class Lookup_Type(StrEnum):
 
 # TODO (#220): Split Lookup into 2 classes
 class Lookup(SecurityContentObject, abc.ABC):
-    default_match: Optional[bool] = None
+    # We need to make sure that this is converted to a string because we widely
+    # use the string "False" in our lookup content.  However, PyYAML reads this
+    # as a BOOL and this causes parsing to fail. As such, we will always
+    # convert this to a string if it is passed as a bool
+    default_match: Annotated[
+        str, BeforeValidator(lambda dm: str(dm).lower() if isinstance(dm, bool) else dm)
+    ] = Field(
+        default="",
+        description="This field is given a default value of ''"
+        "because it is the default value specified in the transforms.conf "
+        "docs. Giving it a type of str rather than str | None simplifies "
+        "the typing for the field.",
+    )
     # Per the documentation for transforms.conf, EXACT should not be specified in this list,
     # so we include only WILDCARD and CIDR
     match_type: list[Annotated[str, Field(pattern=r"(^WILDCARD|CIDR)\(.+\)$")]] = Field(
@@ -88,7 +101,7 @@ class Lookup(SecurityContentObject, abc.ABC):
 
         # All fields custom to this model
         model = {
-            "default_match": "true" if self.default_match is True else "false",
+            "default_match": self.default_match,
             "match_type": self.match_type_to_conf_format,
             "min_matches": self.min_matches,
             "max_matches": self.max_matches,

{contentctl-5.0.5 → contentctl-5.2.0}/contentctl/output/attack_nav_output.py

@@ -1,5 +1,5 @@
-from typing import List, Union
 import pathlib
+from typing import List, Union
 
 from contentctl.objects.detection import Detection
 from contentctl.output.attack_nav_writer import AttackNavWriter
@@ -10,14 +10,21 @@ class AttackNavOutput:
         self, detections: List[Detection], output_path: pathlib.Path
     ) -> None:
         techniques: dict[str, dict[str, Union[List[str], int]]] = {}
+
         for detection in detections:
             for tactic in detection.tags.mitre_attack_id:
                 if tactic not in techniques:
                     techniques[tactic] = {"score": 0, "file_paths": []}
 
-                detection_url = f"https://github.com/splunk/security_content/blob/develop/detections/{detection.source}/{detection.file_path.name}"
-                techniques[tactic]["score"] += 1
-                techniques[tactic]["file_paths"].append(detection_url)
+                detection_type = detection.source
+                detection_id = detection.id
+
+                # Store all three pieces of information separately
+                detection_info = f"{detection_type}|{detection_id}|{detection.name}"
+
+                techniques[tactic]["score"] = techniques[tactic].get("score", 0) + 1
+                if isinstance(techniques[tactic]["file_paths"], list):
+                    techniques[tactic]["file_paths"].append(detection_info)
 
         """
         for detection in objects:

contentctl-5.2.0/contentctl/output/attack_nav_writer.py

@@ -0,0 +1,83 @@
+import json
+import pathlib
+from typing import List, Union
+
+VERSION = "4.5"
+NAME = "Detection Coverage"
+DESCRIPTION = "Security Content Detection Coverage"
+DOMAIN = "enterprise-attack"
+
+
+class AttackNavWriter:
+    @staticmethod
+    def writeAttackNavFile(
+        mitre_techniques: dict[str, dict[str, Union[List[str], int]]],
+        output_path: pathlib.Path,
+    ) -> None:
+        max_count = max(
+            (technique["score"] for technique in mitre_techniques.values()), default=0
+        )
+
+        layer_json = {
+            "versions": {"attack": "16", "navigator": "5.1.0", "layer": VERSION},
+            "name": NAME,
+            "description": DESCRIPTION,
+            "domain": DOMAIN,
+            "techniques": [],
+            "gradient": {
+                "colors": ["#ffffff", "#66b1ff", "#096ed7"],
+                "minValue": 0,
+                "maxValue": max_count,
+            },
+            "filters": {
+                "platforms": [
+                    "Windows",
+                    "Linux",
+                    "macOS",
+                    "Network",
+                    "AWS",
+                    "GCP",
+                    "Azure",
+                    "Azure AD",
+                    "Office 365",
+                    "SaaS",
+                ]
+            },
+            "layout": {
+                "layout": "side",
+                "showName": True,
+                "showID": True,
+                "showAggregateScores": False,
+            },
+            "legendItems": [
+                {"label": "No detections", "color": "#ffffff"},
+                {"label": "Has detections", "color": "#66b1ff"},
+            ],
+            "showTacticRowBackground": True,
+            "tacticRowBackground": "#dddddd",
+            "selectTechniquesAcrossTactics": True,
+        }
+
+        for technique_id, data in mitre_techniques.items():
+            links = []
+            for detection_info in data["file_paths"]:
+                # Split the detection info into its components
+                detection_type, detection_id, detection_name = detection_info.split("|")
+
+                # Construct research website URL (without the name)
+                research_url = (
+                    f"https://research.splunk.com/{detection_type}/{detection_id}/"
+                )
+
+                links.append({"label": detection_name, "url": research_url})
+
+            layer_technique = {
+                "techniqueID": technique_id,
+                "score": data["score"],
+                "enabled": True,
+                "links": links,
+            }
+            layer_json["techniques"].append(layer_technique)
+
+        with open(output_path, "w") as outfile:
+            json.dump(layer_json, outfile, ensure_ascii=False, indent=4)

{contentctl-5.0.5 → contentctl-5.2.0}/contentctl/output/templates/transforms.j2

@@ -7,8 +7,8 @@ filename = {{ lookup.app_filename.name  }}
 collection = {{ lookup.collection }}
 external_type = kvstore
 {% endif %}
-{% if lookup.default_match is defined and lookup.default_match != None  %}
-default_match = {{ lookup.default_match | lower }}
+{% if lookup.default_match != '' %}
+default_match = {{ lookup.default_match }}
 {% endif %}
 {% if lookup.case_sensitive_match is defined and lookup.case_sensitive_match != None %}
 case_sensitive_match = {{ lookup.case_sensitive_match | lower }}

{contentctl-5.0.5 → contentctl-5.2.0}/pyproject.toml

@@ -1,6 +1,7 @@
 [tool.poetry]
 name = "contentctl"
-version = "5.0.5"
+
+version = "5.2.0"
 
 description = "Splunk Content Control Tool"
 authors = ["STRT <research@splunk.com>"]
@@ -32,7 +33,7 @@ gitpython = "^3.1.43"
 setuptools = ">=69.5.1,<76.0.0"
 
 [tool.poetry.group.dev.dependencies]
-ruff = "^0.9.2"
+ruff = "^0.9.10"
 
 [build-system]
 requires = ["poetry-core>=1.0.0"]