dt-extensions-sdk 1.1.14__tar.gz → 1.1.16__tar.gz

{dt_extensions_sdk-1.1.14 → dt_extensions_sdk-1.1.16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dt-extensions-sdk
-Version: 1.1.14
+Version: 1.1.16
 Project-URL: Documentation, https://github.com/dynatrace-extensions/dt-extensions-python-sdk#readme
 Project-URL: Issues, https://github.com/dynatrace-extensions/dt-extensions-python-sdk/issues
 Project-URL: Source, https://github.com/dynatrace-extensions/dt-extensions-python-sdk

{dt_extensions_sdk-1.1.14 → dt_extensions_sdk-1.1.16}/docs/conf.py

@@ -88,7 +88,7 @@ html_theme_options = {
     "logo": "dt-sdk-logo.png",
     "logo_alt": "dt-sdk",
     "logo_url": "/",
-    "github_url": "https://github.com/dynatrace-extensions/dt-extensions-python-sdk",
+    "github_url": "https://github.com/dynatrace-extensions/dt-extensions-python-sdk/tree/main/docs/",
     "footer_links": ",".join(
         [
             "Dynatrace|https://dynatrace.com/",

dt_extensions_sdk-1.1.16/docs/guides/building.rst

@@ -0,0 +1,129 @@
+Building Extensions
+###################
+
+| This guide provides some best practices on:
+| 
+
+* Building extensions
+* Python dependencies
+* CI systems, offline installs
+
+Native dependencies
+===================
+
+| Some python libraries require "native" dependencies, they are not written in pure python and usually contain C, C++, Rust or other compiled languages code.
+| This means that they might be compiled for a very specific version of python, or for a specific operating system.
+|
+
+| Examples:
+|
+
+* **requests** requires **charset_normalizer**, a **native dependency**
+
+
+| If you navigate to the `charset-normalizer pypi page`_ you will see dozens of different wheel files.
+| Each one of these files is compiled for a different version of python, and for a different operating system.
+|
+| Your extension will run on a Dynatrace **Activegate** or **OneAgent**, which is a Linux or Windows machine, and it has a specific version of python.
+| This means that your extension must be built on a machine that has the same version of python as the Activegate.
+|
+| At this time, Dynatrace extensions run on **python 3.10**.
+|
+| When you build the extension with **dt-sdk build**, it downloads the dependencies **whl** files and places them in the lib folder of the extension.  
+| To obtain whl files for different a operating system than what the build machine is, the SDK provides the **--extra-platform** flag.
+|
+| In summary, when building from Windows, you should use:
+|
+
+.. code-block:: bash
+
+    dt-sdk build --extra-platform manylinux2014_x86_64
+
+
+| To get the correct extra wheel files for linux. Note, **manylinux2014_x86_64** works for several packages, but not all of them.
+| You need to investigate the dependencies of your extension to find the correct extra platform if that is the case.
+|
+| When building from Linux, you should use:
+|
+
+
+.. code-block:: bash
+
+     dt-sdk build --extra-platform win_amd64
+
+
+| To get the correct extra wheel files for Windows.  
+|
+
+PyPI Access
+===========
+
+| When building extensions, the SDK downloads the dependencies from PyPI.
+|
+| In some organizations, you are not allowed to access the internet from the build machine.
+| In most cases you will have either:
+| 
+
+* A local PyPI mirror
+* A directory with all the wheel files present
+
+|
+| Both of these solutions can be used with the SDK.
+|
+
+PyPI Mirror
+"""""""""""
+
+| Suppose you have a local PyPi server running on http://my-pypi-server:8080.
+|
+| To use it with the SDK, run the build command as:
+|
+
+.. code-block:: bash
+
+     PIP_INDEX_URL=http://my-pypi-server:8080/simple PIP_TRUSTED_HOST=my-pypi-server dt-sdk build
+
+
+| This will tell the SDK to use the local PyPI server to download the dependencies. 
+| The SDK uses **pip** under the covers, so all the environment variables that **pip** supports can be used with the SDK.
+|
+| Note, that assumes the build machine is a linux machine. If you are building from Windows on Powershell, you can use:
+|
+
+.. code-block:: bash
+
+    $ENV:PIP_INDEX_URL="http://my-pypi-server:8080/simple"; $ENV:PIP_TRUSTED_HOST="my-pypi-server"; dt-sdk build
+
+
+
+
+
+Local Directory
+"""""""""""""""
+
+
+| Another option is to manually download the different whl files you need, and place them in a directory on the build machine.
+| In that case, that directory can be used as the source for the dependencies.
+|
+
+.. code-block:: bash
+
+    dt-sdk build --find-links /path/to/whl/files
+
+
+| This will tell the SDK to use the directory as the source for the dependencies.
+|
+
+Musl vs libc
+============
+
+| Extensions run on `libc`_ based systems, like Ubuntu, CentOS, Windows, etc.
+| You should not use a  `musl`_ based system, like Alpine, to build extensions.
+|
+| This means that if you are using a docker container to build the extension, you should use the **python:3.10** image, or any other image that is based on a `libc`_ system.
+|
+| The reason for this is that a **musl** based system will download native whl files that are not compatible with **libc** based systems.
+
+.. _charset-normalizer pypi page: https://pypi.org/project/charset-normalizer/#files
+.. _musl: https://musl.libc.org/
+.. _libc: https://en.wikipedia.org/wiki/C_standard_library

{dt_extensions_sdk-1.1.14 → dt_extensions_sdk-1.1.16}/docs/guides/extension_structure.rst

@@ -33,7 +33,7 @@ Here is what a sample extension definition looks like:
 
     name: custom:my-extension
     version: 0.0.1
-    minDynatraceVersion: "1.253"
+    minDynatraceVersion: "1.285"
     author:
       name: "Dynatrace"
 
@@ -41,7 +41,7 @@ Here is what a sample extension definition looks like:
       runtime:
         module: my_extension
         version:
-          min: "3.9"
+          min: "3.10"
 
       activation:
         remote:

{dt_extensions_sdk-1.1.14 → dt_extensions_sdk-1.1.16}/docs/index.rst

@@ -273,6 +273,7 @@ Documentation
 
    guides/installation
    guides/extension_structure
+   guides/building
    guides/migration
 
 .. toctree::

{dt_extensions_sdk-1.1.14 → dt_extensions_sdk-1.1.16}/dynatrace_extension/__about__.py

@@ -2,4 +2,4 @@
 #
 # SPDX-License-Identifier: MIT
 
-__version__ = "1.1.14"
+__version__ = "1.1.16"

{dt_extensions_sdk-1.1.14 → dt_extensions_sdk-1.1.16}/dynatrace_extension/cli/create/extension_template/extension/extension.yaml.template

@@ -1,6 +1,6 @@
 name: %extension-prefix%%extension-name%
 version: 0.0.1
-minDynatraceVersion: "1.253"
+minDynatraceVersion: "1.285"
 author:
   name: "Dynatrace"
 
@@ -8,10 +8,10 @@ python:
   runtime:
     module: %extension_name%
     version:
-      min: "3.9"
+      min: "3.10"
 
   activation:
     remote:
       path: activationSchema.json
     local:
-      path: activationSchema.json
+      path: activationSchema.json

dt_extensions_sdk-1.1.16/dynatrace_extension/cli/create/extension_template/setup.py.template

@@ -0,0 +1,28 @@
+from pathlib import Path
+from setuptools import setup, find_packages
+
+
+def find_version() -> str:
+    version = "0.0.1"
+    extension_yaml_path = Path(__file__).parent / "extension" / "extension.yaml"
+    try:
+        with open(extension_yaml_path, encoding="utf-8") as f:
+            for line in f:
+                if line.startswith("version"):
+                    version = line.split(" ")[-1].strip("\"")
+                    break
+    except Exception:
+        pass
+    return version
+
+
+setup(name="%extension_name%",
+      version=find_version(),
+      description="%Extension_Name% python EF2 extension",
+      author="Dynatrace",
+      packages=find_packages(),
+      python_requires=">=3.10",
+      include_package_data=True,
+      install_requires=["dt-extensions-sdk"],
+      extras_require={"dev": ["dt-extensions-sdk[cli]"]},
+      )

{dt_extensions_sdk-1.1.14 → dt_extensions_sdk-1.1.16}/dynatrace_extension/sdk/communication.py

@@ -10,11 +10,12 @@ import random
 import sys
 import time
 from abc import ABC, abstractmethod
+from collections import deque
 from dataclasses import dataclass
 from enum import Enum
 from itertools import islice
 from pathlib import Path
-from typing import Any, Iterable, List, TypeVar
+from typing import Any, Dict, Iterable, List, TypeVar, Union
 
 from .vendor.mureq.mureq import HTTPException, Response, request
 
@@ -22,6 +23,8 @@ CONTENT_TYPE_JSON = "application/json;charset=utf-8"
 CONTENT_TYPE_PLAIN = "text/plain;charset=utf-8"
 COUNT_METRIC_ITEMS_DICT = TypeVar("COUNT_METRIC_ITEMS_DICT", str, List[str])
 MAX_MINT_LINES_PER_REQUEST = 1000
+MAX_LOG_EVENTS_PER_REQUEST = 50_000
+MAX_LOG_REQUEST_SIZE = 5_000_000
 HTTP_BAD_REQUEST = 400
 
 
@@ -94,7 +97,7 @@ class CommunicationClient(ABC):
         pass
 
     @abstractmethod
-    def send_events(self, event: dict | list[dict], eec_enrichment: bool) -> dict | None:
+    def send_events(self, event: dict | list[dict], eec_enrichment: bool) -> list[Union[dict | None]]:
         pass
 
     @abstractmethod
@@ -283,19 +286,26 @@ class HttpClient(CommunicationClient):
             responses.append(mint_response)
         return responses
 
-    def send_events(self, events: dict | list[dict], eec_enrichment: bool = True) -> dict | None:
+    def send_events(self, events: dict | list[dict], eec_enrichment: bool = True) -> list[dict | None]:
         self.logger.debug(f"Sending log events: {events}")
-        event_data = json.dumps(events).encode("utf-8")
-        try:
-            # EEC returns empty body on success
-            return self._make_request(
-                self._events_url,
-                "POST",
-                event_data,
-                extra_headers={"Content-Type": CONTENT_TYPE_JSON, "eec-enrichment": str(eec_enrichment).lower()},
-            ).json()
-        except json.JSONDecodeError:
-            return None
+
+        responses = []
+        batches = divide_logs_into_batches([events] if isinstance(events, dict) else events)
+
+        for batch in batches:
+            try:
+                encoded_batch = json.dumps(batch).encode("utf-8")
+                eec_response = self._make_request(
+                    self._events_url,
+                    "POST",
+                    encoded_batch,
+                    extra_headers={"Content-Type": CONTENT_TYPE_JSON, "eec-enrichment": str(eec_enrichment).lower()},
+                ).json()
+                responses.append(eec_response)
+            except json.JSONDecodeError:
+                responses.append(None)
+
+        return responses
 
     def send_sfm_metrics(self, mint_lines: list[str]) -> MintResponse:
         mint_data = "\n".join(mint_lines).encode("utf-8")
@@ -448,6 +458,46 @@ def divide_into_chunks(iterable: Iterable, chunk_size: int) -> Iterable:
             return
         yield subset
 
+def divide_logs_into_batches(logs: list[dict]):
+    """
+    Yield successive batches from a list of log events, according to sizing limitations
+    imposed by the EEC: 5 MB payload, 50,000 events
+
+    :param logs: The list of log events
+    """
+    events_left = len(logs)
+    events = deque(logs)
+
+    batch = []
+    batch_size = 0
+    batch_items = 0
+
+    while events_left > 0:
+        if batch_items == MAX_LOG_EVENTS_PER_REQUEST:
+            yield batch
+            batch = []
+            batch_size = 0
+            batch_items = 0
+            continue
+
+        event = events.popleft()
+        events_left -= 1
+
+        if event is not None:
+            event = json.dumps(event).encode("utf-8")
+            event_size = len(event)
+
+            if batch_size + event_size >= MAX_LOG_REQUEST_SIZE:
+                yield batch
+                batch = [event]
+                batch_size = event_size
+                batch_items = 1
+            else:
+                batch.append(event)
+                batch_size += event_size
+                batch_items += 1
+    else:
+        yield batch
 
 @dataclass
 class MintResponse:

{dt_extensions_sdk-1.1.14 → dt_extensions_sdk-1.1.16}/dynatrace_extension/sdk/extension.py

@@ -9,6 +9,7 @@ import sys
 import threading
 import time
 from argparse import ArgumentParser
+from collections import deque
 from concurrent.futures import ThreadPoolExecutor
 from datetime import datetime, timedelta, timezone
 from enum import Enum
@@ -79,14 +80,15 @@ class DtEventType(str, Enum):
         https://docs.dynatrace.com/docs/dynatrace-api/environment-api/events-v2/post-event
     """
 
+    AVAILABILITY_EVENT = "AVAILABILITY_EVENT"
     CUSTOM_INFO = "CUSTOM_INFO"
     CUSTOM_ALERT = "CUSTOM_ALERT"
     CUSTOM_ANNOTATION = "CUSTOM_ANNOTATION"
     CUSTOM_CONFIGURATION = "CUSTOM_CONFIGURATION"
     CUSTOM_DEPLOYMENT = "CUSTOM_DEPLOYMENT"
-    MARKED_FOR_TERMINATION = "MARKED_FOR_TERMINATION"
     ERROR_EVENT = "ERROR_EVENT"
-    AVAILABILITY_EVENT = "AVAILABILITY_EVENT"
+    MARKED_FOR_TERMINATION = "MARKED_FOR_TERMINATION"
+    PERFORMANCE_EVENT = "PERFORMANCE_EVENT"
     RESOURCE_CONTENTION_EVENT = "RESOURCE_CONTENTION_EVENT"
 
 
@@ -995,14 +997,16 @@ class Extension:
             self._metrics.extend(lines)
 
     def _send_events_internal(self, events: Union[dict, List[dict]]):
-        response = self._client.send_events(events, self.log_event_enrichment)
-        with self._internal_callbacks_results_lock:
-            self._internal_callbacks_results[self._send_events.__name__] = Status(StatusValue.OK)
-            if not response or "error" not in response or "message" not in response["error"]:
-                return
-            self._internal_callbacks_results[self._send_events.__name__] = Status(
-                StatusValue.GENERIC_ERROR, response["error"]["message"]
-            )
+        responses = self._client.send_events(events, self.log_event_enrichment)
+
+        for response in responses:
+            with self._internal_callbacks_results_lock:
+                self._internal_callbacks_results[self._send_events.__name__] = Status(StatusValue.OK)
+                if not response or "error" not in response or "message" not in response["error"]:
+                    return
+                self._internal_callbacks_results[self._send_events.__name__] = Status(
+                    StatusValue.GENERIC_ERROR, response["error"]["message"]
+                )
 
     def _send_events(self, events: Union[dict, List[dict]]):
         self._internal_executor.submit(self._send_events_internal, events)
@@ -1011,9 +1015,8 @@ class Extension:
         self._client.send_dt_event(event)
 
     def get_version(self) -> str:
-        """Return the version of extensions sdk library."""
-
-        return __version__
+        """Return the extension version."""
+        return self.activation_config.version
 
     @property
     def techrule(self) -> str:

dt_extensions_sdk-1.1.16/test.py

@@ -0,0 +1,23 @@
+import os
+import time
+from concurrent.futures import ProcessPoolExecutor
+
+from dynatrace_extension import Extension
+
+
+class Mass(Extension):
+    def initialize(self):
+        with ProcessPoolExecutor(max_workers=4) as metric_executor:
+            f = metric_executor.submit(self.thing)
+        print(f.result())
+
+    @staticmethod
+    def thing():
+        time.sleep(5)
+        current_pid = os.getpid()
+        with open(r"D:\workspace\repos\github\dt-extensions-python-sdk\test.txt", "w") as f:
+            f.write(f"Hello, world! {current_pid}")
+
+if __name__ == "__main__":
+    m = Mass()
+    m.run()

dt_extensions_sdk-1.1.16/test.txt

@@ -0,0 +1 @@
+Hello, world! 21160

{dt_extensions_sdk-1.1.14 → dt_extensions_sdk-1.1.16}/tests/cli/test_types.py

@@ -8,7 +8,7 @@ from dynatrace_extension.cli.schema import ExtensionYaml
 VALID_YAML = """
 name: custom:mulesoft-cloudhub
 version: 0.0.1
-minDynatraceVersion: "1.902"
+minDynatraceVersion: "1.285"
 author:
   name: "Dynatrace"
 
@@ -16,7 +16,7 @@ python:
   runtime:
     module: mulesoft_cloudhub
     version:
-      min: "3.9"
+      min: "3.10"
 
   activation:
     remote:
@@ -38,7 +38,7 @@ class TestTypes(TestCase):
         assert extension.min_dynatrace_version == "1.902"
         assert extension.author.name == "Dynatrace"
         assert extension.python.runtime.module == "mulesoft_cloudhub"
-        assert extension.python.runtime.version.min_version == "3.9"
+        assert extension.python.runtime.version.min_version == "3.10"
         assert extension.python.activation.remote.path == "activationSchema.json"
         assert extension.python.activation.local is None
 

dt_extensions_sdk-1.1.14/dynatrace_extension/cli/create/extension_template/setup.py.template

@@ -1,12 +0,0 @@
-from setuptools import setup, find_packages
-
-setup(name="%extension_name%",
-      version="0.0.1",
-      description="%Extension_Name% python EF2 extension",
-      author="Dynatrace",
-      packages=find_packages(),
-      python_requires=">=3.10",
-      include_package_data=True,
-      install_requires=["dt-extensions-sdk"],
-      extras_require={"dev": ["dt-extensions-sdk[cli]"]},
-      )

dt_extensions_sdk-1.1.14/test.py

@@ -1,19 +0,0 @@
-import time
-from concurrent.futures import ProcessPoolExecutor
-from datetime import timedelta
-
-from dynatrace_extension import Extension
-from dynatrace_extension.sdk.callback import WrappedCallback
-
-
-class Mass(Extension):
-    def initialize(self):
-        with ProcessPoolExecutor(max_workers=4) as metric_executor:
-            metric_executor.submit(self.thing)
-
-    def thing(self):
-        print("EITA PORRA")
-
-if __name__ == "__main__":
-    m = Mass()
-    m.run()