kubernetes-watch 0.1.0__tar.gz

kubernetes_watch-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Benyamin Motevalli
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

kubernetes_watch-0.1.0/PKG-INFO

@@ -0,0 +1,134 @@
+Metadata-Version: 2.1
+Name: kubernetes-watch
+Version: 0.1.0
+Summary: 
+License: MIT
+Author: bmotevalli
+Author-email: b.motevalli@gmail.com
+Requires-Python: >=3.11,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Requires-Dist: boto3 (==1.34.68)
+Requires-Dist: cryptography (==42.0.5)
+Requires-Dist: gitpython (==3.1.43)
+Requires-Dist: humps (==0.2.2)
+Requires-Dist: hvac (==2.1.0)
+Requires-Dist: kubernetes (==29.0.0)
+Requires-Dist: prefect (==2.18.0)
+Requires-Dist: pyyaml (==6.0.1)
+Requires-Dist: requests (==2.31.0)
+Description-Content-Type: text/markdown
+
+# kube_watch
+
+# To setup the project:
+
+- `poetry install`
+- `poetry shell`
+
+# To install package to your environment:
+
+python setup.py install
+
+
+# Description
+The kube_watch library is build on top of <a href='https://docs.prefect.io/latest/'>Prefect</a>. The library is designed to define workflows in a declaritive and flexible fashion. Originally, workflows in Prefect are defined via decorators such as @flow and @task. In kube_watch, workflows can be defined in a declaritive form via yaml files. The library is mainly focused on running scheduled workflows in kubernetes environment. However, it can easily be extended to be used for any purpose requiring a workflow. The workflow manifest has the following generic structure:
+
+```
+workflow:
+  name: Dummy Workflow
+  runner: concurrent
+  tasks:
+    - name: Task_A
+      module: <module_path>
+      task: <func_name>
+      inputsArgType: arg
+      inputs:
+        parameters:
+          - name: x1
+            value: y1
+          - name: x2
+            value: y2
+          - name: x3
+            type: env
+            value: Y3
+
+    - name: Task_B
+      module: <module_path>
+      task: <func_name>
+      inputsArgType: arg
+      inputs:
+        parameters:
+          - name: xx1
+            value: yy1
+          - name: xx2
+            value: yy2
+      dependency:
+        - taskName: Task_A
+          inputParamName: xx3
+
+    - name: Task_C
+      module: <module_path>
+      task: <func_name>
+      inputsArgType: arg
+      conditional:
+        tasks: ["Task_B"]
+```
+
+
+**runner**: concurrent | sequential: if concurrent selected, tasks will be run concurrently.
+
+**module**: all modules are located in 'modules' directory in kube_watch. This is where you can extend the library and add new tasks / modules. Below modules, there are submodules such as providers, clusters, and logic. Within each of this submodules, specific modules are defined. For example: providers.aws contains a series of tasks related to AWS. In this case, <module_path> = providers.aws. To add new tasks, add a new module with a similar pattern and refer the path in your task block. 
+
+**task**: task is simply the name function that you put in the <module_path>. i.e. as you define a function in a module, you can simply start to use it in your manifests.
+
+**inputArgType**: arg | dict | list: if the task functions accept known-fixed number of parameters, then use arg. 
+
+**dependency**: this block defines dependency of a child task to its parent. If **inputParamName** is defined, then OUTPUT of the parent task is passed to the child with an argument name defined by inputParamName.
+
+**IMPORTATN NOTE**: A strict assumption is that task functions return a single output. If there are cases with multiple output, wrap them into a dictionary and unwrap them in the child task.
+
+**conditional**: These are blocks where you can define when a task runs depending on the outcome of its parent. The parent task should return True or False.
+
+
+Parameters have also a type entry: env | static. static is default value. If type is defined as env, the parameter value is loaded from Environment Variables. In this case, value should be the name of the corresponding env var. 
+
+In above examples:
+
+```
+def Task_A(x1, x2, x3):
+    # do something
+    return output_A
+
+def Task_B(xx1, xx2, xx3):
+    # do something else
+    return output_B
+
+def Task_C():
+    # do another thing
+    return output_C
+```
+
+
+
+# Batch workflows
+kube_watch also enables to run workflows in batch. A separate manifest with following form is required:
+
+batchFlows:
+  runner: sequential
+  items:
+    - path: path_to_flow_A.yaml
+    - path: path_to_flow_B.yaml
+    - path: path_to_flow_C.yaml
+
+
+# cron_app
+The cron_app folder contains an example use case of kube_watch library. The cron_app can be used to deploy a CronJob in a kubernetes environment. The app assumes the manifests are located in a separate repository. It will clone the repo and read the manifests and runs the workflows.
+
+# Connect to a server
+## Start Server
+`prefect server start`
+## To Connect
+To connect to a server, simply set the following environment variable: `PREFECT_API_URL`
+

kubernetes_watch-0.1.0/README.md

@@ -0,0 +1,111 @@
+# kube_watch
+
+# To setup the project:
+
+- `poetry install`
+- `poetry shell`
+
+# To install package to your environment:
+
+python setup.py install
+
+
+# Description
+The kube_watch library is build on top of <a href='https://docs.prefect.io/latest/'>Prefect</a>. The library is designed to define workflows in a declaritive and flexible fashion. Originally, workflows in Prefect are defined via decorators such as @flow and @task. In kube_watch, workflows can be defined in a declaritive form via yaml files. The library is mainly focused on running scheduled workflows in kubernetes environment. However, it can easily be extended to be used for any purpose requiring a workflow. The workflow manifest has the following generic structure:
+
+```
+workflow:
+  name: Dummy Workflow
+  runner: concurrent
+  tasks:
+    - name: Task_A
+      module: <module_path>
+      task: <func_name>
+      inputsArgType: arg
+      inputs:
+        parameters:
+          - name: x1
+            value: y1
+          - name: x2
+            value: y2
+          - name: x3
+            type: env
+            value: Y3
+
+    - name: Task_B
+      module: <module_path>
+      task: <func_name>
+      inputsArgType: arg
+      inputs:
+        parameters:
+          - name: xx1
+            value: yy1
+          - name: xx2
+            value: yy2
+      dependency:
+        - taskName: Task_A
+          inputParamName: xx3
+
+    - name: Task_C
+      module: <module_path>
+      task: <func_name>
+      inputsArgType: arg
+      conditional:
+        tasks: ["Task_B"]
+```
+
+
+**runner**: concurrent | sequential: if concurrent selected, tasks will be run concurrently.
+
+**module**: all modules are located in 'modules' directory in kube_watch. This is where you can extend the library and add new tasks / modules. Below modules, there are submodules such as providers, clusters, and logic. Within each of this submodules, specific modules are defined. For example: providers.aws contains a series of tasks related to AWS. In this case, <module_path> = providers.aws. To add new tasks, add a new module with a similar pattern and refer the path in your task block. 
+
+**task**: task is simply the name function that you put in the <module_path>. i.e. as you define a function in a module, you can simply start to use it in your manifests.
+
+**inputArgType**: arg | dict | list: if the task functions accept known-fixed number of parameters, then use arg. 
+
+**dependency**: this block defines dependency of a child task to its parent. If **inputParamName** is defined, then OUTPUT of the parent task is passed to the child with an argument name defined by inputParamName.
+
+**IMPORTATN NOTE**: A strict assumption is that task functions return a single output. If there are cases with multiple output, wrap them into a dictionary and unwrap them in the child task.
+
+**conditional**: These are blocks where you can define when a task runs depending on the outcome of its parent. The parent task should return True or False.
+
+
+Parameters have also a type entry: env | static. static is default value. If type is defined as env, the parameter value is loaded from Environment Variables. In this case, value should be the name of the corresponding env var. 
+
+In above examples:
+
+```
+def Task_A(x1, x2, x3):
+    # do something
+    return output_A
+
+def Task_B(xx1, xx2, xx3):
+    # do something else
+    return output_B
+
+def Task_C():
+    # do another thing
+    return output_C
+```
+
+
+
+# Batch workflows
+kube_watch also enables to run workflows in batch. A separate manifest with following form is required:
+
+batchFlows:
+  runner: sequential
+  items:
+    - path: path_to_flow_A.yaml
+    - path: path_to_flow_B.yaml
+    - path: path_to_flow_C.yaml
+
+
+# cron_app
+The cron_app folder contains an example use case of kube_watch library. The cron_app can be used to deploy a CronJob in a kubernetes environment. The app assumes the manifests are located in a separate repository. It will clone the repo and read the manifests and runs the workflows.
+
+# Connect to a server
+## Start Server
+`prefect server start`
+## To Connect
+To connect to a server, simply set the following environment variable: `PREFECT_API_URL`

kubernetes_watch-0.1.0/kube_watch/enums/kube.py

@@ -0,0 +1,6 @@
+from enum import Enum
+
+
+class Hosts(str, Enum):
+    LOCAL = 'local'
+    REMOTE = 'remote'

kubernetes_watch-0.1.0/kube_watch/enums/logic.py

@@ -0,0 +1,9 @@
+from enum import Enum
+
+class Operations(str, Enum):
+    OR  = 'or'
+    AND = 'and'
+    SUM = 'sum'
+    AVG = 'avg'
+    MAX = 'max'
+    MIN = 'min'

kubernetes_watch-0.1.0/kube_watch/enums/providers.py

@@ -0,0 +1,13 @@
+from enum import Enum
+
+class Providers(str, Enum):
+    AWS   = "aws"
+    AZURE = "azure"
+    GCP   = "gcp"
+    VAULT = "vault"
+
+
+class AwsResources(str, Enum):
+    ECR = "ecr" # elastic container registry
+    S3  = "s3"
+    IAM = "iam" 

kubernetes_watch-0.1.0/kube_watch/enums/workflow.py

@@ -0,0 +1,17 @@
+from enum import Enum
+
+class ParameterType(str, Enum):
+    STATIC = 'static'
+    FROM_ENV = 'env'
+
+
+class TaskRunners(str, Enum):
+    SEQUENTIAL = 'sequential'
+    CONCURRENT = 'concurrent'
+    DASK       = 'dask'
+    RAY        = 'ray'
+
+
+class TaskInputsType(str, Enum):
+    ARG = 'arg'
+    DICT = 'dict'

kubernetes_watch-0.1.0/kube_watch/models/common.py

@@ -0,0 +1,17 @@
+from pydantic import BaseModel, ConfigDict
+from humps.camel import case
+
+def to_camel(string):
+    if string == "id":
+        return "_id"
+    if string.startswith("_"):  # "_id"
+        return string
+    return case(string)
+
+class CamelModel(BaseModel):
+    """
+    Replacement for pydanitc BaseModel which simply adds a camel case alias to every field
+    NOTE: This has been updated for Pydantic 2 to remove some common encoding helpers
+    """
+
+    model_config = ConfigDict(alias_generator=to_camel, populate_by_name=True)

kubernetes_watch-0.1.0/kube_watch/models/workflow.py

@@ -0,0 +1,55 @@
+from typing import List, Optional, Dict, Any
+from kube_watch.enums.workflow import ParameterType, TaskRunners, TaskInputsType
+from kube_watch.enums.logic import Operations
+
+from .common import CamelModel
+
+class Parameter(CamelModel):
+    name: str
+    value: Any
+    type: Optional[ParameterType] = ParameterType.STATIC
+
+class Artifact(CamelModel):
+    path: str
+
+class Inputs(CamelModel):
+    parameters: Optional[List[Parameter]] = []
+    artifacts: Optional[List[Artifact]] = []
+
+class Dependency(CamelModel):
+    taskName: str
+    inputParamName: Optional[str] = None
+
+class Condition(CamelModel):
+    tasks: List[str]
+    operation: Optional[Operations] = Operations.AND
+
+class Task(CamelModel):
+    module: str
+    task: str
+    name: str
+    inputsArgType: Optional[TaskInputsType] = TaskInputsType.ARG
+    inputs: Optional[Inputs] = None
+    dependency: Optional[List[Dependency]]  = None
+    conditional: Optional[Condition]  = None
+    outputs: Optional[List[str]]  = None
+
+class WorkflowConfig(CamelModel):
+    name: str
+    runner: TaskRunners = TaskRunners.CONCURRENT
+    tasks: List[Task]
+
+class WorkflowOutput(CamelModel):
+    flow_run: Any
+    config: Any
+
+class BatchFlowItem(CamelModel):
+    path: str
+
+class BatchFlowConfig(CamelModel):
+    # Only possible runners are concurrent and sequential
+    runner: TaskRunners = TaskRunners.CONCURRENT
+    items: List[BatchFlowItem]
+
+
+

kubernetes_watch-0.1.0/kube_watch/modules/clusters/kube.py

@@ -0,0 +1,186 @@
+from prefect import get_run_logger
+from typing import List
+from kubernetes import config
+from kubernetes import client
+from kubernetes.client.rest import ApiException
+import base64
+import datetime
+
+from kube_watch.enums.kube import Hosts
+
+logger = get_run_logger()
+
+    
+def setup(host=Hosts.REMOTE, context=None):
+    if host == Hosts.LOCAL:
+        # Running outside a Kubernetes cluster (e.g., local development)
+        config.load_kube_config(context=context)  # You can specify the context here if necessary   
+    else:
+        # Running inside a Kubernetes cluster
+        config.load_incluster_config()
+
+
+
+def create_or_update_configmap(config_name, namespace, data):
+    """
+    Create or update a ConfigMap in a specified namespace if the data is different.
+    
+    :param config_name: The name of the ConfigMap.
+    :param namespace: The namespace of the ConfigMap.
+    :param data: A dictionary containing the data for the ConfigMap.
+    :return: True if the ConfigMap was created or updated, False otherwise.
+    """
+    v1 = client.CoreV1Api()
+    configmap_metadata = client.V1ObjectMeta(name=config_name, namespace=namespace)
+    configmap = client.V1ConfigMap(api_version="v1", kind="ConfigMap", metadata=configmap_metadata, data=data)
+    
+    try:
+        existing_configmap = v1.read_namespaced_config_map(name=config_name, namespace=namespace)
+        # Compare the existing ConfigMap's data with the new data
+        if existing_configmap.data == data:
+            logger.info("No update needed for ConfigMap: {}".format(config_name))
+            return False
+        else:
+            # Data is different, update the ConfigMap
+            api_response = v1.replace_namespaced_config_map(name=config_name, namespace=namespace, body=configmap)
+            logger.info("ConfigMap updated. Name: {}".format(api_response.metadata.name))
+            return True
+    except ApiException as e:
+        if e.status == 404:  # ConfigMap not found, create it
+            try:
+                api_response = v1.create_namespaced_config_map(namespace=namespace, body=configmap)
+                logger.info("ConfigMap created. Name: {}".format(api_response.metadata.name))
+                return {'trigger_restart': True}
+            except ApiException as e:
+                logger.error("Exception when creating ConfigMap: {}".format(e))
+                raise ValueError
+        else:
+            logger.error("Failed to get or create ConfigMap: {}".format(e))
+            raise ValueError
+        
+
+def create_or_update_secret(secret_name, namespace, data, secret_type = None):
+    """
+    Create or update a Secret in a specified namespace if the data is different.
+    
+    :param name: The name of the Secret.
+    :param namespace: The namespace of the Secret.
+    :param data: A dictionary containing the data for the Secret. Values must be strings (not Base64 encoded).
+    :return: True if the secret was created or updated, False otherwise.
+    """
+    if secret_type == None:
+        secret_type = "Opaque"
+
+    v1 = client.CoreV1Api()
+    secret_metadata = client.V1ObjectMeta(name=secret_name, namespace=namespace)
+    secret = client.V1Secret(
+        api_version="v1", 
+        kind="Secret", 
+        metadata=secret_metadata, 
+        string_data=data,
+        type=secret_type
+    )
+    
+    try:
+        existing_secret = v1.read_namespaced_secret(name=secret_name, namespace=namespace)
+        # Encode the new data to compare with the existing Secret
+        encoded_data = {k: base64.b64encode(v.encode()).decode() for k, v in data.items()}
+        
+        # Check if the existing secret's data matches the new data
+        if existing_secret.data == encoded_data:
+            logger.info("No update needed for Secret: {}".format(secret_name))
+            return False
+        else:
+            # Data is different, update the Secret
+            api_response = v1.replace_namespaced_secret(name=secret_name, namespace=namespace, body=secret)
+            logger.info("Secret updated. Name: {}".format(api_response.metadata.name))
+            return True
+
+    except ApiException as e:
+        if e.status == 404:  # Secret not found, create it
+            try:
+                api_response = v1.create_namespaced_secret(namespace=namespace, body=secret)
+                logger.info("Secret created. Name: {}".format(api_response.metadata.name))
+                return {'trigger_restart':  True}
+            except ApiException as e:
+                logger.error("Exception when creating Secret: {}".format(e))
+                raise ValueError
+        else:
+            logger.error("Failed to get or create Secret: {}".format(e))
+            raise ValueError
+        
+
+def get_kubernetes_secret(secret_name, namespace):
+    # Assuming that the Kubernetes configuration is already set
+    v1 = client.CoreV1Api()
+    try:
+        secret = v1.read_namespaced_secret(secret_name, namespace)
+        # Decoding the base64 encoded data
+        decoded_data = {key: base64.b64decode(value).decode('utf-8') for key, value in secret.data.items()}
+        return decoded_data
+    except ApiException as e:
+        logger.error(f"Failed to get secret: {e}")
+        return None
+    
+
+def restart_deployment(deployment, namespace):
+    """
+    Trigger a rollout restart of a deployment in a specified namespace.
+
+    :param name: The name of the deployment.
+    :param namespace: The namespace of the deployment.
+    """ 
+
+    v1 = client.AppsV1Api()
+    body = {
+        'spec': {
+            'template': {
+                'metadata': {
+                    'annotations': {
+                        'kubectl.kubernetes.io/restartedAt': datetime.datetime.utcnow().isoformat()
+                    }
+                }
+            }
+        }
+    }
+    try:
+        api_response = v1.patch_namespaced_deployment(name=deployment, namespace=namespace, body=body)
+        logger.info(f"Deployment restarted. Name: {api_response.metadata.name}")
+    except ApiException as e:
+        logger.error(f"Exception when restarting deployment: {e}")
+
+
+def has_mismatch_image_digest(repo_digest, label_selector, namespace):
+    """
+    Check all pods in the given namespace and matching the label selector for any
+    mismatch between the latest image digest and the current image digest.
+    
+    parameters:
+    - namespace: The namespace to search for pods.
+    - label_selector: The label selector to identify the relevant pods.
+    - repo_digest: The latest image digest to compare against.
+    
+    Returns:
+    - True if any pod is found with an image digest mismatch.
+    - False if all pods match the latest image digest.
+    """
+    core_v1_api         = client.CoreV1Api()
+
+    # Fetch pods based on namespace and label selector
+    pods = core_v1_api.list_namespaced_pod(namespace, label_selector=label_selector)
+    
+    # Iterate over pods and their containers
+    for pod in pods.items:
+        for container_status in pod.status.container_statuses:
+            current_image_id = container_status.image_id
+            # Check for digest mismatch
+            if current_image_id.split('@')[-1] != repo_digest:
+                logger.info(f"Mismatch found in pod: {pod.metadata.name}, container: {container_status.name}")
+                logger.info(f"Repo digest: {repo_digest}")
+                logger.info(f"Curr digest: {current_image_id.split('@')[-1]}")
+                return True
+    
+    logger.info("Images are in-sync.")
+    logger.info(f"Repo digest: {repo_digest}")
+    logger.info(f"Curr digest: {current_image_id.split('@')[-1]}")
+    return False

kubernetes_watch-0.1.0/kube_watch/modules/logic/actions.py

@@ -0,0 +1,26 @@
+import subprocess
+import os
+from prefect import get_run_logger
+logger = get_run_logger()
+
+def run_standalone_script(package_name, package_run, package_exec):
+    script_dir = os.path.dirname(os.path.realpath(__file__))
+    # script_path = os.path.join(script_dir, package_name.replace('.', os.sep))
+    target_dir = os.path.join(script_dir, os.pardir, os.pardir, *package_name.split('.'))
+
+    # Change the current working directory to the script directory
+    full_command = f"{package_run} {os.path.join(target_dir, package_exec)}"
+
+    # Execute the command
+    try:
+        result = subprocess.run(full_command, shell=True, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+        if result.stdout:
+            logger.info(result.stdout)
+        if result.stderr:
+            logger.error(result.stderr)
+        # logger.info(f"Output: {result.stdout}")
+        result.check_returncode()
+    except subprocess.CalledProcessError as e:
+        # All logs should have already been handled above, now just raise an exception
+        logger.error("The subprocess encountered an error: %s", e)
+        raise Exception("Subprocess failed with exit code {}".format(e.returncode))

kubernetes_watch-0.1.0/kube_watch/modules/logic/checks.py

@@ -0,0 +1,8 @@
+
+
+def dicts_has_diff(dict_a, dict_b):
+    return dict_a != dict_b
+
+
+def remove_keys(d, keys):
+    return {k: v for k, v in d.items() if k not in keys}

kubernetes_watch-0.1.0/kube_watch/modules/logic/load.py

@@ -0,0 +1,8 @@
+import os
+from prefect import get_run_logger
+logger = get_run_logger()
+
+def load_secrets_to_env(data):
+    for key, value in data.items():
+        os.environ[key] = value
+        # logger.info(f"ENV VAR: {key} loaded")

kubernetes_watch-0.1.0/kube_watch/modules/logic/merge.py

@@ -0,0 +1,31 @@
+from typing import Any, List, Dict
+from kube_watch.enums.logic import Operations
+
+
+def merge_logical_outputs(inp_dict: Dict):
+    if 'operation' not in inp_dict.keys():
+        raise TypeError("Missing required parameters: 'operation'")
+    operation = inp_dict.get('operation')
+    del inp_dict['operation']
+
+    inputs = [v for k,v in inp_dict.items()]
+    return merge_logical_list(inputs, operation)
+
+
+def merge_logical_list(inp_list: List, operation: Operations):
+    if operation == Operations.OR:
+        return any(inp_list)
+    if operation == Operations.AND:
+        return all(inp_list)
+    raise ValueError("Invalid logical operation")
+
+
+def partial_dict_update(orig_data, new_data):
+    """
+    This function is used when some key value pairs in orig_data should
+    be updated from new_data.
+    """
+    for k, v in new_data.items():
+        orig_data[k] = v
+
+    return orig_data