zrb 0.5.1__tar.gz → 0.6.0__tar.gz

{zrb-0.5.1 → zrb-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: zrb
-Version: 0.5.1
+Version: 0.6.0
 Summary: A Framework to Enhance Your Workflow
 Author-email: Go Frendi Gunawan <gofrendiasgard@gmail.com>
 Requires-Python: >=3.10.0

{zrb-0.5.1 → zrb-0.6.0}/docs/concepts/README.md

@@ -12,6 +12,7 @@
 
 # Concepts
 
+- [Project](project.md)
 - [Runner, Group, and Task](runner-group-and-task.md)
 - [Task Lifecycle](task-lifecycle.md)
 - [Task Upstream](task-upstream.md)
@@ -28,6 +29,7 @@
   - [RecurringTask](specialized-tasks/recurring-task.md)
   - [Checker](specialized-tasks/checker.md)
   - [FlowTask](specialized-tasks/flow-task.md)
+- [Copying Task](copying-task.md)
 - [Extending Task](extending-task.md)
 - [Extending CmdTask](extending-cmd-task.md)
 

zrb-0.6.0/docs/concepts/copying-task.md

@@ -0,0 +1,68 @@
+🔖 [Table of Contents](../README.md) / [Concepts](README.md)
+
+# Copying Task
+
+While building your workflow, you might notice that some of your tasks resemble each other in many ways.
+
+Zrb allows you to copy a Task and modify a limited set of attributes.
+
+The following are the most commonly used methods when you copy a Task:
+
+- `copy(self)`
+- `set_name(self, new_name: str)`
+- `set_description(self, new_description: str)`
+- `set_icon(self, new_icon: str)`
+- `set_color(self, new_color: str)`
+- `set_should_execute(self, should_execute: str)`
+- `set_retry(self, new_retry: int)`
+- `set_retry_interval(self, new_retry_interval: int)`
+- `set_checking_interval(self, new_checking_retry_interval: int)`
+- `insert_checker(self, *checkers: AnyTask)`
+- `add_checker(self, *checkers: AnyTask)`
+- `insert_upstream(self, *upstreams: AnyTask)`
+- `add_upstream(self, *upstreams: AnyTask)`
+- `insert_input(self, *inputs: AnyInput)`
+- `add_input(self, *inputs: AnyInput)`
+- `insert_env(self, *envs: Env)`
+- `add_env(self, *envs: Env)`
+- `insert_env_file(self, *env_files: EnvFile)`
+- `add_env_file(self, *env_files: EnvFile)`
+
+Let's see the following example:
+
+```python
+from zrb import runner, CmdTask, BoolInput
+
+dbt_run = CmdTask(
+    name='dbt-run',
+    cmd='dbt run'
+)
+
+# Copying dbt run, make it skippable
+skippable_dbt_run = dbt_run.copy()
+skippable_dbt_run.add_input(BoolInput(name='dbt-run', default=True))
+skippable_dbt_run.set_should_execute('{{ input.dbt_run }}')
+
+# register dbt-run
+runner.register(dbt_run)
+
+# Make dbt-test depends on skippable dbt run
+dbt_test = CmdTask(
+    name='dbt-test',
+    cmd='dbt test',
+    upstreams=[skippable_dbt_run]
+)
+runner.register(dbt_test)
+```
+
+In the example, you create a task named `dbt-run`. You then copy the task into a variable named `skippable_dbt_run`. This new task has additional input and will be executed based on the input value.
+
+Finally, you create another task named `dbt-test`. This task depends on `skippable_dbt_run`.
+
+You see that despite the option to skip the execution, `dbt_run` and `skippable_dbt_run` do the same thing, and you don't need to define them twice.
+
+# Next
+
+For more flexibility, you can extend [Task](extending-task.md) and [CmdTask](extending-cmd-task.md)
+
+🔖 [Table of Contents](../README.md) / [Concepts](README.md)

zrb-0.6.0/docs/concepts/environments.md

@@ -0,0 +1,342 @@
+🔖 [Table of Contents](../README.md) / [Concepts](README.md)
+
+# Environments
+
+<div align="center">
+  <img src="../_images/emoji/palm_tree.png"/>
+  <p>
+    <sub>
+      Save the Earth. It's the only planet with chocolate!
+    </sub>
+  </p>
+</div>
+
+
+You can use `envs` and `env_files` attributes to define Task Environments.
+
+Let's see the following example:
+
+```python
+from zrb import runner, Env, CmdTask
+
+hello = CmdTask(
+    name='hello',
+    envs=[
+        Env(
+            name='USER_NAME',
+            os_name='USER',
+            default='Employee'
+        )
+    ],
+    cmd='echo Hello $USER_NAME',
+)
+runner.register(hello)
+```
+
+In the example, you define an environment named `USER_NAME`. This environment is linked to the operating system `USER` environment. If the value of the `USER` environment is not set, Zrb will use the default value `Employee`.
+
+If you don't provide `os_name`, Zrb will link your environment to the Operating System environment environment with the same name.
+
+You can set `os_name` to an empty string (i.e., `''`) to avoid association with the operating system's environment. If you set `os_name` to empty, then Zrb will always use the `default` value of the environment.
+
+To access the environment variable, you can use `$USER_NAME`. Alternatively, you can also use the Jinja Template `{{ env.USER_NAME }}`.
+
+As for `@python_task`, you can access Task Environment using `kwargs['_task'].get_env_map()`. Let's see the following example:
+
+
+```python
+from zrb import runner, Env, python_task, Task
+
+@python_task(
+    name='hello',
+    envs=[
+        Env(
+            name='USER_NAME',
+            os_name='USER',
+            default='Employee'
+        )
+    ],
+    runner=runner
+)
+def hello(*args, **kwargs) -> str:
+    task: Task = kwargs.get('_task')
+    env_map = task.get_env_map()
+    user_name = env_map.get('USER_NAME')
+    return f'Hello {user_name}'
+```
+
+Notice that Zrb is not altering your `os.environ`, so you cannot use a typical `os.getenv('USER_NAME')` here. This behavior is intentional since we want every Task to be isolated from each other.
+
+There are some parameters you can use to define an `Env`.
+
+- __name__: Name of the environment variable (required).
+- __os_name__: Name of OS environment (optional, default=`None`)
+    - If set to `None`, Zrb will link the environment variable to the OS environment.
+    - If set to an empty string (i.e., `''`), Zrb will not link the environment variable to the OS's environment.
+    - If set to a non-empty string, Zrb will link the environment variable to the OS's environment corresponding to this value.
+- __default__: Default value of the environment variable (optional, default: `None`).
+- __should_render__: Whether the environment variable should be rendered as a Jinja template (optional, default: `True`).
+
+
+# Using Environment File
+
+<div align="center">
+  <img src="../_images/emoji/desert_island.png"/>
+  <p>
+    <sub>
+      An island is just a sea's attempt at a mountain peak joke.
+    </sub>
+  </p>
+</div>
+
+
+As with `cmd` and `cmd_path` in `CmdTask`, Zrb also allows you to load external environment files. You can use `env_files` attributes for this.
+
+Unlike `cmd_path` and `cmd`, `env` and `env_files` are complementing each other. You can use both attributes in a single Task. As a general rule, `env` will override `env_file`.
+
+Suppose you have an environment file as follows:
+
+```bash
+# file-name: .env
+DBT_PROFILE=dbt-profile
+```
+
+You can load the environments defined in the file as follows:
+
+
+```python
+from zrb import runner, Env, EnvFile, CmdTask
+import os
+
+CURRENT_DIR = os.dirname(__file__)
+
+hello = CmdTask(
+    name='hello',
+    envs=[
+        Env(
+            name='USER_NAME',
+            os_name='USER',
+            default='Employee'
+        )
+    ],
+    env_files=[
+        EnvFile(path=os.path.join(CURRENT_DIR, '.env'))
+    ],
+    cmd=[
+        'echo "Username: $USER_NAME"',
+        'echo "DBT Profile: $DBT_PROFILE"'
+    ]
+)
+runner.register(hello)
+```
+
+As for `@python_task`, you can use the same methods
+
+
+```python
+from zrb import runner, Env, EnvFile, python_task, Task
+import os
+
+CURRENT_DIR = os.dirname(__file__)
+
+@python_task(
+    name='hello',
+    envs=[
+        Env(
+            name='USER_NAME',
+            os_name='USER',
+            default='Employee'
+        )
+    ],
+    env_files=[
+        EnvFile(path=os.path.join(CURRENT_DIR, '.env'))
+    ],
+    runner=runner
+)
+def hello(*args, **kwargs) -> str:
+    task: Task = kwargs.get('_task')
+    env_map = task.get_env_map()
+    user_name = env_map.get('USER_NAME')
+    dbt_profile = env_map.get('DBT_PROFILE')
+    return '\n'.join([
+        f'Username: {user_name}',
+        f'DBT Profile: {dbt_profile}'
+    ])
+```
+
+There are some parameters you can use to define an `EnvFile`.
+
+- __env_file__: Name of the environment file (required).
+- __prefix__: Custom prefix for environment's os_name (optional, default=`None`)
+- __should_render__: Whether the environment variable should be rendered as a Jinja template (optional, default: `True`).
+
+# Environment Cascading
+
+<div align="center">
+  <img src="../_images/emoji/fountain.png"/>
+  <p>
+    <sub>
+      Cascading: Nature's way of saying, 'Let's take this step by step, but faster!'
+    </sub>
+  </p>
+</div>
+
+
+We usually work on multiple environments (e.g., `dev`, `staging`, and `production`).
+
+With environment cascading, you can switch your working environment seamlessly by setting up `ZRB_ENV`.
+
+Let's say you need an environment named `BASE_URL`, and you need to check whether the `BASE_URL` is accessible or not.
+
+The value of the `BASE_URL` depends on which working environment you are currently in.
+- For production, `BASE_URL` should be `https://your-company.com`
+- For staging, `BASE_URL` should be `https://staging.your-company.com`
+- For dev, `BASE_URL` should be `https://dev.your-company.com`
+- For any other working environment, `BASE_URL` should be `http://localhost:8080`
+
+To do this, let's make an environment file `.env`:
+
+```bash
+export BASE_URL=http://localhost:8080
+export DEV_BASE_URL=https://dev.your-company.com
+export STAGING_BASE_URL=https://staging.your-company.com
+export PROD_BASE_URL=https://your-company.com
+```
+
+Notice how you define several `BASE_URL` with `DEV`, `STAGING`, and `PROD` prefixes. You will need these prefixes to distinguish one working environment from another.
+
+Once you create the environment files, you can use a `python_task` to check the accessibility of the `BASE_URL`.
+
+```python
+from zrb import runner, python_task, Task, Env
+
+@python_task(
+    name='check',
+    envs=[
+        Env(name='BASE_URL')
+    ],
+    runner=runner
+)
+def check(*args, **kwargs):
+    task: Task = kwargs.get('_task')
+    env_map = task.get_env_map()
+    url = env_map.get('BASE_URL')
+    task.print_out(f'Checking {url}')
+    try:
+        response = requests.get(url)
+        if response.status_code == 200:
+            return True
+        task.print_err('Non OK status')
+        return False
+    except requests.ConnectionError:
+        task.print_err('Connection error')
+        return False
+    except requests.RequestException:
+        task.print_err('Request exception')
+        return False
+    except Exception:
+        task.print_err('Unknown exception')
+        return False
+```
+
+Now you can load the environment file and switch between working environments
+
+```bash
+# Load the environments from .env
+source .env
+
+export ZRB_ENV=LOCAL
+zrb check # Will check on http://localhost:8080
+
+export ZRB_ENV=PROD
+zrb check # Will check on https://your-company.com
+
+export ZRB_ENV=STAGING
+zrb check # Will check on https://staging.your-company.com
+
+export ZRB_ENV=DEV
+zrb check # Will check on https://dev.your-company.com
+
+export ZRB_ENV=UNKNOWN
+zrb check # Will check on http://localhost:8080
+```
+
+You can see that the value of `BASE_URL` will be used if Zrb cannot find any environment matching the `ZRB_ENV` prefix (i.e., we don't have `LOCAL_BASE_URL` and `UNKNOWN_BASE_URL` in our `.env`).
+
+To see the more detailed visualization, please look at the following table:
+
+| ZRB_ENV | BASE_URL                           | <ZRB_ENV>_BASE_URL                                    | Final                            |
+|---------|------------------------------------|-------------------------------------------------------|----------------------------------|
+| PROD    | BASE_URL=http://localhost:8080     | PROD_BASE_URL=**https://your-company.com**            | https://your-company.com         |
+| STAGING | BASE_URL=http://localhost:8080     | STAGING_BASE_URL=**https://staging.your-company.com** | https://staging.your-company.com |
+| DEV     | BASE_URL=http://localhost:8080     | DEV_BASE_URL=**https://dev.your-company.com**         | https://dev.your-company.com     |
+| LOCAL   | BASE_URL=**http://localhost:8080** | LOCAL_BASE_URL=                                       | http://localhost:8080            |
+| UNKNOWN | BASE_URL=**http://localhost:8080** | UNKNOWN_BASE_URL=                                     | http://localhost:8080            |
+
+Environment cascading is handy if you need to work with multiple working environments.
+
+# Conflicting Environment
+
+If you declare multiple environments with the same name, the latter overrides the previous ones. Also, `envs` takes greater priority than `env_files`. That's mean that you can always override `env_files` with `envs`.
+
+
+Suppose you have two environment files, `a.env`, and `b.env`:
+
+```bash
+# file-name: a.env
+
+export PLANET=Mercury
+export ELEMENT=Air
+export GOD=Hermes
+```
+
+```bash
+# file-name: b.env
+
+export PLANET=Mars
+export ELEMENT=Fire
+```
+
+You also have the following Task definition:
+
+```bash
+from zrb import runner, CmdTask, Env, EnvFile
+
+show = CmdTask(
+    name='show',
+    env_files=[
+        EnvFile(path='a.env'),
+        EnvFile(path='b.env'), # b.env override a.env
+    ],
+    envs=[
+        Env(name='PLANET', os_name='', default='Venus') # Override Mercury and Mars
+        Env(name='PLANET', os_name='', default='Jupiter') # Override Venus 
+    ],
+    cmd=[
+        'echo "GOD: $GOD"', # Hermes
+        'echo "PLANET: $PLANET"', # Jupiter
+        'echo "ELEMENT: $ELEMENT"' # Fire
+    ]
+)
+runner.register(show)
+```
+
+You can see the following output:
+
+- `GOD: Hermes`. There is no `env` named `GOD`, and the only environment file defined `GOD` is `a.env`
+- `PLANET: Jupiter`. There are two `env`s named `PLANET`. The latter overrides the first, and both override `env_file`s.
+- `ELEMENT: Fire`. There is no `env` named `ELEMENT`. As for `env_file`s, `b.env` overrides `a.env`.
+
+For more detailed visualization, please look at the following table:
+
+| Environment | `env_files[0]` a.env | `env_files[1]` b.env | `envs[0]` | `envs[1]`   | Final   |
+|-------------|----------------------|----------------------|-----------|-------------|---------|
+| GOD         | **Hermes**           |                      |           |             | Hermes  |
+| PLANET      | Mercury              | Mars                 | Venus     | **Jupiter** | Jupiter |
+| ELEMENT     | Air                  | **Fire**             |           |             | Fire    |
+
+# Next
+
+Next, you can learn about inter-task communication using [`xcom`](xcom.md).
+
+🔖 [Table of Contents](../README.md) / [Concepts](README.md)

zrb-0.6.0/docs/concepts/extending-cmd-task.md

@@ -0,0 +1,128 @@
+🔖 [Table of Contents](../README.md) / [Concepts](README.md)
+
+# Extending CmdTask
+
+Aside from all methods described in the [Extending Task Section](extending-task.md), there are some other methods you can use to extend CmdTask:
+
+- `get_cmd_script(self, *args: Any, **kwargs: Any)`
+
+
+## SlackPrintTask
+
+```python
+from typing import Any, Optional, Iterable, Union, Callable
+from zrb import (
+    runner, Group, AnyTask, CmdTask, AnyInput, StrInput, Env, EnvFile
+)
+
+import jsons
+import os
+
+
+class SlackPrintTask(CmdTask):
+
+    def __init__(
+        self,
+        name: str,
+        group: Optional[Group] = None,
+        inputs: Iterable[AnyInput] = [],
+        envs: Iterable[Env] = [],
+        env_files: Iterable[EnvFile] = [],
+        icon: Optional[str] = None,
+        color: Optional[str] = None,
+        description: str = '',
+        slack_channel_id: str = '',
+        slack_app_token: str = '',
+        message: str = '',
+        upstreams: Iterable[AnyTask] = [],
+        checkers: Iterable[AnyTask] = [],
+        checking_interval: Union[float, int] = 0,
+        retry: int = 2,
+        retry_interval: Union[float, int] = 1,
+        should_execute: Union[bool, str, Callable[..., bool]] = True,
+    ):
+        CmdTask.__init__(
+            self,
+            name=name,
+            group=group,
+            inputs=inputs,
+            envs=envs,
+            env_files=env_files,
+            icon=icon,
+            color=color,
+            description=description,
+            upstreams=upstreams,
+            checkers=checkers,
+            checking_interval=checking_interval,
+            retry=retry,
+            retry_interval=retry_interval,
+            should_execute=should_execute
+        )
+        self._slack_channel_id = slack_channel_id
+        self._slack_app_token = slack_app_token
+        self._message = message
+
+    def inject_envs(self):
+        self.add_env(
+            Env(
+                name='CHANNEL_ID', os_name='',
+                default=self.render_str(self._slack_channel_id)
+            ),
+            Env(
+                name='TOKEN', os_name='',
+                default=self.render_str(self._slack_app_token)
+            ),
+            Env(
+                name='MESSAGE', os_name='',
+                default=self.render_str(self._message)
+            )
+        )
+
+    def get_cmd_script(self, *args: Any, **kwargs: Any):
+        # contruct json payload and replace all `"` with `\\"`
+        json_payload = jsons.dumps({
+            'channel': '$CHANNEL_ID',
+            'blocks': [{
+                'type': 'section',
+                'text': {
+                    'type': 'mrkdwn',
+                    'text': '$MESSAGE'
+                }
+            }]
+        }).replace('"', '\\"')
+        # send payload to slack API
+        return ' '.join([
+            'curl -H "Content-type: application/json"',
+            f'--data "{json_payload}"',
+            '-H "Authorization: Bearer $TOKEN"',
+            '-X POST https://slack.com/api/chat.postMessage'
+        ])
+```
+
+Here is how you can use `SlackPrintTask`:
+
+```python
+say_hi = SlackPrintTask(
+    name='say-hi',
+    inputs=[
+        StrInput(name='message', default='Hello world')
+    ],
+    slack_channel_id=os.getenv('SLACK_CHANNEL_ID'),
+    slack_app_token=os.getenv('SLACK_APP_TOKEN'),
+    message='{{ input.message }}',
+)
+runner.register(say_hi)
+```
+
+You can run the Task by invoking the following command.
+
+```bash
+zrb say-hi
+```
+
+# TODO
+
+Add more examples.
+
+
+🔖 [Table of Contents](../README.md) / [Concepts](README.md)