dyngle 0.7.0__tar.gz → 1.0.1__tar.gz

{dyngle-0.7.0 → dyngle-1.0.1}/PACKAGE.md

@@ -1,12 +1,8 @@
 # Dyngle
 
-Use cases
-
-- A task runner 
-- A lightweight workflow engine
-- A replacement for Make in Python projects
-- A replacement for short functions in RC files
-- Freedom from quirky Bash syntax
+An experimantal, lightweight, easily configurable workflow engine for
+automating development, operations, data processing, and content management
+tasks.
 
 Technical foundations
 
@@ -41,16 +37,18 @@ dyngle run hello
 
 ## Configuration
 
-Dyngle reads configuration from YAML files. You can specify the config file location using:
+Dyngle reads configuration from YAML files. Specify the config file location using any of the following (in order of precedence):
 
-- `--config` command line option
-- `DYNGLE_CONFIG` environment variable  
-- `.dyngle.yml` in current directory
-- `~/.dyngle.yml` in home directory
+1. A `--config` command line option, OR
+2. A `DYNGLE_CONFIG` environment variable, OR
+3. `.dyngle.yml` in current directory, OR
+4. `~/.dyngle.yml` in home directory
 
 ## Operations
 
-Operations are defined under `dyngle:` in the configuration. In its simplest form, an Operation takes the form of a YAML array defining the steps, as a system command with space-separated arguments. In that sense, a Dyngle operation looks something akin to a "PHONY" Make target, a short Bash script, or a CI/CD job. As a serious example, consider the `init` operation from the Dyngle configuration delivered with the project's source code.
+Operations are defined under `dyngle:` in the configuration. In its simplest form, an Operation is a YAML array defining the Steps, as system commands with space-separated arguments. In that sense, a Dyngle operation looks something akin to a phony Make target, a short Bash script, or a CI/CD job.
+
+As a serious example, consider the `init` operation from the Dyngle configuration delivered with the project's source code.
 
 ```yaml
 dyngle:
@@ -61,11 +59,11 @@ dyngle:
       - .venv/bin/pip install --upgrade pip poetry
 ```
 
-The elements of the YAML array _look_ like lines of Bash, but Dyngle replaces the shell with YAML-based flow control and Python-based expressions (described below). So shell-specific operators such as `|`, `>`, and `$VARIABLES` won't work.
+The elements of the YAML array _look_ like lines of Bash, but Dyngle processes them directly as system commands, allowing for template substitution and Python expression evaluation (described below). So shell-specific syntax such as `|`, `>`, and `$VARIABLE` won't work.
 
 ## Data and Templates
 
-Dyngle maintains a block of Data throughout an operation, which is a set of named values (Python `dict`, YAML "mapping"). The values are usually strings but can also be other data types that are valid in both YAML and Python.
+Dyngle maintains a block of "Live Data" throughout an operation, which is a set of named values (Python `dict`, YAML "mapping"). The values are usually strings but can also be other data types that are valid in both YAML and Python.
 
 The `dyngle run` command feeds the contents of stdin to the Operation as Data, by converting a YAML mapping to named Python values. The values may be substituted into commands or arguments in Steps using double-curly-bracket syntax (`{{` and `}}`) similar to Jinja2.
 
@@ -92,7 +90,7 @@ Hello Francis!
 
 ## Expressions
 
-Operations may contain Expressions, written in Python, that can be referenced in operation steps using the same syntax as for Data. In the case of a naming conflict, an Expression takes precedence over Data with the same name. Expressions can reference names in the Data directly.
+Operations may contain Expressions, written in Python, that can be referenced in Operation Step Templates using the same syntax as for Data. In the case of a naming conflict, an Expression takes precedence over Data with the same name. Expressions can reference names in the Data directly.
 
 Expressions may be defined in either of two ways in the configuration:
 
@@ -142,8 +140,6 @@ YAML keys can contain hyphens, which are fully supported in Dyngle. To reference
 - Reference the name using underscores instead of hyphens (they are automatically replaced), OR
 - Use the built-in special-purpose `resolve()` function (which can also be used to reference other expressions)
 
-
-
 ```yaml
 dyngle:
   expressions:
@@ -207,26 +203,31 @@ dyngle:
         - echo "Hello {{name}}"
 ```
 
-## Data assignment operator
+## Passing values between Steps in an Operation
+
+The Steps parser supports two special operators designed to move data between Steps in an explicit way.
 
-The Steps parser supports one special operator which assigns the output (stdout) of its command to a Data field for use in subsequent steps.
+- The data assignment operator (`=>`) assigns the contents of stdout from the command to an element in the data
+- The data input operator (`->`) assigns the value of an element in the data (or an evaluated expression) to stdin for the command
 
-The operator is `=>` and must go after the command and its arguments. Follow the operator with the name of the Data key to assign.
+The operators must appear in order in the step and must be isolated with whitespace, i.e.
 
-Example:
+```
+<input-variable-name> -> <command and arguments> => <output-variable-name>
+```
+
+Here we get into more useful functionality, where commands can be strung together in meaningful ways without the need for Bash.
 
 ```yaml
 dyngle:
   operations:
-    today:
-      expressions:
-        just-the-date: resolve('full-date')[0:10]
-      steps:
-        - date => full-date
-        - echo "Today is {{just-the-date}}"
+    weather:
+      - curl -s "https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.41&current_weather=true" => weather-data
+      - weather-data -> jq -j '.current_weather.temperature' => temperature
+      - echo "It's {{temperature}} degrees out there!"
 ```
 
-(Note Dyngle does provide Python date operations in the Expression namespace, which might provide a better way to perform the same operation as this example, but it suffices to demonstrate assignment)
+If names overlap, data items populated using the data assignment operator take precedence over expressions and data in the original input from the beginning of the Operation.
 
 ## Lifecycle
 
@@ -235,7 +236,7 @@ The lifecycle of an operation is:
 1. Load Data if it exists from YAML on stdin (if no tty)
 2. Find the named Operation in the configuration
 2. Perform template rendering on the first Step, using Data and Expressions
-3. Execute the Step in a subprocess
+3. Execute the Step in a subprocess, passing in an input value and populating an output value in the Data
 4. Continue with the next Step
 
 Note that operations in the config are _not_ full shell lines. They are passed directly to the system.

{dyngle-0.7.0 → dyngle-1.0.1}/PKG-INFO

@@ -1,27 +1,25 @@
 Metadata-Version: 2.3
 Name: dyngle
-Version: 0.7.0
+Version: 1.0.1
 Summary: Run lightweight local workflows
 License: MIT
 Author: Steampunk Wizard
 Author-email: dyngle@steamwiz.io
-Requires-Python: >=3.11,<3.12
+Requires-Python: >=3.11,<4.0
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Requires-Dist: requests (>=2.32.3,<3.0.0)
-Requires-Dist: wizlib (>=3.1.4,<4.0.0)
+Requires-Dist: wizlib (>=3.3.8,<3.4.0)
 Description-Content-Type: text/markdown
 
 # Dyngle
 
-Use cases
-
-- A task runner 
-- A lightweight workflow engine
-- A replacement for Make in Python projects
-- A replacement for short functions in RC files
-- Freedom from quirky Bash syntax
+An experimantal, lightweight, easily configurable workflow engine for
+automating development, operations, data processing, and content management
+tasks.
 
 Technical foundations
 
@@ -56,16 +54,18 @@ dyngle run hello
 
 ## Configuration
 
-Dyngle reads configuration from YAML files. You can specify the config file location using:
+Dyngle reads configuration from YAML files. Specify the config file location using any of the following (in order of precedence):
 
-- `--config` command line option
-- `DYNGLE_CONFIG` environment variable  
-- `.dyngle.yml` in current directory
-- `~/.dyngle.yml` in home directory
+1. A `--config` command line option, OR
+2. A `DYNGLE_CONFIG` environment variable, OR
+3. `.dyngle.yml` in current directory, OR
+4. `~/.dyngle.yml` in home directory
 
 ## Operations
 
-Operations are defined under `dyngle:` in the configuration. In its simplest form, an Operation takes the form of a YAML array defining the steps, as a system command with space-separated arguments. In that sense, a Dyngle operation looks something akin to a "PHONY" Make target, a short Bash script, or a CI/CD job. As a serious example, consider the `init` operation from the Dyngle configuration delivered with the project's source code.
+Operations are defined under `dyngle:` in the configuration. In its simplest form, an Operation is a YAML array defining the Steps, as system commands with space-separated arguments. In that sense, a Dyngle operation looks something akin to a phony Make target, a short Bash script, or a CI/CD job.
+
+As a serious example, consider the `init` operation from the Dyngle configuration delivered with the project's source code.
 
 ```yaml
 dyngle:
@@ -76,11 +76,11 @@ dyngle:
       - .venv/bin/pip install --upgrade pip poetry
 ```
 
-The elements of the YAML array _look_ like lines of Bash, but Dyngle replaces the shell with YAML-based flow control and Python-based expressions (described below). So shell-specific operators such as `|`, `>`, and `$VARIABLES` won't work.
+The elements of the YAML array _look_ like lines of Bash, but Dyngle processes them directly as system commands, allowing for template substitution and Python expression evaluation (described below). So shell-specific syntax such as `|`, `>`, and `$VARIABLE` won't work.
 
 ## Data and Templates
 
-Dyngle maintains a block of Data throughout an operation, which is a set of named values (Python `dict`, YAML "mapping"). The values are usually strings but can also be other data types that are valid in both YAML and Python.
+Dyngle maintains a block of "Live Data" throughout an operation, which is a set of named values (Python `dict`, YAML "mapping"). The values are usually strings but can also be other data types that are valid in both YAML and Python.
 
 The `dyngle run` command feeds the contents of stdin to the Operation as Data, by converting a YAML mapping to named Python values. The values may be substituted into commands or arguments in Steps using double-curly-bracket syntax (`{{` and `}}`) similar to Jinja2.
 
@@ -107,7 +107,7 @@ Hello Francis!
 
 ## Expressions
 
-Operations may contain Expressions, written in Python, that can be referenced in operation steps using the same syntax as for Data. In the case of a naming conflict, an Expression takes precedence over Data with the same name. Expressions can reference names in the Data directly.
+Operations may contain Expressions, written in Python, that can be referenced in Operation Step Templates using the same syntax as for Data. In the case of a naming conflict, an Expression takes precedence over Data with the same name. Expressions can reference names in the Data directly.
 
 Expressions may be defined in either of two ways in the configuration:
 
@@ -157,8 +157,6 @@ YAML keys can contain hyphens, which are fully supported in Dyngle. To reference
 - Reference the name using underscores instead of hyphens (they are automatically replaced), OR
 - Use the built-in special-purpose `resolve()` function (which can also be used to reference other expressions)
 
-
-
 ```yaml
 dyngle:
   expressions:
@@ -222,26 +220,31 @@ dyngle:
         - echo "Hello {{name}}"
 ```
 
-## Data assignment operator
+## Passing values between Steps in an Operation
+
+The Steps parser supports two special operators designed to move data between Steps in an explicit way.
 
-The Steps parser supports one special operator which assigns the output (stdout) of its command to a Data field for use in subsequent steps.
+- The data assignment operator (`=>`) assigns the contents of stdout from the command to an element in the data
+- The data input operator (`->`) assigns the value of an element in the data (or an evaluated expression) to stdin for the command
 
-The operator is `=>` and must go after the command and its arguments. Follow the operator with the name of the Data key to assign.
+The operators must appear in order in the step and must be isolated with whitespace, i.e.
 
-Example:
+```
+<input-variable-name> -> <command and arguments> => <output-variable-name>
+```
+
+Here we get into more useful functionality, where commands can be strung together in meaningful ways without the need for Bash.
 
 ```yaml
 dyngle:
   operations:
-    today:
-      expressions:
-        just-the-date: resolve('full-date')[0:10]
-      steps:
-        - date => full-date
-        - echo "Today is {{just-the-date}}"
+    weather:
+      - curl -s "https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.41&current_weather=true" => weather-data
+      - weather-data -> jq -j '.current_weather.temperature' => temperature
+      - echo "It's {{temperature}} degrees out there!"
 ```
 
-(Note Dyngle does provide Python date operations in the Expression namespace, which might provide a better way to perform the same operation as this example, but it suffices to demonstrate assignment)
+If names overlap, data items populated using the data assignment operator take precedence over expressions and data in the original input from the beginning of the Operation.
 
 ## Lifecycle
 
@@ -250,7 +253,7 @@ The lifecycle of an operation is:
 1. Load Data if it exists from YAML on stdin (if no tty)
 2. Find the named Operation in the configuration
 2. Perform template rendering on the first Step, using Data and Expressions
-3. Execute the Step in a subprocess
+3. Execute the Step in a subprocess, passing in an input value and populating an output value in the Data
 4. Continue with the next Step
 
 Note that operations in the config are _not_ full shell lines. They are passed directly to the system.

{dyngle-0.7.0 → dyngle-1.0.1}/dyngle/__init__.py

@@ -7,8 +7,8 @@ from wizlib.ui_handler import UIHandler
 
 from dyngle.command import DyngleCommand
 from dyngle.error import DyngleError
-from dyngle.expression import expression
-from dyngle.operation import Operation
+from dyngle.model.expression import expression
+from dyngle.model.operation import Operation
 
 
 class DyngleApp(WizApp):

{dyngle-0.7.0 → dyngle-1.0.1}/dyngle/command/run_command.py

@@ -4,8 +4,8 @@ from wizlib.parser import WizParser
 from yaml import safe_load
 
 from dyngle.command import DyngleCommand
-from dyngle.expression import expression
-from dyngle.template import Template
+from dyngle.model.expression import expression
+from dyngle.model.template import Template
 from dyngle.error import DyngleError
 
 

{dyngle-0.7.0/dyngle → dyngle-1.0.1/dyngle/model}/expression.py

@@ -1,7 +1,8 @@
 from typing import Callable
 
 from dyngle.error import DyngleError
-from dyngle.safe_path import SafePath
+from dyngle.model.live_data import LiveData
+from dyngle.model.safe_path import SafePath
 
 from datetime import datetime as datetime, date, timedelta
 import math
@@ -9,7 +10,7 @@ import json
 import re
 import yaml
 
-from dyngle.template import Template
+from dyngle.model.template import Template
 
 
 def formatted_datetime(dt: datetime, format_string=None) -> str:
@@ -113,10 +114,33 @@ def _evaluate(expression: str, locals: dict) -> str:
     return str(result)
 
 
+# The 'expression' function returns the expression object itself, which is
+# really just a function.
+
 def expression(text: str) -> Callable[[dict], str]:
-    def evaluate(data: dict = None) -> str:
-        items = data.items() if data else ()
-        locals = {k.replace('-', '_'): v for k, v in items}
-        def resolve(s): return Template.resolve(s, data)
-        return _evaluate(text, locals | {'resolve': resolve})
-    return evaluate
+    """Generate an expression, which is a function based on a string
+    expression"""
+
+    def definition(live_data: LiveData | dict | None = None) -> str:
+        """The expression function itself"""
+
+        # Allow for blankness and testability
+        live_data = LiveData(live_data)
+
+        # Translate names to underscore-separated instead of hyphen-separated
+        # so they work within the Python namespace.
+
+        items = live_data.items() if live_data else ()
+        locals = LiveData({k.replace('-', '_'): v for k, v in items})
+
+        # Create a resolve function which allows references using the hyphen
+        # syntax too
+
+        def resolve(key):
+            return live_data.resolve(key)
+        locals = locals | {'resolve': resolve}
+
+        # Perform the Python eval, expanded above
+        return _evaluate(text, locals)
+
+    return definition

dyngle-1.0.1/dyngle/model/live_data.py

@@ -0,0 +1,22 @@
+from collections import UserDict
+
+from dyngle.error import DyngleError
+
+
+class LiveData(UserDict):
+
+    def resolve(self, key: str):
+        """Given a key (which might be dot-separated), return
+        the value (which might include evaluating expressions)."""
+
+        parts = key.split('.')
+        current = self.data
+        for part in parts:
+            if part not in current:
+                raise DyngleError(
+                    f"Invalid expression or data reference '{key}'")
+            current = current[part]
+        if callable(current):
+            return current(self)
+        else:
+            return current

{dyngle-0.7.0/dyngle → dyngle-1.0.1/dyngle/model}/operation.py

@@ -5,7 +5,8 @@ import shlex
 import subprocess
 
 from dyngle.error import DyngleError
-from dyngle.template import Template
+from dyngle.model.live_data import LiveData
+from dyngle.model.template import Template
 
 
 @dataclass
@@ -16,12 +17,12 @@ class Operation:
     steps: list
 
     def run(self, data: dict, global_expressions: dict):
-        # The data dict is mutable
-        steps = self.steps
         expressions = global_expressions | self.local_expressions
-        for markup in steps:
+        # Data takes precedence if names match
+        live_data = LiveData(expressions) | data
+        for markup in self.steps:
             step = Step(markup)
-            step.run(data, expressions)
+            step.run(live_data)
 
 
 STEP_PATTERN = re.compile(
@@ -46,10 +47,12 @@ class Step:
         self.input, self.command_template, self.output = \
             parse_step(self.markup)
 
-    def run(self, data, expressions):
-        command = [Template(word).render(data, expressions)
+    def run(self, live_data: LiveData):
+        command = [Template(word).render(live_data).strip()
                    for word in self.command_template]
         pipes = {}
+        if self.input:
+            pipes["input"] = live_data.resolve(self.input)
         if self.output:
             pipes['stdout'] = subprocess.PIPE
         result = subprocess.run(command, text=True, **pipes)
@@ -57,4 +60,4 @@ class Step:
             raise DyngleError(
                 f'Step failed with code {result.returncode}: {self.markup}')
         if self.output:
-            data[self.output] = result.stdout
+            live_data[self.output] = result.stdout.rstrip()

dyngle-1.0.1/dyngle/model/template.py

@@ -0,0 +1,30 @@
+from dataclasses import dataclass
+from functools import partial
+import re
+
+from dyngle.error import DyngleError
+from dyngle.model.live_data import LiveData
+
+
+PATTERN = re.compile(r'\{\{\s*([^}]+)\s*\}\}')
+
+
+@dataclass
+class Template:
+
+    template: str
+
+    def render(self, live_data: LiveData | dict | None = None) -> str:
+        """Render the template with the provided LiveData (raw data and
+        expressions)."""
+
+        live_data = LiveData(live_data)
+        resolver = partial(self._resolve, live_data=live_data)
+        return PATTERN.sub(resolver, self.template)
+
+    def _resolve(self, match, *, live_data: LiveData):
+        """Resolve a single name/path from the template. The argument is a
+        merge of the raw data and the expressions, either of which are valid
+        substitutions."""
+        key = match.group(1).strip()
+        return live_data.resolve(key)

{dyngle-0.7.0 → dyngle-1.0.1}/pyproject.toml

@@ -8,11 +8,11 @@ description = "Run lightweight local workflows"
 authors = ["Steampunk Wizard <dyngle@steamwiz.io>"]
 license = "MIT"
 readme = "PACKAGE.md"
-version = "0.7.0"
+version = "1.0.1"
 
 [tool.poetry.dependencies]
-python = "~3.11"
-wizlib = "^3.1.4"
+python = "^3.11"
+wizlib = "~3.3.8"
 requests = "^2.32.3"
 
 [tool.poetry.group.dev.dependencies]

dyngle-0.7.0/dyngle/template.py

@@ -1,56 +0,0 @@
-from dataclasses import dataclass
-from functools import partial
-import re
-
-from dyngle.error import DyngleError
-
-
-PATTERN = re.compile(r'\{\{\s*([^}]+)\s*\}\}')
-
-
-@dataclass
-class Template:
-
-    template: str
-
-    def render(self, data: dict = None, expressions: dict = None) -> str:
-        """Render the template with the provided data and expressions.
-
-        Parameters
-        ----------
-        data : dict
-            String data to insert
-        expressions : dict
-            Functions to call with data
-
-        Returns
-        -------
-        str
-            Template rendered with expression resolution and values inserted.
-        """
-
-        data = data if data else {}
-        expressions = expressions if expressions else {}
-        resolver = partial(self._resolve, live_data=data | expressions)
-        return PATTERN.sub(resolver, self.template)
-
-    def _resolve(self, match, *, live_data: dict):
-        """Resolve a single name/path from the template. The argument is a
-        merge of the raw data and the expressions, either of which are valid
-        substitutions."""
-        key = match.group(1).strip()
-        return self.resolve(key, live_data)
-
-    @staticmethod
-    def resolve(key: str, live_data: dict):
-        parts = key.split('.')
-        current = live_data
-        for part in parts:
-            if part not in current:
-                raise DyngleError(
-                    f"Invalid expression or data reference '{key}'")
-            current = current[part]
-        if callable(current):
-            return current(live_data)
-        else:
-            return current