taskcluster-taskgraph 11.2.4__tar.gz → 12.0.0__tar.gz

taskcluster_taskgraph-12.0.0/.readthedocs.yaml

@@ -0,0 +1,14 @@
+---
+version: 2
+build:
+    os: ubuntu-22.04
+    tools:
+        python: "3.12"
+    commands:
+        - asdf plugin add uv
+        - asdf install uv latest
+        - asdf global uv latest
+        - uv sync --only-dev
+        - uv run -m sphinx -T -b html -d docs/_build/doctrees -D language=en docs $READTHEDOCS_OUTPUT/html
+sphinx:
+    configuration: docs/conf.py

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Change Log
 
+## [12.0.0] - 2024-10-25
+
+### Added
+
+- New `matrix` transforms added to allow easily splitting tasks
+- The `-p/--parameters` flag now supports `index=<index>` as a means of discovering parameters
+
+### Fixed
+
+- BREAKING CHANGE: `target_tasks_method` filter no longer implicitly added if other filters are used
+- An OSError when generating docker contexts when many files were involved
+- A bug when attempting to find prior actions in Github action tasks
+- The `when.files-changed` optimization no longer implicitly adds unrelated `tasks-from` files
+
+### Changed
+
+- BREAKING CHANGE: docker image task labels changed from `build-docker-image` to `docker-image`
+
 ## [11.2.4] - 2024-10-16
 
 ### Fixed

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: taskcluster-taskgraph
-Version: 11.2.4
+Version: 12.0.0
 Summary: Build taskcluster taskgraphs
 Project-URL: Repository, https://github.com/taskcluster/taskgraph
 Project-URL: Issues, https://github.com/taskcluster/taskgraph/issues

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/docs/howto/run-locally.rst

@@ -78,6 +78,8 @@ following formats are accepted:
   and used.
 * A value of ``project=<project>``. The ``parameters.yml`` artifact from the
   latest decision task on ``<project>`` will be downloaded and used.
+* A value of ``index=<index>``. The ``parameters.yml`` artifact will be
+  downloaded from the decision task pointed to by the specified index path.
 * Path to a directory containing multiple parameter files. Any ``.yml`` file in
   the directory will be considered a parameter set.
 

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/docs/index.rst

@@ -70,7 +70,7 @@ of tasks, and submit them all to Taskcluster. But you can also test out the
    taskgraph full
 
 You'll notice that ``taskgraph init`` has created a couple of tasks for us
-already, namely ``build-docker-image-linux`` and ``hello-world``.
+already, namely ``docker-image-linux`` and ``hello-world``.
 
 .. note::
 

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/docs/reference/migrations.rst

@@ -3,6 +3,27 @@ Migration Guide
 
 This page can help when migrating Taskgraph across major versions.
 
+11.x -> 12.x
+------------
+
+* Add ``target_tasks_method`` to the front of the ``filters`` parameter wherever
+  you are also using a custom filter.
+
+  For example, if you are passing in:
+
+  .. code-block:: yaml
+
+     filters: ["my_custom_filter"]
+
+  Change it to:
+
+  .. code-block:: yaml
+
+     filters: ["target_tasks_method", "my_custom_filter"]
+
+  No action is necessary if the ``filters`` parameter was empty.
+* Change all references from ``build-docker-image`` to ``docker-image``.
+
 10.x -> 11.x
 ------------
 

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/docs/reference/transforms/chunking.rst

@@ -1,7 +1,7 @@
-.. _chunking:
+.. _chunking transforms:
 
-Chunking Tasks
-==============
+Chunking Transforms
+===================
 
 The :mod:`taskgraph.transforms.chunking` module contains transforms that aid
 in splitting a single entry in a ``kind`` into multiple tasks. This is often

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/docs/reference/transforms/from_deps.rst

@@ -1,7 +1,7 @@
-.. _from_deps:
+.. _from_deps transforms:
 
-From Dependencies
-=================
+From Dependencies Transforms
+============================
 
 The :mod:`taskgraph.transforms.from_deps` transforms can be used to create
 tasks based on the kind dependencies, filtering on common attributes like the

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/docs/reference/transforms/index.rst

@@ -8,6 +8,7 @@ provides, read below to learn how to use them.
 
 .. toctree::
 
-   from_deps
-   task_context
-   chunking
+   From Dependencies <from_deps>
+   Task Context <task_context>
+   Matrix <matrix>
+   Chunking <chunking>

taskcluster_taskgraph-12.0.0/docs/reference/transforms/matrix.rst

@@ -0,0 +1,200 @@
+.. _matrix transforms:
+
+Matrix Transforms
+=================
+
+The :mod:`taskgraph.transforms.matrix` transforms can be used to split a base
+task into many subtasks based on a defined matrix.
+
+These transforms are useful if you need to have many tasks that are very
+similar except for some small configuration differences.
+
+Usage
+-----
+
+Add the transform to the ``transforms`` key in your ``kind.yml`` file:
+
+.. code-block:: yaml
+
+   transforms:
+     - taskgraph.transforms.matrix
+     # ...
+
+Then create a ``matrix`` section in your task definition, e.g:
+
+.. code-block:: yaml
+
+   tasks:
+     test:
+       matrix:
+         os: ["win", "mac", "linux"]
+
+       # rest of task definition
+
+This will split the ``test`` task into three; ``test-win``, ``test-mac`` and
+``test-linux``.
+
+Matrix with Multiple Rows
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can add as many rows as you like to the matrix, and every combination of
+tasks will be generated. For example, the following matrix:
+
+.. code-block:: yaml
+
+   tasks:
+     test:
+       matrix:
+         os: ["win", "mac", "linux"]
+         python: ["py312", "py311"]
+
+       # rest of task definition
+
+Will generate these tasks:
+
+- ``test-win-py312``
+- ``test-win-py311``
+- ``test-mac-py312``
+- ``test-mac-py311``
+- ``test-linux-py312``
+- ``test-linux-py311``
+
+Note that the name of the tasks will be built based on the order of rows in the
+matrix.
+
+Substituting Matrix Context into the Task Definition
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Of course these tasks will be identical, so you'll want to change other parts
+of the task definition based on the matrix values.
+
+Substituting Values in Yaml
+```````````````````````````
+
+The simplest way to change a matrix task's definition, is to use the built-in
+yaml substitution:
+
+.. code-block:: yaml
+
+   tasks:
+     test:
+       matrix:
+         os: ["win", "mac", "linux"]
+       description: Run {matrix[os]} tests
+       worker-type: {matrix[os]}-worker
+
+Limiting Substitution
+'''''''''''''''''''''
+
+By default, all keys and values in the task definition will be checked for
+substitution parameters. But in some cases, it might be desirable to limit which
+keys get substituted, such as when using the ``matrix`` transforms alongside
+other transforms that perform substitution, such as the
+:mod:`~taskgraph.transforms.task_context` or
+:mod:`~taskgraph.transforms.chunking` transforms.
+
+To limit the fields that will be evaluated for substitution, you can pass in the
+``substitution-fields`` config:
+
+.. code-block:: yaml
+
+   tasks:
+     test:
+       matrix:
+         substitution-fields: ["worker-type"]
+         os: ["win"]
+       description: Run {matrix[os]} tests
+       worker-type: {matrix[os]}-worker
+
+In the example above, ``worker-type`` will evaluate to ``win-worker``, whereas
+the description will be the literal string ``Run {matrix[os]} tests``. Dot
+notation can be used in ``substitution-fields`` to limit substitution to some
+sub configuration of the task definition.
+
+Substituting Values in a Later Transform
+````````````````````````````````````````
+
+For more advanced cases, you may wish to use a later transform to act on the
+result of the matrix evaluation. To accomplish this, the ``matrix`` transforms
+will set a ``matrix`` attribute that contains all matrix values applicable to
+the task.
+
+For example, let's say you have a ``kind.yml`` like:
+
+.. code-block:: yaml
+
+   transforms:
+     - taskgraph.transforms.matrix
+     - custom_taskgraph.transforms.custom
+     # ...
+
+   tasks:
+     test:
+       matrix:
+         os: ["win", "mac", "linux"]
+
+Then in your ``custom.py`` transform file, you could add:
+
+.. code-block:: python
+
+   @transforms.add
+   def set_worker_type_and_description(config, tasks):
+       for task in tasks:
+           matrix = task["attributes"]["matrix"]
+           task["description"] = f"Run {matrix['os']} tests"
+           task["worker-type"] = f"{matrix['os']}-worker"
+           yield task
+
+This example will yield the exact same result as the yaml example above, but it
+allows for more complex logic.
+
+Excluding Matrix Combinations
+-----------------------------
+
+Sometimes you might not want to generate *every* possible combination of tasks,
+and there may be some you wish to exclude. This can be accomplished using the
+``exclude`` config:
+
+.. code-block:: yaml
+
+   tasks:
+     test:
+       matrix:
+         os: ["win", "mac"]
+         arch: ["x86", "arm64"]
+         python: ["py312", "py311"]
+         exclude:
+           - os: mac
+             arch: x86
+           - os: win
+             arch: arm64
+             python: py311
+
+This will cause all combinations where ``os == mac and arch == x86`` to be
+skipped, as well as the specific combination where ``os == win and arch ==
+arm64 and python == py311``. This means the following tasks will be generated:
+
+  * test-win-x86-py311
+  * test-win-x86-py312
+  * test-win-arm64-py312
+  * test-mac-arm64-py311
+  * test-mac-arm64-py312
+
+Customizing Task Names
+----------------------
+
+By default, the ``matrix`` transforms will append each matrix value to the
+task's name, separated by a dash. If some other format is desired, you can specify
+the ``set-name`` config:
+
+.. code-block:: yaml
+
+   tasks:
+     test:
+       matrix:
+         set-name: "test-{matrix[os]}/{matrix[python]}"
+         os: ["win"]
+         python: ["py312"]
+
+Instead of creating a task with the name ``test-win-py312``, the name will be
+``test-win/py312``.

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/docs/reference/transforms/task_context.rst

@@ -1,7 +1,7 @@
-.. _task_context:
+.. _task_context transforms:
 
-Task Context
-============
+Task Context Transforms
+=======================
 
 The :mod:`taskgraph.transforms.task_context` transform can be used to
 substitute values into any field in a task with data that is not known

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/__init__.py

@@ -2,7 +2,7 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
-__version__ = "11.2.4"
+__version__ = "12.0.0"
 
 # Maximum number of dependencies a single task can have
 # https://docs.taskcluster.net/docs/reference/platform/queue/api#createTask

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/actions/util.py

@@ -64,12 +64,19 @@ def fetch_graph_and_labels(parameters, graph_config, task_group_id=None):
                     raise
                 logger.debug(f"No label-to-taskid.json found for {task_id}: {e}")
 
-        namespace = "{}.v2.{}.pushlog-id.{}.actions".format(
+        # for backwards compatibility, look up actions via pushlog-id
+        pushlog_namespace = "{}.v2.{}.pushlog-id.{}.actions".format(
             graph_config["trust-domain"],
             parameters["project"],
             parameters["pushlog_id"],
         )
-        for task_id in list_tasks(namespace):
+        # ... but also look by revision, since github doesn't have pushlog-id
+        rev_namespace = "{}.v2.{}.revision.{}.actions".format(
+            graph_config["trust-domain"],
+            parameters["project"],
+            parameters["head_rev"],
+        )
+        for task_id in set(list_tasks(pushlog_namespace) + list_tasks(rev_namespace)):
             fetches.append(e.submit(fetch_action, task_id))
 
         # Similarly for cron tasks..

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/docker.py

@@ -47,7 +47,7 @@ def get_image_digest(image_name):
         strict=False,
     )
     tasks = load_tasks_for_kind(params, "docker-image")
-    task = tasks[f"build-docker-image-{image_name}"]
+    task = tasks[f"docker-image-{image_name}"]
     return task.attributes["cached_task"]["digest"]
 
 
@@ -61,7 +61,7 @@ def load_image_by_name(image_name, tag=None):
         strict=False,
     )
     tasks = load_tasks_for_kind(params, "docker-image")
-    task = tasks[f"build-docker-image-{image_name}"]
+    task = tasks[f"docker-image-{image_name}"]
 
     indexes = task.optimization.get("index-search", [])
     task_id = IndexSearch().should_replace_task(task, {}, None, indexes)

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/generator.py

@@ -269,9 +269,9 @@ class TaskGraphGenerator:
         logger.debug(f"Dumping parameters:\n{repr(parameters)}")
 
         filters = parameters.get("filters", [])
-        # Always add legacy target tasks method until we deprecate that API.
-        if "target_tasks_method" not in filters:
-            filters.insert(0, "target_tasks_method")
+        if not filters:
+            # Default to target_tasks_method if none specified.
+            filters.append("target_tasks_method")
         filters = [filter_tasks.filter_task_functions[f] for f in filters]
 
         yield self.verify("parameters", parameters)

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/parameters.py

@@ -193,7 +193,7 @@ class Parameters(ReadOnlyDict):
         if spec is None:
             return "defaults"
 
-        if any(spec.startswith(s) for s in ("task-id=", "project=")):
+        if any(spec.startswith(s) for s in ("task-id=", "project=", "index=")):
             return spec
 
         result = urlparse(spec)
@@ -327,16 +327,19 @@ def load_parameters_file(
         task_id = None
         if spec.startswith("task-id="):
             task_id = spec.split("=")[1]
-        elif spec.startswith("project="):
-            if trust_domain is None:
-                raise ValueError(
-                    "Can't specify parameters by project "
-                    "if trust domain isn't supplied.",
+        elif spec.startswith("project=") or spec.startswith("index="):
+            if spec.startswith("project="):
+                if trust_domain is None:
+                    raise ValueError(
+                        "Can't specify parameters by project "
+                        "if trust domain isn't supplied.",
+                    )
+                index = "{trust_domain}.v2.{project}.latest.taskgraph.decision".format(
+                    trust_domain=trust_domain,
+                    project=spec.split("=")[1],
                 )
-            index = "{trust_domain}.v2.{project}.latest.taskgraph.decision".format(
-                trust_domain=trust_domain,
-                project=spec.split("=")[1],
-            )
+            else:
+                index = spec.split("=")[1]
             task_id = find_task_id(index)
 
         if task_id:

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/task.py

@@ -17,7 +17,7 @@ class Task:
     - task: the task definition (JSON-able dictionary)
     - optimization: optimization to apply to the task (see taskgraph.optimize)
     - dependencies: tasks this one depends on, in the form {name: label}, for example
-      {'build': 'build-linux64/opt', 'docker-image': 'build-docker-image-desktop-test'}
+      {'build': 'build-linux64/opt', 'docker-image': 'docker-image-desktop-test'}
     - soft_dependencies: tasks this one may depend on if they are available post
       optimisation. They are set as a list of tasks label.
     - if_dependencies: only run this task if at least one of these dependencies

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/transforms/cached_tasks.py

@@ -14,10 +14,7 @@ transforms = TransformSequence()
 
 def order_tasks(config, tasks):
     """Iterate image tasks in an order where parent tasks come first."""
-    if config.kind == "docker-image":
-        kind_prefix = "build-docker-image-"
-    else:
-        kind_prefix = config.kind + "-"
+    kind_prefix = config.kind + "-"
 
     pending = deque(tasks)
     task_labels = {task["label"] for task in pending}

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/transforms/docker_image.py

@@ -131,7 +131,7 @@ def fill_template(config, tasks):
         # include some information that is useful in reconstructing this task
         # from JSON
         taskdesc = {
-            "label": "build-docker-image-" + image_name,
+            "label": "docker-image-" + image_name,
             "description": description,
             "attributes": {
                 "image_name": image_name,
@@ -193,7 +193,7 @@ def fill_template(config, tasks):
 
         if parent:
             deps = taskdesc.setdefault("dependencies", {})
-            deps["parent"] = f"build-docker-image-{parent}"
+            deps["parent"] = f"docker-image-{parent}"
             worker["env"]["PARENT_TASK_ID"] = {
                 "task-reference": "<parent>",
             }

taskcluster_taskgraph-12.0.0/src/taskgraph/transforms/matrix.py

@@ -0,0 +1,112 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+"""
+Transforms used to split one task definition into many tasks, governed by a
+matrix defined in the definition.
+"""
+
+from copy import deepcopy
+from textwrap import dedent
+
+from voluptuous import Extra, Optional, Required
+
+from taskgraph.transforms.base import TransformSequence
+from taskgraph.util.schema import Schema
+from taskgraph.util.templates import substitute_task_fields
+
+MATRIX_SCHEMA = Schema(
+    {
+        Required("name"): str,
+        Optional("matrix"): {
+            Optional(
+                "exclude",
+                description=dedent(
+                    """
+                Exclude the specified combination(s) of matrix values from the
+                final list of tasks.
+
+                If only a subset of the possible rows are present in the
+                exclusion rule, then *all* combinations including that subset
+                subset will be excluded.
+                """.lstrip()
+                ),
+            ): [{str: str}],
+            Optional(
+                "set-name",
+                description=dedent(
+                    """
+                Sets the task name to the specified format string.
+
+                Useful for cases where the default of joining matrix values by
+                a dash is not desired.
+                """.lstrip()
+                ),
+            ): str,
+            Optional(
+                "substitution-fields",
+                description=dedent(
+                    """
+                List of fields in the task definition to substitute matrix values into.
+
+                If not specified, all fields in the task definition will be
+                substituted.
+                """
+                ),
+            ): [str],
+            Extra: [str],
+        },
+        Extra: object,
+    },
+)
+"""Schema for matrix transforms."""
+
+transforms = TransformSequence()
+transforms.add_validate(MATRIX_SCHEMA)
+
+
+def _resolve_matrix(tasks, key, values, exclude):
+    for task in tasks:
+        for value in values:
+            new_task = deepcopy(task)
+            new_task["name"] = f"{new_task['name']}-{value}"
+
+            matrix = new_task.setdefault("attributes", {}).setdefault("matrix", {})
+            matrix[key] = value
+
+            for rule in exclude:
+                if all(matrix.get(k) == v for k, v in rule.items()):
+                    break
+            else:
+                yield new_task
+
+
+@transforms.add
+def split_matrix(config, tasks):
+    for task in tasks:
+        if "matrix" not in task:
+            yield task
+            continue
+
+        matrix = task.pop("matrix")
+        set_name = matrix.pop("set-name", None)
+        fields = matrix.pop("substitution-fields", task.keys())
+        exclude = matrix.pop("exclude", {})
+
+        new_tasks = [task]
+        for key, values in matrix.items():
+            new_tasks = _resolve_matrix(new_tasks, key, values, exclude)
+
+        for new_task in new_tasks:
+            if set_name:
+                if "name" not in fields:
+                    fields.append("name")
+                new_task["name"] = set_name
+
+            substitute_task_fields(
+                new_task,
+                fields,
+                matrix=new_task["attributes"]["matrix"],
+            )
+            yield new_task

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/transforms/run/__init__.py

@@ -121,7 +121,9 @@ def rewrite_when_to_optimization(config, tasks):
         files_changed = when.get("files-changed")
 
         # implicitly add task config directory.
-        files_changed.append(f"{config.path}/**")
+        files_changed.append(f"{config.path}/kind.yml")
+        if task.get("task-from") and task["task-from"] != "kind.yml":
+            files_changed.append(f"{config.path}/{task['task-from']}")
 
         # "only when files changed" implies "skip if files have not changed"
         task["optimization"] = {"skip-unless-changed": files_changed}

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/transforms/task.py

@@ -371,7 +371,7 @@ def build_docker_worker_payload(config, task, task_def):
     if isinstance(image, dict):
         if "in-tree" in image:
             name = image["in-tree"]
-            docker_image_task = "build-docker-image-" + image["in-tree"]
+            docker_image_task = "docker-image-" + image["in-tree"]
             assert "docker-image" not in task.get(
                 "dependencies", ()
             ), "docker-image key in dependencies object is reserved"

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/transforms/task_context.py

@@ -4,7 +4,7 @@ from voluptuous import ALLOW_EXTRA, Any, Optional, Required
 
 from taskgraph.transforms.base import TransformSequence
 from taskgraph.util.schema import Schema
-from taskgraph.util.templates import deep_get, substitute
+from taskgraph.util.templates import deep_get, substitute_task_fields
 from taskgraph.util.yaml import load_yaml
 
 SCHEMA = Schema(
@@ -113,12 +113,5 @@ def render_task(config, tasks):
             subs.setdefault("name", task["name"])
 
         # Now that we have our combined context, we can substitute.
-        for field in fields:
-            container, subfield = task, field
-            while "." in subfield:
-                f, subfield = subfield.split(".", 1)
-                container = container[f]
-
-            container[subfield] = substitute(container[subfield], **subs)
-
+        substitute_task_fields(task, fields, **subs)
         yield task

{taskcluster_taskgraph-11.2.4 → taskcluster_taskgraph-12.0.0}/src/taskgraph/util/docker.py

@@ -149,7 +149,7 @@ def stream_context_tar(topsrcdir, context_dir, out_file, image_name=None, args=N
         for f in files:
             source_path = os.path.join(root, f)
             archive_path = source_path[len(context_dir) + 1 :]
-            archive_files[archive_path] = open(source_path, "rb")
+            archive_files[archive_path] = source_path
 
     # Parse Dockerfile for special syntax of extra files to include.
     content = []