taskcluster-taskgraph 22.0.0__tar.gz → 23.0.0__tar.gz

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/.pre-commit-config.yaml

@@ -13,13 +13,13 @@ repos:
         exclude: template
       - id: check-added-large-files
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.15.8
+    rev: v0.15.12
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
       - id: ruff-format
   - repo: https://github.com/astral-sh/uv-pre-commit
-    rev: 0.11.2
+    rev: 0.11.8
     hooks:
       - id: uv-lock
   - repo: https://github.com/adrienverge/yamllint.git

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/CHANGELOG.md

@@ -1,5 +1,21 @@
 # Change Log
 
+## [23.0.0] - 2026-05-04
+
+### Added
+
+- Support for defining schemas as dicts (in addition to the class-based approach)
+- New `disabled_actions` graph configuration to prevent specific actions from being generated
+
+### Changed
+
+- BREAKING CHANGE: Converted parameter schema from voluptuous to msgspec
+
+### Fixed
+
+- `taskgraph full -J --tasks <regex>` now displays full dependencies instead of a filtered subset
+- `fetch-content` now percent-encodes artifact names, fixing downloads of artifacts containing spaces or other special characters
+
 ## [22.0.0] - 2026-04-16
 
 ### Fixed

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: taskcluster-taskgraph
-Version: 22.0.0
+Version: 23.0.0
 Summary: Build taskcluster taskgraphs
 Project-URL: Repository, https://github.com/taskcluster/taskgraph
 Project-URL: Issues, https://github.com/taskcluster/taskgraph/issues

taskcluster_taskgraph-23.0.0/docs/howto/define-schemas.rst

@@ -0,0 +1,156 @@
+Define Schemas
+==============
+
+:doc:`Transforms </concepts/transforms>` can define schemas to validate task data at each step of the
+pipeline. Taskgraph uses `msgspec`_ under the hood, and provides a
+:class:`~taskgraph.util.schema.Schema` base class that integrates with
+:meth:`TransformSequence.add_validate() <taskgraph.transforms.base.TransformSequence.add_validate>`.
+
+There are two ways to define a schema: the **class based** approach and the
+**dict based** approach. Both produce equivalent results; which one you prefer
+is a matter of style.
+
+.. _msgspec: https://jcristharif.com/msgspec/
+
+
+Class Based Schemas
+-------------------
+
+Subclass :class:`~taskgraph.util.schema.Schema` and declare fields as class
+attributes with type annotations:
+
+.. code-block:: python
+
+   from typing import Optional
+   from taskgraph.transforms.base import TransformSequence
+   from taskgraph.util.schema import Schema
+
+   class MySubConfig(Schema):
+       total_num: int
+       fields: list[str] = []
+
+   class MySchema(Schema, forbid_unknown_fields=False):
+       config: Optional[MySubConfig] = None
+
+   transforms = TransformSequence()
+   transforms.add_validate(MySchema)
+
+A few things to note:
+
+- Field names use ``snake_case`` in Python but are **automatically renamed to
+  ``kebab-case``** in YAML. So ``total_num`` in Python matches
+  ``total-num`` in YAML.
+- ``Optional[T]`` fields default to ``None`` unless you supply an explicit
+  default.
+- Fields without a default are **required**.
+- ``forbid_unknown_fields=True`` (the default) causes validation to fail if the
+  task data contains keys that are not declared in the schema. Set it to
+  ``False`` on outer schemas so that fields belonging to later transforms are
+  not rejected.
+
+
+Dict Based Schemas
+------------------
+
+Call :meth:`Schema.from_dict() <taskgraph.util.schema.Schema.from_dict>` with a
+dictionary mapping field names to ``type`` or ``(type, default)`` tuples:
+
+.. code-block:: python
+
+   from typing import Optional, Union
+   from taskgraph.transforms.base import TransformSequence
+   from taskgraph.util.schema import Schema
+
+   MySchema = Schema.from_dict(
+       {
+           "config": Schema.from_dict(
+               {
+                   "total-num": int,
+                   "fields": list[str] = []
+               },
+               optional=True,
+           ),
+       },
+       forbid_unknown_fields=False,
+   )
+
+   transforms = TransformSequence()
+   transforms.add_validate(MySchema)
+
+This example is equivalent to the first example. One advantage with the dict based approach
+is that you can write keys in **kebab-case** directly.
+
+Field specifications follow these rules:
+
+- A bare type (e.g. ``str``) means the field is required.
+- ``Optional[T]`` means the field is optional and defaults to ``None``.
+- A ``(type, default)`` tuple supplies an explicit default, e.g.
+  ``(list[str], [])``.
+
+Keyword arguments to ``from_dict`` are forwarded to ``msgspec.defstruct``.
+The most commonly used ones are ``name`` (for better error messages) and
+``forbid_unknown_fields``.
+
+.. note::
+   ``Schema.from_dict`` does **not** apply ``rename="kebab"`` automatically,
+   because you can express the kebab-case names directly in the dict keys.
+   Underscores in dict keys stay as underscores and dashes become valid
+   kebab-case field names.
+
+
+Nesting Schemas
+---------------
+
+Both approaches support nesting:
+
+.. code-block:: python
+
+   # Class-based nesting
+   class Inner(Schema):
+       value: str
+
+   class Outer(Schema, forbid_unknown_fields=False, kw_only=True):
+       inner: Optional[Inner] = None
+
+   # Dict-based nesting
+   Outer = Schema.from_dict(
+       {
+           "inner": Schema.from_dict({"value": str}, optional=True),
+       },
+       forbid_unknown_fields=False,
+   )
+
+Pass ``optional=True`` to ``from_dict`` to make the whole nested schema
+optional. This is necessary as function calls are not allowed in type
+annotations, so ``Optional[Schema.from_dict(...)]`` is not valid Python.
+
+
+Mutually Exclusive Fields
+-------------------------
+
+Use the ``exclusive`` keyword to declare groups of fields where at most one
+may be set at a time:
+
+.. code-block:: python
+
+   # Class-based
+   class MySchema(Schema, exclusive=[["field_a", "field_b"]]):
+       field_a: Optional[str] = None
+       field_b: Optional[str] = None
+
+   # Dict-based
+   MySchema = Schema.from_dict(
+       {
+           "field-a": Optional[str],
+           "field-b": Optional[str],
+       },
+       exclusive=[["field_a", "field_b"]],
+   )
+
+``exclusive`` takes a list of groups, where each group is a list of field
+names (Python ``snake_case``). A validation error is raised if more than one
+field in a group is set.
+
+.. note::
+   When using ``exclusive`` with the dict-based approach, refer to fields by
+   their Python attribute names (``snake_case``), not their YAML keys.

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/docs/howto/index.rst

@@ -10,6 +10,7 @@ A collection of how-to guides.
    run-locally
    debugging
    bootstrap-taskgraph
+   define-schemas
    resolve-keyed-by
    use-fetches
    docker

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/docs/reference/migrations.rst

@@ -3,6 +3,13 @@ Migration Guide
 
 This page can help when migrating Taskgraph across major versions.
 
+22.x -> 23.x
+------------
+
+* :func:`taskgraph.parameters.extend_parameters_schema` now requires a msgspec
+  ``Schema`` subclass instead of a voluptuous schema dict. Convert any custom
+  parameter schemas to msgspec.
+
 21.x -> 22.x
 ------------
 

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/pyproject.toml

@@ -1,7 +1,7 @@
 ### Project
 [project]
 name = "taskcluster-taskgraph"
-version = "22.0.0"
+version = "23.0.0"
 description = "Build taskcluster taskgraphs"
 readme = "README.rst"
 authors = [

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/src/taskgraph/actions/registry.py

@@ -273,12 +273,27 @@ def render_actions_json(parameters, graph_config, decision_task_id):
             artifact.
     """
     assert isinstance(parameters, Parameters), "requires instance of Parameters"
+
+    all_actions = _get_actions(graph_config)
+    disabled = set(graph_config["taskgraph"].get("disabled-actions") or [])
+    known_cb_names = {action.cb_name for action in all_actions}
+    unknown = disabled - known_cb_names
+    if unknown:
+        raise ValueError(
+            "Unknown action(s) in `taskgraph.disabled-actions`: "
+            f"{sorted(unknown)}. Known actions: {sorted(known_cb_names)}"
+        )
+
     actions = []
-    for action in sorted(_get_actions(graph_config), key=lambda action: action.order):
+    for action in sorted(all_actions, key=lambda action: action.order):
+        if action.cb_name in disabled:
+            continue
+
         action = action.action_builder(parameters, graph_config, decision_task_id)
         if action:
             assert is_json(action), "action must be a JSON compatible object"
             actions.append(action)
+
     return {
         "version": 1,
         "variables": {},

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/src/taskgraph/config.py

@@ -57,6 +57,9 @@ class TaskgraphConfig(Schema):
     index_path_regexes: Optional[list[str]] = None
     # Configuration related to the 'run' transforms.
     run: Optional[RunConfig] = None
+    # List of action `cb_name`s to omit from `actions.json`,
+    # e.g. `["retrigger", "retrigger-disabled", "rerun"]`.
+    disabled_actions: Optional[list[str]] = None
 
     def __post_init__(self):
         # Validate repositories has at least 1 entry (was All(..., Length(min=1)))

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/src/taskgraph/parameters.py

@@ -10,16 +10,17 @@ from datetime import datetime
 from io import BytesIO
 from pprint import pformat
 from subprocess import CalledProcessError
+from typing import Optional, Union
 from unittest.mock import Mock
 from urllib.parse import urlparse
 from urllib.request import urlopen
 
 import mozilla_repo_urls
-from voluptuous import ALLOW_EXTRA, Any, Optional, Required, Schema
+import msgspec
 
 from taskgraph.util import json, yaml
 from taskgraph.util.readonlydict import ReadOnlyDict
-from taskgraph.util.schema import validate_schema
+from taskgraph.util.schema import Schema, validate_schema
 from taskgraph.util.taskcluster import find_task_id, get_artifact_url
 from taskgraph.util.vcs import get_repository
 
@@ -30,43 +31,50 @@ class ParameterMismatch(Exception):
 
 #: Schema for base parameters.
 #: Please keep this list sorted and in sync with docs/reference/parameters.rst
-base_schema = Schema(
+base_schema = Schema.from_dict(
     {
-        Required("base_repository"): str,
-        Optional("base_ref"): str,
-        Required("base_rev"): str,
-        Required("build_date"): int,
-        Required("build_number"): int,
-        Required("do_not_optimize"): [str],
-        Required("enable_always_target"): Any(bool, [str]),
-        Required("existing_tasks"): {str: str},
-        Required("files_changed"): [str],
-        Required("filters"): [str],
-        Required("head_ref"): str,
-        Required("head_repository"): str,
-        Required("head_rev"): str,
-        Required("head_tag"): str,
-        Required("level"): str,
-        Required("moz_build_date"): str,
-        Required("next_version"): Any(str, None),
-        Required("optimize_strategies"): Any(str, None),
-        Required("optimize_target_tasks"): bool,
-        Required("owner"): str,
-        Required("project"): str,
-        Required("pushdate"): int,
-        Required("pushlog_id"): str,
-        Required("repository_type"): str,
+        "base_repository": str,
+        "base_ref": Optional[str],
+        "base_rev": str,
+        "build_date": int,
+        "build_number": int,
+        "do_not_optimize": list[str],
+        "enable_always_target": Union[bool, list[str]],
+        "existing_tasks": dict[str, str],
+        "files_changed": list[str],
+        "filters": list[str],
+        "head_ref": str,
+        "head_repository": str,
+        "head_rev": str,
+        "head_tag": str,
+        "level": str,
+        "moz_build_date": str,
+        "next_version": Optional[str],
+        "optimize_strategies": Optional[str],
+        "optimize_target_tasks": bool,
+        "owner": str,
+        "project": str,
+        "pushdate": int,
+        "pushlog_id": str,
+        "repository_type": str,
         # target-kinds is not included, since it should never be
         # used at run-time
-        Required("target_tasks_method"): str,
-        Required("tasks_for"): str,
-        Required("version"): Any(str, None),
-        Optional("code-review"): {
-            Required("phabricator-build-target"): str,
-        },
-    }
+        "target_tasks_method": str,
+        "tasks_for": str,
+        "version": Optional[str],
+        "code-review": Schema.from_dict(
+            {"phabricator-build-target": str},
+            name="CodeReviewConfig",
+            optional=True,
+        ),
+    },
+    name="BaseParametersSchema",
+    forbid_unknown_fields=False,
+    kw_only=True,
 )
 
+_parameter_extensions: list = []
+
 
 def get_contents(path):
     with open(path) as fh:
@@ -83,11 +91,21 @@ def _get_defaults(repo_root=None):
     repo_path = repo_root or os.getcwd()
     try:
         repo = get_repository(repo_path)
-    except RuntimeError:
-        # Use fake values if no repo is detected.
-        repo = Mock(branch="", head_rev="", tool="git")
+        # Resolve git-backed attributes eagerly so any subprocess failures
+        # (e.g. Windows "dubious ownership" when safe.directory isn't honored)
+        # are caught by the except below instead of escaping later.
+        branch = repo.branch
+        head_rev = repo.head_rev
+        tool = repo.tool
+        files_changed = repo.get_changed_files("AM")
+    except (RuntimeError, CalledProcessError):
+        # Use fake values if no repo is detected or git refuses to operate.
+        repo = Mock()
         repo.get_url.return_value = ""
-        repo.get_changed_files.return_value = []
+        branch = ""
+        head_rev = ""
+        tool = "git"
+        files_changed = []
 
     try:
         repo_url = repo.get_url()
@@ -110,11 +128,11 @@ def _get_defaults(repo_root=None):
         "do_not_optimize": [],
         "enable_always_target": True,
         "existing_tasks": {},
-        "files_changed": lambda: repo.get_changed_files("AM"),
+        "files_changed": files_changed,
         "filters": ["target_tasks_method"],
-        "head_ref": repo.branch or repo.head_rev,
+        "head_ref": branch or head_rev,
         "head_repository": repo_url,
-        "head_rev": repo.head_rev,
+        "head_rev": head_rev,
         "head_tag": "",
         "level": "3",
         "moz_build_date": datetime.now().strftime("%Y%m%d%H%M%S"),
@@ -125,7 +143,7 @@ def _get_defaults(repo_root=None):
         "project": project,
         "pushdate": int(time.time()),
         "pushlog_id": "0",
-        "repository_type": repo.tool,
+        "repository_type": tool,
         "target_tasks_method": "default",
         "tasks_for": "",
         "version": get_version(repo_path),
@@ -143,19 +161,27 @@ def extend_parameters_schema(schema, defaults_fn=None):
     graph-configuration.
 
     Args:
-        schema (Schema): The voluptuous.Schema object used to describe extended
-            parameters.
+        schema: A msgspec ``Schema`` subclass describing extended parameters.
         defaults_fn (function): A function which takes no arguments and returns a
             dict mapping parameter name to default value in the
             event strict=False (optional).
     """
-    global base_schema
     global defaults_functions
-    base_schema = base_schema.extend(schema)
+    if not (isinstance(schema, type) and issubclass(schema, msgspec.Struct)):
+        raise TypeError(
+            "extend_parameters_schema requires a msgspec Schema subclass; "
+            f"got {type(schema).__name__}"
+        )
+    _parameter_extensions.append(schema)
     if defaults_fn:
         defaults_functions.append(defaults_fn)
 
 
+def _schema_key_names(schema) -> set:
+    """Return the data-level field names declared by a parameters schema."""
+    return {f.encode_name for f in msgspec.structs.fields(schema)}
+
+
 class Parameters(ReadOnlyDict):
     """An immutable dictionary with nicer KeyError messages on failure"""
 
@@ -214,11 +240,30 @@ class Parameters(ReadOnlyDict):
         return kwargs
 
     def check(self):
-        schema = (
-            base_schema if self.strict else base_schema.extend({}, extra=ALLOW_EXTRA)
-        )
+        data = dict(self.copy())
         try:
-            validate_schema(schema, self.copy(), "Invalid parameters:")
+            # Validate core fields against just the subset of data owned by the
+            # base schema. Extension keys are validated separately below, and a
+            # strict-mode check rejects anything unknown to either.
+            base_keys = _schema_key_names(base_schema)
+            base_data = {k: v for k, v in data.items() if k in base_keys}
+            validate_schema(base_schema, base_data, "Invalid parameters:")
+
+            # Validate each registered extension against the keys it declares.
+            allowed = set(base_keys)
+            for ext in _parameter_extensions:
+                ext_keys = _schema_key_names(ext)
+                allowed |= ext_keys
+                ext_data = {k: data[k] for k in ext_keys if k in data}
+                validate_schema(ext, ext_data, "Invalid parameters:")
+
+            # Strict mode: reject any data key not covered by base or extensions.
+            if self.strict:
+                unknown = sorted(set(data) - allowed)
+                if unknown:
+                    raise Exception(
+                        "Invalid parameters:\nunknown keys: " + ", ".join(unknown)
+                    )
         except Exception as e:
             raise ParameterMismatch(str(e))
 

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/src/taskgraph/run-task/fetch-content

@@ -626,7 +626,7 @@ def fetch_and_extract(url, dest_dir, extract=True, sha256=None, size=None):
     the destination directory.
     """
 
-    basename = urllib.parse.urlparse(url).path.split("/")[-1]
+    basename = urllib.parse.unquote(urllib.parse.urlparse(url).path.split("/")[-1])
     dest_path = dest_dir / basename
 
     download_to_path(url, dest_path, sha256=sha256, size=size)
@@ -920,16 +920,17 @@ def command_task_artifacts(args):
         sha256 = None
         if fetch.get("verify-hash"):
             sha256 = get_hash(fetch, root_url)
+        encoded_artifact = urllib.parse.quote(fetch["artifact"])
         if fetch["artifact"].startswith("public/"):
             path = "task/{task}/artifacts/{artifact}".format(
-                task=fetch["task"], artifact=fetch["artifact"]
+                task=fetch["task"], artifact=encoded_artifact
             )
             url = api(root_url, "queue", "v1", path)
         else:
             url = ("{proxy_url}/api/queue/v1/task/{task}/artifacts/{artifact}").format(
                 proxy_url=os.environ["TASKCLUSTER_PROXY_URL"],
                 task=fetch["task"],
-                artifact=fetch["artifact"],
+                artifact=encoded_artifact,
             )
         downloads.append((url, extdir, fetch["extract"], sha256))
 

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/src/taskgraph/taskgraph.py

@@ -44,13 +44,10 @@ class TaskGraph:
 
     def to_json(self):
         "Return a JSON-able object representing the task graph, as documented"
-        named_links_dict = self.graph.named_links_dict()
         # this dictionary may be keyed by label or by taskid, so let's just call it 'key'
         tasks = {}
         for key in self.graph.visit_postorder():
             tasks[key] = self.tasks[key].to_json()
-            # overwrite dependencies with the information in the taskgraph's edges.
-            tasks[key]["dependencies"] = named_links_dict.get(key, {})
         return tasks
 
     @classmethod

{taskcluster_taskgraph-22.0.0 → taskcluster_taskgraph-23.0.0}/src/taskgraph/transforms/from_deps.py

@@ -22,46 +22,46 @@ from taskgraph.util.dependencies import GROUP_BY_MAP, get_dependencies
 from taskgraph.util.schema import Schema, validate_schema
 from taskgraph.util.set_name import SET_NAME_MAP
 
-
-class FromDepsConfig(Schema):
-    # Limit dependencies to specified kinds (defaults to all kinds in
-    # `kind-dependencies`).
-    #
-    # The first kind in the list is the "primary" kind. The
-    # dependency of this kind will be used to derive the label
-    # and copy attributes (if `copy-attributes` is True).
-    kinds: Optional[list[str]] = None
-    # Set-name function (dynamic: validated at runtime against SET_NAME_MAP).
-    set_name: Optional[Union[bool, str, dict[str, object]]] = None
-    # Limit dependencies to tasks whose attributes match
-    # using :func:`~taskgraph.util.attributes.attrmatch`.
-    with_attributes: Optional[dict[str, Union[list, str]]] = None
-    # Group cross-kind dependencies using the given group-by
-    # function. One task will be created for each group. If not
-    # specified, the 'single' function will be used which creates
-    # a new task for each individual dependency.
-    group_by: Optional[Union[str, dict[str, object]]] = None
-    # If True, copy attributes from the dependency matching the
-    # first kind in the `kinds` list (whether specified explicitly
-    # or taken from `kind-dependencies`).
-    copy_attributes: Optional[bool] = None
-    # If true (the default), there must be only a single unique task
-    # for each kind in a dependency group. Setting this to false
-    # disables that requirement.
-    unique_kinds: Optional[bool] = None
-    # If present, a `fetches` entry will be added for each task
-    # dependency. Attributes of the upstream task may be used as
-    # substitution values in the `artifact` or `dest` values of the
-    # `fetches` entry.
-    fetches: Optional[dict[str, list[FetchesEntrySchema]]] = None
-
-
-#: Schema for from_deps transforms
-class FromDepsSchema(Schema, forbid_unknown_fields=False, kw_only=True):
-    from_deps: FromDepsConfig
-
-
-FROM_DEPS_SCHEMA = FromDepsSchema
+FROM_DEPS_SCHEMA = Schema.from_dict(
+    {
+        "from-deps": Schema.from_dict(
+            {
+                # Limit dependencies to specified kinds (defaults to all kinds in
+                # `kind-dependencies`).
+                #
+                # The first kind in the list is the "primary" kind. The
+                # dependency of this kind will be used to derive the label
+                # and copy attributes (if `copy-attributes` is True).
+                "kinds": Optional[list[str]],
+                # Set-name function (dynamic: validated at runtime against SET_NAME_MAP).
+                "set-name": Optional[Union[bool, str, dict[str, object]]],
+                # Limit dependencies to tasks whose attributes match
+                # using :func:`~taskgraph.util.attributes.attrmatch`.
+                "with-attributes": Optional[dict[str, Union[list, str]]],
+                # Group cross-kind dependencies using the given group-by
+                # function. One task will be created for each group. If not
+                # specified, the 'single' function will be used which creates
+                # a new task for each individual dependency.
+                "group-by": Optional[Union[str, dict[str, object]]],
+                # If True, copy attributes from the dependency matching the
+                # first kind in the `kinds` list (whether specified explicitly
+                # or taken from `kind-dependencies`).
+                "copy-attributes": Optional[bool],
+                # If true (the default), there must be only a single unique task
+                # for each kind in a dependency group. Setting this to false
+                # disables that requirement.
+                "unique-kinds": Optional[bool],
+                # If present, a `fetches` entry will be added for each task
+                # dependency. Attributes of the upstream task may be used as
+                # substitution values in the `artifact` or `dest` values of the
+                # `fetches` entry.
+                "fetches": Optional[dict[str, list[FetchesEntrySchema]]],
+            },
+        ),
+    },
+    name="FromDepsSchema",
+    forbid_unknown_fields=False,
+)
 
 transforms = TransformSequence()
 transforms.add_validate(FROM_DEPS_SCHEMA)