tangle-cli 0.1.18__tar.gz → 0.1.20__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (103) hide show
  1. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/PKG-INFO +87 -4
  2. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/README.md +86 -3
  3. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/__init__.py +1 -1
  4. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/args_container.py +478 -39
  5. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_compiler.py +115 -34
  6. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipelines.py +11 -0
  7. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/cfg.py +84 -26
  8. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/compiler_context.py +8 -0
  9. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/errors.py +11 -0
  10. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/schema_validation.py +158 -8
  11. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/pyproject.toml +1 -1
  12. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/pyproject.toml.orig +1 -1
  13. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/api_cli.py +0 -0
  14. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/api_schema.py +0 -0
  15. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/api_transport.py +0 -0
  16. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/artifacts.py +0 -0
  17. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/artifacts_cli.py +0 -0
  18. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/authenticated_identity.py +0 -0
  19. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/cli.py +0 -0
  20. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/cli_helpers.py +0 -0
  21. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/cli_options.py +0 -0
  22. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/client.py +0 -0
  23. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/component_from_func.py +0 -0
  24. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/component_generator.py +0 -0
  25. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/component_inspector.py +0 -0
  26. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/component_publisher.py +0 -0
  27. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/components_cli.py +0 -0
  28. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/dynamic_discovery_client.py +0 -0
  29. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/generated_model_extensions.py +0 -0
  30. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/handler.py +0 -0
  31. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/hydration_trust.py +0 -0
  32. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/logger.py +0 -0
  33. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/models.py +0 -0
  34. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/module_bundler.py +0 -0
  35. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/openapi/__init__.py +0 -0
  36. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/openapi/codegen.py +0 -0
  37. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/openapi/parser.py +0 -0
  38. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_dehydrator.py +0 -0
  39. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_hydrator.py +0 -0
  40. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_run_annotations.py +0 -0
  41. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_run_details.py +0 -0
  42. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_run_manager.py +0 -0
  43. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_run_search.py +0 -0
  44. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_runner.py +0 -0
  45. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_runs_cli.py +0 -0
  46. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_spec_utils.py +0 -0
  47. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipeline_validation.py +0 -0
  48. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/pipelines_cli.py +0 -0
  49. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/published_components_cli.py +0 -0
  50. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/py.typed +0 -0
  51. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/__init__.py +0 -0
  52. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/dynamic_data.py +0 -0
  53. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/emit.py +0 -0
  54. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/graph.py +0 -0
  55. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/ids.py +0 -0
  56. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/pipeline.py +0 -0
  57. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/placeholders.py +0 -0
  58. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/publish.py +0 -0
  59. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/raw.py +0 -0
  60. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/ref.py +0 -0
  61. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/registered.py +0 -0
  62. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/subpipeline.py +0 -0
  63. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/task.py +0 -0
  64. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/task_env.py +0 -0
  65. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/trace.py +0 -0
  66. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/python_pipeline/types.py +0 -0
  67. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/quickstart.py +0 -0
  68. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/schemas/__init__.py +0 -0
  69. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/schemas/dehydrated_pipeline_schema.json +0 -0
  70. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/schemas/pipeline_schema.json +0 -0
  71. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/secrets.py +0 -0
  72. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/secrets_cli.py +0 -0
  73. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/utils.py +0 -0
  74. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/packages/tangle-cli/src/tangle_cli/version_manager.py +0 -0
  75. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/OSS-CONVENTIONS.md +0 -0
  76. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/PORT-README.md +0 -0
  77. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/SKILL.md +0 -0
  78. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/agents/auth-wizard.md +0 -0
  79. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/agents/builder.md +0 -0
  80. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/agents/debugger.md +0 -0
  81. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/agents/reporter.md +0 -0
  82. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/agents/researcher.md +0 -0
  83. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/agents/reviewer.md +0 -0
  84. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/agents/scenario-builder.md +0 -0
  85. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/data-sources.md +0 -0
  86. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/event-log.md +0 -0
  87. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/example-scenarios/01-mslr-ranking.md +0 -0
  88. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/example-scenarios/02-text-classification.md +0 -0
  89. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/example-scenarios/INDEX.md +0 -0
  90. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/iterating-on-runs.md +0 -0
  91. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/knowledge-corpus.md +0 -0
  92. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/secrets.md +0 -0
  93. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/setup.md +0 -0
  94. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/step-0-initialize.md +0 -0
  95. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/step-1-analyze.md +0 -0
  96. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/step-2-hypothesize.md +0 -0
  97. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/step-3-submit.md +0 -0
  98. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/step-4-monitor.md +0 -0
  99. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/step-5-evaluate.md +0 -0
  100. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/step-6-synthesize.md +0 -0
  101. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/step-7-decide.md +0 -0
  102. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/tangle-tools.md +0 -0
  103. {tangle_cli-0.1.18 → tangle_cli-0.1.20}/skills/tangent/references/uploading-artifacts.md +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.3
2
2
  Name: tangle-cli
3
- Version: 0.1.18
3
+ Version: 0.1.20
4
4
  Summary: CLI for Tangle, the open-source ML pipeline orchestration platform
5
5
  Author: Alexey Volkov, Tangle authors, Volv Grebennikov
6
6
  Author-email: Alexey Volkov <alexey.volkov@ark-kun.com>, Volv Grebennikov <volv.grebennikov@shopify.com>
@@ -83,7 +83,7 @@ API-backed commands commonly accept these options. Explicit CLI options win over
83
83
  | `--token`, `TANGLE_API_TOKEN` | Bearer token shorthand. |
84
84
  | `--auth-header`, `TANGLE_API_AUTH_HEADER`, `TANGLE_AUTH_HEADER` | Full `Authorization` value such as `Bearer ...` or `Basic ...`. |
85
85
  | `-H`, `--header`, `TANGLE_API_HEADERS` | Extra headers. Repeatable as CLI flags; env accepts a JSON object or newline-separated `Name: value` entries. |
86
- | `--config` | YAML/JSON defaults. Many commands accept a single object, a list of objects, or `_defaults` + `configs`, optionally wrapped in a top-level `_select` environment selector. |
86
+ | `--config` | YAML/JSON defaults. Many commands accept a single object, a list of objects, or `_defaults` + `configs`, optionally wrapped in a top-level `_select` environment selector. Values may be read from the environment with `{_env: NAME}`. |
87
87
  | `--log-type` | SDK progress logs: `console`, `none`, or `file`. Logs go to stderr or a temp log file so structured stdout stays parseable. |
88
88
  | `TANGLE_VERBOSE=1` | Redacted HTTP request/response diagnostics only. This is separate from normal progress logging. |
89
89
  | `--ca-bundle` | Global CLI flag: path to a PEM CA bundle used as the TLS trust store for every transport. Overrides `TANGLE_API_CA_BUNDLE`. Place before the subcommand. |
@@ -258,7 +258,7 @@ For generated `tangle api` commands, config keys use generated CLI parameter nam
258
258
 
259
259
  ### Environment-selected configs (`_select`)
260
260
 
261
- Any command that accepts `--config` can pick one of several config documents from an environment variable by making `_select` the top-level node:
261
+ Any command that accepts `--config`, and any Python pipeline `config.yaml` (see [Environment-selected pipeline config](#environment-selected-pipeline-config-_select-_env)), can pick one of several config documents from an environment variable by making `_select` the top-level node:
262
262
 
263
263
  ```yaml
264
264
  _shared: &shared
@@ -297,7 +297,7 @@ With `default`, an unset variable or a value matching no case resolves to that b
297
297
 
298
298
  Rules:
299
299
 
300
- - `_select` is the only newly reserved key, and it is an exact key name rather than a prefix. Configs without `_select` are unchanged byte-for-byte and semantically.
300
+ - `_select` is an exact reserved key name rather than a prefix (the only other reserved key is the [`_env`](#environment-variable-values-_env) value directive). Configs without `_select` are unchanged byte-for-byte and semantically.
301
301
  - At a `_select` node, only `_select` and other underscore-prefixed helper keys (YAML anchor holders such as `_shared`) may appear; ordinary sibling keys are rejected.
302
302
  - `_select` accepts only `env`, `cases`, and the optional `default`. There are no aliases: `else`, `fallback`, and `defaults` are rejected.
303
303
  - The selector shape, the `env` name, every `cases` key, and every case and `default` branch are validated before the environment is read, so a malformed selector fails identically in every environment. Branches are checked as complete config documents with the same rules the loader applies to a whole file, and nested selectors are validated recursively.
@@ -306,6 +306,41 @@ Rules:
306
306
  - Matching uses `os.environ[NAME]` exactly: case sensitive, with no trimming, case folding, or interpolation.
307
307
  - Fallback exists only where it is authored. Without `default`, an unset variable or an unmatched value is an error; there is never an implicit default or implicit production branch. The raw environment value is never echoed — diagnostics list only the configured case names.
308
308
 
309
+ ### Environment-variable values (`_env`)
310
+
311
+ Any config value — in `--config` files and in [Python pipeline configs](#environment-selected-pipeline-config-_select-_env) — can be read from an environment variable with the `_env` value directive, so a checked-in config can reference a secret or per-machine value without containing it:
312
+
313
+ ```yaml
314
+ base_url: https://api.prod
315
+ token: {_env: TANGLE_PROD_TOKEN}
316
+ header:
317
+ - {_env: GATEWAY_HEADER}
318
+ - "X-Team: search"
319
+ limit: {_env: RUN_LIMIT, default: 10}
320
+ ```
321
+
322
+ - The directive is exactly `{_env: NAME}` or `{_env: NAME, default: <scalar>}`. Any other key beside `_env` — including underscore-prefixed keys and aliases such as `fallback` — is rejected, and `NAME` follows the same rule as `_select.env` (`[A-Za-z_][A-Za-z0-9_]*`).
323
+ - It may appear anywhere a value may appear: under a config key, in `_defaults`, in `configs` entries, and inside nested maps and lists. A document or config-entry mapping is never itself a directive, so an `_env:` helper/anchor key at the top level of a config object is unaffected.
324
+ - A missing variable without `default` fails closed with the variable name, the key path (for example `configs[1].token`), and the config file. An empty string counts as set and is used as-is.
325
+ - The resolved value is always a string, and a `default` is stringified the same way: numbers and booleans use their JSON spelling (`10`, `1.5`, `true`), dates use ISO format, and `null`/maps/lists are rejected (quote `''` for an empty default). A field therefore has one type whether or not the variable is set, and the command's usual conversion (JSON fields, repeatable options, enums, typed converters) applies downstream in both cases.
326
+ - Diagnostics name the variable, never its value, and no directive value is logged. There is no `${VAR}` string interpolation.
327
+
328
+ With `_select`, selection happens first, and `_env` applies to the selected document:
329
+
330
+ ```yaml
331
+ _select:
332
+ env: TANGLE_ENV
333
+ cases:
334
+ prod:
335
+ token: {_env: TANGLE_PROD_TOKEN}
336
+ dev:
337
+ token: {_env: TANGLE_DEV_TOKEN, default: dev-token}
338
+ ```
339
+
340
+ Every `_env` directive — in every case and `default` branch, and in helper sections — is shape-checked before any variable is read, so a malformed directive fails identically in every environment. Only the directives in the selected document's config entries and `_defaults` are looked up; a dormant branch never requires its variables. `_select` itself is unchanged.
341
+
342
+ Precedence per field is **CLI > config > environment > default**: an explicit CLI value wins, then the config key (including a value read through `_env`), then an environment variable the command has opted that field into (see [`EnvField`](#shared-cli-helpers-and-logging)), then the default. As before, a CLI value equal to the option default is indistinguishable from an omitted one.
343
+
309
344
  ## API schema cache and dynamic commands
310
345
 
311
346
  Refresh the local schema cache for a live backend with:
@@ -535,8 +570,54 @@ def greeting_pipeline(who: In[str], cfg) -> Out[str]:
535
570
 
536
571
  `In[T]` parameters become runtime graph inputs. A single `-> Out[T]` return exposes one graph output; use `@pipeline(output_name=...)` to name that output. For multiple outputs, define a frozen dataclass subclass of `Outputs` with `Out[T]` fields and return an instance. A pipeline that accepts a `cfg` parameter reads `config.yaml` (or the path passed via `@pipeline(config="...")`) at compile time, with `--override key=value` values overlaid by the compile command.
537
572
 
573
+ ##### Environment-selected pipeline config (`_select`, `_env`)
574
+
575
+ A pipeline `config.yaml` — root or child, including one broadcast with `propagate_config=True` — is resolved by the same loader as `--config` files (see [`_select`](#environment-selected-configs-_select) and [`_env`](#environment-variable-values-_env)), with one shape rule: the selected document, and every case and `default` branch, must be a mapping. `_defaults`/`configs` have no special meaning in a pipeline config; they are ordinary keys.
576
+
577
+ ```yaml
578
+ _select:
579
+ env: TANGLE_ENV
580
+ cases:
581
+ prod: {dataset: prod_ds, batch_size: 500, token: {_env: PROD_TOKEN}}
582
+ dev: {dataset: dev_ds, batch_size: 10}
583
+ default: {dataset: local_ds, batch_size: 1}
584
+ ```
585
+
586
+ - Resolution happens first; overrides are layered on the result. Precedence is `--override` / `.override_config` / `propagate_config` broadcast > selected branch. An `.override_config` key must exist in the child's *selected* branch, and a broadcast carries resolved values, never `_select`/`_env` nodes. Each child resolves its own config.
587
+ - `_select` branch values keep their native YAML types (`batch_size` is an `int`). `_env` values are strings (a `default` is stringified) and are **not** YAML-coerced the way raw `--override` strings are: an environment value is opaque (`007` stays `"007"`, `no` stays `"no"`). Convert explicitly in pipeline code (`int(cfg.limit)`), or put typed values in `_select` branches.
588
+ - Every branch is structure-checked in every environment, including the `template_file:` rejection, and a dormant branch never requires its variables. An unset or unmatched selector without `default` fails closed without echoing the value.
589
+ - Compile identity follows the resolved values. For a config that uses `_select` or `_env`, the child sidecar name (`<child>-<hash8>.yaml`) includes a digest of the resolved config, so different selections that produce different values never share a sidecar and identical values keep the same name. Configs without `_select`/`_env` keep their existing names.
590
+
538
591
  Task IDs default from the left-hand variable name at the call site, converted to title case. If there is no simple left-hand variable, or if you want a stable explicit label, call `.named("Task Id")` before invoking the task. Use `.bind(...)` to pre-fill task arguments and `.with_annotations({...})` to add per-task annotations.
539
592
 
593
+ ##### Root pipeline annotations
594
+
595
+ `@pipeline(annotations={...})` writes the compiled pipeline's root `metadata.annotations` block. A caller that compiles programmatically can supply the block instead — typically from its own per-environment config file, so the values do not have to be hard-coded in source:
596
+
597
+ ```python
598
+ from tangle_cli.pipelines import compile_pipeline_file
599
+
600
+ compile_pipeline_file(
601
+ "pipeline.py",
602
+ "pipeline.yaml",
603
+ pipeline_annotations={"environment": "staging", "owner": "search-platform"},
604
+ )
605
+ ```
606
+
607
+ The same keyword exists on `tangle_cli.pipeline_compiler.compile_pipeline` and on `PipelineCompiler.compile_file`. There is no CLI flag: the source route already exists, and what the keyword adds is a programmatic/config route for the part of the block that varies by environment.
608
+
609
+ Semantics:
610
+
611
+ - **Per-key merge, caller wins.** `@pipeline(annotations={"author": "a", "version": "1.0"})` compiled with `pipeline_annotations={"version": "2.0", "environment": "staging"}` emits all three keys, with `version: "2.0"`. Source keys the caller does not mention are preserved, so invariants stay in source and only the varying subset is passed in.
612
+ - **Omitted or `{}` is a no-op**, byte for byte — an empty mapping is not a destructive clear of the source block.
613
+ - **Root only.** `subpipeline` children never inherit it, so child subgraph sidecar names, bytes, and component digests are unaffected. A child that wants annotations declares its own.
614
+ - **Descriptive only.** Root metadata is not read by the orchestrator, so it cannot influence placement, routing, scheduling, or run identity. Use pipeline-run annotations for anything execution-bearing.
615
+ - **`str -> str`, validated up front.** A non-mapping argument, a non-string key or value, an empty key, a `system/`-prefixed key (reserved by Tangle), or a template delimiter (`{{`, `{%`, `{#`) in a key or value raises `InvalidPipelineAnnotationsError` (a `CompileError`) before anything is imported or written. Annotations usually come from an untrusted config file, so the diagnostics name the key and the type and never echo a value. Values are baked into the compiled YAML and the stored pipeline definition: labels only, never secrets.
616
+
617
+ The rules live in one place, `tangle_cli.schema_validation`: `check_annotations(mapping, policy=..., error_cls=...)` applied under a named `AnnotationPolicy`. `CALLER_ANNOTATION_POLICY` is the strict input policy described above; `DOCUMENT_ANNOTATION_POLICY` is the lenient policy every pipeline document is validated against (scalar-or-null values, no key rules), matching the schema and hand-authored YAML. Only the caller-supplied input surface is strict: existing documents are accepted exactly as before, and the document check still runs on the merged result, so annotations reaching the output by any route are validated.
618
+
619
+ A distribution that reads these annotations from its own config file should call `check_annotations(mapping, policy=CALLER_ANNOTATION_POLICY, error_cls=...)` at config-parse time — passing its own error type and adding the config path and key to the message — so one user mistake produces one diagnostic instead of two competing ones. The compiler's own call is then the backstop for anything arriving by another route.
620
+
540
621
  ##### Conditional task execution
541
622
 
542
623
  Pipeline inputs used as conditions are ordinary `In[str]` values; there is no special conditional input annotation. Pass the value through the reserved task-call metadata keyword `is_enabled=`:
@@ -966,6 +1047,8 @@ Use these for generic downstream behavior such as alternate storage, extra annot
966
1047
 
967
1048
  `cli_options.py` centralizes shared Cyclopts annotations such as `BaseUrlOption`, `TokenOption`, `AuthHeaderOption`, `HeaderOption`, `ConfigOption`, and `LogTypeOption`. `cli_helpers.py` centralizes config loading, JSON printing, credential-isolation helpers, and the native-safe `LazyTangleApiClient` proxy. `logger.py` provides `ConsoleLogger`, `NullLogger`, `CaptureLogger`, `logger_for_log_type(...)`, and `run_with_logging(...)`.
968
1049
 
1050
+ `ArgsContainer.load(...)` field specs are tuples (see `ArgsContainer._resolve`). To give one field an environment tier, wrap its unchanged spec: `token=EnvField("TANGLE_PROD_TOKEN", (token, None))`. The field then resolves CLI > config > `os.environ["TANGLE_PROD_TOKEN"]` > default; the raw string (empty counts as set) goes through the spec's usual converter, and a conversion error names the variable without echoing its value. Nothing is mapped automatically: fields that are not wrapped never read the environment. `args.origin(name)` reports where each field came from — `cli`, `config`, `env:NAME`, or `default` — without the value.
1051
+
969
1052
  Use these helpers for new SDK commands so top-level imports remain native-free, `--config` behavior stays consistent, credentials from config do not accidentally mix with ambient environment auth, and progress logs stay off structured stdout.
970
1053
 
971
1054
  ## Development checks
@@ -57,7 +57,7 @@ API-backed commands commonly accept these options. Explicit CLI options win over
57
57
  | `--token`, `TANGLE_API_TOKEN` | Bearer token shorthand. |
58
58
  | `--auth-header`, `TANGLE_API_AUTH_HEADER`, `TANGLE_AUTH_HEADER` | Full `Authorization` value such as `Bearer ...` or `Basic ...`. |
59
59
  | `-H`, `--header`, `TANGLE_API_HEADERS` | Extra headers. Repeatable as CLI flags; env accepts a JSON object or newline-separated `Name: value` entries. |
60
- | `--config` | YAML/JSON defaults. Many commands accept a single object, a list of objects, or `_defaults` + `configs`, optionally wrapped in a top-level `_select` environment selector. |
60
+ | `--config` | YAML/JSON defaults. Many commands accept a single object, a list of objects, or `_defaults` + `configs`, optionally wrapped in a top-level `_select` environment selector. Values may be read from the environment with `{_env: NAME}`. |
61
61
  | `--log-type` | SDK progress logs: `console`, `none`, or `file`. Logs go to stderr or a temp log file so structured stdout stays parseable. |
62
62
  | `TANGLE_VERBOSE=1` | Redacted HTTP request/response diagnostics only. This is separate from normal progress logging. |
63
63
  | `--ca-bundle` | Global CLI flag: path to a PEM CA bundle used as the TLS trust store for every transport. Overrides `TANGLE_API_CA_BUNDLE`. Place before the subcommand. |
@@ -232,7 +232,7 @@ For generated `tangle api` commands, config keys use generated CLI parameter nam
232
232
 
233
233
  ### Environment-selected configs (`_select`)
234
234
 
235
- Any command that accepts `--config` can pick one of several config documents from an environment variable by making `_select` the top-level node:
235
+ Any command that accepts `--config`, and any Python pipeline `config.yaml` (see [Environment-selected pipeline config](#environment-selected-pipeline-config-_select-_env)), can pick one of several config documents from an environment variable by making `_select` the top-level node:
236
236
 
237
237
  ```yaml
238
238
  _shared: &shared
@@ -271,7 +271,7 @@ With `default`, an unset variable or a value matching no case resolves to that b
271
271
 
272
272
  Rules:
273
273
 
274
- - `_select` is the only newly reserved key, and it is an exact key name rather than a prefix. Configs without `_select` are unchanged byte-for-byte and semantically.
274
+ - `_select` is an exact reserved key name rather than a prefix (the only other reserved key is the [`_env`](#environment-variable-values-_env) value directive). Configs without `_select` are unchanged byte-for-byte and semantically.
275
275
  - At a `_select` node, only `_select` and other underscore-prefixed helper keys (YAML anchor holders such as `_shared`) may appear; ordinary sibling keys are rejected.
276
276
  - `_select` accepts only `env`, `cases`, and the optional `default`. There are no aliases: `else`, `fallback`, and `defaults` are rejected.
277
277
  - The selector shape, the `env` name, every `cases` key, and every case and `default` branch are validated before the environment is read, so a malformed selector fails identically in every environment. Branches are checked as complete config documents with the same rules the loader applies to a whole file, and nested selectors are validated recursively.
@@ -280,6 +280,41 @@ Rules:
280
280
  - Matching uses `os.environ[NAME]` exactly: case sensitive, with no trimming, case folding, or interpolation.
281
281
  - Fallback exists only where it is authored. Without `default`, an unset variable or an unmatched value is an error; there is never an implicit default or implicit production branch. The raw environment value is never echoed — diagnostics list only the configured case names.
282
282
 
283
+ ### Environment-variable values (`_env`)
284
+
285
+ Any config value — in `--config` files and in [Python pipeline configs](#environment-selected-pipeline-config-_select-_env) — can be read from an environment variable with the `_env` value directive, so a checked-in config can reference a secret or per-machine value without containing it:
286
+
287
+ ```yaml
288
+ base_url: https://api.prod
289
+ token: {_env: TANGLE_PROD_TOKEN}
290
+ header:
291
+ - {_env: GATEWAY_HEADER}
292
+ - "X-Team: search"
293
+ limit: {_env: RUN_LIMIT, default: 10}
294
+ ```
295
+
296
+ - The directive is exactly `{_env: NAME}` or `{_env: NAME, default: <scalar>}`. Any other key beside `_env` — including underscore-prefixed keys and aliases such as `fallback` — is rejected, and `NAME` follows the same rule as `_select.env` (`[A-Za-z_][A-Za-z0-9_]*`).
297
+ - It may appear anywhere a value may appear: under a config key, in `_defaults`, in `configs` entries, and inside nested maps and lists. A document or config-entry mapping is never itself a directive, so an `_env:` helper/anchor key at the top level of a config object is unaffected.
298
+ - A missing variable without `default` fails closed with the variable name, the key path (for example `configs[1].token`), and the config file. An empty string counts as set and is used as-is.
299
+ - The resolved value is always a string, and a `default` is stringified the same way: numbers and booleans use their JSON spelling (`10`, `1.5`, `true`), dates use ISO format, and `null`/maps/lists are rejected (quote `''` for an empty default). A field therefore has one type whether or not the variable is set, and the command's usual conversion (JSON fields, repeatable options, enums, typed converters) applies downstream in both cases.
300
+ - Diagnostics name the variable, never its value, and no directive value is logged. There is no `${VAR}` string interpolation.
301
+
302
+ With `_select`, selection happens first, and `_env` applies to the selected document:
303
+
304
+ ```yaml
305
+ _select:
306
+ env: TANGLE_ENV
307
+ cases:
308
+ prod:
309
+ token: {_env: TANGLE_PROD_TOKEN}
310
+ dev:
311
+ token: {_env: TANGLE_DEV_TOKEN, default: dev-token}
312
+ ```
313
+
314
+ Every `_env` directive — in every case and `default` branch, and in helper sections — is shape-checked before any variable is read, so a malformed directive fails identically in every environment. Only the directives in the selected document's config entries and `_defaults` are looked up; a dormant branch never requires its variables. `_select` itself is unchanged.
315
+
316
+ Precedence per field is **CLI > config > environment > default**: an explicit CLI value wins, then the config key (including a value read through `_env`), then an environment variable the command has opted that field into (see [`EnvField`](#shared-cli-helpers-and-logging)), then the default. As before, a CLI value equal to the option default is indistinguishable from an omitted one.
317
+
283
318
  ## API schema cache and dynamic commands
284
319
 
285
320
  Refresh the local schema cache for a live backend with:
@@ -509,8 +544,54 @@ def greeting_pipeline(who: In[str], cfg) -> Out[str]:
509
544
 
510
545
  `In[T]` parameters become runtime graph inputs. A single `-> Out[T]` return exposes one graph output; use `@pipeline(output_name=...)` to name that output. For multiple outputs, define a frozen dataclass subclass of `Outputs` with `Out[T]` fields and return an instance. A pipeline that accepts a `cfg` parameter reads `config.yaml` (or the path passed via `@pipeline(config="...")`) at compile time, with `--override key=value` values overlaid by the compile command.
511
546
 
547
+ ##### Environment-selected pipeline config (`_select`, `_env`)
548
+
549
+ A pipeline `config.yaml` — root or child, including one broadcast with `propagate_config=True` — is resolved by the same loader as `--config` files (see [`_select`](#environment-selected-configs-_select) and [`_env`](#environment-variable-values-_env)), with one shape rule: the selected document, and every case and `default` branch, must be a mapping. `_defaults`/`configs` have no special meaning in a pipeline config; they are ordinary keys.
550
+
551
+ ```yaml
552
+ _select:
553
+ env: TANGLE_ENV
554
+ cases:
555
+ prod: {dataset: prod_ds, batch_size: 500, token: {_env: PROD_TOKEN}}
556
+ dev: {dataset: dev_ds, batch_size: 10}
557
+ default: {dataset: local_ds, batch_size: 1}
558
+ ```
559
+
560
+ - Resolution happens first; overrides are layered on the result. Precedence is `--override` / `.override_config` / `propagate_config` broadcast > selected branch. An `.override_config` key must exist in the child's *selected* branch, and a broadcast carries resolved values, never `_select`/`_env` nodes. Each child resolves its own config.
561
+ - `_select` branch values keep their native YAML types (`batch_size` is an `int`). `_env` values are strings (a `default` is stringified) and are **not** YAML-coerced the way raw `--override` strings are: an environment value is opaque (`007` stays `"007"`, `no` stays `"no"`). Convert explicitly in pipeline code (`int(cfg.limit)`), or put typed values in `_select` branches.
562
+ - Every branch is structure-checked in every environment, including the `template_file:` rejection, and a dormant branch never requires its variables. An unset or unmatched selector without `default` fails closed without echoing the value.
563
+ - Compile identity follows the resolved values. For a config that uses `_select` or `_env`, the child sidecar name (`<child>-<hash8>.yaml`) includes a digest of the resolved config, so different selections that produce different values never share a sidecar and identical values keep the same name. Configs without `_select`/`_env` keep their existing names.
564
+
512
565
  Task IDs default from the left-hand variable name at the call site, converted to title case. If there is no simple left-hand variable, or if you want a stable explicit label, call `.named("Task Id")` before invoking the task. Use `.bind(...)` to pre-fill task arguments and `.with_annotations({...})` to add per-task annotations.
513
566
 
567
+ ##### Root pipeline annotations
568
+
569
+ `@pipeline(annotations={...})` writes the compiled pipeline's root `metadata.annotations` block. A caller that compiles programmatically can supply the block instead — typically from its own per-environment config file, so the values do not have to be hard-coded in source:
570
+
571
+ ```python
572
+ from tangle_cli.pipelines import compile_pipeline_file
573
+
574
+ compile_pipeline_file(
575
+ "pipeline.py",
576
+ "pipeline.yaml",
577
+ pipeline_annotations={"environment": "staging", "owner": "search-platform"},
578
+ )
579
+ ```
580
+
581
+ The same keyword exists on `tangle_cli.pipeline_compiler.compile_pipeline` and on `PipelineCompiler.compile_file`. There is no CLI flag: the source route already exists, and what the keyword adds is a programmatic/config route for the part of the block that varies by environment.
582
+
583
+ Semantics:
584
+
585
+ - **Per-key merge, caller wins.** `@pipeline(annotations={"author": "a", "version": "1.0"})` compiled with `pipeline_annotations={"version": "2.0", "environment": "staging"}` emits all three keys, with `version: "2.0"`. Source keys the caller does not mention are preserved, so invariants stay in source and only the varying subset is passed in.
586
+ - **Omitted or `{}` is a no-op**, byte for byte — an empty mapping is not a destructive clear of the source block.
587
+ - **Root only.** `subpipeline` children never inherit it, so child subgraph sidecar names, bytes, and component digests are unaffected. A child that wants annotations declares its own.
588
+ - **Descriptive only.** Root metadata is not read by the orchestrator, so it cannot influence placement, routing, scheduling, or run identity. Use pipeline-run annotations for anything execution-bearing.
589
+ - **`str -> str`, validated up front.** A non-mapping argument, a non-string key or value, an empty key, a `system/`-prefixed key (reserved by Tangle), or a template delimiter (`{{`, `{%`, `{#`) in a key or value raises `InvalidPipelineAnnotationsError` (a `CompileError`) before anything is imported or written. Annotations usually come from an untrusted config file, so the diagnostics name the key and the type and never echo a value. Values are baked into the compiled YAML and the stored pipeline definition: labels only, never secrets.
590
+
591
+ The rules live in one place, `tangle_cli.schema_validation`: `check_annotations(mapping, policy=..., error_cls=...)` applied under a named `AnnotationPolicy`. `CALLER_ANNOTATION_POLICY` is the strict input policy described above; `DOCUMENT_ANNOTATION_POLICY` is the lenient policy every pipeline document is validated against (scalar-or-null values, no key rules), matching the schema and hand-authored YAML. Only the caller-supplied input surface is strict: existing documents are accepted exactly as before, and the document check still runs on the merged result, so annotations reaching the output by any route are validated.
592
+
593
+ A distribution that reads these annotations from its own config file should call `check_annotations(mapping, policy=CALLER_ANNOTATION_POLICY, error_cls=...)` at config-parse time — passing its own error type and adding the config path and key to the message — so one user mistake produces one diagnostic instead of two competing ones. The compiler's own call is then the backstop for anything arriving by another route.
594
+
514
595
  ##### Conditional task execution
515
596
 
516
597
  Pipeline inputs used as conditions are ordinary `In[str]` values; there is no special conditional input annotation. Pass the value through the reserved task-call metadata keyword `is_enabled=`:
@@ -940,6 +1021,8 @@ Use these for generic downstream behavior such as alternate storage, extra annot
940
1021
 
941
1022
  `cli_options.py` centralizes shared Cyclopts annotations such as `BaseUrlOption`, `TokenOption`, `AuthHeaderOption`, `HeaderOption`, `ConfigOption`, and `LogTypeOption`. `cli_helpers.py` centralizes config loading, JSON printing, credential-isolation helpers, and the native-safe `LazyTangleApiClient` proxy. `logger.py` provides `ConsoleLogger`, `NullLogger`, `CaptureLogger`, `logger_for_log_type(...)`, and `run_with_logging(...)`.
942
1023
 
1024
+ `ArgsContainer.load(...)` field specs are tuples (see `ArgsContainer._resolve`). To give one field an environment tier, wrap its unchanged spec: `token=EnvField("TANGLE_PROD_TOKEN", (token, None))`. The field then resolves CLI > config > `os.environ["TANGLE_PROD_TOKEN"]` > default; the raw string (empty counts as set) goes through the spec's usual converter, and a conversion error names the variable without echoing its value. Nothing is mapped automatically: fields that are not wrapped never read the environment. `args.origin(name)` reports where each field came from — `cli`, `config`, `env:NAME`, or `default` — without the value.
1025
+
943
1026
  Use these helpers for new SDK commands so top-level imports remain native-free, `--config` behavior stays consistent, credentials from config do not accidentally mix with ambient environment auth, and progress logs stay off structured stdout.
944
1027
 
945
1028
  ## Development checks
@@ -14,6 +14,6 @@ from tangle_cli.dynamic_discovery_client import TangleDynamicDiscoveryClient
14
14
  try:
15
15
  __version__ = metadata_version("tangle-cli")
16
16
  except PackageNotFoundError:
17
- __version__ = "0.1.18"
17
+ __version__ = "0.1.20"
18
18
 
19
19
  __all__ = ["TangleDynamicDiscoveryClient", "__version__"]