repro-lambda 0.4.2__tar.gz → 0.5.1__tar.gz

{repro_lambda-0.4.2 → repro_lambda-0.5.1}/.github/workflows/build.yml

@@ -25,6 +25,14 @@ on:
         required: false
         default: ""
         description: S3 bucket name for prod Lambda artifacts (master push only).
+    secrets:
+      sources_token:
+        required: false
+        description: >-
+          Token with read access to the private release repos referenced by any
+          github_release [[lambda.source]]. Exported as REPRO_LAMBDA_SOURCES_TOKEN for the
+          build. Omit when every source is a public https URL. Generic by design - the
+          reusable workflow never names a consumer repo.
 
 jobs:
   detect-arches:
@@ -64,6 +72,16 @@ jobs:
       - uses: actions/checkout@v7
       - uses: astral-sh/setup-uv@v8.2.0
 
+      # Content-addressed source cache, scoped per-arch (a source's sha256 is its key, so a
+      # re-pin changes the manifest hash -> fresh fetch; restore-keys reuses the prior cache).
+      - name: Cache fetched sources
+        uses: actions/cache@v4
+        with:
+          path: builds/.sources-cache
+          key: repro-sources-${{ matrix.arch }}-${{ hashFiles(inputs.manifest_path) }}
+          restore-keys: |
+            repro-sources-${{ matrix.arch }}-
+
       - name: Configure AWS credentials (dev)
         uses: aws-actions/configure-aws-credentials@v6
         with:
@@ -73,6 +91,7 @@ jobs:
       - name: Build (dev bucket)
         env:
           REPRO_LAMBDA_BUCKET: ${{ inputs.dev_bucket }}
+          REPRO_LAMBDA_SOURCES_TOKEN: ${{ secrets.sources_token }}
         run: uvx --from repro-lambda repro-lambda build --manifest "${{ inputs.manifest_path }}" --arch "${{ matrix.arch }}"
 
       - name: Verify reproducible (PR only)
@@ -90,6 +109,7 @@ jobs:
         if: github.ref == 'refs/heads/master' && inputs.aws_prod_role_arn != ''
         env:
           REPRO_LAMBDA_BUCKET: ${{ inputs.prod_bucket }}
+          REPRO_LAMBDA_SOURCES_TOKEN: ${{ secrets.sources_token }}
         run: uvx --from repro-lambda repro-lambda build --manifest "${{ inputs.manifest_path }}" --arch "${{ matrix.arch }}"
 
       - name: Commit catalog drift (master only, dev bot)

{repro_lambda-0.4.2 → repro_lambda-0.5.1}/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## v0.5.1 - 2026-06-21
+
+### Fixed
+- Send a `User-Agent` header on every source fetch. The GitHub REST API rejects requests with no User-Agent (HTTP 403), which broke `github_release` source resolution (the asset lookup against `api.github.com`). The header is now `repro-lambda/<version>` on all requests (harmless for plain `https` sources, required for the API). Does not affect any content hash (request headers are not part of artifact identity).
+
+## v0.5.0 - 2026-06-21
+
+### Added
+- Declarative sources DSL: `[[lambda.source]]` fetches a pinned external artifact into the package before the container build, replacing consumer-side download scripts. Two source types - `github_release` (`repo`/`tag`/`asset`, resolved via the GitHub API) and `https` (a direct `url`). Each source is PINNED: `sha256` is verified before extraction; `extract` is `zip`/`tar.gz`/`none`; an optional `member` extracts a single file or a (versioned) directory subtree to `dest` (package-root-relative); `executable` sets +x. Source names are required and unique per lambda; dest collisions (including tree overlaps and overlaps with the staged source) are refused.
+- `version_from` (single-level): a source can derive its `version` at lock time from an asdf-style `key value` line in a referenced source's file (read relative to that source's member-stripped tree). `{version}` is substituted into `url`/`tag`/`asset`/`member`. `version_from` is a lock input and never participates in the content hash.
+- `lock --sources` re-resolves `version_from`, re-downloads each source, recomputes its sha256, and rewrites `lambdas.toml` in place with tomlkit (comments preserved, atomic). It is idempotent: a run that changes nothing leaves the file byte-for-byte unchanged (no spurious PR). `lock` keeps regenerating requirements locks too; use `--no-requirements` / `--no-sources` to scope it.
+- The reusable `build.yml` gains a generic `sources_token` secret (exported as `REPRO_LAMBDA_SOURCES_TOKEN`, used only for `github_release` API calls) and an arch-scoped `actions/cache` for the content-addressed source cache.
+
+### Security
+- SSRF-hardened fetcher: HTTPS-only with a manual redirect loop; each hop resolves the hostname, refuses any non-global-unicast / reserved / private / loopback / link-local / IPv4-mapped address, and connects to that pinned IP while verifying the TLS cert against the original hostname (no second DNS lookup a rebind could poison). `Authorization` is stripped on any cross-host redirect (e.g. the GitHub API -> asset host), so a private token never leaks. Download size is capped and log URLs are redacted (userinfo + query stripped).
+- Hardened extraction: sha256 is verified before any archive is opened; entries with absolute / `..` / symlink / hardlink / device / fifo paths are rejected; total bytes, entry count, and per-entry bytes are bounded (decompression-bomb limits, with streaming tar reads); files are written to a temp tree then promoted with normalized mtime + perms. The sha256-keyed cache is re-verified on every use (anti cache-poisoning).
+- Content hashing folds the RESOLVED source metadata (not the bytes), so the artifact key is computable offline and a `member`/`extract`/`dest`/`sha256`/version change re-keys, while a re-fetch alone does not.
+
 ## v0.4.2 - 2026-06-21
 
 ### Added

{repro_lambda-0.4.2 → repro_lambda-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: repro-lambda
-Version: 0.4.2
+Version: 0.5.1
 Summary: Build reproducible AWS Lambda packages outside Terraform, optimized for terraform-aws-lambda by serverless.tf.
 Project-URL: Homepage, https://github.com/antonbabenko/repro-lambda
 Project-URL: Repository, https://github.com/antonbabenko/repro-lambda
@@ -20,6 +20,7 @@ Classifier: Topic :: Software Development :: Build Tools
 Requires-Python: >=3.11
 Requires-Dist: boto3>=1.34
 Requires-Dist: repro-zipfile>=0.3.1
+Requires-Dist: tomlkit>=0.13
 Requires-Dist: typer>=0.12
 Provides-Extra: dev
 Requires-Dist: moto[s3]>=5; extra == 'dev'

{repro_lambda-0.4.2 → repro_lambda-0.5.1}/SETUP.md

@@ -330,6 +330,82 @@ The resolved per-lambda builder (base-image digest + include/exclude lists +
 builder version) folds into the content hash, so changing an override re-keys
 that lambda's artifact while leaving the others untouched.
 
+## Declarative sources — `[[lambda.source]]`
+
+A lambda can bundle pinned external artifacts (a release tarball, a vendored CLI
+binary) fetched at build time, instead of a hand-rolled download script. Each
+`[[lambda.source]]` is fully pinned and fetched + verified + extracted into the
+package before the container build:
+
+```toml
+[[lambda]]
+logical_name      = "app"
+source_dir        = "src/app"
+requirements_lock = "src/app/requirements.${arch}.lock"
+runtime           = "python3.13"
+arch              = "arm64"
+handler           = "app.lambda_handler"
+
+# A private GitHub release tarball -> staged at package path "vendor".
+[[lambda.source]]
+name    = "vendor"
+type    = "github_release"
+repo    = "owner/vendor-tool"
+tag     = "vendor-v{version}"
+asset   = "vendor-{version}.tar.gz"
+sha256  = "<64-hex; written by `repro-lambda lock`>"
+extract = "tar.gz"
+member  = "vendor-{version}"   # map the versioned top dir to dest
+dest    = "vendor"
+version = "1.4.0"              # bump this one line; lock re-pins everything
+
+# A public binary whose version is derived from the vendor release's .tool-versions.
+[[lambda.source]]
+name    = "terraform"
+type    = "https"
+url     = "https://releases.hashicorp.com/terraform/{version}/terraform_{version}_linux_arm64.zip"
+sha256  = "<64-hex; written by lock>"
+extract = "zip"
+member  = "terraform"
+dest    = "bin/terraform"
+executable = true
+version = "1.9.0"
+[lambda.source.version_from]
+source = "vendor"          # read from the vendor source's extracted tree
+file   = ".tool-versions"  # relative to its member-stripped root
+key    = "terraform"
+```
+
+- **Pinning.** `sha256` is verified before the archive is opened. `extract` is
+  `zip` / `tar.gz` / `none`. `member` extracts one file or a directory subtree to
+  `dest`; omit it to extract the whole archive under `dest`. Source names are
+  unique per lambda and dests may not overlap each other or the staged source.
+- **`version_from`** (single-level) derives a source's `version` from an asdf-style
+  `key value` line in another source's file, so bumping the root `version`
+  cascades to dependents. It is a lock input - it never affects the artifact hash.
+- **`repro-lambda lock`** re-resolves `version_from`, re-downloads, recomputes each
+  `sha256`, and rewrites this file (comment-preserving, atomic, idempotent). Run it
+  after bumping a `version`. Pass `REPRO_LAMBDA_SOURCES_TOKEN` for private
+  `github_release` sources.
+- **Security.** Fetches are HTTPS-only with an SSRF guard (no private/loopback/
+  link-local/metadata IPs), strip `Authorization` on cross-host redirects, verify
+  sha256 before extraction, and reject path-traversal / link / device entries with
+  decompression-bomb bounds. See `src/repro_lambda/sources.py`.
+
+In CI, pass the token to the reusable `build.yml` as the `sources_token` secret:
+
+```yaml
+jobs:
+  build:
+    uses: antonbabenko/repro-lambda/.github/workflows/build.yml@v0
+    with:
+      manifest_path: lambdas.toml
+      aws_dev_role_arn: arn:aws:iam::<account>:role/<dev-builder-role>
+      dev_bucket: <env>-my-lambda-artifacts
+    secrets:
+      sources_token: ${{ secrets.MY_RELEASE_TOKEN }}
+```
+
 ## Terraform consumer — `s3_existing_package`
 
 In the Terraform that creates your Lambda function, point at the artifact in

{repro_lambda-0.4.2 → repro_lambda-0.5.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "repro-lambda"
-version = "0.4.2"
+version = "0.5.1"
 description = "Build reproducible AWS Lambda packages outside Terraform, optimized for terraform-aws-lambda by serverless.tf."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -21,7 +21,7 @@ dependencies = [
   "typer>=0.12",
   "repro-zipfile>=0.3.1",
   "boto3>=1.34",
-  
+  "tomlkit>=0.13",
 ]
 
 [project.optional-dependencies]

{repro_lambda-0.4.2 → repro_lambda-0.5.1}/src/repro_lambda/__init__.py

@@ -1,3 +1,3 @@
 """repro-lambda — reproducible AWS Lambda packaging outside Terraform."""
 
-__version__ = "0.4.2"
+__version__ = "0.5.1"

{repro_lambda-0.4.2 → repro_lambda-0.5.1}/src/repro_lambda/build.py

@@ -88,6 +88,7 @@ def compute_sha_for(
             payload_exec=[(ef.dest, ef.executable) for ef in spec.extra_files],
             include_patterns=builder.include_patterns,
             exclude_patterns=builder.exclude_patterns,
+            sources=spec.sources,
         )
 
 
@@ -100,6 +101,8 @@ def build_one(
     catalog: Catalog,
     source_commit: str,
     dry_run: bool = False,
+    sources_token: str | None = None,
+    sources_cache: Path | None = None,
 ) -> BuildOutcome:
     """Build one lambda end-to-end. Returns BuildOutcome with sha + cache verdict."""
     builder = resolve_builder(builder, spec)
@@ -132,6 +135,7 @@ def build_one(
             payload_exec=[(ef.dest, ef.executable) for ef in spec.extra_files],
             include_patterns=builder.include_patterns,
             exclude_patterns=builder.exclude_patterns,
+            sources=spec.sources,
         )
         bucket_key = f"lambdas/{spec.logical_name}/{sha}.zip"
 
@@ -143,6 +147,18 @@ def build_one(
             _record(catalog, spec, sha, source_commit, builder)
             return BuildOutcome(BuildResult.CACHE_HIT, sha, bucket_key)
 
+        # Cache miss only: fetch + verify + extract declarative sources into the staged
+        # tree (post-filter; collides loudly with already-staged source/payload files).
+        if spec.sources:
+            from repro_lambda.sources import fetch_sources
+
+            fetch_sources(
+                sources=spec.sources,
+                dest_root=stage_dir / "source",
+                cache_dir=sources_cache or (repo_root / "builds" / ".sources-cache"),
+                github_token=sources_token,
+            )
+
         out_zip = stage_dir / "lambda.zip"
         if spec.package_manager == "pip":
             build_python_lambda(

{repro_lambda-0.4.2 → repro_lambda-0.5.1}/src/repro_lambda/cli.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import os
 import subprocess
 from pathlib import Path
 from typing import Annotated
@@ -128,6 +129,8 @@ def build(
     except subprocess.CalledProcessError:
         source_commit = "unknown"
 
+    sources_token = os.environ.get("REPRO_LAMBDA_SOURCES_TOKEN") or None
+
     summary = []
     for spec in selected:
         outcome = build_one(
@@ -138,6 +141,7 @@ def build(
             catalog=catalog,
             source_commit=source_commit,
             dry_run=dry_run,
+            sources_token=sources_token,
         )
         summary.append(
             {
@@ -261,13 +265,39 @@ def promote(
 @app.command()
 def lock(
     manifest: Annotated[Path, typer.Option("--manifest", "-m")] = Path("lambdas.toml"),
+    requirements: Annotated[
+        bool,
+        typer.Option(
+            "--requirements/--no-requirements",
+            help="Regenerate per-arch requirements.${arch}.lock via uv pip compile.",
+        ),
+    ] = True,
+    sources: Annotated[
+        bool,
+        typer.Option(
+            "--sources/--no-sources",
+            help="Re-resolve version_from + re-pin [[lambda.source]] sha256 in the manifest.",
+        ),
+    ] = True,
 ) -> None:
-    """Regenerate per-arch requirements.${arch}.lock files via `uv pip compile`."""
+    """Lock pinned inputs: per-arch requirements locks and/or declarative source pins."""
     from repro_lambda.manifest import load_manifest
 
     parsed = load_manifest(manifest)
     repo_root = manifest.parent.resolve()
 
+    if requirements:
+        _lock_requirements(parsed, repo_root)
+
+    if sources:
+        from repro_lambda.source_locker import lock_sources
+
+        token = os.environ.get("REPRO_LAMBDA_SOURCES_TOKEN") or None
+        changed = lock_sources(manifest, token)
+        typer.echo(f"lock sources: {'updated ' + str(manifest) if changed else 'no changes'}")
+
+
+def _lock_requirements(parsed, repo_root: Path) -> None:
     for spec in parsed.lambdas:
         if spec.package_manager == "npm":
             typer.echo(

{repro_lambda-0.4.2 → repro_lambda-0.5.1}/src/repro_lambda/hasher.py

@@ -5,7 +5,7 @@ from __future__ import annotations
 import hashlib
 from pathlib import Path
 
-from repro_lambda.manifest import LambdaSpec
+from repro_lambda.manifest import LambdaSpec, Source
 
 
 def _sha256_file(path: Path) -> str:
@@ -27,6 +27,7 @@ def compute_content_hash(
     payload_exec: list[tuple[str, bool]] | None = None,
     include_patterns: list[str] | None = None,
     exclude_patterns: list[str] | None = None,
+    sources: tuple[Source, ...] | None = None,
 ) -> str:
     """
     sha256 over: sorted (relative-path, sha256(content)) tuples for the staged tree
@@ -46,6 +47,12 @@ def compute_content_hash(
     not re-key) and only when not None, so an explicit empty list (replace-with-empty)
     hashes differently from an unset/None filter. Callers passing None omit the
     section entirely, preserving hashes for code paths that do not resolve a builder.
+
+    sources are the declarative [[lambda.source]] entries. Their RESOLVED metadata
+    (the {version}-substituted url/tag/asset/member plus type/repo/extract/dest/
+    executable/sha256), sorted by name, folds in - NOT the downloaded bytes, so the
+    hash stays download-free. version_from is a lock input and never folded; a
+    member/extract/dest/sha256 change re-keys. Omitted entirely when there are none.
     """
     h = hashlib.sha256()
 
@@ -102,4 +109,22 @@ def compute_content_hash(
         for dest, executable in sorted(payload_exec):
             h.update(f"{dest}={int(executable)}\n".encode())
 
+    # Declarative sources: fold resolved metadata only (no bytes), sorted by name.
+    # version_from is intentionally excluded (it is a lock input, not artifact identity).
+    if sources:
+        h.update(b"---sources---\n")
+        for s in sorted(sources, key=lambda s: s.name):
+            h.update(f"name={s.name}\n".encode())
+            h.update(f"type={s.type}\n".encode())
+            h.update(f"repo={s.repo}\n".encode())
+            h.update(f"url={s.resolved_url}\n".encode())
+            h.update(f"tag={s.resolved_tag}\n".encode())
+            h.update(f"asset={s.resolved_asset}\n".encode())
+            h.update(f"member={s.resolved_member or ''}\n".encode())
+            h.update(f"extract={s.extract}\n".encode())
+            h.update(f"dest={s.dest}\n".encode())
+            h.update(f"executable={int(s.executable)}\n".encode())
+            h.update(f"sha256={s.sha256}\n".encode())
+            h.update(b"--\n")
+
     return h.hexdigest()