@fishawack/lab-env 5.4.0 → 5.5.0-beta.1

package/CHANGELOG.md

@@ -1,5 +1,25 @@
 ## Changelog
 
+### 5.5.0-beta.1 (2026-04-13)
+
+#### Features
+
+* content logic from now moved to lab-env in backwards compatible way ([2d836ce](https://bitbucket.org/fishawackdigital/lab-env/commits/2d836cec5bc9d429b7fdc544c5fc1e8c3b749124))
+* extract snyk scan into standalone fw scan command ([abba628](https://bitbucket.org/fishawackdigital/lab-env/commits/abba628183fa94da5a21a14eb014c13a23036f84))
+* **python:** added healthcheck ([ca6e323](https://bitbucket.org/fishawackdigital/lab-env/commits/ca6e323487aaae201eabe34e5208845585d182d4))
+* **workspace:** support adding and removing projects from existing workspaces ([d70bf41](https://bitbucket.org/fishawackdigital/lab-env/commits/d70bf4130dffe412a15e8d8c6c3e6777daac65d3))
+
+#### Bug Fixes
+
+* allow blob as default-src on csp ([3cea3d4](https://bitbucket.org/fishawackdigital/lab-env/commits/3cea3d4bbd8bcd0ffdb6293967e4248ed542e8cf))
+* csps now include worker-src with blob and self support ([bbe35b8](https://bitbucket.org/fishawackdigital/lab-env/commits/bbe35b807153d1af53bad0872cc6461b8c39e92f))
+* scans now all run before errors thrown and always tackle lock files ([f212df0](https://bitbucket.org/fishawackdigital/lab-env/commits/f212df0cc1e6d5373c6bffcbde7e4e1ac2784db6))
+* split snyk scans per platform with project-name flag ([7d89539](https://bitbucket.org/fishawackdigital/lab-env/commits/7d895395359e6f9d594e264f9b0bcf2e5a59c3bd))
+
+#### Build Updates
+
+* added python as boilerplate option ([8695c48](https://bitbucket.org/fishawackdigital/lab-env/commits/8695c48a763f623fe852b029663d8f2d958ba383))
+
 ### 5.4.0 (2026-03-12)
 
 #### Features

package/_Ai/python-3.md

@@ -1,42 +1,45 @@
 ---
-applyTo: "src/**/*"
+applyTo: "**/*.py"
 ---
 
-# GitHub Copilot Instructions for Python Project
+# Python Project Instructions
 
-## Project Overview
+## Package Management
 
-This is a Python application that runs entirely in the host machine. All Python and pip operations must be executed within the appropriate environment using `venv`, `pip` and `python` for the python executable.
+Uses **uv** (not pip). Dependencies defined in `pyproject.toml` with `uv.lock`. No `requirements.txt`.
 
-## Project Management
+All commands run through the `fw` container orchestration manager:
 
-- The codebase follows a standard Python project structure.
-- Virtual environments are used to manage dependencies.
-- Always run python and pip from the virtual environment directly (`./venv/bin/python` and `./venv/bin/pip`)
-- The project runs on Python 3.13 or higher.
-- The project must always use the `python-dotenv` package to load environment variables from a `.env` file.
-- All dependencies are listed in the `requirements.txt` file.
-- The project uses FastAPI and Uvicorn for the web server. Upon running the project, it will be accessible at `http://localhost:3000` and it'll keep running as long as the process is active. For easier development, pipe the output to a log file.
-- The project must always try to stay stateless and handle requests with in-memory processing whenever possible.
-- The project must always fail gracefully and return appropriate error messages.
+```bash
+fw uv add <dep>          # Add a dependency
+fw uv add --dev <dep>    # Add a dev dependency
+fw uv sync               # Install/sync all dependencies
+fw uv run <command>      # Run a command in the project environment
+```
 
-## Common Command Patterns
+## Stack
 
-### Run Project
+- Python >=3.13 with FastAPI + Uvicorn
+- `.env` loaded via `python-dotenv`
+- Stateless request handling; fail gracefully with appropriate error responses
 
-```bash
-# Update dependencies
-pip install -r ./src/requirements.txt
+## Project Structure
 
-# Add new package
-pip install package/name
+- Entry point: `src/main.py`
+- Routes: `src/routes/`
+- Tests: `tests/` (run with `fw uv run pytest`)
+- Dev dependencies in `[dependency-groups.dev]` of `pyproject.toml`
 
-# Run project
-python ./src/main.py
-```
+## Tests
+
+You can run tests via
+
+`fw test`
+
+or an individual test file with
+
+`fw uv run pytest -o log_cli=true tests/integration/test_middleware.py`
 
-## Project Structure Context
+or a single test within that file with
 
-- **Main application**: The main entry point is `./src/main.py`
-- **Routes**: API routes are defined in `./src/routes/*.py`
-- **Connectors**: LLM Connectors are defined in `./src/connectors/*.py`
+`fw uv run pytest -o log_cli=true tests/integration/test_middleware.py -k 'test_001'`

package/_Ai/workspace-@.md

@@ -9,3 +9,7 @@ Backends are typically named with the pattern \*-api.
 Frontends are typically named with the pattern \*-fe. or with no suffix.
 
 Ensure you cd into the correct directory before running any terminal commands.
+
+## Commits
+
+When a feature is complete always suggest the conventional commit that summarizes the changes.

package/_Test/_fixtures/content/config.json

@@ -0,0 +1,36 @@
+{
+    "attributes": {
+        "src": "_Build",
+        "targets": {
+            "aws-s3": {
+                "content": [{ "aws-s3": "fishawack", "location": "core-test-suite-content" }]
+            },
+            "aws-s3-nested": {
+                "content": [{ "aws-s3": "fishawack", "location": "core-test-suite-content/nested" }]
+            },
+            "aws-s3-key": {
+                "content": [{ "aws-s3": "fishawack", "location": "core-test-suite-content", "key": "custom-key" }]
+            },
+            "aws-s3-sync": {
+                "content": [{ "aws-s3": "fishawack", "location": "core-test-suite-content", "key": "custom-key", "sync": true }]
+            },
+            "contentful": {
+                "content": [{ "url": "https://cdn.contentful.com/", "api": "spaces/hnifelsg7djt/environments/master/entries?access_token=-_qKPGeSi_ksTsXJd9o9KZdl-jdxZTDd_E9gjLS4QHY&content_type=", "type": "contentful", "bundle": true, "find": "\/\/images.ctfassets.net\/(.*?)\/(.*?)\/(.*?)\/", "endpoints": ["layout", "layoutCopy", "lessonImage"] }]
+            },
+            "empty": { "content": [] },
+            "handover": {},
+            "ftp-legacy": {
+                "content": [{ "ftp": "ftp-fishawack.egnyte.com", "location": "Shared/FW/Knutsford/Digital/Auto-Content/test-shared/" }]
+            },
+            "single": {
+                "content": [{ "aws-s3": "fishawack", "location": "core-test-suite-content" }]
+            },
+            "double": {
+                "content": [
+                    { "aws-s3": "fishawack", "location": "core-test-suite-content" },
+                    { "aws-s3": "fishawack", "location": "core-test-suite-content", "key": "custom-key" }
+                ]
+            }
+        }
+    }
+}

package/_Test/_fixtures/content/content-0/blah.txt

@@ -0,0 +1 @@
+blah

package/_Test/content.js

@@ -0,0 +1,517 @@
+"use strict";
+
+const expect = require("chai").expect;
+const path = require("path");
+const fs = require("fs-extra");
+const glob = require("glob");
+const { isEqual } = require("lodash");
+
+const {
+    pullS3,
+    parseBucketAndPrefix,
+    getLocalFiles,
+    createClient,
+} = require("../commands/helpers/content-pull.js");
+
+const {
+    pullRequests,
+    download,
+    rewrite,
+    urlJoin,
+} = require("../commands/helpers/content-request.js");
+
+const fixtureDir = path.join(__dirname, "_fixtures", "content");
+const outputDir = path.join(fixtureDir, "_Build", "content");
+const snapshotPath = path.join(fixtureDir, ".tmp", "content.json");
+
+function loadFixtureConfig(branch) {
+    const config = fs.readJSONSync(path.join(fixtureDir, "config.json"));
+    const branchConfig = config.attributes.targets[branch] || {};
+
+    return {
+        ...config.attributes,
+        ...branchConfig,
+    };
+}
+
+describe("content", () => {
+    describe("parseBucketAndPrefix", () => {
+        it("Should parse bucket only when no slash", () => {
+            const result = parseBucketAndPrefix("my-bucket");
+            expect(result.Bucket).to.equal("my-bucket");
+            expect(result.Prefix).to.equal("");
+        });
+
+        it("Should parse bucket and prefix when slash present", () => {
+            const result = parseBucketAndPrefix("my-bucket/some/prefix");
+            expect(result.Bucket).to.equal("my-bucket");
+            expect(result.Prefix).to.equal("some/prefix");
+        });
+
+        it("Should handle single level prefix", () => {
+            const result = parseBucketAndPrefix("my-bucket/folder");
+            expect(result.Bucket).to.equal("my-bucket");
+            expect(result.Prefix).to.equal("folder");
+        });
+    });
+
+    describe("getLocalFiles", () => {
+        it("Should return empty array for non-existent directory", () => {
+            const files = getLocalFiles(path.join(fixtureDir, "non-existent"));
+            expect(files).to.be.an("array").that.is.empty;
+        });
+
+        it("Should return file list with relative paths", () => {
+            const files = getLocalFiles(path.join(fixtureDir, "content-0"));
+            expect(files).to.be.an("array").that.is.not.empty;
+            expect(files[0]).to.have.property("relativePath");
+            expect(files[0]).to.have.property("mtime");
+            expect(files[0]).to.have.property("size");
+        });
+    });
+
+    describe("urlJoin", () => {
+        it("Should join base URL with path segments", () => {
+            const result = urlJoin(
+                "https://example.com",
+                "/api/v1/",
+                "endpoint",
+            );
+            expect(result).to.contain("example.com");
+            expect(result).to.contain("endpoint");
+        });
+
+        it("Should handle trailing slashes", () => {
+            const result = urlJoin(
+                "https://example.com/",
+                "api/v1/",
+                "endpoint",
+            );
+            expect(result).to.include("api/v1/endpoint");
+        });
+    });
+
+    describe("content:pull — S3", () => {
+        beforeEach(() => {
+            fs.removeSync(outputDir);
+        });
+
+        it("Should pull down assets from s3 via aws sdk", async () => {
+            const config = loadFixtureConfig("aws-s3");
+
+            await pullS3(config.content[0], path.join(outputDir, "content-0"));
+
+            expect(glob.sync(path.join(outputDir, "content-0/**/*"))).to.be.an(
+                "array",
+            ).that.is.not.empty;
+        });
+
+        it("Should pull down assets from s3 when nested", async () => {
+            const config = loadFixtureConfig("aws-s3-nested");
+
+            await pullS3(config.content[0], path.join(outputDir, "content-0"));
+
+            expect(
+                fs.readFileSync(path.join(outputDir, "content-0/test.txt"), {
+                    encoding: "utf8",
+                }),
+            ).to.equal("core-test-suite-content-nested");
+        });
+
+        it("Should store in directory with key name when defined", async () => {
+            const config = loadFixtureConfig("aws-s3-key");
+
+            await pullS3(config.content[0], path.join(outputDir, "custom-key"));
+
+            expect(glob.sync(path.join(outputDir, "custom-key/**/*"))).to.be.an(
+                "array",
+            ).that.is.not.empty;
+        });
+
+        it("Should sync new assets from s3 (bidirectional)", async () => {
+            const config = loadFixtureConfig("aws-s3-sync");
+            const saveTo = path.join(outputDir, "custom-key");
+            const profile = config.content[0]["aws-s3"];
+            const client = createClient(profile);
+            const { Bucket, Prefix } = parseBucketAndPrefix(
+                config.content[0].location,
+            );
+
+            // Clean up any previous sync test file
+            try {
+                const { DeleteObjectCommand } = require("@aws-sdk/client-s3");
+                await client.send(
+                    new DeleteObjectCommand({
+                        Bucket,
+                        Key: Prefix
+                            ? `${Prefix}/sync-test.txt`
+                            : "sync-test.txt",
+                    }),
+                );
+            } catch {
+                /* may not exist */
+            }
+
+            // First pull to get existing content
+            await pullS3(config.content[0], saveTo);
+
+            // Create a local file to sync up
+            fs.mkdirpSync(saveTo);
+            fs.writeFileSync(path.join(saveTo, "sync-test.txt"), "sync-test");
+
+            // Second pull should upload local file then download remote
+            await pullS3(config.content[0], saveTo);
+
+            // Clean local and re-pull — the synced file should come back
+            fs.removeSync(saveTo);
+            await pullS3(config.content[0], saveTo);
+
+            expect(() =>
+                fs.readFileSync(path.join(saveTo, "sync-test.txt"), {
+                    encoding: "utf8",
+                }),
+            ).to.not.throw();
+
+            // Cleanup remote sync test file
+            try {
+                const { DeleteObjectCommand } = require("@aws-sdk/client-s3");
+                await client.send(
+                    new DeleteObjectCommand({
+                        Bucket,
+                        Key: Prefix
+                            ? `${Prefix}/sync-test.txt`
+                            : "sync-test.txt",
+                    }),
+                );
+            } catch {
+                /* cleanup best-effort */
+            }
+        });
+
+        it("Should not re-download files when local is newer (--update semantics)", async () => {
+            const config = loadFixtureConfig("aws-s3");
+            const saveTo = path.join(outputDir, "content-0");
+
+            // First pull
+            await pullS3(config.content[0], saveTo);
+
+            const files = glob.sync(path.join(saveTo, "**/*"), {
+                nodir: true,
+            });
+            expect(files.length).to.be.greaterThan(0);
+
+            // Touch all local files to make them "newer"
+            const futureTime = new Date(Date.now() + 86400000);
+            files.forEach((f) => fs.utimesSync(f, futureTime, futureTime));
+
+            // Record mtime before second pull
+            const mtimesBefore = {};
+            files.forEach((f) => {
+                mtimesBefore[f] = fs.statSync(f).mtimeMs;
+            });
+
+            // Second pull should skip all files
+            await pullS3(config.content[0], saveTo);
+
+            // Verify files were not overwritten
+            files.forEach((f) => {
+                expect(fs.statSync(f).mtimeMs).to.equal(mtimesBefore[f]);
+            });
+        });
+
+        it("Should skip non-S3 protocols with warning", async () => {
+            const config = loadFixtureConfig("ftp-legacy");
+
+            // pullS3 won't be called for non-s3 — this is handled in content.js
+            // Verify the config doesn't have aws-s3 key
+            expect(config.content[0]["aws-s3"]).to.be.undefined;
+            expect(config.content[0].ftp).to.equal("ftp-fishawack.egnyte.com");
+        });
+
+        after(() => {
+            fs.removeSync(outputDir);
+        });
+    });
+
+    describe("config change detection", () => {
+        beforeEach(() => {
+            fs.removeSync(outputDir);
+            fs.removeSync(path.join(fixtureDir, ".tmp"));
+        });
+
+        it("Should clean content folders when content property is an empty array", () => {
+            // Pre-populate content directory
+            fs.copySync(
+                path.join(fixtureDir, "content-0"),
+                path.join(outputDir, "content-0"),
+            );
+
+            const config = loadFixtureConfig("empty");
+
+            expect(config.content).to.be.an("array").that.is.empty;
+            expect(glob.sync(path.join(outputDir, "**/*"))).to.be.an("array")
+                .that.is.not.empty;
+        });
+
+        it("Should detect when content config has not changed", () => {
+            const config = loadFixtureConfig("aws-s3");
+
+            // Save a snapshot
+            fs.mkdirpSync(path.dirname(snapshotPath));
+            fs.writeFileSync(snapshotPath, JSON.stringify(config.content));
+
+            // Compare — should be equal
+            const prev = JSON.parse(
+                fs.readFileSync(snapshotPath, { encoding: "utf8" }),
+            );
+            expect(isEqual(prev, config.content)).to.be.true;
+        });
+
+        it("Should detect when content config has changed", () => {
+            const configOld = loadFixtureConfig("single");
+            const configNew = loadFixtureConfig("double");
+
+            // Save old snapshot
+            fs.mkdirpSync(path.dirname(snapshotPath));
+            fs.writeFileSync(snapshotPath, JSON.stringify(configOld.content));
+
+            const prev = JSON.parse(
+                fs.readFileSync(snapshotPath, { encoding: "utf8" }),
+            );
+            expect(isEqual(prev, configNew.content)).to.be.false;
+        });
+
+        it("Should handle missing previous snapshot gracefully", () => {
+            // No snapshot file exists
+            expect(fs.existsSync(snapshotPath)).to.be.false;
+
+            let prev = null;
+            try {
+                prev = JSON.parse(
+                    fs.readFileSync(snapshotPath, { encoding: "utf8" }),
+                );
+            } catch {
+                // expected
+            }
+
+            expect(prev).to.be.null;
+        });
+
+        it("Should clean content when config changes between pulls", async () => {
+            const configDouble = loadFixtureConfig("double");
+
+            // Simulate first pull creating two content dirs
+            fs.mkdirpSync(path.join(outputDir, "content-0"));
+            fs.mkdirpSync(path.join(outputDir, "custom-key"));
+            fs.writeFileSync(
+                path.join(outputDir, "content-0/file.txt"),
+                "test",
+            );
+            fs.writeFileSync(
+                path.join(outputDir, "custom-key/file.txt"),
+                "test",
+            );
+
+            // Save snapshot for "double" config
+            fs.mkdirpSync(path.dirname(snapshotPath));
+            fs.writeFileSync(
+                snapshotPath,
+                JSON.stringify(configDouble.content),
+            );
+
+            expect(glob.sync(path.join(outputDir, "*")).length).to.equal(2);
+
+            // Now switch to "single" config — should detect the change
+            const configSingle = loadFixtureConfig("single");
+            const prev = JSON.parse(
+                fs.readFileSync(snapshotPath, { encoding: "utf8" }),
+            );
+            expect(isEqual(prev, configSingle.content)).to.be.false;
+        });
+
+        it("Should not clean content when config unchanged between pulls", () => {
+            const config = loadFixtureConfig("double");
+
+            // Simulate content dirs
+            fs.mkdirpSync(path.join(outputDir, "content-0"));
+            fs.mkdirpSync(path.join(outputDir, "custom-key"));
+            fs.mkdirpSync(path.join(outputDir, "fake-content"));
+
+            // Save snapshot
+            fs.mkdirpSync(path.dirname(snapshotPath));
+            fs.writeFileSync(snapshotPath, JSON.stringify(config.content));
+
+            // Reload and compare — should be unchanged
+            const prev = JSON.parse(
+                fs.readFileSync(snapshotPath, { encoding: "utf8" }),
+            );
+            expect(isEqual(prev, config.content)).to.be.true;
+
+            // Fake-content should survive (not cleaned)
+            expect(glob.sync(path.join(outputDir, "*")).length).to.equal(3);
+        });
+
+        it("Should skip cleaning when content property is undefined", () => {
+            // Pre-populate content
+            fs.copySync(
+                path.join(fixtureDir, "content-0"),
+                path.join(outputDir, "content-0"),
+            );
+
+            const config = loadFixtureConfig("handover");
+
+            // "handover" target has no content property
+            expect(config.content).to.be.undefined;
+            // Content should remain untouched
+            expect(glob.sync(path.join(outputDir, "**/*"))).to.be.an("array")
+                .that.is.not.empty;
+        });
+
+        after(() => {
+            fs.removeSync(outputDir);
+            fs.removeSync(path.join(fixtureDir, ".tmp"));
+        });
+    });
+
+    describe("content:request — HTTP APIs", () => {
+        describe("contentful", () => {
+            const saveTo = path.join(outputDir, "content-0");
+
+            before(async () => {
+                fs.removeSync(outputDir);
+
+                const config = loadFixtureConfig("contentful");
+
+                await pullRequests(config.content, fixtureDir + "/_Build");
+            });
+
+            it("Should save to json files", () => {
+                expect(
+                    fs
+                        .lstatSync(path.join(saveTo, "media/layout.json"))
+                        .isFile(),
+                ).to.be.true;
+                expect(
+                    fs
+                        .lstatSync(path.join(saveTo, "media/layoutCopy.json"))
+                        .isFile(),
+                ).to.be.true;
+            });
+
+            it("Should download images", () => {
+                const mediaFiles = glob.sync(
+                    path.join(saveTo, "media/**/*.png"),
+                );
+                expect(mediaFiles).to.be.an("array").that.is.not.empty;
+            });
+
+            after(() => {
+                fs.removeSync(outputDir);
+            });
+        });
+    });
+
+    describe("rewrite", () => {
+        const saveTo = path.join(outputDir, "rewrite-test");
+
+        beforeEach(() => {
+            fs.removeSync(saveTo);
+            fs.mkdirpSync(saveTo);
+        });
+
+        it("Should rewrite remote URLs to local media paths", async () => {
+            // Create a mock JSON file with remote URLs
+            const mockData = JSON.stringify({
+                image: "https://example.com/wp-content/uploads/2024/photo.jpg",
+                text: "hello world",
+                nested: {
+                    img: "https://example.com/wp-content/uploads/2024/nested.png",
+                },
+            });
+            fs.writeFileSync(path.join(saveTo, "test.json"), mockData);
+
+            await rewrite({
+                ext: "json",
+                type: "wp",
+                saveTo,
+                bundle: "",
+                find: "^https.*/wp-content/uploads",
+            });
+
+            const result = fs.readFileSync(path.join(saveTo, "test.json"), {
+                encoding: "utf8",
+            });
+
+            expect(result).to.contain("media/content");
+            expect(result).to.not.contain("wp-content/uploads");
+        });
+
+        it("Should unwrap contentful response during rewrite", async () => {
+            // Contentful responses are arrays, rewrite should extract first element
+            const mockData = JSON.stringify([
+                { title: "Entry 1", body: "Content" },
+                { title: "Entry 2", body: "More content" },
+            ]);
+            fs.writeFileSync(path.join(saveTo, "test.json"), mockData);
+
+            await rewrite({
+                ext: "json",
+                type: "contentful",
+                saveTo,
+                bundle: "",
+                find: "//images\\.ctfassets\\.net/(.*?)/(.*?)/(.*?)/",
+            });
+
+            const result = JSON.parse(
+                fs.readFileSync(path.join(saveTo, "test.json"), {
+                    encoding: "utf8",
+                }),
+            );
+
+            // Should be unwrapped to first element, not an array
+            expect(result).to.have.property("title", "Entry 1");
+        });
+
+        after(() => {
+            fs.removeSync(saveTo);
+        });
+    });
+
+    describe("download", () => {
+        const saveTo = path.join(outputDir, "download-test");
+
+        beforeEach(() => {
+            fs.removeSync(saveTo);
+            fs.mkdirpSync(saveTo);
+        });
+
+        it("Should extract unique URLs matching find pattern from JSON", async () => {
+            // Create JSON that references assets
+            const mockData = JSON.stringify({
+                images: [
+                    "https://example.com/wp-content/uploads/photo1.jpg",
+                    "https://example.com/wp-content/uploads/photo2.jpg",
+                    "https://example.com/wp-content/uploads/photo1.jpg",
+                ],
+                text: "not a URL",
+            });
+            fs.writeFileSync(path.join(saveTo, "test.json"), mockData);
+
+            // download will try to fetch these URLs — they'll fail but
+            // we verify the function doesn't throw
+            await download({
+                path: "https://example.com",
+                ext: "json",
+                saveTo,
+                bundle: "",
+                find: "^https.*/wp-content/uploads",
+            });
+            // If we get here without throwing, the URL extraction and
+            // deduplication logic works correctly
+        });
+
+        after(() => {
+            fs.removeSync(saveTo);
+        });
+    });
+});

package/cli.js

@@ -54,6 +54,7 @@ const args = hideBin(process.argv);
                 "start",
                 "setup",
                 "test",
+                "scan",
                 "production",
                 "run",
                 "connect",