@projectdochelp/s3te 3.1.2 → 3.1.3

package/README.md

@@ -59,7 +59,9 @@ The website bucket contains the finished result that visitors actually receive t
 <details>
   <summary>What happens when I deploy?</summary>
 
-`s3te deploy` validates the project, packages the AWS runtime, creates or updates one persistent CloudFormation environment stack, creates one temporary CloudFormation deploy stack for packaging artifacts, synchronizes your current source files into the code bucket, and removes the temporary stack again after the real deploy run.
+`s3te deploy` loads the validated project configuration, packages the AWS runtime, creates or updates one persistent CloudFormation environment stack, creates one temporary CloudFormation deploy stack for packaging artifacts, synchronizes your current source files into the code bucket, and removes the temporary stack again after the real deploy run.
+
+That source sync is not limited to Lambda code. It includes your `.html`, `.part`, CSS, JavaScript, images and other project files so the running AWS stack can react to source changes inside the code bucket.
 
 The persistent environment stack contains the long-lived AWS resources such as buckets, Lambda functions, DynamoDB tables, CloudFront distributions and the runtime manifest parameter. The temporary deploy stack exists only so CloudFormation can consume the packaged Lambda artifacts cleanly.
 
@@ -171,6 +173,9 @@ The default scaffold creates:
 mywebsite/
   package.json
   s3te.config.json
+  .github/
+    workflows/
+      s3te-sync.yml
   app/
     part/
       head.part
@@ -187,6 +192,8 @@ mywebsite/
     extensions.json
 ```
 
+The generated `.github/workflows/s3te-sync.yml` is the default CI path for GitHub-based source publishing into the S3TE code bucket. It is scaffolded once and then left alone on later `s3te init` runs unless you use `--force`.
+
 </details>
 
 <details>
@@ -238,13 +245,15 @@ npx s3te deploy --env dev
 
 `deploy` creates or updates the persistent environment stack, uses a temporary deploy stack for packaged Lambda artifacts, synchronizes the source project into the code bucket, and removes the temporary stack again when the deploy finishes.
 
+After the first successful deploy, use `s3te sync --env dev` for regular template, partial, asset and source updates when the infrastructure itself did not change.
+
 If you left `route53HostedZoneId` out of the config, the last DNS step stays manual: point your domain at the created CloudFront distribution after deploy.
 
 </details>
 
 ## Usage
 
-Once the project is installed, your everyday loop is deliberately small: edit templates, validate, render locally, then deploy.
+Once the project is installed, your everyday loop splits into two paths: deploy when infrastructure changes, sync when only project sources changed.
 
 ### Daily Workflow
 
@@ -255,15 +264,24 @@ Once the project is installed, your everyday loop is deliberately small: edit te
 2. If you use content-driven tags without Webiny, edit `offline/content/en.json` or `offline/content/items.json`.
 3. Validate and render locally.
 4. Run your tests.
-5. Deploy when the result looks right.
+5. Use `deploy` for the first installation or after infrastructure/config/runtime changes.
+6. Use `sync` for day-to-day source publishing into the code bucket.
 
 ```bash
 npx s3te validate
 npx s3te render --env dev
 npx s3te test
+npx s3te sync --env dev
+```
+
+Use a full deploy only when needed:
+
+```bash
 npx s3te deploy --env dev
 ```
 
+Once Webiny is installed and the stack is deployed with Webiny enabled, CMS content changes are picked up in AWS through the DynamoDB stream integration. Those content changes do not require another `sync` or `deploy`.
+
 </details>
 
 ### CLI Commands
@@ -278,6 +296,7 @@ npx s3te deploy --env dev
 | `s3te render --env <name>` | Renders locally into `offline/S3TELocal/preview/<env>/...`. |
 | `s3te test` | Runs the project tests from `offline/tests/`. |
 | `s3te package --env <name>` | Builds the AWS deployment artifacts without deploying them yet. |
+| `s3te sync --env <name>` | Uploads current project sources into the configured code buckets. |
 | `s3te doctor --env <name>` | Checks local machine and AWS access before deploy. |
 | `s3te deploy --env <name>` | Deploys or updates the AWS environment and syncs source files. |
 | `s3te migrate` | Updates older project configs and can retrofit Webiny into an existing S3TE project. |
@@ -394,7 +413,100 @@ npx s3te doctor --env prod
 npx s3te deploy --env prod
 ```
 
-That deploy updates the existing environment stack and adds the Webiny mirror resources to it. You do not need a fresh S3TE installation.
+That deploy updates the existing environment stack and adds the Webiny mirror resources to it. You do not need a fresh S3TE installation. After that, Webiny content changes flow through the deployed AWS resources automatically; only template or asset changes still need `s3te sync --env <name>`.
+
+</details>
+
+<details>
+  <summary>GitHub Actions source publishing</summary>
+
+If your team works through GitHub instead of running `s3te sync` locally, the scaffold already includes `.github/workflows/s3te-sync.yml`.
+
+That workflow is meant for source publishing only:
+
+- it validates the project
+- it uploads `app/...` and `part/...` into the S3TE code bucket
+- the resulting S3 events trigger the deployed Lambda pipeline in AWS
+
+Use a full `deploy` only when the infrastructure, environment config, or runtime package changes.
+
+Before the workflow can run, do this once:
+
+1. Run the first real `npx s3te deploy --env <name>` so the code bucket already exists.
+2. In AWS IAM, create an access key for a CI user that may sync only the S3TE code bucket for that environment.
+3. In GitHub open `Settings -> Secrets and variables -> Actions -> Secrets`.
+4. Add these repository secrets:
+   - `AWS_ACCESS_KEY_ID`
+   - `AWS_SECRET_ACCESS_KEY`
+5. Open `.github/workflows/s3te-sync.yml` and adjust:
+   - the branch under `on.push.branches`
+   - `aws-region`
+   - `npx s3te sync --env dev` to your target environment such as `prod` or `test`
+
+Minimal IAM policy example for one code bucket:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["s3:ListBucket"],
+      "Resource": ["arn:aws:s3:::dev-website-code-mywebsite"]
+    },
+    {
+      "Effect": "Allow",
+      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
+      "Resource": ["arn:aws:s3:::dev-website-code-mywebsite/*"]
+    }
+  ]
+}
+```
+
+For non-production environments or additional variants, use the derived code bucket names from your config, for example `test-website-code-mywebsite` or `app-code-mywebsite`.
+
+The scaffolded workflow looks like this:
+
+```yaml
+name: S3TE Sync
+on:
+  workflow_dispatch:
+  push:
+    branches: ["main"]
+    paths:
+      - "app/**"
+      - "package.json"
+      - "package-lock.json"
+      - ".github/workflows/s3te-sync.yml"
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: npm
+      - name: Install dependencies
+        shell: bash
+        run: |
+          if [ -f package-lock.json ]; then
+            npm ci
+          else
+            npm install
+          fi
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: eu-central-1
+      - run: npx s3te validate
+      - run: npx s3te sync --env dev
+```
 
 </details>
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@projectdochelp/s3te",
-  "version": "3.1.2",
+  "version": "3.1.3",
   "description": "CLI, render core, AWS adapter, and testkit for S3TemplateEngine projects",
   "repository": {
     "type": "git",

package/packages/aws-adapter/src/deploy.mjs

@@ -10,6 +10,7 @@ import { ensureAwsCliAvailable, ensureAwsCredentials, runAwsCli } from "./aws-cl
 import { resolveRequestedFeatures } from "./features.mjs";
 import { buildAwsRuntimeManifest, extractStackOutputsMap } from "./manifest.mjs";
 import { packageAwsProject } from "./package.mjs";
+import { syncPreparedSources } from "./sync.mjs";
 import { buildTemporaryDeployStackTemplate } from "./template.mjs";
 
 function normalizeRelative(projectDir, targetPath) {
@@ -365,20 +366,15 @@ export async function deployAwsProject({
       stdio
     });
 
-    const syncedCodeBuckets = [];
-    if (!noSync) {
-      for (const [variantName, variantConfig] of Object.entries(runtimeConfig.variants)) {
-        const syncDir = path.join(projectDir, packaged.manifest.syncDirectories[variantName]);
-        await runAwsCli(["s3", "sync", syncDir, `s3://${variantConfig.codeBucket}`], {
-          region: runtimeConfig.awsRegion,
+    const syncedCodeBuckets = noSync
+      ? []
+      : (await syncPreparedSources({
+          projectDir,
+          runtimeConfig,
+          syncDirectories: packaged.manifest.syncDirectories,
           profile,
-          cwd: projectDir,
-          stdio,
-          errorCode: "ADAPTER_ERROR"
-        });
-        syncedCodeBuckets.push(variantConfig.codeBucket);
-      }
-    }
+          stdio
+        })).syncedCodeBuckets;
 
     const distributions = [];
     for (const [variantName, variantConfig] of Object.entries(runtimeConfig.variants)) {

package/packages/aws-adapter/src/index.mjs

@@ -5,3 +5,4 @@ export { ensureAwsCliAvailable, ensureAwsCredentials, runAwsCli } from "./aws-cl
 export { getConfiguredFeatures, resolveRequestedFeatures } from "./features.mjs";
 export { packageAwsProject } from "./package.mjs";
 export { deployAwsProject } from "./deploy.mjs";
+export { stageProjectSources, syncPreparedSources, syncAwsProject } from "./sync.mjs";

package/packages/aws-adapter/src/sync.mjs

@@ -0,0 +1,155 @@
+import fs from "node:fs/promises";
+import path from "node:path";
+
+import { buildEnvironmentRuntimeConfig } from "../../core/src/index.mjs";
+import { ensureAwsCliAvailable, ensureAwsCredentials, runAwsCli } from "./aws-cli.mjs";
+
+async function ensureDirectory(targetDir) {
+  await fs.mkdir(targetDir, { recursive: true });
+}
+
+async function removeDirectory(targetDir) {
+  await fs.rm(targetDir, { recursive: true, force: true });
+}
+
+async function listFiles(rootDir, currentDir = rootDir) {
+  const entries = await fs.readdir(currentDir, { withFileTypes: true });
+  const files = [];
+
+  for (const entry of entries) {
+    const fullPath = path.join(currentDir, entry.name);
+    if (entry.isDirectory()) {
+      files.push(...await listFiles(rootDir, fullPath));
+      continue;
+    }
+
+    if (entry.isFile()) {
+      files.push(path.relative(rootDir, fullPath).replace(/\\/g, "/"));
+    }
+  }
+
+  return files.sort();
+}
+
+async function copyDirectory(sourceDir, targetDir) {
+  const files = await listFiles(sourceDir);
+  for (const relativePath of files) {
+    const sourcePath = path.join(sourceDir, relativePath);
+    const targetPath = path.join(targetDir, relativePath);
+    await ensureDirectory(path.dirname(targetPath));
+    await fs.copyFile(sourcePath, targetPath);
+  }
+}
+
+function normalizeRelative(projectDir, targetPath) {
+  return path.relative(projectDir, targetPath).replace(/\\/g, "/");
+}
+
+async function stageVariantSources(projectDir, runtimeConfig, variantName, syncRoot) {
+  const variantConfig = runtimeConfig.variants[variantName];
+  const variantRoot = path.join(syncRoot, variantName);
+  await removeDirectory(variantRoot);
+  await ensureDirectory(variantRoot);
+
+  await copyDirectory(path.join(projectDir, variantConfig.partDir), path.join(variantRoot, "part"));
+  await copyDirectory(path.join(projectDir, variantConfig.sourceDir), path.join(variantRoot, variantName));
+
+  return variantRoot;
+}
+
+export async function stageProjectSources({
+  projectDir,
+  config,
+  environment,
+  outDir
+}) {
+  const runtimeConfig = buildEnvironmentRuntimeConfig(config, environment);
+  const syncRoot = outDir
+    ? path.join(projectDir, outDir)
+    : path.join(projectDir, "offline", "IAAS", "sync", environment);
+
+  await ensureDirectory(syncRoot);
+
+  const syncDirectories = {};
+  for (const variantName of Object.keys(runtimeConfig.variants)) {
+    const variantRoot = await stageVariantSources(projectDir, runtimeConfig, variantName, syncRoot);
+    syncDirectories[variantName] = normalizeRelative(projectDir, variantRoot);
+  }
+
+  return {
+    runtimeConfig,
+    syncRoot: normalizeRelative(projectDir, syncRoot),
+    syncDirectories
+  };
+}
+
+export async function syncPreparedSources({
+  projectDir,
+  runtimeConfig,
+  syncDirectories,
+  profile,
+  stdio = "pipe",
+  ensureAwsCliAvailableFn = ensureAwsCliAvailable,
+  ensureAwsCredentialsFn = ensureAwsCredentials,
+  runAwsCliFn = runAwsCli
+}) {
+  await ensureAwsCliAvailableFn({ cwd: projectDir });
+  await ensureAwsCredentialsFn({
+    region: runtimeConfig.awsRegion,
+    profile,
+    cwd: projectDir
+  });
+
+  const syncedCodeBuckets = [];
+  for (const [variantName, variantConfig] of Object.entries(runtimeConfig.variants)) {
+    const syncDir = path.join(projectDir, syncDirectories[variantName]);
+    await runAwsCliFn(["s3", "sync", syncDir, `s3://${variantConfig.codeBucket}`, "--delete"], {
+      region: runtimeConfig.awsRegion,
+      profile,
+      cwd: projectDir,
+      stdio,
+      errorCode: "ADAPTER_ERROR"
+    });
+    syncedCodeBuckets.push(variantConfig.codeBucket);
+  }
+
+  return {
+    syncedCodeBuckets
+  };
+}
+
+export async function syncAwsProject({
+  projectDir,
+  config,
+  environment,
+  outDir,
+  profile,
+  stdio = "pipe",
+  ensureAwsCliAvailableFn = ensureAwsCliAvailable,
+  ensureAwsCredentialsFn = ensureAwsCredentials,
+  runAwsCliFn = runAwsCli
+}) {
+  const prepared = await stageProjectSources({
+    projectDir,
+    config,
+    environment,
+    outDir
+  });
+
+  const synced = await syncPreparedSources({
+    projectDir,
+    runtimeConfig: prepared.runtimeConfig,
+    syncDirectories: prepared.syncDirectories,
+    profile,
+    stdio,
+    ensureAwsCliAvailableFn,
+    ensureAwsCredentialsFn,
+    runAwsCliFn
+  });
+
+  return {
+    syncRoot: prepared.syncRoot,
+    syncDirectories: prepared.syncDirectories,
+    syncedCodeBuckets: synced.syncedCodeBuckets
+  };
+}

package/packages/cli/bin/s3te.mjs

@@ -11,6 +11,7 @@ import {
   renderProject,
   runProjectTests,
   scaffoldProject,
+  syncProject,
   validateProject
 } from "../src/project.mjs";
 
@@ -82,6 +83,7 @@ function printHelp() {
     "  render\n" +
     "  test\n" +
     "  package\n" +
+    "  sync\n" +
     "  deploy\n" +
     "  doctor\n" +
     "  migrate\n"
@@ -281,6 +283,37 @@ async function main() {
     return;
   }
 
+  if (command === "sync") {
+    const loaded = await loadConfigForCommand(cwd, options.config);
+    if (!loaded.ok) {
+      if (wantsJson) {
+        printJson("sync", false, loaded.warnings, loaded.errors, startedAt);
+      }
+      process.exitCode = 2;
+      return;
+    }
+    if (!options.env) {
+      process.stderr.write("sync requires --env <name>\n");
+      process.exitCode = 1;
+      return;
+    }
+
+    const report = await syncProject(cwd, loaded.config, {
+      environment: asArray(options.env)[0],
+      outDir: options["out-dir"],
+      profile: options.profile,
+      stdio: wantsJson ? "pipe" : "inherit"
+    });
+
+    if (wantsJson) {
+      printJson("sync", true, [], [], startedAt, report);
+      return;
+    }
+
+    process.stdout.write(`Synced project sources to ${report.syncedCodeBuckets.length} code bucket(s)\n`);
+    return;
+  }
+
   if (command === "deploy") {
     const loaded = await loadConfigForCommand(cwd, options.config);
     if (!loaded.ok) {

package/packages/cli/src/project.mjs

@@ -15,7 +15,8 @@ import {
   deployAwsProject,
   ensureAwsCliAvailable,
   ensureAwsCredentials,
-  packageAwsProject
+  packageAwsProject,
+  syncAwsProject
 } from "../../aws-adapter/src/index.mjs";
 
 import {
@@ -233,6 +234,52 @@ function schemaTemplate() {
   };
 }
 
+function githubSyncWorkflowTemplate() {
+  return `name: S3TE Sync
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+    paths:
+      - "app/**"
+      - "package.json"
+      - "package-lock.json"
+      - ".github/workflows/s3te-sync.yml"
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: npm
+      - name: Install dependencies
+        shell: bash
+        run: |
+          if [ -f package-lock.json ]; then
+            npm ci
+          else
+            npm install
+          fi
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          aws-access-key-id: \${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: \${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: eu-central-1
+      - name: Validate project
+        run: npx s3te validate
+      - name: Sync project sources to the S3TE code bucket
+        run: npx s3te sync --env dev
+`;
+}
+
 async function fileExists(targetPath) {
   try {
     await fs.stat(targetPath);
@@ -482,6 +529,7 @@ export async function scaffoldProject(projectDir, options = {}) {
   await ensureDirectory(path.join(projectDir, "offline", "tests"));
   await ensureDirectory(path.join(projectDir, "offline", "content"));
   await ensureDirectory(path.join(projectDir, "offline", "schemas"));
+  await ensureDirectory(path.join(projectDir, ".github", "workflows"));
   await ensureDirectory(path.join(projectDir, ".vscode"));
 
   const projectPackageJson = {
@@ -491,6 +539,7 @@ export async function scaffoldProject(projectDir, options = {}) {
     scripts: {
       validate: "s3te validate",
       render: "s3te render --env dev",
+      sync: "s3te sync --env dev",
       test: "s3te test"
     }
   };
@@ -530,6 +579,7 @@ export async function scaffoldProject(projectDir, options = {}) {
   await writeProjectPackageJson(path.join(projectDir, "package.json"), projectPackageJson, scaffoldOptions, force);
   await writeProjectConfigJson(path.join(projectDir, "s3te.config.json"), config, scaffoldOptions, force);
   await writeProjectFile(path.join(projectDir, "offline", "schemas", "s3te.config.schema.json"), JSON.stringify(schemaTemplate(), null, 2) + "\n", force, true);
+  await writeProjectFile(path.join(projectDir, ".github", "workflows", "s3te-sync.yml"), githubSyncWorkflowTemplate(), force);
   await writeProjectFile(path.join(projectDir, "app", "part", "head.part"), "<meta charset='utf-8'>\n<title>My S3TE Site</title>\n", force);
   await writeProjectFile(path.join(projectDir, "app", variant, "index.html"), "<!doctype html>\n<html lang=\"<lang>2</lang>\">\n  <head>\n    <part>head.part</part>\n  </head>\n  <body>\n    <h1>Hello from S3TemplateEngine</h1>\n  </body>\n</html>\n", force);
   await writeProjectFile(path.join(projectDir, "offline", "content", `${language}.json`), "[]\n", force);
@@ -674,6 +724,17 @@ export async function deployProject(projectDir, config, options = {}) {
   });
 }
 
+export async function syncProject(projectDir, config, options = {}) {
+  return syncAwsProject({
+    projectDir,
+    config,
+    environment: options.environment,
+    outDir: options.outDir,
+    profile: options.profile,
+    stdio: options.stdio ?? "pipe"
+  });
+}
+
 export async function doctorProject(projectDir, configPath, options = {}) {
   const checks = [];
   const majorVersion = Number(process.versions.node.split(".")[0]);