@tandem-language-exchange/content-store 1.0.0

package/README.md

@@ -0,0 +1,335 @@
+# Content Store
+
+Syncs content from **Contentful** and **Sanity** into versioned JSON bundles on **S3**. Provides both a CLI for one-off or scheduled syncs and an Express REST API for on-demand triggering.
+
+## Architecture
+
+The codebase is split into three areas so that the npm-published SDK never ships server dependencies (Express, Contentful SDK, Sanity client, dotenv, etc.):
+
+```
+src/
+├── shared/                      # Shared between SDK and server
+│   ├── types.ts                 # S3Config, CMSProvider
+│   └── s3.ts                    # ContentStore class (S3 operations)
+├── sdk/                         # npm package surface — importable by consumers
+│   ├── client.ts                # ContentStoreSDK (fetchBundles + queryBundle)
+│   ├── index.ts                 # SDK entry point + re-exports
+│   └── README.md                # SDK usage documentation
+├── server/                      # Express REST API + CLI + CMS sync engine
+│   ├── config.ts                # Env-based runtime configuration
+│   ├── server.ts                # Express app + endpoints
+│   ├── cli.ts                   # Commander CLI
+│   ├── middleware/
+│   │   └── auth.ts              # Bearer token auth
+│   ├── adapters/                # CMS adapter pattern
+│   │   ├── types.ts             # CMSAdapter interface
+│   │   ├── contentful.ts        # Contentful (batched pagination + envelope stripping)
+│   │   ├── sanity.ts            # Sanity (GROQ queries)
+│   │   └── index.ts             # createAdapter factory
+│   └── sync/
+│       ├── retry.ts             # Exponential backoff with jitter & 429 handling
+│       └── engine.ts            # Orchestrates fetch → upload → copy-latest
+└── index.ts                     # npm entry — re-exports SDK + shared only
+```
+
+When published to npm, only `dist/index.*`, `dist/sdk/`, and `dist/shared/` are included (controlled by the `files` field in `package.json`). Everything under `dist/server/` is excluded.
+
+### S3 Object Naming
+
+Each sync produces two objects per content type:
+
+| Object | Purpose |
+| --- | --- |
+| `content-{cms}-{contentType}-{timestamp}.json` | Versioned snapshot (retained indefinitely) |
+| `content-{cms}-{contentType}.json` | Latest pointer (overwritten on each sync) |
+
+For example, syncing the `gridLayout` type from Contentful at Unix timestamp `1743400000` creates:
+
+```
+content-contentful-gridLayout-1743400000.json   ← versioned
+content-contentful-gridLayout.json              ← latest (copy of above)
+```
+
+---
+
+## Getting Started Locally
+
+### 1. Clone this repo from Azure DevOps
+
+```bash
+git clone git@ssh.dev.azure.com:v3/tripod-technology/content-store/content-store
+cd content-store
+```
+
+### 2. Setup NPM Auth
+
+The project depends on the private `@tandem-web/web-azure-appconfig-sync` package. Authenticate with the npm registry before installing:
+
+```bash
+//registry.npmjs.org/:_authToken={NPM_TOKEN}
+registry=https://registry.npmjs.org/
+```
+
+This token can be found in the Azure Pipelines:
+
+```
+[Project] > Pipelines > Library > Variable Groups > azure-appconfig-sync > NPM_TOKEN
+```
+
+### 3. Install Dependencies
+
+Requires **Node >= 24.1** and **npm >= 11.3** (see `engines` in `package.json`).
+
+```bash
+npm install
+```
+
+### 4. Setup Azure App Config Syncing
+
+This allows you to pull environment variables down from Azure App Configuration, and also switch environments easily.
+
+You need to create a file in the root of this project called `.env.appconfig` with the connection strings for Azure App Configuration for each environment:
+
+```.env.appconfig
+AZURE_APP_CONFIG_CONNECTION_STRING_STAGING=[string]
+AZURE_APP_CONFIG_CONNECTION_STRING_PRODUCTION=Endpoint=[string]
+AZURE_APP_CONFIG_CONNECTION_STRING_LOCAL=Endpoint=[string]
+```
+
+These connection strings can be found in the Azure Pipelines:
+
+```
+[Project] > Pipelines > Library > Variable Groups > azure-appconfig-sync
+```
+
+*See the [docs](https://github.com/viveme/web-azure-appconfig-sync) for full the config needed of this package*
+
+Run the following command to sync the `local` config:
+
+
+```bash
+npm run sync-appconfig:local
+```
+
+Also, run the following command to sync the `staging` config:
+
+```bash
+npm run sync-appconfig:staging
+```
+
+### 5. Configuring Content Types
+
+Content types to sync from Contentful are defined directly in `src/server/config.ts`:
+
+```typescript
+contentTypes: [
+  'gridLayout',
+  'iconWithText',
+  // Add or remove content type IDs here.
+  // Leave empty to sync every content type in the space.
+]
+```
+
+Sanity syncs all document types by default (excluding internal `system.*` and `sanity.*` types). You can override this at runtime via the `--types` CLI flag or the `contentTypes` field in the REST API request body.
+
+---
+
+## Running Local Development Server
+
+Starts the Express server with hot-reload via `tsx watch`:
+
+```bash
+npm run dev
+```
+
+The server listens on `http://localhost:4000` (or the port set by `PORT`).
+
+Available endpoints:
+
+| Method | Path | Auth | Description |
+| --- | --- | --- | --- |
+| `GET` | `/health` | None | Health check |
+| `POST` | `/sync` | Bearer token | Trigger a content sync |
+
+---
+
+## Building & Running Locally
+
+Compile TypeScript to `dist/`:
+
+```bash
+npm run build
+```
+
+Run the compiled server:
+
+```bash
+npm start
+```
+
+---
+
+## Docker Build
+
+Run the following command to build the the app in a docker container locally:
+
+```bash
+npm run build:docker
+```
+
+CI builds are handled by Azure Pipelines (`deploy/staging-pipeline.yml` and `deploy/production-pipeline.yml`), which:
+
+1. Authenticate to npm and install `@tandem-web/web-azure-appconfig-sync`
+2. Fetch environment variables from Azure App Configuration into `.env`
+3. Build and tag the Docker image
+4. Push to Azure Container Registry (`tandemstaging.azurecr.io` / `tandemproduction.azurecr.io`)
+5. Force a new ECS service deployment on the `nextgen-staging` or `nextgen-production` cluster
+
+---
+
+## Running from CLI
+
+All CLI commands are invoked through `tsx src/server/cli.ts` under the hood. Convenience npm scripts are provided for common operations.
+
+### `sync` — push CMS content to S3
+
+**Sync all configured Contentful content types:**
+
+```bash
+npm run sync:contentful
+```
+
+**Sync all Sanity document types:**
+
+```bash
+npm run sync:sanity
+```
+
+**Sync specific content types only:**
+
+```bash
+npm run sync -- --cms contentful --types gridLayout,iconWithText
+npm run sync -- --cms sanity --types article,author
+```
+
+The process exits with code `0` on full success and `1` if any content type failed to sync. A JSON summary is printed to stdout on completion.
+
+### `fetch` — download bundles from S3 to the local filesystem
+
+Downloads the latest version of each requested content type bundle from S3 and writes them as JSON files to a local directory.
+
+```bash
+npm run fetch -- --cms contentful --types gridLayout,page
+```
+
+| Flag | Required | Default | Description |
+| --- | --- | --- | --- |
+| `--cms <provider>` | Yes | | `contentful` or `sanity` |
+| `--types <types>` | Yes | | Comma-separated content types |
+| `--output <dir>` | No | `./content-cache` | Local directory to write bundle files to |
+
+Files are written as `content-{cms}-{contentType}.json` inside the output directory.
+
+### `query` — query a local bundle
+
+Reads a previously fetched bundle from disk and prints filtered results as JSON.
+
+```bash
+npm run query -- --cms contentful --type gridLayout \
+  --fields '{"columns":"2"}' \
+  --select title,bodyBefore \
+  --limit 5 \
+  --include 2
+```
+
+| Flag | Required | Default | Description |
+| --- | --- | --- | --- |
+| `--cms <provider>` | Yes | | `contentful` or `sanity` |
+| `--type <type>` | Yes | | Content type to query |
+| `--output <dir>` | No | `./content-cache` | Directory where bundles are stored |
+| `--fields <json>` | No | | JSON filter object (e.g. `'{"columns":"2"}'`) |
+| `--select <props>` | No | | Comma-separated properties to include in results |
+| `--limit <n>` | No | | Maximum number of results |
+| `--include <n>` | No | | Depth of nested references to include (see [SDK README](src/sdk/README.md)) |
+
+---
+
+## Calling the REST API
+
+All requests to `POST /sync` must include a valid bearer token in the `Authorization` header. The token is the value of the `CONTENT_STORE_API_TOKEN` environment variable.
+
+**Sync all configured Contentful types:**
+
+```bash
+curl -X POST http://localhost:3000/sync \
+  -H "Authorization: Bearer <CONTENT_STORE_API_TOKEN>" \
+  -H "Content-Type: application/json" \
+  -d '{"cms": "contentful"}'
+```
+
+**Sync specific types:**
+
+```bash
+curl -X POST http://localhost:3000/sync \
+  -H "Authorization: Bearer <CONTENT_STORE_API_TOKEN>" \
+  -H "Content-Type: application/json" \
+  -d '{"cms": "sanity", "contentTypes": ["article", "author"]}'
+```
+
+**Response format:**
+
+```json
+{
+  "cms": "contentful",
+  "timestamp": 1743400000,
+  "entries": [
+    {
+      "contentType": "gridLayout",
+      "itemCount": 42,
+      "versionedKey": "content-contentful-gridLayout-1743400000.json",
+      "latestKey": "content-contentful-gridLayout.json"
+    }
+  ],
+  "errors": []
+}
+```
+
+| Status | Meaning |
+| --- | --- |
+| `200` | All content types synced successfully |
+| `207` | Partial success (some types failed — check `errors` array) |
+| `400` | Invalid CMS provider |
+| `401` | Missing or malformed Authorization header |
+| `403` | Invalid bearer token |
+| `500` | Unexpected server error |
+
+**Health check (no auth required):**
+
+```bash
+curl http://localhost:3000/health
+```
+
+---
+
+## Additional Considerations
+
+### Rate Limiting & Retry Behaviour
+
+Contentful's Content Delivery/Preview API enforces rate limits. The sync engine wraps every API call in a retry loop with:
+
+- **Exponential backoff** — delay doubles on each attempt (1 s → 2 s → 4 s → …)
+- **Random jitter** — prevents thundering-herd effects when multiple syncs run concurrently
+- **429 awareness** — if the API returns a `429 Too Many Requests` with an `x-contentful-ratelimit-reset` header, that value is used as the wait time instead of computed backoff
+- **Configurable limits** — `RETRY_MAX_RETRIES`, `RETRY_BASE_DELAY_MS`, and `RETRY_MAX_DELAY_MS` can be tuned via env vars
+
+### Contentful Pagination (Batched Sync)
+
+Contentful's `getEntries` endpoint returns a maximum of 1,000 items per request. The Contentful adapter automatically pages through the full result set using `skip` and `limit`, collecting all entries before uploading a single consolidated JSON file to S3.
+
+### S3 Versioning Strategy
+
+Every sync writes a timestamped object and then copies it to the "latest" key. This means:
+
+- Consumers that always read `content-{cms}-{type}.json` get the most recent data without knowing the timestamp.
+- Older snapshots (`content-{cms}-{type}-{timestamp}.json`) remain in the bucket and can be used for auditing, rollback, or diffing.
+- To manage storage costs over time, consider adding an S3 lifecycle rule to expire versioned objects older than a chosen retention period.

package/dist/index.d.ts

@@ -0,0 +1,5 @@
+export { ContentStoreSDK } from './sdk/index.js';
+export type { SDKConfig, FetchBundlesOptions, QueryOptions } from './sdk/index.js';
+export { fetchBundles, queryBundle, trimDepth } from './shared/bundles.js';
+export { ContentStore } from './shared/s3.js';
+export type { CMSProvider, S3Config } from './shared/types.js';

package/dist/index.js

@@ -0,0 +1,4 @@
+export { ContentStoreSDK } from './sdk/index.js';
+export { fetchBundles, queryBundle, trimDepth } from './shared/bundles.js';
+export { ContentStore } from './shared/s3.js';
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAEjD,OAAO,EAAE,YAAY,EAAE,WAAW,EAAE,SAAS,EAAE,MAAM,qBAAqB,CAAC;AAC3E,OAAO,EAAE,YAAY,EAAE,MAAM,gBAAgB,CAAC"}

package/dist/sdk/client.d.ts

@@ -0,0 +1,24 @@
+import type { S3Config, CMSProvider } from '../shared/types.js';
+import { type FetchBundlesOptions, type QueryOptions } from '../shared/bundles.js';
+export type { FetchBundlesOptions, QueryOptions };
+export interface SDKConfig {
+    s3: S3Config;
+    /** Directory where bundle JSON files are saved on the local filesystem. */
+    outputDir: string;
+}
+export declare class ContentStoreSDK {
+    private store;
+    private outputDir;
+    constructor(config: SDKConfig);
+    /**
+     * Downloads the latest bundles from S3 and writes them as JSON files
+     * to `outputDir`.
+     *
+     * @returns A map of contentType to absolute file path.
+     */
+    fetchBundles(options: FetchBundlesOptions): Promise<Record<string, string>>;
+    /**
+     * Queries a previously fetched bundle from the local filesystem.
+     */
+    queryBundle(cms: CMSProvider, contentType: string, options?: QueryOptions): Promise<unknown[]>;
+}

package/dist/sdk/client.js

@@ -0,0 +1,26 @@
+import { ContentStore } from '../shared/s3.js';
+import { fetchBundles, queryBundle, } from '../shared/bundles.js';
+export class ContentStoreSDK {
+    store;
+    outputDir;
+    constructor(config) {
+        this.store = new ContentStore(config.s3);
+        this.outputDir = config.outputDir;
+    }
+    /**
+     * Downloads the latest bundles from S3 and writes them as JSON files
+     * to `outputDir`.
+     *
+     * @returns A map of contentType to absolute file path.
+     */
+    async fetchBundles(options) {
+        return fetchBundles(this.store, this.outputDir, options);
+    }
+    /**
+     * Queries a previously fetched bundle from the local filesystem.
+     */
+    async queryBundle(cms, contentType, options = {}) {
+        return queryBundle(this.outputDir, cms, contentType, options);
+    }
+}
+//# sourceMappingURL=client.js.map

package/dist/sdk/client.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"client.js","sourceRoot":"","sources":["../../src/sdk/client.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC;AAC/C,OAAO,EACL,YAAY,EACZ,WAAW,GAGZ,MAAM,sBAAsB,CAAC;AAU9B,MAAM,OAAO,eAAe;IAClB,KAAK,CAAe;IACpB,SAAS,CAAS;IAE1B,YAAY,MAAiB;QAC3B,IAAI,CAAC,KAAK,GAAG,IAAI,YAAY,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;QACzC,IAAI,CAAC,SAAS,GAAG,MAAM,CAAC,SAAS,CAAC;IACpC,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,YAAY,CAChB,OAA4B;QAE5B,OAAO,YAAY,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,SAAS,EAAE,OAAO,CAAC,CAAC;IAC3D,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,WAAW,CACf,GAAgB,EAChB,WAAmB,EACnB,UAAwB,EAAE;QAE1B,OAAO,WAAW,CAAC,IAAI,CAAC,SAAS,EAAE,GAAG,EAAE,WAAW,EAAE,OAAO,CAAC,CAAC;IAChE,CAAC;CACF"}

package/dist/sdk/index.d.ts

@@ -0,0 +1,5 @@
+export { ContentStoreSDK } from './client.js';
+export type { SDKConfig, FetchBundlesOptions, QueryOptions } from './client.js';
+export { fetchBundles, queryBundle, trimDepth } from '../shared/bundles.js';
+export { ContentStore } from '../shared/s3.js';
+export type { S3Config, CMSProvider } from '../shared/types.js';

package/dist/sdk/index.js

@@ -0,0 +1,4 @@
+export { ContentStoreSDK } from './client.js';
+export { fetchBundles, queryBundle, trimDepth } from '../shared/bundles.js';
+export { ContentStore } from '../shared/s3.js';
+//# sourceMappingURL=index.js.map

package/dist/sdk/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/sdk/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,eAAe,EAAE,MAAM,aAAa,CAAC;AAE9C,OAAO,EAAE,YAAY,EAAE,WAAW,EAAE,SAAS,EAAE,MAAM,sBAAsB,CAAC;AAC5E,OAAO,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC"}

package/dist/shared/bundles.d.ts

@@ -0,0 +1,41 @@
+import type { CMSProvider } from './types.js';
+import { ContentStore } from './s3.js';
+export interface FetchBundlesOptions {
+    cms: CMSProvider;
+    contentTypes: string[];
+}
+export interface QueryOptions {
+    /** Filter items by matching top-level property values (exact equality, or array for IN). */
+    fields?: Record<string, unknown>;
+    /** Properties to include in each result object. Omit to return all properties. */
+    select?: string[];
+    /** Maximum number of items to return. */
+    limit?: number;
+    /**
+     * How many levels deep to return.
+     *  - 1 = the item's own scalar properties only; nested objects/refs are nulled.
+     *  - 2 = the item including its direct references; refs inside those are nulled.
+     *  - 3 = three levels deep, and so on.
+     *  - Omit to return the full depth.
+     */
+    include?: number;
+}
+/**
+ * Trims nested object depth.
+ *
+ * `remaining` represents how many levels the current object is allowed.
+ *  - Scalar properties are always kept.
+ *  - Nested objects / arrays-of-objects consume one level. When `remaining`
+ *    drops to 1 they are replaced with `null` (no budget left for refs).
+ */
+export declare function trimDepth(value: unknown, remaining: number): unknown;
+/**
+ * Downloads the latest bundles from S3 and writes them as JSON files to `outputDir`.
+ *
+ * @returns A map of contentType to absolute file path.
+ */
+export declare function fetchBundles(store: ContentStore, outputDir: string, options: FetchBundlesOptions): Promise<Record<string, string>>;
+/**
+ * Queries a previously fetched bundle from the local filesystem.
+ */
+export declare function queryBundle(outputDir: string, cms: CMSProvider, contentType: string, options?: QueryOptions): Promise<unknown[]>;

package/dist/shared/bundles.js

@@ -0,0 +1,94 @@
+import fs from 'node:fs/promises';
+import path from 'node:path';
+/**
+ * Trims nested object depth.
+ *
+ * `remaining` represents how many levels the current object is allowed.
+ *  - Scalar properties are always kept.
+ *  - Nested objects / arrays-of-objects consume one level. When `remaining`
+ *    drops to 1 they are replaced with `null` (no budget left for refs).
+ */
+export function trimDepth(value, remaining) {
+    if (value === null || value === undefined || typeof value !== 'object') {
+        return value;
+    }
+    if (Array.isArray(value)) {
+        return value.map((item) => trimDepth(item, remaining));
+    }
+    const obj = value;
+    const result = {};
+    for (const [k, v] of Object.entries(obj)) {
+        if (v === null || v === undefined || typeof v !== 'object') {
+            result[k] = v;
+        }
+        else if (remaining <= 1) {
+            result[k] = null;
+        }
+        else if (Array.isArray(v)) {
+            result[k] = v.map((item) => {
+                if (item !== null && typeof item === 'object' && !Array.isArray(item)) {
+                    return trimDepth(item, remaining - 1);
+                }
+                return item;
+            });
+        }
+        else {
+            result[k] = trimDepth(v, remaining - 1);
+        }
+    }
+    return result;
+}
+/**
+ * Downloads the latest bundles from S3 and writes them as JSON files to `outputDir`.
+ *
+ * @returns A map of contentType to absolute file path.
+ */
+export async function fetchBundles(store, outputDir, options) {
+    const { cms, contentTypes } = options;
+    await fs.mkdir(outputDir, { recursive: true });
+    const result = {};
+    await Promise.all(contentTypes.map(async (contentType) => {
+        const key = store.buildLatestKey(cms, contentType);
+        const data = await store.download(key);
+        const filePath = path.resolve(outputDir, `content-${cms}-${contentType}.json`);
+        await fs.writeFile(filePath, JSON.stringify(data, null, 2), 'utf-8');
+        result[contentType] = filePath;
+    }));
+    return result;
+}
+/**
+ * Queries a previously fetched bundle from the local filesystem.
+ */
+export async function queryBundle(outputDir, cms, contentType, options = {}) {
+    const filePath = path.resolve(outputDir, `content-${cms}-${contentType}.json`);
+    const raw = await fs.readFile(filePath, 'utf-8');
+    let items = JSON.parse(raw);
+    if (options.fields) {
+        const filters = options.fields;
+        items = items.filter((item) => Object.entries(filters).every(([key, expected]) => {
+            const actual = item[key];
+            if (Array.isArray(expected))
+                return expected.includes(actual);
+            return actual === expected;
+        }));
+    }
+    if (options.limit !== undefined && options.limit > 0) {
+        items = items.slice(0, options.limit);
+    }
+    if (options.include !== undefined) {
+        items = items.map((item) => trimDepth(item, options.include));
+    }
+    if (options.select?.length) {
+        const keys = options.select;
+        items = items.map((item) => {
+            const picked = {};
+            for (const k of keys) {
+                if (k in item)
+                    picked[k] = item[k];
+            }
+            return picked;
+        });
+    }
+    return items;
+}
+//# sourceMappingURL=bundles.js.map

package/dist/shared/bundles.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"bundles.js","sourceRoot":"","sources":["../../src/shared/bundles.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,MAAM,kBAAkB,CAAC;AAClC,OAAO,IAAI,MAAM,WAAW,CAAC;AA0B7B;;;;;;;GAOG;AACH,MAAM,UAAU,SAAS,CAAC,KAAc,EAAE,SAAiB;IACzD,IAAI,KAAK,KAAK,IAAI,IAAI,KAAK,KAAK,SAAS,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QACvE,OAAO,KAAK,CAAC;IACf,CAAC;IAED,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC;QACzB,OAAO,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,EAAE,SAAS,CAAC,CAAC,CAAC;IACzD,CAAC;IAED,MAAM,GAAG,GAAG,KAAgC,CAAC;IAC7C,MAAM,MAAM,GAA4B,EAAE,CAAC;IAE3C,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;QACzC,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,KAAK,SAAS,IAAI,OAAO,CAAC,KAAK,QAAQ,EAAE,CAAC;YAC3D,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;QAChB,CAAC;aAAM,IAAI,SAAS,IAAI,CAAC,EAAE,CAAC;YAC1B,MAAM,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC;QACnB,CAAC;aAAM,IAAI,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;YAC5B,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE;gBACzB,IAAI,IAAI,KAAK,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,CAAC;oBACtE,OAAO,SAAS,CAAC,IAAI,EAAE,SAAS,GAAG,CAAC,CAAC,CAAC;gBACxC,CAAC;gBACD,OAAO,IAAI,CAAC;YACd,CAAC,CAAC,CAAC;QACL,CAAC;aAAM,CAAC;YACN,MAAM,CAAC,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC,EAAE,SAAS,GAAG,CAAC,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED,OAAO,MAAM,CAAC;AAChB,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,KAAmB,EACnB,SAAiB,EACjB,OAA4B;IAE5B,MAAM,EAAE,GAAG,EAAE,YAAY,EAAE,GAAG,OAAO,CAAC;IACtC,MAAM,EAAE,CAAC,KAAK,CAAC,SAAS,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IAE/C,MAAM,MAAM,GAA2B,EAAE,CAAC;IAE1C,MAAM,OAAO,CAAC,GAAG,CACf,YAAY,CAAC,GAAG,CAAC,KAAK,EAAE,WAAW,EAAE,EAAE;QACrC,MAAM,GAAG,GAAG,KAAK,CAAC,cAAc,CAAC,GAAG,EAAE,WAAW,CAAC,CAAC;QACnD,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC;QACvC,MAAM,QAAQ,GAAG,IAAI,CAAC,OAAO,CAC3B,SAAS,EACT,WAAW,GAAG,IAAI,WAAW,OAAO,CACrC,CAAC;QACF,MAAM,EAAE,CAAC,SAAS,CAAC,QAAQ,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC,EAAE,OAAO,CAAC,CAAC;QACrE,MAAM,CAAC,WAAW,CAAC,GAAG,QAAQ,CAAC;IACjC,CAAC,CAAC,CACH,CAAC;IAEF,OAAO,MAAM,CAAC;AAChB,CAAC;AAED;;GAEG;AACH,MAAM,CAAC,KAAK,UAAU,WAAW,CAC/B,SAAiB,EACjB,GAAgB,EAChB,WAAmB,EACnB,UAAwB,EAAE;IAE1B,MAAM,QAAQ,GAAG,IAAI,CAAC,OAAO,CAC3B,SAAS,EACT,WAAW,GAAG,IAAI,WAAW,OAAO,CACrC,CAAC;IACF,MAAM,GAAG,GAAG,MAAM,EAAE,CAAC,QAAQ,CAAC,QAAQ,EAAE,OAAO,CAAC,CAAC;IACjD,IAAI,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAA8B,CAAC;IAEzD,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;QACnB,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC;QAC/B,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAC5B,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,EAAE,QAAQ,CAAC,EAAE,EAAE;YAChD,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC;YACzB,IAAI,KAAK,CAAC,OAAO,CAAC,QAAQ,CAAC;gBAAE,OAAO,QAAQ,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;YAC9D,OAAO,MAAM,KAAK,QAAQ,CAAC;QAC7B,CAAC,CAAC,CACH,CAAC;IACJ,CAAC;IAED,IAAI,OAAO,CAAC,KAAK,KAAK,SAAS,IAAI,OAAO,CAAC,KAAK,GAAG,CAAC,EAAE,CAAC;QACrD,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,OAAO,CAAC,KAAK,CAAC,CAAC;IACxC,CAAC;IAED,IAAI,OAAO,CAAC,OAAO,KAAK,SAAS,EAAE,CAAC;QAClC,KAAK,GAAG,KAAK,CAAC,GAAG,CACf,CAAC,IAAI,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,EAAE,OAAO,CAAC,OAAQ,CAA4B,CACvE,CAAC;IACJ,CAAC;IAED,IAAI,OAAO,CAAC,MAAM,EAAE,MAAM,EAAE,CAAC;QAC3B,MAAM,IAAI,GAAG,OAAO,CAAC,MAAM,CAAC;QAC5B,KAAK,GAAG,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE;YACzB,MAAM,MAAM,GAA4B,EAAE,CAAC;YAC3C,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;gBACrB,IAAI,CAAC,IAAI,IAAI;oBAAE,MAAM,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;YACrC,CAAC;YACD,OAAO,MAAM,CAAC;QAChB,CAAC,CAAC,CAAC;IACL,CAAC;IAED,OAAO,KAAK,CAAC;AACf,CAAC"}

package/dist/shared/s3.d.ts

@@ -0,0 +1,17 @@
+import type { S3Config } from './types.js';
+export declare class ContentStore {
+    private client;
+    private bucket;
+    constructor(cfg: S3Config);
+    /** content-{cms}-{contentType}-{timestamp}.json */
+    buildVersionedKey(cms: string, contentType: string, timestamp: number): string;
+    /** content-{cms}-{contentType}.json  (always points at the latest version) */
+    buildLatestKey(cms: string, contentType: string): string;
+    upload(key: string, data: unknown): Promise<string>;
+    download(key: string): Promise<unknown>;
+    /**
+     * Copies a versioned object to the "latest" key so that it always reflects
+     * the most recent sync while older timestamped versions are retained.
+     */
+    copyToLatest(sourceKey: string, cms: string, contentType: string): Promise<string>;
+}

package/dist/shared/s3.js

@@ -0,0 +1,54 @@
+import { S3Client, PutObjectCommand, CopyObjectCommand, GetObjectCommand, } from '@aws-sdk/client-s3';
+export class ContentStore {
+    client;
+    bucket;
+    constructor(cfg) {
+        this.client = new S3Client({
+            region: cfg.region,
+            credentials: {
+                accessKeyId: cfg.accessKeyId,
+                secretAccessKey: cfg.secretAccessKey,
+            },
+        });
+        this.bucket = cfg.bucket;
+    }
+    /** content-{cms}-{contentType}-{timestamp}.json */
+    buildVersionedKey(cms, contentType, timestamp) {
+        return `content-${cms}-${contentType}-${timestamp}.json`;
+    }
+    /** content-{cms}-{contentType}.json  (always points at the latest version) */
+    buildLatestKey(cms, contentType) {
+        return `content-${cms}-${contentType}.json`;
+    }
+    async upload(key, data) {
+        await this.client.send(new PutObjectCommand({
+            Bucket: this.bucket,
+            Key: key,
+            Body: JSON.stringify(data, null, 2),
+            ContentType: 'application/json',
+        }));
+        return key;
+    }
+    async download(key) {
+        const response = await this.client.send(new GetObjectCommand({ Bucket: this.bucket, Key: key }));
+        const body = await response.Body?.transformToString();
+        if (!body)
+            throw new Error(`Empty response for key: ${key}`);
+        return JSON.parse(body);
+    }
+    /**
+     * Copies a versioned object to the "latest" key so that it always reflects
+     * the most recent sync while older timestamped versions are retained.
+     */
+    async copyToLatest(sourceKey, cms, contentType) {
+        const latestKey = this.buildLatestKey(cms, contentType);
+        await this.client.send(new CopyObjectCommand({
+            Bucket: this.bucket,
+            CopySource: `${this.bucket}/${sourceKey}`,
+            Key: latestKey,
+            ContentType: 'application/json',
+        }));
+        return latestKey;
+    }
+}
+//# sourceMappingURL=s3.js.map

package/dist/shared/s3.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"s3.js","sourceRoot":"","sources":["../../src/shared/s3.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,QAAQ,EACR,gBAAgB,EAChB,iBAAiB,EACjB,gBAAgB,GACjB,MAAM,oBAAoB,CAAC;AAG5B,MAAM,OAAO,YAAY;IACf,MAAM,CAAW;IACjB,MAAM,CAAS;IAEvB,YAAY,GAAa;QACvB,IAAI,CAAC,MAAM,GAAG,IAAI,QAAQ,CAAC;YACzB,MAAM,EAAE,GAAG,CAAC,MAAM;YAClB,WAAW,EAAE;gBACX,WAAW,EAAE,GAAG,CAAC,WAAW;gBAC5B,eAAe,EAAE,GAAG,CAAC,eAAe;aACrC;SACF,CAAC,CAAC;QACH,IAAI,CAAC,MAAM,GAAG,GAAG,CAAC,MAAM,CAAC;IAC3B,CAAC;IAED,mDAAmD;IACnD,iBAAiB,CAAC,GAAW,EAAE,WAAmB,EAAE,SAAiB;QACnE,OAAO,WAAW,GAAG,IAAI,WAAW,IAAI,SAAS,OAAO,CAAC;IAC3D,CAAC;IAED,8EAA8E;IAC9E,cAAc,CAAC,GAAW,EAAE,WAAmB;QAC7C,OAAO,WAAW,GAAG,IAAI,WAAW,OAAO,CAAC;IAC9C,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,GAAW,EAAE,IAAa;QACrC,MAAM,IAAI,CAAC,MAAM,CAAC,IAAI,CACpB,IAAI,gBAAgB,CAAC;YACnB,MAAM,EAAE,IAAI,CAAC,MAAM;YACnB,GAAG,EAAE,GAAG;YACR,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC;YACnC,WAAW,EAAE,kBAAkB;SAChC,CAAC,CACH,CAAC;QACF,OAAO,GAAG,CAAC;IACb,CAAC;IAED,KAAK,CAAC,QAAQ,CAAC,GAAW;QACxB,MAAM,QAAQ,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,IAAI,CACrC,IAAI,gBAAgB,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,GAAG,EAAE,GAAG,EAAE,CAAC,CACxD,CAAC;QACF,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,iBAAiB,EAAE,CAAC;QACtD,IAAI,CAAC,IAAI;YAAE,MAAM,IAAI,KAAK,CAAC,2BAA2B,GAAG,EAAE,CAAC,CAAC;QAC7D,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAC1B,CAAC;IAED;;;OAGG;IACH,KAAK,CAAC,YAAY,CAChB,SAAiB,EACjB,GAAW,EACX,WAAmB;QAEnB,MAAM,SAAS,GAAG,IAAI,CAAC,cAAc,CAAC,GAAG,EAAE,WAAW,CAAC,CAAC;QACxD,MAAM,IAAI,CAAC,MAAM,CAAC,IAAI,CACpB,IAAI,iBAAiB,CAAC;YACpB,MAAM,EAAE,IAAI,CAAC,MAAM;YACnB,UAAU,EAAE,GAAG,IAAI,CAAC,MAAM,IAAI,SAAS,EAAE;YACzC,GAAG,EAAE,SAAS;YACd,WAAW,EAAE,kBAAkB;SAChC,CAAC,CACH,CAAC;QACF,OAAO,SAAS,CAAC;IACnB,CAAC;CACF"}

package/dist/shared/types.d.ts

@@ -0,0 +1,7 @@
+export type CMSProvider = 'contentful' | 'sanity';
+export interface S3Config {
+    bucket: string;
+    region: string;
+    accessKeyId: string;
+    secretAccessKey: string;
+}

package/dist/shared/types.js

@@ -0,0 +1,2 @@
+export {};
+//# sourceMappingURL=types.js.map

package/dist/shared/types.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.js","sourceRoot":"","sources":["../../src/shared/types.ts"],"names":[],"mappings":""}

package/package.json

@@ -0,0 +1,53 @@
+{
+  "name": "@tandem-language-exchange/content-store",
+  "version": "1.0.0",
+  "type": "module",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "exports": {
+    ".": {
+      "import": "./dist/index.js",
+      "types": "./dist/index.d.ts"
+    }
+  },
+  "files": [
+    "dist/index.*",
+    "dist/sdk/",
+    "dist/shared/"
+  ],
+  "engines": {
+    "npm": "^11.3.0",
+    "node": "^24.1.0"
+  },
+  "scripts": {
+    "build": "tsc",
+    "dev": "tsx watch src/server/cli.ts serve",
+    "start": "node dist/server/cli.js serve",
+    "sync": "tsx src/server/cli.ts sync",
+    "sync-appconfig:staging": "sync-appconfig env=staging project=content-store",
+    "sync-appconfig:production": "sync-appconfig env=production project=content-store",
+    "sync-appconfig:local": "sync-appconfig env=local project=content-store",
+    "sync:contentful": "tsx src/server/cli.ts sync --cms contentful",
+    "sync:sanity": "tsx src/server/cli.ts sync --cms sanity",
+    "fetch": "tsx src/server/cli.ts fetch",
+    "query": "tsx src/server/cli.ts query",
+    "lint": "eslint .",
+    "lint:fix": "eslint --fix ."
+  },
+  "dependencies": {
+    "@aws-sdk/client-s3": "^3.700.0",
+    "@sanity/client": "^7.20.0",
+    "commander": "^13.1.0",
+    "contentful": "11.10.3",
+    "dotenv": "16.4.5",
+    "express": "^5.1.0",
+    "nodemon": "^3.1.14"
+  },
+  "devDependencies": {
+    "@tandem-web/web-azure-appconfig-sync": "1.0.9",
+    "@types/express": "^5.0.2",
+    "@types/node": "22.0.0",
+    "tsx": "^4.19.0",
+    "typescript": "^5.9.3"
+  }
+}