@3sixty/node 0.0.0-stage → 0.20.0-next.0

package/docs/commands/vault-key-create.md

@@ -0,0 +1,58 @@
+# vault-key-create
+
+Create a vault key for an identity.
+
+## Tenancy
+
+Available in single-tenant and multi-tenant mode, with the same behaviour in both.
+
+## Usage
+
+```text
+vault-key-create: Create a vault key for an identity
+
+env-prefix: (string, optional)
+Prefix to use for standard .env files e.g. TWIN_.
+
+identity: (string, did, required)
+The DID to associate the key with.
+
+key-id: (string, required)
+The ID of the key.
+
+key-type: (string, default: 'Ed25519', optional, options: [Ed25519, ChaCha20Poly1305])
+The type of the key.
+
+overwrite-mode: (string, default: 'skip', optional, options: [skip, overwrite, error])
+The mode to use when a key with the same id already exists.
+
+load-env: (string, optional)
+Comma separated list of paths to .env files to read input parameters from.
+
+output-json: (string, optional)
+Path to a .json file to store the command output.
+
+output-env: (string, optional)
+Path to a .env file to store the command output.
+
+output-env-prefix: (string, optional)
+Prefix to use for variables in the output .env file.
+
+Example: vault-key-create --identity="did:iota:...." --key-id="my-key" --key-type="Ed25519"
+```
+
+## Examples
+
+Create the authentication signing key for the node:
+
+```shell
+3sixty-node vault-key-create --load-env="node-identity.env" --identity=!NODE_DID --key-type=Ed25519 --key-id=!TWIN_AUTH_SIGNING_KEY_ID --overwrite-mode=skip --output-json="node-auth-key.json" --output-env="node-auth-key.env"
+```
+
+Create a blob storage encryption key for the organisation:
+
+```shell
+3sixty-node vault-key-create --load-env="organization-identity.env" --identity=!ORGANIZATION_DID --key-type=ChaCha20Poly1305 --key-id=!TWIN_BLOB_STORAGE_ENCRYPTION_KEY_ID --overwrite-mode=skip --output-json="organization-blob-encryption.json"
+```
+
+See the [usage guide](../usage.md) for running the node and the full list of commands.

package/docs/commands/vault-key-import.md

@@ -0,0 +1,43 @@
+# vault-key-import
+
+Import a vault key for an identity.
+
+## Tenancy
+
+Available in single-tenant and multi-tenant mode, with the same behaviour in both.
+
+## Usage
+
+```text
+vault-key-import: Import a vault key for an identity
+
+env-prefix: (string, optional)
+Prefix to use for standard .env files e.g. TWIN_.
+
+identity: (string, did, required)
+The DID to associate the key with.
+
+key-id: (string, required)
+The ID of the key.
+
+key-type: (string, default: 'Ed25519', optional, options: [Ed25519, ChaCha20Poly1305])
+The type of the key.
+
+private-key-hex: (string, required)
+The private key in hex format.
+
+load-env: (string, optional)
+Comma separated list of paths to .env files to read input parameters from.
+
+Example: vault-key-import --identity="did:iota:...." --key-id="my-key" --key-type=Ed25519 --private-key-hex="15...a2"
+```
+
+## Examples
+
+Import an existing authentication signing key:
+
+```shell
+3sixty-node vault-key-import --load-env="node-identity.env,node-auth-key.env" --identity=!NODE_DID --key-id=!TWIN_AUTH_SIGNING_KEY_ID --key-type=!KEY_TYPE --private-key-hex=!PRIVATE_KEY_HEX
+```
+
+See the [usage guide](../usage.md) for running the node and the full list of commands.

package/docs/commands/vault-key-remove.md

@@ -0,0 +1,37 @@
+# vault-key-remove
+
+Remove a vault key for an identity.
+
+## Tenancy
+
+Available in single-tenant and multi-tenant mode, with the same behaviour in both.
+
+## Usage
+
+```text
+vault-key-remove: Remove a vault key for an identity
+
+env-prefix: (string, optional)
+Prefix to use for standard .env files e.g. TWIN_.
+
+identity: (string, did, required)
+The DID the key is associated with.
+
+key-id: (string, required)
+The ID of the key.
+
+load-env: (string, optional)
+Comma separated list of paths to .env files to read input parameters from.
+
+Example: vault-key-remove --identity="did:iota:...." --key-id="my-key"
+```
+
+## Examples
+
+Remove a vault key:
+
+```shell
+3sixty-node vault-key-remove --load-env="node-identity.env" --identity=!NODE_DID --key-id="my-key"
+```
+
+See the [usage guide](../usage.md) for running the node and the full list of commands.

package/docs/commands/vault-key-update.md

@@ -0,0 +1,40 @@
+# vault-key-update
+
+Replace the key material of an existing vault key for an identity.
+
+## Tenancy
+
+Available in single-tenant and multi-tenant mode, with the same behaviour in both. The key keeps its existing type, only the key material is replaced.
+
+## Usage
+
+```text
+vault-key-update: Replace the key material of an existing vault key for an identity
+
+env-prefix: (string, optional)
+Prefix to use for standard .env files e.g. TWIN_.
+
+identity: (string, did, required)
+The DID the key is associated with.
+
+key-id: (string, required)
+The ID of the key.
+
+private-key-hex: (string, required)
+The replacement private key in hex format, the key type is kept.
+
+load-env: (string, optional)
+Comma separated list of paths to .env files to read input parameters from.
+
+Example: vault-key-update --identity="did:iota:...." --key-id="my-key" --private-key-hex="15...a2"
+```
+
+## Examples
+
+Replace the material of an existing key:
+
+```shell
+3sixty-node vault-key-update --load-env="node-identity.env" --identity=!NODE_DID --key-id="my-key" --private-key-hex="0x..."
+```
+
+See the [usage guide](../usage.md) for running the node and the full list of commands.

package/docs/configuration.md

@@ -0,0 +1,234 @@
+# Node Configuration
+
+## Available Example Configurations
+
+Four ready-to-use example files are provided in `apps/node/`. Copy the relevant file to `.env` as a starting point:
+
+| File                                | Role                                                                      |
+| ----------------------------------- | ------------------------------------------------------------------------- |
+| `.env.example-local-entity-storage` | Full-featured node using local file/entity storage                        |
+| `.env.example-aws-ec2`              | Full-featured node deployed on Amazon AWS EC2 with DynamoDB               |
+| `.env.example-iota-testnet`         | Full-featured node connected to the IOTA testnet                          |
+| `.env.example-federated-catalogue`  | Standalone federated catalogue server (no dataspace or rights management) |
+
+## Extension Loading
+
+The 3Sixty Node supports loading extensions dynamically to extend its functionality. Extensions can be loaded from multiple sources using protocol-based syntax.
+
+### Extension Loading Syntax
+
+Extensions are configured via the `TWIN_EXTENSIONS` environment variable. Multiple extensions can be specified as a comma-separated list.
+
+#### Local File Extensions
+
+Load extensions from the local filesystem:
+
+```bash
+# Relative path
+TWIN_EXTENSIONS="./my-extension.mjs"
+
+# Absolute path
+TWIN_EXTENSIONS="/path/to/extension.mjs"
+```
+
+#### NPM Package Extensions
+
+Load extensions from npm packages. The package will be automatically downloaded and installed if not already cached:
+
+```bash
+# Basic npm package
+TWIN_EXTENSIONS="npm:@3sixty/identity-management-service"
+
+# Scoped package
+TWIN_EXTENSIONS="npm:@org/custom-extension"
+```
+
+**Note:** Version pinning (e.g., `npm:@3sixty/pkg@1.2.3`) is planned for a future release. Currently, the latest version is installed.
+
+#### HTTPS URL Extensions
+
+Load extensions from remote HTTPS URLs:
+
+```bash
+TWIN_EXTENSIONS="https://example.twin.org/extensions/my-module.mjs"
+```
+
+**⚠️ SECURITY WARNING:** Loading code from HTTPS URLs introduces security risks. Only load extensions from trusted sources that you control or verify.
+
+Security features:
+
+- Only HTTPS is allowed (HTTP is rejected)
+- Default 10 MB size limit to prevent DoS attacks
+- Downloaded modules are cached locally
+- Security warning is displayed when loading
+
+#### Multiple Extensions
+
+Load multiple extensions from different sources:
+
+```bash
+TWIN_EXTENSIONS="@3sixty/identity-management-service,npm:@3sixty/custom-service,./local-extension.mjs,https://example.com/remote.mjs"
+```
+
+### Extension Configuration Options
+
+#### Maximum Download Size
+
+Configure the maximum size (in MB) for HTTPS extension downloads:
+
+```bash
+TWIN_EXTENSIONS_MAX_SIZE_MB=10  # Default is 10 MB
+```
+
+#### Cache Management
+
+By default, downloaded extensions are cached in `.tmp/extensions/` directory to speed up subsequent startups. To clear the cache on each startup:
+
+```bash
+TWIN_EXTENSIONS_CLEAR_CACHE=true  # Default is false
+```
+
+#### Custom Cache Directory
+
+You can configure a custom directory for extension caching:
+
+```bash
+TWIN_EXTENSIONS_CACHE_DIRECTORY="cache"  # Default is ".tmp"
+```
+
+This allows you to:
+
+- Use a dedicated partition with more space
+- Separate cache by environment (dev/staging/prod)
+- Follow corporate storage policies
+- Create persistent cache directories for better performance
+
+### Extension API
+
+Extensions must export specific lifecycle methods to integrate with the node:
+
+#### `extensionInitialise(envVars, nodeEngineConfig)`
+
+Called during configuration phase. Allows the extension to modify environment variables and engine configuration.
+
+```typescript
+export async function extensionInitialise(
+  envVars: INodeEnvironmentVariables,
+  nodeEngineConfig: INodeEngineConfig
+): Promise<void> {
+  // Modify configuration here
+}
+```
+
+#### `extensionInitialiseEngine(engineCore)`
+
+Called after the engine is created but before the server starts.
+
+```typescript
+export async function extensionInitialiseEngine(engineCore: IEngineCore): Promise<void> {
+  // Register components, set up services, etc.
+}
+```
+
+#### `extensionInitialiseEngineServer(engineCore, engineServer)`
+
+Called after the engine server is created but before it starts listening.
+
+```typescript
+export async function extensionInitialiseEngineServer(
+  engineCore: IEngineCore,
+  engineServer: IEngineServer
+): Promise<void> {
+  // Add routes, configure server, etc.
+}
+```
+
+#### `extensionShutdown()`
+
+Called when the node is shutting down. Clean up resources here.
+
+```typescript
+export async function extensionShutdown(): Promise<void> {
+  // Clean up resources
+}
+```
+
+### Example Extension
+
+```typescript
+// my-extension.mjs
+export async function extensionInitialise(envVars, nodeEngineConfig) {
+  console.log('Extension initializing with config');
+}
+
+export async function extensionInitialiseEngine(engineCore) {
+  engineCore.logInfo('Custom extension loaded');
+}
+
+export async function extensionInitialiseEngineServer(engineCore, engineServer) {
+  // Add custom routes
+  engineServer.addRestRoute({
+    path: '/custom/endpoint',
+    method: 'GET',
+    handler: async () => ({ body: { message: 'Hello from extension!' } })
+  });
+}
+
+export async function extensionShutdown() {
+  console.log('Extension shutting down');
+}
+```
+
+### Cache Directory Structure
+
+Downloaded extensions are stored in:
+
+```text
+<execution-directory>/<cache-directory>/extensions/
+  ├── npm/                      # NPM packages
+  │   └── node_modules/
+  │       └── @3sixty/...
+  └── https/                    # HTTPS downloads
+      └── <hashed-filename>.mjs
+```
+
+Where `<cache-directory>` is:
+
+- `.tmp` by default
+- Configurable via `TWIN_EXTENSIONS_CACHE_DIRECTORY`
+
+Examples:
+
+- Default: `./.tmp/extensions/`
+- Custom: `./cache/extensions/` (if `TWIN_EXTENSIONS_CACHE_DIRECTORY="cache"`)
+- Absolute: `/var/cache/twin-node/extensions/` (if `TWIN_EXTENSIONS_CACHE_DIRECTORY="/var/cache/twin-node"`)
+
+### Security Best Practices
+
+1. **Prefer npm: protocol** over https: for third-party extensions (npm has package verification and malware scanning)
+2. **Use local files** (`./extension.mjs`) for maximum security and control
+3. **Verify HTTPS sources** before using - only load from domains you trust
+4. **Review extension code** before deployment, especially for HTTPS sources
+5. **Monitor cache directory** - review what's been downloaded to `.tmp/extensions/`
+6. **Use environment-specific configuration** - different extensions for dev/staging/prod
+
+### Troubleshooting
+
+#### Extension fails to load
+
+- Check that npm is installed and accessible in PATH (required for npm: protocol)
+- Verify the extension file/URL is accessible
+- Check `.tmp/extensions/` cache for partially downloaded files
+- Review node logs for specific error messages
+
+#### Clear extension cache manually
+
+```bash
+rm -rf .tmp/extensions
+```
+
+#### NPM package not found
+
+- Verify the package name is correct
+- Check network connectivity
+- Ensure npm registry is accessible

package/docs/deployment-docker.md

@@ -0,0 +1,142 @@
+# Node Deployment Docker
+
+The 3Sixty Node can be run as a Docker container, either from the published image or from an image you build yourself.
+
+## Published Image
+
+Images are published to GitHub Packages as [ghcr.io/3sixtyglobal/node](https://github.com/3sixtyglobal/node/pkgs/container/3sixty-node) for `linux/amd64` and `linux/arm64`, with the following tags:
+
+| Tag       | Description                                      |
+| --------- | ------------------------------------------------ |
+| `latest`  | The most recent production release.              |
+| `next`    | The most recent prerelease.                      |
+| `<x.y.z>` | A specific version, for example `0.10.1-next.6`. |
+
+```shell
+docker pull ghcr.io/3sixtyglobal/node:latest
+```
+
+## Building the Image
+
+The dockerfile is in `apps/node/deploy/dockerfile`. It builds `@3sixty/node-core` from the repository source and packs it, then installs that package and the other dependencies with pnpm, merges the translation messages, and removes the development dependencies. The repository root forms the build context, and `apps/node/deploy/dockerfile.dockerignore` keeps your local `node_modules`, build output and `.env` files out of the image.
+
+Build the image from the repository root:
+
+```shell
+docker build -t 3sixty-node -f apps/node/deploy/dockerfile . --load
+```
+
+> **Note**: The `--load` flag is required when using Docker Buildx to ensure the image is loaded into your local Docker registry. Without it, the image will only exist in the build cache.
+
+To build for both of the published platforms use a Buildx builder with the `docker-container` driver:
+
+```shell
+docker buildx create --name 3sixty-multiplatform-builder --driver docker-container --use
+docker buildx build --platform linux/amd64,linux/arm64 -t 3sixty-node -f apps/node/deploy/dockerfile .
+```
+
+The examples below use the locally built `3sixty-node` image, replace it with `ghcr.io/3sixtyglobal/node:<tag>` to use the published image.
+
+## Configuration
+
+The node reads its configuration from `/app/.env` in the container, and from environment variables, which take precedence over the file. The image does not contain a `.env`, so start from one of the example configurations described in [Node Configuration](./configuration.md) and mount it into the container:
+
+```shell
+-v /home/twin-node/.env:/app/.env:ro
+```
+
+Mount the file rather than passing it with `--env-file`. Docker does not remove the quotes around values in an env file, so the quoted values used in the example configurations would be read incorrectly.
+
+The image sets the following environment variables, which override the same settings in `/app/.env`:
+
+| Variable                 | Value             | Description                                    |
+| ------------------------ | ----------------- | ---------------------------------------------- |
+| `TWIN_HOST`              | `0.0.0.0`         | Listen on all interfaces inside the container. |
+| `TWIN_PORT`              | `3000`            | The port the API server listens on.            |
+| `TWIN_STORAGE_FILE_ROOT` | `/twin-node/data` | The root directory for `file` storage.         |
+
+Change any of them with `-e`, for example `-e TWIN_PORT=8080`.
+
+## Data Persistence
+
+When storage is configured to use the `file` connector, its data and the engine state in `engine-state.json` are written to `/twin-node/data`. Mount a volume or a host directory there so that the data is kept when the container is replaced:
+
+```shell
+-v /home/twin-node/data:/twin-node/data
+```
+
+The same data directory must be mounted for the bootstrap commands and for the running node.
+
+## Bootstrapping
+
+The node will not start until it has been bootstrapped. A container started without a node identity stops with:
+
+```shell
+The node identity is enabled in config but is not set, please set it using the "node-identity-set" CLI command.
+```
+
+The CLI commands run from the same image by passing the command after `node src/index.js`, with the same configuration and data directory that the node will use. For a development node the [bootstrap-dev](./commands/bootstrap-dev.md) command creates the node identity, organisation identity and admin user in a single step:
+
+```shell
+docker run --rm \
+  -v /home/twin-node/.env:/app/.env:ro \
+  -v /home/twin-node/.env.bootstrap-dev:/app/.env.bootstrap-dev:ro \
+  -v /home/twin-node/data:/twin-node/data \
+  3sixty-node node src/index.js bootstrap-dev --load-env=.env.bootstrap-dev
+```
+
+The values it reads are described in [bootstrap-dev](./commands/bootstrap-dev.md), and `.env.example-bootstrap-dev` is a starting point for the file. Any value that is not provided is generated. Generated mnemonics and passwords are only shown in the command output, so record them.
+
+To bootstrap a production node step by step, or to manage identities, tenants and users later, run the relevant commands from [Node Usage](./usage.md) in the same way. Use `help` to list the commands available in the image:
+
+```shell
+docker run --rm 3sixty-node node src/index.js help
+```
+
+## Running the Node
+
+Once bootstrapped, start the node with the same configuration and data directory:
+
+```shell
+docker run -d --name 3sixty-node \
+  -v /home/twin-node/.env:/app/.env:ro \
+  -v /home/twin-node/data:/twin-node/data \
+  -p 3000:3000 \
+  3sixty-node
+```
+
+Follow the output with `docker logs -f 3sixty-node`. When the node is ready you should see:
+
+```shell
+INFO [2026-09-24T04:57:23.320Z] EngineCore Engine has started
+INFO [2026-09-24T04:57:23.346Z] FastifyWebServer Building Web Server
+INFO [2026-09-24T04:57:23.384Z] FastifyWebServer Starting Web Server with binding "0.0.0.0" on port "3000"
+INFO [2026-09-24T04:57:23.392Z] FastifyWebServer The Web Server started on http://127.0.0.1:3000
+```
+
+You should now be able to access the server in the browser at [http://localhost:3000/info](http://localhost:3000/info), which returns something similar to:
+
+```json
+{
+  "name": "3Sixty Node",
+  "version": "0.10.1-next.6"
+}
+```
+
+The request and response are also shown in the logs:
+
+```shell
+INFO [2026-09-24T04:57:25.150Z] LoggingProcessor ===> GET /info {}
+INFO [2026-09-24T04:57:25.150Z] LoggingProcessor <=== 200 GET /info duration: 722µs {"body":{"name":"3Sixty Node","version":"0.10.1-next.6"}}
+```
+
+## Stopping the Node
+
+The node responds to the standard termination signals, so `docker stop 3sixty-node` stops the components gracefully before the container exits:
+
+```shell
+INFO [2026-09-24T05:00:09.380Z] EngineCore Components have stopped
+INFO [2026-09-24T05:00:09.380Z] EngineCore Engine has stopped
+```
+
+Starting the container again with `docker start 3sixty-node` reuses the existing engine state, so the bootstrap does not need to be repeated.