@pixlcore/xyplug-s3 1.0.0

package/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Joseph Huckaby
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,359 @@
+<p align="center"><img src="https://raw.githubusercontent.com/pixlcore/xyplug-s3/refs/heads/main/logo.png" height="128" alt="S3"/></p>
+<h1 align="center">S3 Toolbox</h1>
+
+An AWS S3 event plugin for the [xyOps Workflow Automation System](https://xyops.io). It can upload, download, move, copy, list, grep, and delete files in S3 buckets, and is designed to work naturally with xyOps job input and output files.
+
+## Requirements
+
+- `Node.js`
+- `npx`
+- AWS credentials with permission to access your target S3 bucket
+
+## Environment Variables
+
+Create a [Secret Vault](https://xyops.io/docs/secrets) in xyOps and assign this Plugin to it. Add:
+
+- `AWS_ACCESS_KEY_ID`
+- `AWS_SECRET_ACCESS_KEY`
+
+These should be credentials for an IAM user or role with the minimum S3 permissions needed for your workflow.
+
+## Data Collection
+
+This plugin does not collect, store, or transmit user data anywhere except to the configured AWS S3 service endpoint. AWS may log requests according to their own policies.
+
+## Overview
+
+The plugin exposes a toolset with seven tools:
+
+- [Upload Files](#upload-files)
+- [Download Files](#download-files)
+- [Delete Files](#delete-files)
+- [Move Files](#move-files)
+- [Copy Files](#copy-files)
+- [List Files](#list-files)
+- [Grep Files](#grep-files)
+
+All tools operate on a single configured bucket and region per job run.
+
+## Common Parameters
+
+These parameters are always present regardless of the selected tool:
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `Region ID` | Yes | AWS region containing the bucket, e.g. `us-east-1`. |
+| `Bucket Name` | Yes | The S3 bucket to operate on. |
+
+## General Notes
+
+- `Remote Path` values are S3 key prefixes. Leave blank to operate at the bucket root.
+- For folder-like prefixes, it is best to include a trailing slash, e.g. `incoming/` or `logs/2026/03/`.
+- `Filename Pattern` uses glob-style matching and is applied to filenames, not full directory paths.
+- `Older Than` and `Newer Than` can be specified as raw seconds or friendly text like `7 days`, `12 hours`, or `30 minutes`.
+- `Maximum Files` is especially useful when combined with `Sort`.
+- Progress is reported back to xyOps while large transfers are in progress.
+
+## Tool Reference
+
+### Upload Files
+
+Uploads local files into S3.
+
+In normal xyOps usage, if you leave `Local Path` blank, the plugin uploads files from the job temp directory. This means the easiest way to upload files to S3 is to attach them as job inputs or pass them from upstream workflow steps. The user does not need to know where those files live on disk, because xyOps and xySat handle that automatically.
+
+If you do want to upload from a specific directory on the server, you can set `Local Path` explicitly.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `Local Path` | No | Base local directory to upload from. Leave blank to use the job temp directory and any job input files already placed there by xyOps. |
+| `Filename Pattern` | No | Optional glob to limit which local files are uploaded. |
+| `Remote Path` | No | Base S3 prefix to upload into. Leave blank for the bucket root. |
+| `Compress Files (Gzip)` | No | Gzip-compress files during upload. The plugin also appends `.gz` to the destination filename. |
+| `Custom S3 Params` | No | JSON object passed to S3 for uploaded objects, for settings such as `ACL`, `StorageClass`, `ContentType`, or `CacheControl`. |
+
+Notes:
+
+- Compression is streamed directly to S3. Files are not written to a temporary `.gz` file first, so disk usage stays low.
+- The response returns the uploaded file list in job `data`.
+
+Example custom S3 params:
+
+```json
+{
+	"ACL": "public-read",
+	"StorageClass": "STANDARD_IA"
+}
+```
+
+Example static asset upload params:
+
+```json
+{
+	"ContentType": "text/css",
+	"CacheControl": "public, max-age=86400"
+}
+```
+
+### Download Files
+
+Downloads files from S3 to the local machine running the job.
+
+By default, if you leave `Local Path` blank, files are downloaded into the job temp directory. Also by default, `Attach Files` is enabled, which means the downloaded files are attached to the job output and become available to downstream workflow steps.
+
+You can override either behavior:
+
+- Set `Local Path` to save files somewhere else.
+- Uncheck `Attach Files` if you want local files only and do not want them attached to job outputs.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `Remote Path` | No | Base S3 prefix to download from. Leave blank for the bucket root. |
+| `Filename Pattern` | No | Optional glob to limit which remote files are downloaded. |
+| `Local Path` | No | Destination directory on local disk. Leave blank to use the xyOps job temp directory. |
+| `Decompress Files (Gunzip)` | No | Automatically gunzip downloaded files and strip a trailing `.gz` from the local filename when present. |
+| `Delete Files` | No | Delete the remote S3 files after they are successfully downloaded. |
+| `Attach Files` | No | Attach downloaded files to the job output. Enabled by default. |
+| `Maximum Files` | No | Limit how many files are downloaded. `0` means no limit. |
+| `Sort Files` | No | Sort before downloading: `newest`, `oldest`, `largest`, or `smallest`. Useful with `Maximum Files`. |
+
+Notes:
+
+- This tool returns both the matched file metadata and the total transferred byte count in job `data`.
+- `Delete Files` turns the tool into a download-and-consume action, which is useful for queue-like workflows.
+
+### Delete Files
+
+Deletes files from S3.
+
+Use this tool when you want to purge objects matching a prefix, filename pattern, age filter, or size/date ordering strategy.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `Remote Path` | No | Base S3 prefix to delete from. Leave blank for the bucket root. |
+| `Filename Pattern` | No | Optional glob to limit which remote files are deleted. |
+| `Older Than` | No | Delete only files older than this relative time, e.g. `7 days` or `3600`. |
+| `Maximum Files` | No | Limit how many files are deleted. `0` means no limit. |
+| `Sort Files` | No | Sort before deleting: `oldest`, `newest`, `largest`, or `smallest`. Useful with `Maximum Files`. |
+| `Dry Run` | No | Preview the matched files without actually deleting anything. |
+
+Notes:
+
+- Start with `Dry Run` enabled when building destructive workflows.
+- This tool returns the matched file metadata and total byte count in job `data`.
+
+### Move Files
+
+Moves files from one S3 path to another, optionally across buckets.
+
+This is ideal for archive pipelines, inbox-to-processed flows, or lifecycle-style workflows controlled by xyOps.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `Remote Path` | No | Base S3 prefix to move from. |
+| `Filename Pattern` | No | Optional glob to limit which remote files are moved. |
+| `Destination Path` | No | Base S3 prefix to move files into. Leave blank to preserve the relative path at the destination root. |
+| `Destination Bucket` | No | Optional destination bucket. Leave blank to keep the source bucket. |
+| `Maximum Files` | No | Limit how many files are moved. `0` means no limit. |
+| `Sort Files` | No | Sort before moving: `oldest`, `newest`, `largest`, or `smallest`. |
+| `Custom S3 Params` | No | JSON object applied to destination objects, e.g. `ACL` or `StorageClass`. |
+| `Dry Run` | No | Preview the matched files without actually moving them. |
+
+Notes:
+
+- Under the hood, an S3 move is a copy followed by a delete.
+- If you need destination metadata like `ACL` or `StorageClass`, specify it in `Custom S3 Params`.
+- This tool returns the matched file metadata and total byte count in job `data`.
+
+### Copy Files
+
+Copies files from one S3 path to another, optionally across buckets.
+
+Use this for replication, fan-out, staging, publishing, or storage-class transitions where you want to keep the source objects intact.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `Remote Path` | No | Base S3 prefix to copy from. |
+| `Filename Pattern` | No | Optional glob to limit which remote files are copied. |
+| `Destination Path` | No | Base S3 prefix to copy files into. Leave blank to preserve the relative path at the destination root. |
+| `Destination Bucket` | No | Optional destination bucket. Leave blank to keep the source bucket. |
+| `Maximum Files` | No | Limit how many files are copied. `0` means no limit. |
+| `Sort Files` | No | Sort before copying: `oldest`, `newest`, `largest`, or `smallest`. |
+| `Custom S3 Params` | No | JSON object applied to destination objects, e.g. `ACL` or `StorageClass`. |
+
+Notes:
+
+- If you want copied objects to have explicit metadata or a different storage class, specify it in `Custom S3 Params`.
+- This tool returns the matched file metadata and total byte count in job `data`.
+
+### List Files
+
+Lists files in S3 and returns metadata without downloading object contents.
+
+This is useful for audits, inventory flows, preflight checks, or driving downstream workflow logic from S3 state.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `Remote Path` | No | Base S3 prefix to list from. Leave blank for the bucket root. |
+| `Filename Pattern` | No | Optional glob to limit which remote files are included. |
+| `Older Than` | No | Include only files older than this relative time. |
+| `Newer Than` | No | Include only files newer than this relative time. |
+| `Maximum Files` | No | Limit how many files are returned. `0` means no limit. |
+| `Sort Files` | No | Sort before returning metadata: `newest`, `oldest`, `largest`, or `smallest`. |
+
+Output:
+
+- `data.files`: Array of file objects containing `key`, `size`, and `mtime`
+- `data.bytes`: Total bytes across all matched files
+
+### Grep Files
+
+Searches inside files stored in S3 using a regular expression.
+
+This tool can stream through large remote files, including gzip-compressed files, and extract only the matching lines. It is especially useful for log processing and incident response workflows.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `Remote Path` | No | Base S3 prefix to search under. Leave blank for the bucket root. |
+| `Filename Pattern` | No | Optional glob to limit which remote files are searched. |
+| `Match Pattern (Regex)` | Yes | JavaScript regular expression pattern used to match lines inside each file. |
+| `Decompress Files (Gunzip)` | No | Automatically gunzip files while searching. Enable this for `.gz` log archives. |
+| `Older Than` | No | Search only files older than this relative time. |
+| `Newer Than` | No | Search only files newer than this relative time. |
+| `Maximum Matches` | No | Stop after this many matching lines have been found. |
+| `Output Format` | No | Return results as structured JSON (`data`) or write them into an attached text file (`file`). |
+
+Output behavior:
+
+- `JSON Data`: Returns `data.count` and `data.matches`, where each match contains the matched `line` and its source `file` metadata.
+- `Text File`: Writes all matched lines into `matched-lines.txt` and attaches that file to the job output.
+
+Notes:
+
+- Matching is line-based.
+- Searching is streamed, so memory usage stays low even for very large objects.
+- Compressed `.gz` files can be searched without creating decompressed files on disk first.
+
+## Custom S3 Params
+
+The `Upload Files`, `Move Files`, and `Copy Files` tools support `Custom S3 Params`, which is a raw JSON object passed through to S3 when creating destination objects.
+
+Common examples:
+
+```json
+{
+	"ACL": "public-read",
+	"StorageClass": "STANDARD_IA"
+}
+```
+
+```json
+{
+	"ContentType": "application/json",
+	"CacheControl": "public, max-age=300"
+}
+```
+
+Useful keys include:
+
+- `ACL`
+- `StorageClass`
+- `ContentType`
+- `CacheControl`
+- `ContentDisposition`
+- `ContentEncoding`
+- `Metadata`
+
+This is particularly useful when:
+
+- publishing static assets from a workflow
+- copying or moving objects into a lower-cost storage tier
+- setting HTTP headers on web-facing objects
+- forcing a fresh metadata policy on copied objects
+
+## Local Testing
+
+When invoked by xyOps, the plugin expects a single JSON document on STDIN using the xyOps wire protocol. You can simulate this locally by piping JSON into `node index.js`.
+
+Example upload test using the current directory as the local source:
+
+```json
+{
+	"xy": 1,
+	"params": {
+		"region": "us-east-1",
+		"bucket": "my-bucket",
+		"tool": "uploadFiles",
+		"localPath": "./",
+		"filespec": "*.txt",
+		"remotePath": "test-upload/"
+	}
+}
+```
+
+Example download test:
+
+```json
+{
+	"xy": 1,
+	"params": {
+		"region": "us-east-1",
+		"bucket": "my-bucket",
+		"tool": "downloadFiles",
+		"remotePath": "test-upload/",
+		"localPath": "./downloads/",
+		"attach": false
+	}
+}
+```
+
+Example grep test on compressed logs:
+
+```json
+{
+	"xy": 1,
+	"params": {
+		"region": "us-east-1",
+		"bucket": "my-bucket",
+		"tool": "grepFiles",
+		"remotePath": "logs/",
+		"filespec": "*.log.gz",
+		"match": "ERROR|FATAL",
+		"decompress": true,
+		"output": "data",
+		"maxLines": 100
+	}
+}
+```
+
+Run any of the above like this:
+
+```sh
+export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
+export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_ACCESS_KEY"
+cat sample.json | node index.js
+```
+
+Or without a file:
+
+```sh
+echo '{"xy":1,"params":{"region":"us-east-1","bucket":"my-bucket","tool":"listFiles","remotePath":"incoming/","filespec":"*.csv"}}' | node index.js
+```
+
+## Output Summary
+
+Depending on the selected tool, the plugin returns structured job `data` such as:
+
+- uploaded file paths
+- downloaded / moved / copied / deleted file metadata
+- total byte counts
+- grep match counts and matched lines
+
+For file-producing tools, the plugin can also attach local files to the xyOps job output:
+
+- `Download Files`: attaches downloaded files when `Attach Files` is enabled
+- `Grep Files`: attaches `matched-lines.txt` when `Output Format` is set to `Text File`
+
+## License
+
+MIT

package/index.js

@@ -0,0 +1,224 @@
+#!/usr/bin/env node
+
+// S3 Utility Plugin for xyOps
+// Copyright (c) 2026 PixlCore LLC
+// MIT License
+
+const fs = require('fs');
+const Path = require('path');
+const S3 = require('s3-api');
+const Perf = require('pixl-perf');
+const picomatch = require('picomatch');
+
+const app = {
+	
+	async run() {
+		// read job from STDIN
+		const chunks = [];
+		for await (const chunk of process.stdin) chunks.push(chunk);
+		this.job = JSON.parse( chunks.join('').trim() );
+		this.params = this.job.params;
+		
+		console.log(`Setting up S3 with bucket: ${this.params.bucket} in region: ${this.params.region}...`);
+		
+		// setup s3 instance
+		this.s3 = new S3({
+			bucket: this.params.bucket,
+			region: this.params.region
+		});
+		
+		// setup logging hook
+		this.s3.attachLogAgent({
+			debug: function(level, msg, data) {
+				console.log( msg );
+			},
+			error: function(code, msg, data) {
+				console.log( "Error: " + code + ": " + msg );
+			}
+		});
+		
+		// setup perf hook
+		this.perf = new Perf();
+		this.perf.begin();
+		this.s3.attachPerfAgent( this.perf );
+		
+		// jump to tool handler
+		let func = 'tool_' + this.params.tool;
+		if (!this[func]) return this.fatal('tool', "Unknown tool: " + this.params.tool);
+		
+		// let the handler do the rest of the work
+		await this[func]();
+	},
+	
+	async tool_uploadFiles() {
+		// upload files to s3
+		let { files } = await this.s3.uploadFiles({ 
+			localPath: this.params.localPath || './', 
+			remotePath: this.params.remotePath, 
+			filespec: picomatch.makeRe( this.params.filespec || '*' ),
+			compress: this.params.compress || false,
+			suffix: this.params.compress ? '.gz' : undefined,
+			params: this.params.s3params || {},
+			progress: this.progress
+		});
+		
+		this.sendFinalResponse({ 
+			code: 0, 
+			data: { files } 
+		});
+	},
+	
+	async tool_downloadFiles() {
+		// download files from s3
+		let { files, bytes } = await this.s3.downloadFiles({ 
+			remotePath: this.params.remotePath, 
+			localPath: this.params.localPath || './', 
+			filespec: picomatch.makeRe( this.params.filespec || '*' ),
+			decompress: this.params.decompress || false,
+			strip: this.params.decompress ? /\.gz$/ : undefined,
+			max: this.params.max || 0,
+			sort: this.params.sort || '',
+			delete: this.params.delete || false,
+			progress: this.progress
+		});
+		
+		this.sendFinalResponse({ 
+			code: 0,
+			files: this.params.attach ? files.map( file => Path.resolve( this.params.localPath || './', Path.basename(file.key) ) ) : [], 
+			data: { files, bytes } 
+		});
+	},
+	
+	async tool_deleteFiles() {
+		// delete files in s3
+		let { files, bytes } = await this.s3.deleteFiles({ 
+			remotePath: this.params.remotePath, 
+			filespec: picomatch.makeRe( this.params.filespec || '*' ),
+			older: this.params.older || '',
+			max: this.params.max || 0,
+			sort: this.params.sort || '',
+			dry: this.params.dry || false,
+			progress: this.progress
+		});
+		
+		this.sendFinalResponse({ 
+			code: 0,
+			files: files.map( file => Path.basename(file.key) ), 
+			data: { files, bytes } 
+		});
+	},
+	
+	async tool_moveFiles() {
+		// move files in s3
+		let { files, bytes } = await this.s3.moveFiles({ 
+			remotePath: this.params.remotePath, 
+			destPath: this.params.destPath, 
+			bucket: this.params.destBucket || '',
+			filespec: picomatch.makeRe( this.params.filespec || '*' ),
+			max: this.params.max || 0,
+			sort: this.params.sort || '',
+			params: this.params.s3params || {},
+			dry: this.params.dry || false,
+			progress: this.progress
+		});
+		
+		this.sendFinalResponse({ 
+			code: 0,
+			data: { files, bytes } 
+		});
+	},
+	
+	async tool_copyFiles() {
+		// copy files in s3
+		let { files, bytes } = await this.s3.copyFiles({ 
+			remotePath: this.params.remotePath, 
+			destPath: this.params.destPath, 
+			bucket: this.params.destBucket || '',
+			filespec: picomatch.makeRe( this.params.filespec || '*' ),
+			max: this.params.max || 0,
+			sort: this.params.sort || '',
+			params: this.params.s3params || {},
+			progress: this.progress
+		});
+		
+		this.sendFinalResponse({ 
+			code: 0,
+			data: { files, bytes } 
+		});
+	},
+	
+	async tool_listFiles() {
+		// list files in s3
+		let { files, bytes } = await this.s3.list({ 
+			remotePath: this.params.remotePath, 
+			filespec: picomatch.makeRe( this.params.filespec || '*' ),
+			older: this.params.older || '',
+			newer: this.params.newer || '',
+			max: this.params.max || 0,
+			sort: this.params.sort || '',
+			progress: this.progress
+		});
+		
+		this.sendFinalResponse({ 
+			code: 0,
+			data: { files, bytes } 
+		});
+	},
+	
+	async tool_grepFiles() {
+		// grep files in s3
+		let mode = this.params.output;
+		let matches = [];
+		let count = 0;
+		
+		await this.s3.grepFiles({ 
+			remotePath: this.params.remotePath, 
+			filespec: picomatch.makeRe( this.params.filespec || '*' ),
+			match: new RegExp( this.params.match || '.+' ),
+			decompress: this.params.decompress || false,
+			maxLines: this.params.maxLines || 0,
+			older: this.params.older || '',
+			newer: this.params.newer || '',
+			
+			iterator: function(line, file) {
+				count++;
+				if (mode == 'data') matches.push({ file, line });
+				else fs.appendFileSync( 'matched-lines.txt', line + "\n" );
+			}
+		});
+		
+		this.sendFinalResponse({ 
+			code: 0,
+			files: (mode == 'file') ? ['matched-lines.txt'] : null,
+			data: { count, matches } 
+		});
+	},
+	
+	progress(prog) {
+		// send progress updates to xyops
+		if (prog.loaded && prog.total) {
+			console.log( JSON.stringify({ xy: 1, progress: prog.loaded / prog.total }) );
+		}
+	},
+	
+	fatal(code, description) {
+		// Emit an error response and exit.
+		return this.sendFinalResponse({ code, description });
+	},
+	
+	sendFinalResponse(payload) {
+		// Emit a final XYWP message and exit.
+		payload.xy = 1;
+		if (this.perf) {
+			this.perf.end();
+			payload.perf = this.perf.metrics();
+		}
+		process.stdout.write(`${JSON.stringify(payload)}\n`, () => process.exit(0));
+	}
+};
+
+app.run().catch((err) => {
+	// Catch-all handler for unexpected errors.
+	console.error( err );
+	return app.fatal("error", err && err.message ? err.message : "Unknown error");
+});

package/package.json

@@ -0,0 +1,31 @@
+{
+	"name": "@pixlcore/xyplug-s3",
+	"version": "1.0.0",
+	"description": "An AWS S3 utility plugin for the xyOps workflow automation system.",
+	"author": "Joseph Huckaby <jhuckaby@pixlcore.com>",
+	"homepage": "https://github.com/pixlcore/xyplug-s3",
+	"license": "MIT",
+	"bin": {
+		"xyplug-s3": "index.js"
+	},
+	"repository": {
+		"type": "git",
+		"url": "https://github.com/pixlcore/xyplug-s3"
+	},
+	"bugs": {
+		"url": "https://github.com/pixlcore/xyplug-s3/issues"
+	},
+	"keywords": [
+		"xyops",
+		"s3",
+		"aws"
+	],
+	"dependencies": {
+		"picomatch": "4.0.4",
+		"pixl-perf": "^1.0.9",
+		"s3-api": "^2.0.22"
+	},
+	"publishConfig": {
+		"access": "public"
+	}
+}

package/xyops.json

@@ -0,0 +1,445 @@
+{
+	"type": "xypdf",
+	"description": "xyOps Portable Data Object",
+	"version": "1.0",
+	"items": [
+		{
+			"type": "plugin",
+			"data": {
+				"id": "pmn6zec9v4vn5e7s3",
+				"title": "S3 Toolbox",
+				"enabled": true,
+				"type": "event",
+				"command": "npx -y @pixlcore/xyplug-s3@1.0.0",
+				"script": "",
+				"kill": "parent",
+				"notes": "Upload, download, move, copy, list, grep, and delete your files in S3 buckets.",
+				"icon": "aws",
+				"env": { 
+					"AWS_ACCESS_KEY_ID": "Your AWS `accessKeyId` credential.", 
+					"AWS_SECRET_ACCESS_KEY": "Your AWS `secretAccessKey` credential." 
+				},
+				"params": [
+					{
+						"id": "region",
+						"title": "Region ID",
+						"type": "text",
+						"value": "us-east-1",
+						"caption": "The AWS Region ID where your bucket resides, e.g. `us-east-1`.",
+						"required": true
+					},
+					{
+						"id": "bucket",
+						"title": "Bucket Name",
+						"type": "text",
+						"value": "",
+						"caption": "The name of your S3 bucket, e.g. `mybucket01`.",
+						"required": true
+					},
+					{
+						"id": "tool",
+						"title": "Tool Select",
+						"type": "toolset",
+						"caption": "",
+						"data": {
+							"tools": [
+								{
+									"id": "uploadFiles",
+									"title": "Upload Files",
+									"description": "Upload job input files to an S3 bucket.",
+									"fields": [
+										{
+											"id": "localPath",
+											"title": "Local Path",
+											"type": "text",
+											"value": "",
+											"caption": "The local path to find files under.  Leave blank to use the job temp directory and job input files."
+										},
+										{
+											"id": "filespec",
+											"title": "Filename Pattern",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the input files using a glob pattern, applied to the filenames."
+										},
+										{
+											"id": "remotePath",
+											"title": "Remote Path",
+											"type": "text",
+											"value": "",
+											"caption": "The base S3 path to store files under.  Leave blank to store at the root level."
+										},
+										{
+											"id": "compress",
+											"title": "Compress Files (Gzip)",
+											"type": "checkbox",
+											"value": false,
+											"caption": "Optionally gzip-compress all files before upload.  This also adds a `.gz` filename suffix."
+										},
+										{
+											"id": "s3params",
+											"title": "Custom S3 Params",
+											"type": "json",
+											"value": {},
+											"caption": "Optionally specify parameters to the S3 API, for e.g. ACL and Storage Class."
+										}
+									]
+								},
+								{
+									"id": "downloadFiles",
+									"title": "Download Files",
+									"description": "Download S3 files and attach to the job output.",
+									"fields": [
+										{
+											"id": "remotePath",
+											"title": "Remote Path",
+											"type": "text",
+											"value": "",
+											"caption": "The base S3 path to fetch files from.  Leave blank to fetch files at the root level."
+										},
+										{
+											"id": "filespec",
+											"title": "Filename Pattern",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the remote files using a glob pattern, applied to the filenames."
+										},
+										{
+											"id": "localPath",
+											"title": "Local Path",
+											"type": "text",
+											"value": "",
+											"caption": "The local path to save files in.  Leave blank to use the job temp directory."
+										},
+										{
+											"id": "decompress",
+											"title": "Decompress Files (Gunzip)",
+											"type": "checkbox",
+											"value": false,
+											"caption": "Optionally decompress all gzipped files after download.  This also strips the `.gz` filename suffix if found."
+										},
+										{
+											"id": "delete",
+											"title": "Delete Files",
+											"type": "checkbox",
+											"value": false,
+											"caption": "Optionally delete all remote S3 files after successful download."
+										},
+										{
+											"id": "attach",
+											"title": "Attach Files",
+											"type": "checkbox",
+											"value": true,
+											"caption": "Optionally attach all downloaded files to the job output."
+										},
+										{
+											"id": "max",
+											"title": "Maximum Files",
+											"type": "text",
+											"variant": "number",
+											"value": 0,
+											"caption": "Optionally set a maximum number of files to download."
+										},
+										{
+											"id": "sort",
+											"title": "Sort Files",
+											"type": "select",
+											"value": "Newest First [newest], Oldest First [oldest], Largest First [largest], Smallest First [smallest]",
+											"caption": "Optionally sort the files before downloading.  Useful when combined with maximum."
+										}
+									]
+								},
+								{
+									"id": "deleteFiles",
+									"title": "Delete Files",
+									"description": "Delete remote S3 files.",
+									"fields": [
+										{
+											"id": "remotePath",
+											"title": "Remote Path",
+											"type": "text",
+											"value": "",
+											"caption": "The base S3 path to fetch files from.  Leave blank to fetch files at the root level."
+										},
+										{
+											"id": "filespec",
+											"title": "Filename Pattern",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the remote files using a glob pattern, applied to the filenames."
+										},
+										{
+											"id": "older",
+											"title": "Older Than",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the S3 files based on their mod date, so they must be older than the specified number of seconds (or specify a string like `7 days`)."
+										},
+										{
+											"id": "max",
+											"title": "Maximum Files",
+											"type": "text",
+											"variant": "number",
+											"value": 0,
+											"caption": "Optionally set a maximum limit on the number of files to delete."
+										},
+										{
+											"id": "sort",
+											"title": "Sort Files",
+											"type": "select",
+											"value": "Oldest First [oldest], Newest First [newest], Largest First [largest], Smallest First [smallest]",
+											"caption": "Optionally sort the files before deleting.  Useful when combined with maximum."
+										},
+										{
+											"id": "dry",
+											"title": "Dry Run",
+											"type": "checkbox",
+											"value": false,
+											"caption": "When checked, will perform a 'dry run' and not actually perform the deletes."
+										}
+									]
+								},
+								{
+									"id": "moveFiles",
+									"title": "Move Files",
+									"description": "Move remote S3 files to another path and/or bucket.",
+									"fields": [
+										{
+											"id": "remotePath",
+											"title": "Remote Path",
+											"type": "text",
+											"value": "",
+											"caption": "The base S3 path to move files from.  Leave blank to move files from the root level."
+										},
+										{
+											"id": "filespec",
+											"title": "Filename Pattern",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the remote files using a glob pattern, applied to the filenames."
+										},
+										{
+											"id": "destPath",
+											"title": "Destination Path",
+											"type": "text",
+											"value": "",
+											"caption": "The base S3 path to move files to.  Leave blank to keep the same relative path at the destination root."
+										},
+										{
+											"id": "destBucket",
+											"title": "Destination Bucket",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally override the destination bucket name.  Leave blank to use the source bucket configured above."
+										},
+										{
+											"id": "max",
+											"title": "Maximum Files",
+											"type": "text",
+											"variant": "number",
+											"value": 0,
+											"caption": "Optionally set a maximum limit on the number of files to move."
+										},
+										{
+											"id": "sort",
+											"title": "Sort Files",
+											"type": "select",
+											"value": "Oldest First [oldest], Newest First [newest], Largest First [largest], Smallest First [smallest]",
+											"caption": "Optionally sort the files before moving.  Useful when combined with maximum."
+										},
+										{
+											"id": "s3params",
+											"title": "Custom S3 Params",
+											"type": "json",
+											"value": {},
+											"caption": "Optionally specify parameters to the S3 API for the destination objects, for e.g. ACL and Storage Class."
+										},
+										{
+											"id": "dry",
+											"title": "Dry Run",
+											"type": "checkbox",
+											"value": false,
+											"caption": "When checked, will perform a 'dry run' and not actually perform the moves."
+										}
+									]
+								},
+								{
+									"id": "copyFiles",
+									"title": "Copy Files",
+									"description": "Copy remote S3 files to another path and/or bucket.",
+									"fields": [
+										{
+											"id": "remotePath",
+											"title": "Remote Path",
+											"type": "text",
+											"value": "",
+											"caption": "The base S3 path to copy files from.  Leave blank to copy files from the root level."
+										},
+										{
+											"id": "filespec",
+											"title": "Filename Pattern",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the remote files using a glob pattern, applied to the filenames."
+										},
+										{
+											"id": "destPath",
+											"title": "Destination Path",
+											"type": "text",
+											"value": "",
+											"caption": "The base S3 path to copy files to.  Leave blank to keep the same relative path at the destination root."
+										},
+										{
+											"id": "destBucket",
+											"title": "Destination Bucket",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally override the destination bucket name.  Leave blank to use the source bucket configured above."
+										},
+										{
+											"id": "max",
+											"title": "Maximum Files",
+											"type": "text",
+											"variant": "number",
+											"value": 0,
+											"caption": "Optionally set a maximum limit on the number of files to copy."
+										},
+										{
+											"id": "sort",
+											"title": "Sort Files",
+											"type": "select",
+											"value": "Oldest First [oldest], Newest First [newest], Largest First [largest], Smallest First [smallest]",
+											"caption": "Optionally sort the files before copying.  Useful when combined with maximum."
+										},
+										{
+											"id": "s3params",
+											"title": "Custom S3 Params",
+											"type": "json",
+											"value": {},
+											"caption": "Optionally specify parameters to the S3 API for the destination objects, for e.g. ACL and Storage Class."
+										}
+									]
+								},
+								{
+									"id": "listFiles",
+									"title": "List Files",
+									"description": "List remote S3 files with optional filter, and return metadata.",
+									"fields": [
+										{
+											"id": "remotePath",
+											"title": "Remote Path",
+											"type": "text",
+											"value": "",
+											"caption": "The base S3 path to list files from.  Leave blank to list files at the root level."
+										},
+										{
+											"id": "filespec",
+											"title": "Filename Pattern",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the remote files using a glob pattern, applied to the filenames."
+										},
+										{
+											"id": "older",
+											"title": "Older Than",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the S3 files based on their mod date, so they must be older than the specified number of seconds (or specify a string like `7 days`)."
+										},
+										{
+											"id": "newer",
+											"title": "Newer Than",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the S3 files based on their mod date, so they must be newer than the specified number of seconds (or specify a string like `7 days`)."
+										},
+										{
+											"id": "max",
+											"title": "Maximum Files",
+											"type": "text",
+											"variant": "number",
+											"value": 0,
+											"caption": "Optionally set a maximum number of files to return."
+										},
+										{
+											"id": "sort",
+											"title": "Sort Files",
+											"type": "select",
+											"value": "Newest First [newest], Oldest First [oldest], Largest First [largest], Smallest First [smallest]",
+											"caption": "Optionally sort the files before returning metadata."
+										}
+									]
+								},
+								{
+									"id": "grepFiles",
+									"title": "Grep Files",
+									"description": "Search inside remote S3 files for regular expression match.",
+									"fields": [
+										{
+											"id": "remotePath",
+											"title": "Remote Path",
+											"type": "text",
+											"value": "",
+											"caption": "The base S3 path to search under.  Leave blank to search from the root level."
+										},
+										{
+											"id": "filespec",
+											"title": "Filename Pattern",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the remote files using a glob pattern, applied to the filenames."
+										},
+										{
+											"id": "match",
+											"title": "Match Pattern (Regex)",
+											"type": "text",
+											"value": "",
+											"caption": "The regular expression pattern to search for inside each remote file.",
+											"required": true
+										},
+										{
+											"id": "decompress",
+											"title": "Decompress Files (Gunzip)",
+											"type": "checkbox",
+											"value": false,
+											"caption": "Automatically decompress all gzipped files during download before searching their contents."
+										},
+										{
+											"id": "older",
+											"title": "Older Than",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the S3 files based on their mod date, so they must be older than the specified number of seconds (or specify a string like `7 days`)."
+										},
+										{
+											"id": "newer",
+											"title": "Newer Than",
+											"type": "text",
+											"value": "",
+											"caption": "Optionally filter the S3 files based on their mod date, so they must be newer than the specified number of seconds (or specify a string like `7 days`)."
+										},
+										{
+											"id": "maxLines",
+											"title": "Maximum Matches",
+											"type": "text",
+											"variant": "number",
+											"value": 1000,
+											"caption": "Optionally stop after this many matching lines have been found."
+										},
+										{
+											"id": "output",
+											"title": "Output Format",
+											"type": "select",
+											"value": "JSON Data [data], Text File [file]",
+											"caption": "Select whether you want the matched lines returned as job JSON data, or attached as a text file."
+										}
+									]
+								}
+							]
+						}
+					}
+				]
+			}
+		}
+	]
+}