gulp-cli 1.1.8__tar.gz → 1.1.10__tar.gz

{gulp_cli-1.1.8/src/gulp_cli.egg-info → gulp_cli-1.1.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gulp-cli
-Version: 1.1.8
+Version: 1.1.10
 Summary: Command-line client for gULP
 Author-email: Mentat <info@mentat.is>
 Requires-Python: >=3.12
@@ -22,7 +22,7 @@ Dynamic: license-file
 ## ✨ What can you do?
 
 - 🔐 **Authentication** — secure login with token persistence
-- 📥 **Ingestion** — ingest files (single/batch/wildcard), zip archives, with concurrent uploads
+- 📥 **Ingestion** — ingest files (single/batch/wildcard), optionally compressed before upload, with concurrent uploads
 - 🔍 **Querying** — raw OpenSearch queries, Sigma rules, external plugins
 - 🏷️ **Enrichment** — enrich documents, tag/untag, update fields
 - 👥 **User Management** — create users, manage permissions (admin only)

{gulp_cli-1.1.8 → gulp_cli-1.1.10}/README.md

@@ -5,7 +5,7 @@
 ## ✨ What can you do?
 
 - 🔐 **Authentication** — secure login with token persistence
-- 📥 **Ingestion** — ingest files (single/batch/wildcard), zip archives, with concurrent uploads
+- 📥 **Ingestion** — ingest files (single/batch/wildcard), optionally compressed before upload, with concurrent uploads
 - 🔍 **Querying** — raw OpenSearch queries, Sigma rules, external plugins
 - 🏷️ **Enrichment** — enrich documents, tag/untag, update fields
 - 👥 **User Management** — create users, manage permissions (admin only)

{gulp_cli-1.1.8 → gulp_cli-1.1.10}/docs/command-reference.md

@@ -461,7 +461,7 @@ gulp-cli ingest file OPERATION_ID PLUGIN [FILE...] [OPTIONS]
 **Options:**
 
 - `--context TEXT` — Context for ingestion, specify a name to create a new context if it doesn't exist, or an existing `context_id` (default: `sdk_context`)
-- `--plugin-params TEXT` — JSON string with plugin parameters
+- `--plugin-params TEXT` — JSON string with plugin parameters. When it includes `"compressed": true`, gulp-cli bzip2-compresses each input file before upload and the backend decompresses it before plugin ingestion.
 - `--flt TEXT` — JSON object for `GulpIngestionFilter`
 - `--paths-file PATH` — Text file with one file or glob pattern per line. When set, positional `FILE` arguments are optional and ignored if also provided.
 - `--reset-operation` — Delete and recreate the operation before ingest starts
@@ -493,6 +493,9 @@ gulp-cli ingest file my_op win_evtx --paths-file /tmp/file_paths.txt
 # With plugin parameters
 gulp-cli ingest file my_op csv file.csv --plugin-params '{"delimiter":";","encoding":"utf-8"}'
 
+# Compress before upload and let the backend decompress before ingestion
+gulp-cli ingest file my_op win_evtx /path/to/System.evtx --plugin-params '{"compressed": true}'
+
 # Preview mode (no ingestion persisted)
 gulp-cli ingest file my_op win_evtx /path/to/System.evtx --preview
 
@@ -521,7 +524,7 @@ gulp-cli ingest file-to-source SOURCE_ID [FILE...] [OPTIONS]
 **Options:**
 
 - `--plugin TEXT` — Override the plugin associated with the source (requires `--plugin-params`)
-- `--plugin-params TEXT` — JSON object for plugin_params (overrides source defaults; required when `--plugin` is set, `{}` is allowed)
+- `--plugin-params TEXT` — JSON object for plugin_params (overrides source defaults; required when `--plugin` is set, `{}` is allowed). When it includes `"compressed": true`, gulp-cli bzip2-compresses each input file before upload and the backend decompresses it before plugin ingestion.
 - `--flt TEXT` — JSON object for GulpIngestionFilter
 - `--paths-file PATH` — Text file with one file or glob pattern per line. When set, positional `FILE` arguments are optional and ignored if also provided.
 - `--wait` — Wait for completion with progress
@@ -536,38 +539,7 @@ gulp-cli ingest file-to-source source123 /new/System.evtx --wait
 gulp-cli ingest file-to-source source123 '/new/*.evtx'
 gulp-cli ingest file-to-source source123 --paths-file /tmp/file_paths.txt
 gulp-cli ingest file-to-source source123 /new/events.json --plugin json --plugin-params '{"mapping_parameters": {}}'
-```
-
----
-
-#### `zip`
-
-Ingest from a ZIP archive.
-
-> TO BE DEPRECATED...
-
-```bash
-gulp-cli ingest zip OPERATION_ID ZIP_FILE [OPTIONS]
-```
-
-**Arguments:**
-
-- `OPERATION_ID` — Target operation
-- `ZIP_FILE` — Path to ZIP file containing a `metadata.json` in the root, describing the content as specified in gulp's `ingest_zip` docs.
-
-**Options:**
-
-- `--context TEXT` — Context for ingestion, specify a name to create a new context if it doesn't exist, or an existing `context_id` (default: `sdk_context`)
-- `--flt TEXT` — JSON object for `GulpIngestionFilter`
-- `--create-operation` — Create operation automatically when it does not exist
-- `--wait` — Wait for completion
-- `--timeout INTEGER` — Timeout in seconds for `--wait` mode
-
-**Examples:**
-
-```bash
-gulp-cli ingest zip my_op /data/evidence.zip --wait
-gulp-cli ingest zip my_op /data/evidence.zip --create-operation
+gulp-cli ingest file-to-source source123 /new/System.evtx --plugin win_evtx --plugin-params '{"compressed": true}'
 ```
 
 ---
@@ -2330,32 +2302,6 @@ gulp-cli plugin list-ui [OPTIONS]
 
 ---
 
-#### `plugin upload`
-
-Upload a new plugin.
-
-```bash
-gulp-cli plugin upload PLUGIN_FILE [OPTIONS]
-```
-
-#### `plugin download`
-
-Download a plugin.
-
-```bash
-gulp-cli plugin download FILENAME OUTPUT_PATH [OPTIONS]
-```
-
-#### `plugin delete`
-
-Delete a plugin.
-
-```bash
-gulp-cli plugin delete FILENAME [OPTIONS]
-```
-
----
-
 #### `mapping list`
 
 List all mapping files.

{gulp_cli-1.1.8 → gulp_cli-1.1.10}/docs/examples.md

@@ -8,7 +8,7 @@
     - [CSV Data Ingestion with Custom Parameters](#csv-data-ingestion-with-custom-parameters)
     - [JSON Logs Ingestion](#json-logs-ingestion)
     - [Add More Evidence to Existing Source](#add-more-evidence-to-existing-source)
-    - [ZIP Archive Ingestion](#zip-archive-ingestion)
+    - [Compressed File Ingestion](#compressed-file-ingestion)
     - [Create ZIP Archive from Paths and Masks](#create-zip-archive-from-paths-and-masks)
     - [Raw Payload Ingestion](#raw-payload-ingestion)
   - [Request Stats Monitoring Workflows](#request-stats-monitoring-workflows)
@@ -205,14 +205,12 @@ gulp-cli ingest file-to-source data-2026-001 /additional/evidence.evtx --wait
 gulp-cli ingest file-to-source data-2026-001 '/additional/*.evtx' --wait
 ```
 
-### ZIP Archive Ingestion
+### Compressed File Ingestion
 
 ```bash
-# Ingest prepared ZIP evidence archive
-gulp-cli ingest zip incident-001 /evidence/evidence.zip --wait
-
-# Auto-create operation when missing
-gulp-cli ingest zip incident-001 /evidence/evidence.zip --create-operation
+# Compress files with bzip2 before upload; the backend decompresses before ingestion
+gulp-cli ingest file incident-001 win_evtx /evidence/System.evtx \
+  --plugin-params '{"compressed": true}' --wait
 ```
 
 ### Create ZIP Archive from Paths and Masks
@@ -233,9 +231,6 @@ $CASE_ROOT/linux/**/*.log
 EOF
 
 gulp-cli ingest zip-create '$CASE_ROOT/evidence.zip' --paths-file /tmp/evidence_paths.txt --overwrite
-
-# Then ingest the generated ZIP
-gulp-cli ingest zip incident-001 '$CASE_ROOT/evidence.zip' --wait
 ```
 
 ### Raw Payload Ingestion

{gulp_cli-1.1.8 → gulp_cli-1.1.10}/docs/getting-started.md

@@ -207,11 +207,11 @@ This shows a real-time progress bar while documents are being ingested. Per-file
 gulp-cli ingest file-to-source existing-source-id /gulp/samples/win_evtx/Security.evtx --wait
 ```
 
-### Ingest ZIP / Raw Data
+### Compressed / Raw Data
 
 ```bash
-# ZIP archive ingestion
-gulp-cli ingest zip my_investigation /path/to/evidence.zip --wait
+# Compress locally with bzip2 before upload; the backend decompresses before ingestion
+gulp-cli ingest file my_investigation win_evtx /path/to/System.evtx --plugin-params '{"compressed": true}' --wait
 
 # Raw payload ingestion from a file
 gulp-cli ingest raw my_investigation --data-file /tmp/raw_chunk.json --last --wait

{gulp_cli-1.1.8 → gulp_cli-1.1.10}/docs/resource-management.md

@@ -196,58 +196,6 @@ List available UI plugins:
 gulp plugin list-ui
 ```
 
-### Upload Plugin
-
-Upload a custom plugin:
-
-```bash
-# Upload as default plugin
-gulp plugin upload /path/to/my_plugin.py
-
-# Upload as extension plugin
-gulp plugin upload /path/to/my_extension.py --type extension
-
-# Upload as UI plugin
-gulp plugin upload /path/to/my_ui_plugin.py --type ui
-
-# Fail if already exists
-gulp plugin upload /path/to/my_plugin.py --fail-if-exists
-```
-
-Response includes the file path where the plugin was saved on the server.
-
-### Download Plugin
-
-Download a plugin from the server:
-
-```bash
-# Download default plugin
-gulp plugin download win_evtx.py /local/path/win_evtx.py
-
-# Download extension plugin
-gulp plugin download my_extension.py /path/to/save --type extension
-
-# Download UI plugin
-gulp plugin download my_ui.py /path/to/save --type ui
-```
-
-### Delete Plugin
-
-Delete a plugin from the server:
-
-```bash
-# Delete default plugin
-gulp plugin delete my_plugin.py
-
-# Delete extension plugin
-gulp plugin delete my_extension.py --type extension
-
-# Delete UI plugin
-gulp plugin delete my_ui.py --type ui
-```
-
----
-
 ## Mapping File Management
 
 Mapping files define data transformations and field mappings for ingestion plugins.

gulp_cli-1.1.10/docs/stdout-markers.md

@@ -0,0 +1,125 @@
+# Stdout Markers Cheat Sheet
+
+Marker lines are emitted as one unwrapped stdout line:
+
+```text
+GULP_MARKER: [MARKER_NAME] | {"json": "payload"}
+```
+
+Parser rule: match lines starting with `GULP_MARKER: `, split once on ` | `, then JSON-decode the right side.
+
+## Ingest `--wait-log`
+
+These markers are emitted by `ingest file --wait-log`, `ingest file-to-source --wait-log`, and `ingest raw --wait-log`.
+
+### `[MARKER_INGEST_SOURCE_DONE_RECEIVED]`
+
+Emitted when the backend sends `INGEST_SOURCE_DONE`.
+
+Payload:
+- `req_id`: backend request id.
+- `file`: display label for the file/request.
+- `status`: status from the source-done packet or current state.
+- `ingested`: accumulated ingested record count.
+- `skipped`: accumulated skipped record count.
+- `failed`: accumulated failed record count.
+- `errors`: extracted backend errors, usually an array of strings.
+
+### `[MARKER_ONGOING_STATS_RECEIVED]`
+
+Emitted for `STATS_CREATE` or `STATS_UPDATE` while status is not `done`, `failed`, or `canceled`.
+
+Payload:
+- `req_id`: backend request id.
+- `file`: display label for the file/request.
+- `event`: websocket event name, usually `STATS_CREATE` or `STATS_UPDATE`.
+- `status`: current stats status.
+- `ingested`: current authoritative ingested count.
+- `skipped`: current authoritative skipped count.
+- `failed`: current authoritative failed count.
+- `pct`: ingest percentage when provided by the backend, otherwise `null`.
+- `errors`: extracted backend errors, usually an array of strings.
+
+### `[MARKER_DONE_STATS_RECEIVED]`
+
+Emitted for `STATS_CREATE` or `STATS_UPDATE` when status is `done`.
+
+Payload: same fields as `[MARKER_ONGOING_STATS_RECEIVED]`.
+
+### `[MARKER_FAILED_STATS_RECEIVED]`
+
+Emitted for `STATS_CREATE` or `STATS_UPDATE` when status is `failed` or `canceled`.
+
+Also emitted after a backend websocket `ERROR` packet.
+
+Payload:
+- From stats packets: same fields as `[MARKER_ONGOING_STATS_RECEIVED]`.
+- From `ERROR` packets: `req_id`, `file`, `event`, `status`, `ingested`, `skipped`, `failed`, `errors`.
+
+### `[MARKER_BACKEND_EXCEPTION_REPORTED]`
+
+Emitted when the backend sends websocket `ERROR`.
+
+Payload:
+- `req_id`: backend request id.
+- `file`: display label for the file/request.
+- `event`: websocket event name, `ERROR`.
+- `status`: always set to `failed` by the CLI for this request.
+- `errors`: extracted backend errors, usually an array of strings.
+
+### `[MARKER_INGESTION_FINISHED]`
+
+Emitted once after wait-log completion, before the final CLI result JSON/table output.
+
+Payload:
+- `requests_total`: number of request results.
+- `requests_done`: count with status `done`.
+- `requests_failed`: count with status `failed`.
+- `requests_canceled`: count with status `canceled`.
+- `requests_timeout`: count with status `timeout`.
+- `ingested`: accumulated ingested record count.
+- `skipped`: accumulated skipped record count.
+- `failed`: accumulated failed record count.
+
+## `ingest zip-create`
+
+These markers are emitted by local ZIP creation.
+
+### `[MARKER_FILE_ADDED_TO_ZIP]`
+
+Emitted after each ZIP entry is written.
+
+Payload:
+- `zip`: requested output ZIP path.
+- `entry`: ZIP entry name.
+- `source`: source filesystem path for the entry.
+- `file_size`: uncompressed entry size in bytes.
+- `compress_size`: compressed entry size in bytes.
+- `current`: 1-based count of entries added so far.
+- `total`: total entries planned for the ZIP.
+- `percent`: integer progress percentage.
+
+### `[MARKER_ZIP_CREATED_SUCCESSFULLY]`
+
+Emitted once after the ZIP or multipart ZIP volumes are published.
+
+Payload:
+- `zip_files`: array of created archive paths.
+- `files_archived`: number of source files archived. Directory entries are not counted here.
+- `entries_archived`: number of ZIP entries archived, including directory entries.
+
+### `[MARKER_ZIP_CREATE_ERROR]`
+
+Emitted when ZIP creation hits a validation, scan, write, or publish error.
+
+Payload variants:
+- Scan/write skip errors: `path`, `zip`, `error`.
+- Top-level `zip-create` command errors: `zip`, `error`.
+
+### `[MARKER_OTHER_ERROR]`
+
+Emitted with top-level `zip-create` command exceptions after `[MARKER_ZIP_CREATE_ERROR]`.
+
+Payload:
+- `command`: currently `zip-create`.
+- `error`: exception text.

{gulp_cli-1.1.8 → gulp_cli-1.1.10}/src/gulp_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.1.8'
-__version_tuple__ = version_tuple = (1, 1, 8)
+__version__ = version = '1.1.10'
+__version_tuple__ = version_tuple = (1, 1, 10)
 
-__commit_id__ = commit_id = 'g6f1fa3720'
+__commit_id__ = commit_id = 'g015adf27b'