polyswarm-engine 3.1.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (31) hide show
  1. polyswarm_engine-3.1.1/PKG-INFO +361 -0
  2. polyswarm_engine-3.1.1/README.md +337 -0
  3. polyswarm_engine-3.1.1/polyswarm_engine/__init__.py +49 -0
  4. polyswarm_engine-3.1.1/polyswarm_engine/backend.py +302 -0
  5. polyswarm_engine-3.1.1/polyswarm_engine/bidutils.py +69 -0
  6. polyswarm_engine-3.1.1/polyswarm_engine/bounty.py +387 -0
  7. polyswarm_engine-3.1.1/polyswarm_engine/celeryconfig.py +76 -0
  8. polyswarm_engine-3.1.1/polyswarm_engine/cli.py +316 -0
  9. polyswarm_engine-3.1.1/polyswarm_engine/command.py +34 -0
  10. polyswarm_engine-3.1.1/polyswarm_engine/constants.py +37 -0
  11. polyswarm_engine-3.1.1/polyswarm_engine/engine.py +123 -0
  12. polyswarm_engine-3.1.1/polyswarm_engine/exceptions.py +41 -0
  13. polyswarm_engine-3.1.1/polyswarm_engine/log_config.py +104 -0
  14. polyswarm_engine-3.1.1/polyswarm_engine/py.typed +0 -0
  15. polyswarm_engine-3.1.1/polyswarm_engine/settings.py +41 -0
  16. polyswarm_engine-3.1.1/polyswarm_engine/typing.py +125 -0
  17. polyswarm_engine-3.1.1/polyswarm_engine/utils.py +434 -0
  18. polyswarm_engine-3.1.1/polyswarm_engine/wine.py +125 -0
  19. polyswarm_engine-3.1.1/polyswarm_engine/wsgi.py +122 -0
  20. polyswarm_engine-3.1.1/polyswarm_engine.egg-info/PKG-INFO +361 -0
  21. polyswarm_engine-3.1.1/polyswarm_engine.egg-info/SOURCES.txt +29 -0
  22. polyswarm_engine-3.1.1/polyswarm_engine.egg-info/dependency_links.txt +1 -0
  23. polyswarm_engine-3.1.1/polyswarm_engine.egg-info/requires.txt +17 -0
  24. polyswarm_engine-3.1.1/polyswarm_engine.egg-info/top_level.txt +1 -0
  25. polyswarm_engine-3.1.1/pyproject.toml +100 -0
  26. polyswarm_engine-3.1.1/setup.cfg +4 -0
  27. polyswarm_engine-3.1.1/tests/test_bidutils.py +11 -0
  28. polyswarm_engine-3.1.1/tests/test_bounty.py +155 -0
  29. polyswarm_engine-3.1.1/tests/test_engine.py +38 -0
  30. polyswarm_engine-3.1.1/tests/test_utils.py +373 -0
  31. polyswarm_engine-3.1.1/tests/test_wine.py +35 -0
polyswarm_engine-3.1.1/PKG-INFO ADDED
@@ -0,0 +1,361 @@
1
+ Metadata-Version: 2.4
2
+ Name: polyswarm_engine
3
+ Version: 3.1.1
4
+ Summary: Polyswarm engine libarary
5
+ Author-email: Polyswarm Developers <developers@polyswarm.io>
6
+ Project-URL: Homepage, https://github.com/polyswarm/polyswarm_engine
7
+ Requires-Python: <4,>=3.10
8
+ Description-Content-Type: text/markdown
9
+ Requires-Dist: celery
10
+ Requires-Dist: click
11
+ Requires-Dist: click-log
12
+ Requires-Dist: python-json-logger
13
+ Requires-Dist: requests
14
+ Provides-Extra: tests
15
+ Requires-Dist: flake8; extra == "tests"
16
+ Requires-Dist: isort; extra == "tests"
17
+ Requires-Dist: mypy; extra == "tests"
18
+ Requires-Dist: pytest>=7.0; extra == "tests"
19
+ Requires-Dist: types-requests; extra == "tests"
20
+ Requires-Dist: yapf; extra == "tests"
21
+ Provides-Extra: web
22
+ Requires-Dist: flask; extra == "web"
23
+ Requires-Dist: gunicorn; extra == "web"
24
+
25
+
26
+
27
+ ## Analyzers
28
+
29
+ This tool wraps the PolySwarm engine machinery, allowing you (the engine developer)
30
+ to run a custom-built python script, web API, software running on another machine, etc.
31
+
32
+ - Scanner functions should be able to depend on an execution environment which is initialized exactly once for each scanner & shares nothing between individual processes / tasks / threads / etc.
33
+ - Scanners should support initializer / setup() which is run only once, which can write to a safe data store readable (but not necessarily writable) from worker children.
34
+ - Scanner's interface functions should be segregated, with many specific interfaces rather than one general-purpose interface. each of which should have a single responsibility
35
+ - we should depend upon an abstraction of scanner interfaces, not concrete engine function internals.
36
+ - It is better to split scanning up into as many small tasks as there are responsibilities for steps of execution.
37
+ - engines shouldn't tamper with the internal implementation of client-supplied functions (corollary: functions defined by the author should be callable from within their debugger, ipython, e.g. as Sam's set_handler does this) e.g engine functions should be callable "as they are", without setting up external state.
38
+ - Ideally, scanner functions should be pure. They should never keep internal state.
39
+ - Engines should be able to kick-off scans which are entirely autonomous processes, e.g can submit their bounty response without external code waiting on that result (e.g ignoring it's result).
40
+ - Both the author of the engine interface & the users of it should be able to add callbacks which are triggered on errors and success (with the results included).
41
+ - Engines should, in exceptional cases, have the ability to extend the core classes without reimplementing everything from the ground up.
42
+
43
+ ## CLI
44
+
45
+ ### `analyze`
46
+
47
+ The `analyze` command creates a `Bounty` from a file, pipe or URL
48
+
49
+ ```console
50
+ [zv@fedora] Development/polyswarm_engine $ ./examples/nanoav/nanoav.py analyze -v --backend local resource/malicious/pe/*
51
+ Starting NANO Antivirus JSON RPC server...
52
+ OS environment: WINE
53
+ The installation path: /home/zv/Development/microengines/microengines/nanoav/vendor/nanoavsdk
54
+ The executable file path: /home/zv/Development/microengines/microengines/nanoav/vendor/nanoavsdk/bin/nanoavc.exe
55
+ JSON RPC server initialization started (PID: 2142926)...
56
+ JSON RPC server initialization completed...
57
+ -------------------------resource/malicious/pe/Abel.exe-------------------------
58
+ Bounty: {
59
+ "id": 3890646043,
60
+ "artifact_uri": "file:///home/zv/Development/psengine-test/psengine_test/resource/malicious/pe/Abel.exe",
61
+ "artifact_type": "file",
62
+ "sha256": "a709f37b3a50608f2e9830f92ea25da04bfa4f34d2efecfd061de9f29af02427",
63
+ "mimetype": "application/x-dosexec"
64
+ }
65
+ Analysis: {
66
+ "bid": 18446744073709551615,
67
+ "result": "malicious",
68
+ "result_name": "Riskware.Win32.Cain.ecncwa",
69
+ "product": "NANO Antivirus",
70
+ "vendor": "NanoAV",
71
+ "analysis_engine_version": "1.0.146.90945",
72
+ "analysis_definition_version": "0.14.45.23447"
73
+ }
74
+ ------------------------resource/malicious/pe/conficker-------------------------
75
+ Bounty: {
76
+ "id": 474020256,
77
+ "artifact_uri": "file:///home/zv/Development/psengine-test/psengine_test/resource/malicious/pe/conficker",
78
+ "artifact_type": "file",
79
+ "sha256": "523d40c69b0972ddeff0682fcb569e8a346cf10b2894479ab227bbb24e19846e",
80
+ "mimetype": "application/x-dosexec"
81
+ }
82
+ Analysis: {
83
+ "bid": 18446744073709551615,
84
+ "result": "malicious",
85
+ "result_name": "Trojan.Win32.Kido.bvftw",
86
+ "product": "NANO Antivirus",
87
+ "vendor": "NanoAV",
88
+ "analysis_engine_version": "1.0.146.90945",
89
+ "analysis_definition_version": "0.14.45.23447"
90
+ }
91
+ JSON RPC server has finished.
92
+ ```
93
+
94
+ ### `worker`
95
+
96
+ You can start a `celery` worker with the `worker` command
97
+
98
+
99
+ ```console
100
+ [zv@fedora] Development/polyswarm_engine $ ./examples/nanoav/nanoav.py worker
101
+ Starting NANO Antivirus JSON RPC server...
102
+ OS environment: WINE
103
+ The installation path: /home/zv/Development/microengines/microengines/nanoav/vendor/nanoavsdk
104
+ The executable file path: /home/zv/Development/microengines/microengines/nanoav/vendor/nanoavsdk/bin/nanoavc.exe
105
+ JSON RPC server initialization started (PID: 2143537)...
106
+ JSON RPC server initialization completed...
107
+
108
+ -------------- celery@fedora v5.2.1 (dawn-chorus)
109
+ --- ***** -----
110
+ -- ******* ---- Linux-5.14.18-100.fc33.x86_64-x86_64-with-glibc2.32 2022-02-10 17:49:08
111
+ - *** --- * ---
112
+ - ** ---------- [config]
113
+ - ** ---------- .> app: nanoav:0x7f886432ef70
114
+ - ** ---------- .> transport: amqp://guest:**@localhost:5672//
115
+ - ** ---------- .> results: redis://localhost/
116
+ - *** --- * --- .> concurrency: 8 (prefork)
117
+ -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
118
+ --- ***** -----
119
+ -------------- [queues]
120
+ .> celery exchange=celery(direct) key=celery
121
+ ```
122
+
123
+ After you've started a `celery` worker, you can provide `--backend celery` to the `analyze` command t
124
+
125
+ ```console
126
+ [zv@fedora] Development/psengine $ ./examples/nanoav/nanoav.py analyze -v --backend celery resource/malicious/pe/*
127
+ -------------------------resource/malicious/pe/Abel.exe-------------------------
128
+ Bounty: {
129
+ "id": 3890646043,
130
+ "artifact_uri": "file:///home/zv/Development/psengine-test/psengine_test/resource/malicious/pe/Abel.exe",
131
+ "artifact_type": "file",
132
+ "sha256": "a709f37b3a50608f2e9830f92ea25da04bfa4f34d2efecfd061de9f29af02427",
133
+ "mimetype": "application/x-dosexec"
134
+ }
135
+ Analysis: {
136
+ "bid": 18446744073709551615,
137
+ "result": "malicious",
138
+ "result_name": "Riskware.Win32.Cain.ecncwa",
139
+ "product": "NANO Antivirus",
140
+ "vendor": "NanoAV",
141
+ "analysis_engine_version": "1.0.146.90945",
142
+ "analysis_definition_version": "0.14.45.23447"
143
+ }
144
+ ------------------------resource/malicious/pe/conficker-------------------------
145
+ Bounty: {
146
+ "id": 474020256,
147
+ "artifact_uri": "file:///home/zv/Development/psengine-test/psengine_test/resource/malicious/pe/conficker",
148
+ "artifact_type": "file",
149
+ "sha256": "523d40c69b0972ddeff0682fcb569e8a346cf10b2894479ab227bbb24e19846e",
150
+ "mimetype": "application/x-dosexec"
151
+ }
152
+ Analysis: {
153
+ "bid": 18446744073709551615,
154
+ "result": "malicious",
155
+ "result_name": "Trojan.Win32.Kido.bvftw",
156
+ "product": "NANO Antivirus",
157
+ "vendor": "NanoAV",
158
+ "analysis_engine_version": "1.0.146.90945",
159
+ "analysis_definition_version": "0.14.45.23447"
160
+ }
161
+ ```
162
+
163
+ ### `command`
164
+
165
+ You can access all of the commands exposed by an engine through `command`
166
+
167
+ ```console
168
+ [zv@fedora] Development/polyswarm_engine $ ./examples/nanoav/nanoav.py commands --help
169
+ Usage: nanoav.py commands [OPTIONS] COMMAND [ARGS]...
170
+
171
+ Engine commands
172
+
173
+ Options:
174
+ --help Show this message and exit.
175
+
176
+ Commands:
177
+ info Engine & signature versions
178
+ scan_file Scan filename
179
+ #
180
+ ```
181
+
182
+ ### `webhook`
183
+
184
+ `webhook` starts an HTTP server you can submit bounty webhook test requests to
185
+
186
+ ```console
187
+ [zv@fedora] Development/polyswarm_engine $ ./examples/nanoav/nanoav.py webhook --help
188
+ Usage: nanoav.py webhook [OPTIONS]
189
+
190
+ Bounty webhooks
191
+
192
+ Options:
193
+ -p, --port INTEGER Server port
194
+ --backend [local|process|celery]
195
+ [required]
196
+ --help Show this message and exit.
197
+
198
+ ```
199
+
200
+
201
+ ### Config
202
+
203
+ #### Environment Variables
204
+
205
+ ##### `TMPPREFIX`
206
+ `TMPPREFIX` specifies a directory prefix to use for all temporary files created by `polyswarm_engine`. `polyswarm_engine` (and most other Unix programs) will also respect `TMPDIR` and use it's value to denote the scratch area instead of the default.
207
+
208
+ ##### Engine
209
+
210
+ Engines should prefer to specify files (binaries, license files, DLLs, etc.) as paths relative to a well-known base directory variable.
211
+
212
+ ###### `${ENGINE}_PATH`
213
+ Engine executables should be stored here (analogous to `/usr/bin`)
214
+
215
+ ###### `${ENGINE}_SDK_HOME`
216
+ Engine-specific libraries, shared objects & DLLs (analogous to `/usr/lib`)
217
+
218
+ ###### `${ENGINE}_SECRETS_HOME`
219
+ License files and other secrets (analogous to `/usr/bin`)
220
+
221
+ ###### `${ENGINE}_CONFIG_HOME`
222
+ Engine-specific configurations (analogous to `/etc`)
223
+
224
+ ###### `${ENGINE}_CACHE_HOME`
225
+ Non-essential (cached) data (analogous to `/var/cache`)
226
+
227
+ ###### `${ENGINE}_DATA_HOME`
228
+ Engine-specific data files (analogous to `/usr/share`).
229
+
230
+ ###### `${ENGINE}_STATE_HOME`
231
+ Contains state data that should persist between restarts:
232
+
233
+ - Logs
234
+ - Rate-limiting data
235
+ - Telemetry
236
+ - Current application state (if running as a daemon)
237
+
238
+ ###### `${ENGINE}_RUNTIME_DIR`
239
+ Used for non-essential, engine data files such as sockets, named pipes, etc. (analogous to `/var/run`)
240
+
241
+ - Not required to have a default value; warnings should be issued if not set or equivalents provided.
242
+ - Must be owned by the user with an access mode of `0700`.
243
+ - May be subject to periodic cleanup.
244
+ - This directory **MUST** be on a local file system which supports:
245
+ + symbolic links
246
+ + proper permissions
247
+ + file locking
248
+ + sparse files
249
+ + memory mapping
250
+ + file change notifications
251
+
252
+ ###### Files
253
+
254
+ Your configuration should strive to specify particular relative to well-known base directory variables shown above. If your implementation prefers absolute paths to executable or configuration files, you can still use a few well-known names
255
+
256
+ ###### `${ENGINE}_SCANNER`
257
+ Path to the executable used for scanning binaries
258
+
259
+ ###### `${ENGINE}_UPDATER`
260
+ Path to the executable used for updating engine definitions
261
+
262
+ ##### System
263
+
264
+ ##### `TMPDIR`
265
+ `TMPDIR` is the canonical environment variable in POSIX systems that should be used to specify a temporary directory for scratch space. `polyswarm_engine` (and most other Unix programs) will honor this setting and use it's value to denote the scratch area for temporary files instead of the common default of `/tmp`
266
+
267
+ ##### `LC_ALL`
268
+ Locales can be especially confusing when using PE executables from within a POSIX system with WINE. Many WinNT applications are compiled to use UCS-2 / 1252 without regard for locale configuration (in addition to using `CRLF` newlines)
269
+
270
+ ##### `TZ`
271
+ Timezone
272
+
273
+ #### Wine
274
+
275
+ ##### `WINELOADER`
276
+ Specifies the path and name of the Wine binary to use to launch new Windows processes.
277
+
278
+ If not set and not running on Windows, this will look for a file named "wine" in `PATH`
279
+
280
+ ##### `PERSISTENT_WINESERVER`
281
+ If set to any non-`0` value, a `wineserver` is started that stays around forever.
282
+
283
+ `wineserver` is normally launched automatically when starting `wine`, however,
284
+ it is useful to start `wineserver` explicitly with an unlimited persistence
285
+ delay, this avoids the cost of shutting down and starting again when wine-loaded
286
+ programs are launched in quick succession.
287
+
288
+ ###### Killing
289
+
290
+ You can kill a persistent `wineserver` with `--kill`
291
+
292
+ ```console
293
+ swarm@engine:~$ wineserver --kill
294
+ ```
295
+
296
+
297
+ ### Fetching
298
+
299
+ Bounties are delivered with an Artifact URI, defining where an Artifact's
300
+ contents can be found. This can reference an HTTP/HTTPS/data URI and a `file://`
301
+ when using your engine locally:
302
+
303
+ | Scheme | Example URI |
304
+ | --- | ----------- |
305
+ | file | file:///usr/bin/ls |
306
+ | data | data:text/uri-list;base64,aHR0cDovL2dvb2dsZS5jb20v |
307
+ | https | https://resource.polyswarm.io/artifact/b1b249f39beaa9360abe95570560437f41e0a0f8bb7e3c74546078996d80c5ff |
308
+
309
+
310
+
311
+ #### Categories & Mimetypes
312
+
313
+ | Category | Mimetype |
314
+ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
315
+ | android | `application/vnd.android.package-archive` |
316
+ | archives | `application/java-archive` `application/x-tar` `application/zip` `application/x-compressed-zip` `application/gzip` `application/vnd.rar` `application/x-bzip2` `application/x-xz` `application/octet-stream` `application/x-7z-compressed` |
317
+ | elf | `application/x-executable` |
318
+ | flash | `application/x-shockwave-flash` |
319
+ | font | `application/vnd.ms-fontobject` `application/vnd.oasis.opendocument.text` `font/otf` `font/ttf` `font/woff` `font/woff2` |
320
+ | linux | elf + `application/x-cpio` `application/x-rpm` `application/x-dpkg` `application/octet-stream` |
321
+ | mach-o | `application/x-mach-binary` `application/octet-stream` |
322
+ | office (abiword) | `application/x-abiword` |
323
+ | office (etc) | `text/calendar` |
324
+ | office (ms) | `application/msword` `application/vnd.ms-excel` `application/vnd.ms-excel.addin.macroEnabled.12` `application/vnd.ms-excel.sheet.binary.macroEnabled.12` `application/vnd.ms-excel.sheet.macroEnabled.12` `application/vnd.ms-excel.template.macroEnabled.12` `application/vnd.ms-powerpoint` `application/vnd.ms-powerpoint.addin.macroEnabled.12` `application/vnd.ms-powerpoint.presentation.macroEnabled.12` `application/vnd.ms-powerpoint.slideshow.macroEnabled.12` `application/vnd.ms-word.document.macroEnabled.12` `application/vnd.ms-word.template.macroEnabled.12` `application/vnd.openxmlformats-officedocument.presentationml.presentation` `application/vnd.openxmlformats-officedocument.presentationml.slideshow` `application/vnd.openxmlformats-officedocument.presentationml.template` `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` `application/vnd.openxmlformats-officedocument.spreadsheetml.template` `application/vnd.openxmlformats-officedocument.wordprocessingml.document` `application/vnd.openxmlformats-officedocument.wordprocessingml.template` |
325
+ | office (open) | `application/vnd.oasis.opendocument.chart-template` `application/vnd.oasis.opendocument.chart` `application/vnd.oasis.opendocument.database` `application/vnd.oasis.opendocument.formula` `application/vnd.oasis.opendocument.graphics-template` `application/vnd.oasis.opendocument.graphics` `application/vnd.oasis.opendocument.image-template` `application/vnd.oasis.opendocument.image` `application/vnd.oasis.opendocument.presentation-template` `application/vnd.oasis.opendocument.presentation` `application/vnd.oasis.opendocument.spreadsheet-template` `application/vnd.oasis.opendocument.spreadsheet` `application/vnd.oasis.opendocument.text-master` `application/vnd.oasis.opendocument.text-template` `application/vnd.oasis.opendocument.text-web` `application/vnd.oasis.opendocument.text` `application/vnd.openofficeorg.extension` |
326
+ | osx | mach-o + `application/octet-stream` |
327
+ | pdf | `application/pdf` `text/plain` `application/octet-stream` |
328
+ | pe | `application/x-dosexec` |
329
+ | rtf | `application/rtf` `text/plain` |
330
+ | text & script | `application/x-httpd-php` `application/javascript` `application/json` `application/x-sh` `application/x-csh` `text/html` `text/plain` `text/x-python` `text/xml` `text/plain` `application/vnd.coffeescript` |
331
+ | winnt | pe + `text/plain` `application/octet-stream` `image/vnd.microsoft.icon` `text/plain` |
332
+
333
+ #### 1-100 Scale
334
+ | 0-100 Scale | Polyswarm Bid |
335
+ |-------------+---------------|
336
+ | 0 | 0 |
337
+ | 10 | 10 |
338
+ | 20 | 20 |
339
+ | 30 | 30 |
340
+ | 40 | 40 |
341
+ | 50 | 50 |
342
+ | 60 | 60 |
343
+ | 70 | 70 |
344
+ | 80 | 80 |
345
+ | 90 | 90 |
346
+ | 100 | 100 |
347
+
348
+
349
+ #### DNI Scale
350
+ Definition: [https://www.dni.gov/files/documents/ICD/ICD%20203%20Analytic%20Standards.pdf]
351
+
352
+ | DNI Scale | Polyswarm Bid |
353
+ |-----------------------------------------+---------------|
354
+ | Not Specified | Not Specified |
355
+ | Almost No Chance / Remote | 5 |
356
+ | Very Unlikely / Highly Improbable | 15 |
357
+ | Unlikely / Improbable | 30 |
358
+ | Roughly Even Chance / Roughly Even Odds | 50 |
359
+ | Likely / Probable | 70 |
360
+ | Very Likely / Highly Probable | 85 |
361
+ | Almost Certain / Nearly Certain | 95 |