assemblyline-core 4.5.1.dev388__tar.gz → 4.5.1.dev389__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (91) hide show
  1. assemblyline-core-4.5.1.dev389/PKG-INFO +239 -0
  2. assemblyline-core-4.5.1.dev389/README.md +216 -0
  3. assemblyline-core-4.5.1.dev389/assemblyline_core/VERSION +1 -0
  4. assemblyline-core-4.5.1.dev389/assemblyline_core.egg-info/PKG-INFO +239 -0
  5. assemblyline-core-4.5.1.dev388/PKG-INFO +0 -60
  6. assemblyline-core-4.5.1.dev388/README.md +0 -37
  7. assemblyline-core-4.5.1.dev388/assemblyline_core/VERSION +0 -1
  8. assemblyline-core-4.5.1.dev388/assemblyline_core.egg-info/PKG-INFO +0 -60
  9. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/LICENCE.md +0 -0
  10. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/__init__.py +0 -0
  11. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/alerter/__init__.py +0 -0
  12. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/alerter/processing.py +0 -0
  13. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/alerter/run_alerter.py +0 -0
  14. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/archiver/__init__.py +0 -0
  15. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/archiver/run_archiver.py +0 -0
  16. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/badlist_client.py +0 -0
  17. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/dispatching/__init__.py +0 -0
  18. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/dispatching/__main__.py +0 -0
  19. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/dispatching/client.py +0 -0
  20. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/dispatching/dispatcher.py +0 -0
  21. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/dispatching/schedules.py +0 -0
  22. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/dispatching/timeout.py +0 -0
  23. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/expiry/__init__.py +0 -0
  24. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/expiry/run_expiry.py +0 -0
  25. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/ingester/__init__.py +0 -0
  26. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/ingester/__main__.py +0 -0
  27. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/ingester/constants.py +0 -0
  28. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/ingester/ingester.py +0 -0
  29. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/metrics/__init__.py +0 -0
  30. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/metrics/es_metrics.py +0 -0
  31. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/metrics/heartbeat_formatter.py +0 -0
  32. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/metrics/helper.py +0 -0
  33. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/metrics/metrics_server.py +0 -0
  34. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/metrics/run_heartbeat_manager.py +0 -0
  35. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/metrics/run_metrics_aggregator.py +0 -0
  36. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/metrics/run_statistics_aggregator.py +0 -0
  37. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/plumber/__init__.py +0 -0
  38. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/plumber/run_plumber.py +0 -0
  39. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/replay/__init__.py +0 -0
  40. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/replay/client.py +0 -0
  41. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/replay/creator/__init__.py +0 -0
  42. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/replay/creator/run.py +0 -0
  43. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/replay/creator/run_worker.py +0 -0
  44. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/replay/loader/__init__.py +0 -0
  45. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/replay/loader/run.py +0 -0
  46. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/replay/loader/run_worker.py +0 -0
  47. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/replay/replay.py +0 -0
  48. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/safelist_client.py +0 -0
  49. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/scaler/__init__.py +0 -0
  50. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/scaler/collection.py +0 -0
  51. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/scaler/controllers/__init__.py +0 -0
  52. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/scaler/controllers/docker_ctl.py +0 -0
  53. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/scaler/controllers/interface.py +0 -0
  54. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/scaler/controllers/kubernetes_ctl.py +0 -0
  55. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/scaler/run_scaler.py +0 -0
  56. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/scaler/scaler_server.py +0 -0
  57. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/server_base.py +0 -0
  58. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/signature_client.py +0 -0
  59. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/submission_client.py +0 -0
  60. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/tasking_client.py +0 -0
  61. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/updater/__init__.py +0 -0
  62. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/updater/helper.py +0 -0
  63. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/updater/run_updater.py +0 -0
  64. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/vacuum/__init__.py +0 -0
  65. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/vacuum/crawler.py +0 -0
  66. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/vacuum/department_map.py +0 -0
  67. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/vacuum/safelist.py +0 -0
  68. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/vacuum/stream_map.py +0 -0
  69. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/vacuum/worker.py +0 -0
  70. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/workflow/__init__.py +0 -0
  71. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core/workflow/run_workflow.py +0 -0
  72. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core.egg-info/SOURCES.txt +0 -0
  73. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core.egg-info/dependency_links.txt +0 -0
  74. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core.egg-info/requires.txt +0 -0
  75. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/assemblyline_core.egg-info/top_level.txt +0 -0
  76. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/setup.cfg +0 -0
  77. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/setup.py +0 -0
  78. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_alerter.py +0 -0
  79. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_badlist_client.py +0 -0
  80. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_dispatcher.py +0 -0
  81. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_expiry.py +0 -0
  82. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_plumber.py +0 -0
  83. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_replay.py +0 -0
  84. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_safelist_client.py +0 -0
  85. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_scaler.py +0 -0
  86. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_scheduler.py +0 -0
  87. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_signature_client.py +0 -0
  88. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_simulation.py +0 -0
  89. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_vacuum.py +0 -0
  90. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_worker_ingest.py +0 -0
  91. {assemblyline-core-4.5.1.dev388 → assemblyline-core-4.5.1.dev389}/test/test_worker_submit.py +0 -0
@@ -0,0 +1,239 @@
1
+ Metadata-Version: 2.1
2
+ Name: assemblyline-core
3
+ Version: 4.5.1.dev389
4
+ Summary: Assemblyline 4 - Core components
5
+ Home-page: https://github.com/CybercentreCanada/assemblyline-core/
6
+ Author: CCCS Assemblyline development team
7
+ Author-email: assemblyline@cyber.gc.ca
8
+ License: MIT
9
+ Keywords: assemblyline automated malware analysis gc canada cse-cst cse cst cyber cccs
10
+ Classifier: Development Status :: 5 - Production/Stable
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: Topic :: Software Development :: Libraries
13
+ Classifier: License :: OSI Approved :: MIT License
14
+ Classifier: Programming Language :: Python :: 3.7
15
+ Classifier: Programming Language :: Python :: 3.8
16
+ Classifier: Programming Language :: Python :: 3.9
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Description-Content-Type: text/markdown
21
+ Provides-Extra: test
22
+ License-File: LICENCE.md
23
+
24
+ [![Discord](https://img.shields.io/badge/chat-on%20discord-7289da.svg?sanitize=true)](https://discord.gg/GUAy9wErNu)
25
+ [![](https://img.shields.io/discord/908084610158714900)](https://discord.gg/GUAy9wErNu)
26
+ [![Static Badge](https://img.shields.io/badge/github-assemblyline-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline)
27
+ [![Static Badge](https://img.shields.io/badge/github-assemblyline--core-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline-core)
28
+ [![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/CybercentreCanada/assemblyline/core)](https://github.com/CybercentreCanada/assemblyline/issues?q=is:issue+is:open+label:core)
29
+ [![License](https://img.shields.io/github/license/CybercentreCanada/assemblyline-core)](./LICENSE.md)
30
+
31
+ # Assemblyline 4 - Core
32
+
33
+ This repository provides cores services for Assemblyline 4.
34
+
35
+ ## Image variants and tags
36
+
37
+ | **Tag Type** | **Description** | **Example Tag** |
38
+ | :----------: | :----------------------------------------------------------------------------------------------- | :------------------------: |
39
+ | latest | The most recent build (can be unstable). | `latest` |
40
+ | build_type | The type of build used. `dev` is the latest unstable build. `stable` is the latest stable build. | `stable` or `dev` |
41
+ | series | Complete build details, including version and build type: `version.buildType`. | `4.5.stable`, `4.5.1.dev3` |
42
+
43
+ ## Components
44
+
45
+ ### Alerter
46
+
47
+ Create alerts for the different submissions in the system.
48
+
49
+ ```bash
50
+ docker run --name alerter cccs/assemblyline-core python -m assemblyline_core.alerter.run_alerter
51
+ ```
52
+
53
+ ### Archiver
54
+
55
+ Archives submissions and their results & files into the archive.
56
+
57
+ ```bash
58
+ docker run --name archiver cccs/assemblyline-core python -m assemblyline_core.archiver.run_archiver
59
+ ```
60
+
61
+ ### Dispatcher
62
+
63
+ Route the files in the system while a submission is tacking place. Make sure all files during a submission are completed by all required services.
64
+
65
+ ```bash
66
+ docker run --name dispatcher cccs/assemblyline-core python -m assemblyline_core.dispatching
67
+ ```
68
+
69
+ ### Expiry
70
+
71
+ Delete submissions and their results when their time-to-live expires.
72
+
73
+ ```bash
74
+ docker run --name expiry cccs/assemblyline-core python -m assemblyline_core.expiry.run_expiry
75
+ ```
76
+
77
+ ### Ingester
78
+
79
+ Move ingested files from the priority queues to the processing queues.
80
+
81
+ ```bash
82
+ docker run --name ingester cccs/assemblyline-core python -m assemblyline_core.ingester
83
+ ```
84
+
85
+ ### Metrics
86
+
87
+ Generates metrics of the different components in the system.
88
+
89
+ #### Heartbeat Manager
90
+
91
+ ```bash
92
+ docker run --name heartbeat cccs/assemblyline-core python -m assemblyline_core.metrics.run_heartbeat_manager
93
+ ```
94
+
95
+ #### Metrics Aggregator
96
+
97
+ ```bash
98
+ docker run --name metrics cccs/assemblyline-core python -m assemblyline_core.metrics.run_metrics_aggregator
99
+ ```
100
+
101
+ #### Statistics Aggregator
102
+
103
+ ```bash
104
+ docker run --name statistics cccs/assemblyline-core python -m assemblyline_core.metrics.run_statistics_aggregator
105
+ ```
106
+
107
+ ### Scaler
108
+
109
+ Spin up and down services in the system depending on the load.
110
+
111
+ ```bash
112
+ docker run --name scaler cccs/assemblyline-core python -m assemblyline_core.scaler.run_scaler
113
+ ```
114
+
115
+ ### Updater
116
+
117
+ Make sure the different services get their latest update files.
118
+
119
+ ```bash
120
+ docker run --name updater cccs/assemblyline-core python -m assemblyline_core.updater.run_updater
121
+ ```
122
+
123
+ ### Workflow
124
+
125
+ Run the different workflows in the system and apply their labels, priority and status.
126
+
127
+ ```bash
128
+ docker run --name workflow cccs/assemblyline-core python -m assemblyline_core.workflow.run_workflow
129
+ ```
130
+
131
+ ## Documentation
132
+
133
+ For more information about these Assemblyline components, follow this [overview](https://cybercentrecanada.github.io/assemblyline4_docs/overview/architecture/) of the system's architecture.
134
+
135
+ ---
136
+
137
+ # Assemblyline 4 - Core
138
+
139
+ Ce dépôt fournit des services de base pour Assemblyline 4.
140
+
141
+ ## Variantes et étiquettes d'image
142
+
143
+ | **Type d'étiquette** | **Description** | **Exemple d'étiquette** |
144
+ | :------------------: | :--------------------------------------------------------------------------------------------------------------- | :------------------------: |
145
+ | dernière | La version la plus récente (peut être instable). | `latest` |
146
+ | build_type | Le type de compilation utilisé. `dev` est la dernière version instable. `stable` est la dernière version stable. | `stable` ou `dev` |
147
+ | séries | Le détail de compilation utilisé, incluant la version et le type de compilation : `version.buildType`. | `4.5.stable`, `4.5.1.dev3` |
148
+
149
+ ## Composants
150
+
151
+ ### Alerter
152
+
153
+ Crée des alertes pour les différentes soumissions dans le système.
154
+
155
+ ```bash
156
+ docker run --name alerter cccs/assemblyline-core python -m assemblyline_core.alerter.run_alerter
157
+ ```
158
+
159
+ ### Archiver
160
+
161
+ Archivage des soumissions, de leurs résultats et des fichiers dans l'archive.
162
+
163
+ ```bash
164
+ docker run --name archiver cccs/assemblyline-core python -m assemblyline_core.archiver.run_archiver
165
+ ```
166
+
167
+ ### Dispatcher
168
+
169
+ Achemine les fichiers dans le système durant une soumission. S'assure que tous les fichiers de la soumission courante soient complétés par tous les services requis.
170
+
171
+ ```bash
172
+ docker run --name dispatcher cccs/assemblyline-core python -m assemblyline_core.dispatching
173
+ ```
174
+
175
+ ### Expiration
176
+
177
+ Supprimer les soumissions et leurs résultats à l'expiration de leur durée de vie.
178
+
179
+ ```bash
180
+ docker run --name expiry cccs/assemblyline-core python -m assemblyline_core.expiry.run_expiry
181
+ ```
182
+
183
+ ### Ingester
184
+
185
+ Déplace les fichiers ingérés des files d'attente prioritaires vers les files d'attente de traitement.
186
+
187
+ ```bash
188
+ docker run --name ingester cccs/assemblyline-core python -m assemblyline_core.ingester
189
+ ```
190
+
191
+ ### Métriques
192
+
193
+ Génère des métriques des différents composants du système.
194
+
195
+ #### Heartbeat Manager
196
+
197
+ ```bash
198
+ docker run --name heartbeat cccs/assemblyline-core python -m assemblyline_core.metrics.run_heartbeat_manager
199
+ ```
200
+
201
+ #### Agrégateur de métriques
202
+
203
+ ```bash
204
+ docker run --name metrics cccs/assemblyline-core python -m assemblyline_core.metrics.run_metrics_aggregator
205
+ ```
206
+
207
+ ##### Agrégateur de statistiques
208
+
209
+ ```bash
210
+ docker run --name statistics cccs/assemblyline-core python -m assemblyline_core.metrics.run_statistics_aggregator
211
+ ```
212
+
213
+ ### Scaler
214
+
215
+ Augmente et diminue les services dans le système en fonction de la charge.
216
+
217
+ ```bash
218
+ docker run --name scaler cccs/assemblyline-core python -m assemblyline_core.scaler.run_scaler
219
+ ```
220
+
221
+ ### Mise à jour
222
+
223
+ Assure que les différents services reçoivent leurs derniers fichiers de mise à jour.
224
+
225
+ ```bash
226
+ docker run --name updater cccs/assemblyline-core python -m assemblyline_core.updater.run_updater
227
+ ```
228
+
229
+ ### Workflow
230
+
231
+ Exécute les différents flux de travail dans le système et appliquer leurs étiquettes, leur priorité et leur statut.
232
+
233
+ ```bash
234
+ docker run --name workflow cccs/assemblyline-core python -m assemblyline_core.workflow.run_workflow
235
+ ```
236
+
237
+ ## Documentation
238
+
239
+ Pour plus d'informations sur ces composants Assemblyline, suivez ce [overview](https://cybercentrecanada.github.io/assemblyline4_docs/overview/architecture/) de l'architecture du système.
@@ -0,0 +1,216 @@
1
+ [![Discord](https://img.shields.io/badge/chat-on%20discord-7289da.svg?sanitize=true)](https://discord.gg/GUAy9wErNu)
2
+ [![](https://img.shields.io/discord/908084610158714900)](https://discord.gg/GUAy9wErNu)
3
+ [![Static Badge](https://img.shields.io/badge/github-assemblyline-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline)
4
+ [![Static Badge](https://img.shields.io/badge/github-assemblyline--core-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline-core)
5
+ [![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/CybercentreCanada/assemblyline/core)](https://github.com/CybercentreCanada/assemblyline/issues?q=is:issue+is:open+label:core)
6
+ [![License](https://img.shields.io/github/license/CybercentreCanada/assemblyline-core)](./LICENSE.md)
7
+
8
+ # Assemblyline 4 - Core
9
+
10
+ This repository provides cores services for Assemblyline 4.
11
+
12
+ ## Image variants and tags
13
+
14
+ | **Tag Type** | **Description** | **Example Tag** |
15
+ | :----------: | :----------------------------------------------------------------------------------------------- | :------------------------: |
16
+ | latest | The most recent build (can be unstable). | `latest` |
17
+ | build_type | The type of build used. `dev` is the latest unstable build. `stable` is the latest stable build. | `stable` or `dev` |
18
+ | series | Complete build details, including version and build type: `version.buildType`. | `4.5.stable`, `4.5.1.dev3` |
19
+
20
+ ## Components
21
+
22
+ ### Alerter
23
+
24
+ Create alerts for the different submissions in the system.
25
+
26
+ ```bash
27
+ docker run --name alerter cccs/assemblyline-core python -m assemblyline_core.alerter.run_alerter
28
+ ```
29
+
30
+ ### Archiver
31
+
32
+ Archives submissions and their results & files into the archive.
33
+
34
+ ```bash
35
+ docker run --name archiver cccs/assemblyline-core python -m assemblyline_core.archiver.run_archiver
36
+ ```
37
+
38
+ ### Dispatcher
39
+
40
+ Route the files in the system while a submission is tacking place. Make sure all files during a submission are completed by all required services.
41
+
42
+ ```bash
43
+ docker run --name dispatcher cccs/assemblyline-core python -m assemblyline_core.dispatching
44
+ ```
45
+
46
+ ### Expiry
47
+
48
+ Delete submissions and their results when their time-to-live expires.
49
+
50
+ ```bash
51
+ docker run --name expiry cccs/assemblyline-core python -m assemblyline_core.expiry.run_expiry
52
+ ```
53
+
54
+ ### Ingester
55
+
56
+ Move ingested files from the priority queues to the processing queues.
57
+
58
+ ```bash
59
+ docker run --name ingester cccs/assemblyline-core python -m assemblyline_core.ingester
60
+ ```
61
+
62
+ ### Metrics
63
+
64
+ Generates metrics of the different components in the system.
65
+
66
+ #### Heartbeat Manager
67
+
68
+ ```bash
69
+ docker run --name heartbeat cccs/assemblyline-core python -m assemblyline_core.metrics.run_heartbeat_manager
70
+ ```
71
+
72
+ #### Metrics Aggregator
73
+
74
+ ```bash
75
+ docker run --name metrics cccs/assemblyline-core python -m assemblyline_core.metrics.run_metrics_aggregator
76
+ ```
77
+
78
+ #### Statistics Aggregator
79
+
80
+ ```bash
81
+ docker run --name statistics cccs/assemblyline-core python -m assemblyline_core.metrics.run_statistics_aggregator
82
+ ```
83
+
84
+ ### Scaler
85
+
86
+ Spin up and down services in the system depending on the load.
87
+
88
+ ```bash
89
+ docker run --name scaler cccs/assemblyline-core python -m assemblyline_core.scaler.run_scaler
90
+ ```
91
+
92
+ ### Updater
93
+
94
+ Make sure the different services get their latest update files.
95
+
96
+ ```bash
97
+ docker run --name updater cccs/assemblyline-core python -m assemblyline_core.updater.run_updater
98
+ ```
99
+
100
+ ### Workflow
101
+
102
+ Run the different workflows in the system and apply their labels, priority and status.
103
+
104
+ ```bash
105
+ docker run --name workflow cccs/assemblyline-core python -m assemblyline_core.workflow.run_workflow
106
+ ```
107
+
108
+ ## Documentation
109
+
110
+ For more information about these Assemblyline components, follow this [overview](https://cybercentrecanada.github.io/assemblyline4_docs/overview/architecture/) of the system's architecture.
111
+
112
+ ---
113
+
114
+ # Assemblyline 4 - Core
115
+
116
+ Ce dépôt fournit des services de base pour Assemblyline 4.
117
+
118
+ ## Variantes et étiquettes d'image
119
+
120
+ | **Type d'étiquette** | **Description** | **Exemple d'étiquette** |
121
+ | :------------------: | :--------------------------------------------------------------------------------------------------------------- | :------------------------: |
122
+ | dernière | La version la plus récente (peut être instable). | `latest` |
123
+ | build_type | Le type de compilation utilisé. `dev` est la dernière version instable. `stable` est la dernière version stable. | `stable` ou `dev` |
124
+ | séries | Le détail de compilation utilisé, incluant la version et le type de compilation : `version.buildType`. | `4.5.stable`, `4.5.1.dev3` |
125
+
126
+ ## Composants
127
+
128
+ ### Alerter
129
+
130
+ Crée des alertes pour les différentes soumissions dans le système.
131
+
132
+ ```bash
133
+ docker run --name alerter cccs/assemblyline-core python -m assemblyline_core.alerter.run_alerter
134
+ ```
135
+
136
+ ### Archiver
137
+
138
+ Archivage des soumissions, de leurs résultats et des fichiers dans l'archive.
139
+
140
+ ```bash
141
+ docker run --name archiver cccs/assemblyline-core python -m assemblyline_core.archiver.run_archiver
142
+ ```
143
+
144
+ ### Dispatcher
145
+
146
+ Achemine les fichiers dans le système durant une soumission. S'assure que tous les fichiers de la soumission courante soient complétés par tous les services requis.
147
+
148
+ ```bash
149
+ docker run --name dispatcher cccs/assemblyline-core python -m assemblyline_core.dispatching
150
+ ```
151
+
152
+ ### Expiration
153
+
154
+ Supprimer les soumissions et leurs résultats à l'expiration de leur durée de vie.
155
+
156
+ ```bash
157
+ docker run --name expiry cccs/assemblyline-core python -m assemblyline_core.expiry.run_expiry
158
+ ```
159
+
160
+ ### Ingester
161
+
162
+ Déplace les fichiers ingérés des files d'attente prioritaires vers les files d'attente de traitement.
163
+
164
+ ```bash
165
+ docker run --name ingester cccs/assemblyline-core python -m assemblyline_core.ingester
166
+ ```
167
+
168
+ ### Métriques
169
+
170
+ Génère des métriques des différents composants du système.
171
+
172
+ #### Heartbeat Manager
173
+
174
+ ```bash
175
+ docker run --name heartbeat cccs/assemblyline-core python -m assemblyline_core.metrics.run_heartbeat_manager
176
+ ```
177
+
178
+ #### Agrégateur de métriques
179
+
180
+ ```bash
181
+ docker run --name metrics cccs/assemblyline-core python -m assemblyline_core.metrics.run_metrics_aggregator
182
+ ```
183
+
184
+ ##### Agrégateur de statistiques
185
+
186
+ ```bash
187
+ docker run --name statistics cccs/assemblyline-core python -m assemblyline_core.metrics.run_statistics_aggregator
188
+ ```
189
+
190
+ ### Scaler
191
+
192
+ Augmente et diminue les services dans le système en fonction de la charge.
193
+
194
+ ```bash
195
+ docker run --name scaler cccs/assemblyline-core python -m assemblyline_core.scaler.run_scaler
196
+ ```
197
+
198
+ ### Mise à jour
199
+
200
+ Assure que les différents services reçoivent leurs derniers fichiers de mise à jour.
201
+
202
+ ```bash
203
+ docker run --name updater cccs/assemblyline-core python -m assemblyline_core.updater.run_updater
204
+ ```
205
+
206
+ ### Workflow
207
+
208
+ Exécute les différents flux de travail dans le système et appliquer leurs étiquettes, leur priorité et leur statut.
209
+
210
+ ```bash
211
+ docker run --name workflow cccs/assemblyline-core python -m assemblyline_core.workflow.run_workflow
212
+ ```
213
+
214
+ ## Documentation
215
+
216
+ Pour plus d'informations sur ces composants Assemblyline, suivez ce [overview](https://cybercentrecanada.github.io/assemblyline4_docs/overview/architecture/) de l'architecture du système.
@@ -0,0 +1 @@
1
+ 4.5.1.dev389
@@ -0,0 +1,239 @@
1
+ Metadata-Version: 2.1
2
+ Name: assemblyline-core
3
+ Version: 4.5.1.dev389
4
+ Summary: Assemblyline 4 - Core components
5
+ Home-page: https://github.com/CybercentreCanada/assemblyline-core/
6
+ Author: CCCS Assemblyline development team
7
+ Author-email: assemblyline@cyber.gc.ca
8
+ License: MIT
9
+ Keywords: assemblyline automated malware analysis gc canada cse-cst cse cst cyber cccs
10
+ Classifier: Development Status :: 5 - Production/Stable
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: Topic :: Software Development :: Libraries
13
+ Classifier: License :: OSI Approved :: MIT License
14
+ Classifier: Programming Language :: Python :: 3.7
15
+ Classifier: Programming Language :: Python :: 3.8
16
+ Classifier: Programming Language :: Python :: 3.9
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Description-Content-Type: text/markdown
21
+ Provides-Extra: test
22
+ License-File: LICENCE.md
23
+
24
+ [![Discord](https://img.shields.io/badge/chat-on%20discord-7289da.svg?sanitize=true)](https://discord.gg/GUAy9wErNu)
25
+ [![](https://img.shields.io/discord/908084610158714900)](https://discord.gg/GUAy9wErNu)
26
+ [![Static Badge](https://img.shields.io/badge/github-assemblyline-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline)
27
+ [![Static Badge](https://img.shields.io/badge/github-assemblyline--core-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline-core)
28
+ [![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/CybercentreCanada/assemblyline/core)](https://github.com/CybercentreCanada/assemblyline/issues?q=is:issue+is:open+label:core)
29
+ [![License](https://img.shields.io/github/license/CybercentreCanada/assemblyline-core)](./LICENSE.md)
30
+
31
+ # Assemblyline 4 - Core
32
+
33
+ This repository provides cores services for Assemblyline 4.
34
+
35
+ ## Image variants and tags
36
+
37
+ | **Tag Type** | **Description** | **Example Tag** |
38
+ | :----------: | :----------------------------------------------------------------------------------------------- | :------------------------: |
39
+ | latest | The most recent build (can be unstable). | `latest` |
40
+ | build_type | The type of build used. `dev` is the latest unstable build. `stable` is the latest stable build. | `stable` or `dev` |
41
+ | series | Complete build details, including version and build type: `version.buildType`. | `4.5.stable`, `4.5.1.dev3` |
42
+
43
+ ## Components
44
+
45
+ ### Alerter
46
+
47
+ Create alerts for the different submissions in the system.
48
+
49
+ ```bash
50
+ docker run --name alerter cccs/assemblyline-core python -m assemblyline_core.alerter.run_alerter
51
+ ```
52
+
53
+ ### Archiver
54
+
55
+ Archives submissions and their results & files into the archive.
56
+
57
+ ```bash
58
+ docker run --name archiver cccs/assemblyline-core python -m assemblyline_core.archiver.run_archiver
59
+ ```
60
+
61
+ ### Dispatcher
62
+
63
+ Route the files in the system while a submission is tacking place. Make sure all files during a submission are completed by all required services.
64
+
65
+ ```bash
66
+ docker run --name dispatcher cccs/assemblyline-core python -m assemblyline_core.dispatching
67
+ ```
68
+
69
+ ### Expiry
70
+
71
+ Delete submissions and their results when their time-to-live expires.
72
+
73
+ ```bash
74
+ docker run --name expiry cccs/assemblyline-core python -m assemblyline_core.expiry.run_expiry
75
+ ```
76
+
77
+ ### Ingester
78
+
79
+ Move ingested files from the priority queues to the processing queues.
80
+
81
+ ```bash
82
+ docker run --name ingester cccs/assemblyline-core python -m assemblyline_core.ingester
83
+ ```
84
+
85
+ ### Metrics
86
+
87
+ Generates metrics of the different components in the system.
88
+
89
+ #### Heartbeat Manager
90
+
91
+ ```bash
92
+ docker run --name heartbeat cccs/assemblyline-core python -m assemblyline_core.metrics.run_heartbeat_manager
93
+ ```
94
+
95
+ #### Metrics Aggregator
96
+
97
+ ```bash
98
+ docker run --name metrics cccs/assemblyline-core python -m assemblyline_core.metrics.run_metrics_aggregator
99
+ ```
100
+
101
+ #### Statistics Aggregator
102
+
103
+ ```bash
104
+ docker run --name statistics cccs/assemblyline-core python -m assemblyline_core.metrics.run_statistics_aggregator
105
+ ```
106
+
107
+ ### Scaler
108
+
109
+ Spin up and down services in the system depending on the load.
110
+
111
+ ```bash
112
+ docker run --name scaler cccs/assemblyline-core python -m assemblyline_core.scaler.run_scaler
113
+ ```
114
+
115
+ ### Updater
116
+
117
+ Make sure the different services get their latest update files.
118
+
119
+ ```bash
120
+ docker run --name updater cccs/assemblyline-core python -m assemblyline_core.updater.run_updater
121
+ ```
122
+
123
+ ### Workflow
124
+
125
+ Run the different workflows in the system and apply their labels, priority and status.
126
+
127
+ ```bash
128
+ docker run --name workflow cccs/assemblyline-core python -m assemblyline_core.workflow.run_workflow
129
+ ```
130
+
131
+ ## Documentation
132
+
133
+ For more information about these Assemblyline components, follow this [overview](https://cybercentrecanada.github.io/assemblyline4_docs/overview/architecture/) of the system's architecture.
134
+
135
+ ---
136
+
137
+ # Assemblyline 4 - Core
138
+
139
+ Ce dépôt fournit des services de base pour Assemblyline 4.
140
+
141
+ ## Variantes et étiquettes d'image
142
+
143
+ | **Type d'étiquette** | **Description** | **Exemple d'étiquette** |
144
+ | :------------------: | :--------------------------------------------------------------------------------------------------------------- | :------------------------: |
145
+ | dernière | La version la plus récente (peut être instable). | `latest` |
146
+ | build_type | Le type de compilation utilisé. `dev` est la dernière version instable. `stable` est la dernière version stable. | `stable` ou `dev` |
147
+ | séries | Le détail de compilation utilisé, incluant la version et le type de compilation : `version.buildType`. | `4.5.stable`, `4.5.1.dev3` |
148
+
149
+ ## Composants
150
+
151
+ ### Alerter
152
+
153
+ Crée des alertes pour les différentes soumissions dans le système.
154
+
155
+ ```bash
156
+ docker run --name alerter cccs/assemblyline-core python -m assemblyline_core.alerter.run_alerter
157
+ ```
158
+
159
+ ### Archiver
160
+
161
+ Archivage des soumissions, de leurs résultats et des fichiers dans l'archive.
162
+
163
+ ```bash
164
+ docker run --name archiver cccs/assemblyline-core python -m assemblyline_core.archiver.run_archiver
165
+ ```
166
+
167
+ ### Dispatcher
168
+
169
+ Achemine les fichiers dans le système durant une soumission. S'assure que tous les fichiers de la soumission courante soient complétés par tous les services requis.
170
+
171
+ ```bash
172
+ docker run --name dispatcher cccs/assemblyline-core python -m assemblyline_core.dispatching
173
+ ```
174
+
175
+ ### Expiration
176
+
177
+ Supprimer les soumissions et leurs résultats à l'expiration de leur durée de vie.
178
+
179
+ ```bash
180
+ docker run --name expiry cccs/assemblyline-core python -m assemblyline_core.expiry.run_expiry
181
+ ```
182
+
183
+ ### Ingester
184
+
185
+ Déplace les fichiers ingérés des files d'attente prioritaires vers les files d'attente de traitement.
186
+
187
+ ```bash
188
+ docker run --name ingester cccs/assemblyline-core python -m assemblyline_core.ingester
189
+ ```
190
+
191
+ ### Métriques
192
+
193
+ Génère des métriques des différents composants du système.
194
+
195
+ #### Heartbeat Manager
196
+
197
+ ```bash
198
+ docker run --name heartbeat cccs/assemblyline-core python -m assemblyline_core.metrics.run_heartbeat_manager
199
+ ```
200
+
201
+ #### Agrégateur de métriques
202
+
203
+ ```bash
204
+ docker run --name metrics cccs/assemblyline-core python -m assemblyline_core.metrics.run_metrics_aggregator
205
+ ```
206
+
207
+ ##### Agrégateur de statistiques
208
+
209
+ ```bash
210
+ docker run --name statistics cccs/assemblyline-core python -m assemblyline_core.metrics.run_statistics_aggregator
211
+ ```
212
+
213
+ ### Scaler
214
+
215
+ Augmente et diminue les services dans le système en fonction de la charge.
216
+
217
+ ```bash
218
+ docker run --name scaler cccs/assemblyline-core python -m assemblyline_core.scaler.run_scaler
219
+ ```
220
+
221
+ ### Mise à jour
222
+
223
+ Assure que les différents services reçoivent leurs derniers fichiers de mise à jour.
224
+
225
+ ```bash
226
+ docker run --name updater cccs/assemblyline-core python -m assemblyline_core.updater.run_updater
227
+ ```
228
+
229
+ ### Workflow
230
+
231
+ Exécute les différents flux de travail dans le système et appliquer leurs étiquettes, leur priorité et leur statut.
232
+
233
+ ```bash
234
+ docker run --name workflow cccs/assemblyline-core python -m assemblyline_core.workflow.run_workflow
235
+ ```
236
+
237
+ ## Documentation
238
+
239
+ Pour plus d'informations sur ces composants Assemblyline, suivez ce [overview](https://cybercentrecanada.github.io/assemblyline4_docs/overview/architecture/) de l'architecture du système.
@@ -1,60 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: assemblyline-core
3
- Version: 4.5.1.dev388
4
- Summary: Assemblyline 4 - Core components
5
- Home-page: https://github.com/CybercentreCanada/assemblyline-core/
6
- Author: CCCS Assemblyline development team
7
- Author-email: assemblyline@cyber.gc.ca
8
- License: MIT
9
- Keywords: assemblyline automated malware analysis gc canada cse-cst cse cst cyber cccs
10
- Classifier: Development Status :: 5 - Production/Stable
11
- Classifier: Intended Audience :: Developers
12
- Classifier: Topic :: Software Development :: Libraries
13
- Classifier: License :: OSI Approved :: MIT License
14
- Classifier: Programming Language :: Python :: 3.7
15
- Classifier: Programming Language :: Python :: 3.8
16
- Classifier: Programming Language :: Python :: 3.9
17
- Classifier: Programming Language :: Python :: 3.10
18
- Classifier: Programming Language :: Python :: 3.11
19
- Classifier: Programming Language :: Python :: 3.12
20
- Description-Content-Type: text/markdown
21
- Provides-Extra: test
22
- License-File: LICENCE.md
23
-
24
- # Assemblyline 4 - Core services
25
-
26
- This repository provides cores services for Assemblyline 4.
27
-
28
- #### Core Services
29
-
30
- ##### Alerter
31
-
32
- Create alerts for the different submissions in the system.
33
-
34
- ##### Dispatcher
35
-
36
- Route the files in the system while a submission is tacking place. Make sure all files during a submission are completed by all required services.
37
-
38
- ##### Expiry
39
-
40
- Delete submissions and their results when their TTL expires.
41
-
42
- ##### Ingester
43
-
44
- Move ingested files from the priority queues to the processing queues.
45
-
46
- ##### Metrics
47
-
48
- Generates metrics of the different components in the system.
49
-
50
- ##### Scaler
51
-
52
- Spin up and down services in the system depending on the load.
53
-
54
- ##### Updater
55
-
56
- Make sure the different services get their latest update files.
57
-
58
- ##### Workflow
59
-
60
- Run the different workflows in the system and apply their labels, priority and status.
@@ -1,37 +0,0 @@
1
- # Assemblyline 4 - Core services
2
-
3
- This repository provides cores services for Assemblyline 4.
4
-
5
- #### Core Services
6
-
7
- ##### Alerter
8
-
9
- Create alerts for the different submissions in the system.
10
-
11
- ##### Dispatcher
12
-
13
- Route the files in the system while a submission is tacking place. Make sure all files during a submission are completed by all required services.
14
-
15
- ##### Expiry
16
-
17
- Delete submissions and their results when their TTL expires.
18
-
19
- ##### Ingester
20
-
21
- Move ingested files from the priority queues to the processing queues.
22
-
23
- ##### Metrics
24
-
25
- Generates metrics of the different components in the system.
26
-
27
- ##### Scaler
28
-
29
- Spin up and down services in the system depending on the load.
30
-
31
- ##### Updater
32
-
33
- Make sure the different services get their latest update files.
34
-
35
- ##### Workflow
36
-
37
- Run the different workflows in the system and apply their labels, priority and status.
@@ -1 +0,0 @@
1
- 4.5.1.dev388
@@ -1,60 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: assemblyline-core
3
- Version: 4.5.1.dev388
4
- Summary: Assemblyline 4 - Core components
5
- Home-page: https://github.com/CybercentreCanada/assemblyline-core/
6
- Author: CCCS Assemblyline development team
7
- Author-email: assemblyline@cyber.gc.ca
8
- License: MIT
9
- Keywords: assemblyline automated malware analysis gc canada cse-cst cse cst cyber cccs
10
- Classifier: Development Status :: 5 - Production/Stable
11
- Classifier: Intended Audience :: Developers
12
- Classifier: Topic :: Software Development :: Libraries
13
- Classifier: License :: OSI Approved :: MIT License
14
- Classifier: Programming Language :: Python :: 3.7
15
- Classifier: Programming Language :: Python :: 3.8
16
- Classifier: Programming Language :: Python :: 3.9
17
- Classifier: Programming Language :: Python :: 3.10
18
- Classifier: Programming Language :: Python :: 3.11
19
- Classifier: Programming Language :: Python :: 3.12
20
- Description-Content-Type: text/markdown
21
- Provides-Extra: test
22
- License-File: LICENCE.md
23
-
24
- # Assemblyline 4 - Core services
25
-
26
- This repository provides cores services for Assemblyline 4.
27
-
28
- #### Core Services
29
-
30
- ##### Alerter
31
-
32
- Create alerts for the different submissions in the system.
33
-
34
- ##### Dispatcher
35
-
36
- Route the files in the system while a submission is tacking place. Make sure all files during a submission are completed by all required services.
37
-
38
- ##### Expiry
39
-
40
- Delete submissions and their results when their TTL expires.
41
-
42
- ##### Ingester
43
-
44
- Move ingested files from the priority queues to the processing queues.
45
-
46
- ##### Metrics
47
-
48
- Generates metrics of the different components in the system.
49
-
50
- ##### Scaler
51
-
52
- Spin up and down services in the system depending on the load.
53
-
54
- ##### Updater
55
-
56
- Make sure the different services get their latest update files.
57
-
58
- ##### Workflow
59
-
60
- Run the different workflows in the system and apply their labels, priority and status.