neuralui 0.12.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- neuralui-0.12.2/PKG-INFO +679 -0
- neuralui-0.12.2/README.md +658 -0
- neuralui-0.12.2/neuralui/__init__.py +0 -0
- neuralui-0.12.2/neuralui/__main__.py +7 -0
- neuralui-0.12.2/neuralui/actions.py +504 -0
- neuralui-0.12.2/neuralui/checkpkg.py +60 -0
- neuralui-0.12.2/neuralui/export.py +103 -0
- neuralui-0.12.2/neuralui/handlers.py +1885 -0
- neuralui-0.12.2/neuralui/jobs.py +400 -0
- neuralui-0.12.2/neuralui/limits.py +71 -0
- neuralui-0.12.2/neuralui/lint.py +508 -0
- neuralui-0.12.2/neuralui/pkg.py +27 -0
- neuralui-0.12.2/neuralui/pkgproto/neuralui.yaml +195 -0
- neuralui-0.12.2/neuralui/pkgproto/storm/commands/neuralui.query.migrate.storm +12 -0
- neuralui-0.12.2/neuralui/pkgproto/storm/commands/neuralui.query.save.storm +1 -0
- neuralui-0.12.2/neuralui/pkgproto/storm/commands/neuralui.scrape.migrate.storm +20 -0
- neuralui-0.12.2/neuralui/pkgproto/storm/modules/neuralui.storm +121 -0
- neuralui-0.12.2/neuralui/schema/neuralui.schema.json +144 -0
- neuralui-0.12.2/neuralui/security.py +265 -0
- neuralui-0.12.2/neuralui/service.py +342 -0
- neuralui-0.12.2/neuralui/store.py +399 -0
- neuralui-0.12.2/neuralui/version.py +12 -0
- neuralui-0.12.2/neuralui/web/brand/neuralui-compass-primary-180.png +0 -0
- neuralui-0.12.2/neuralui/web/brand/neuralui-compass-primary-light.svg +1 -0
- neuralui-0.12.2/neuralui/web/brand/neuralui-compass-primary.svg +1 -0
- neuralui-0.12.2/neuralui/web/brand/neuralui-compass-small-32.png +0 -0
- neuralui-0.12.2/neuralui/web/brand/neuralui-compass-small-light.svg +1 -0
- neuralui-0.12.2/neuralui/web/brand/neuralui-compass-small.svg +1 -0
- neuralui-0.12.2/neuralui/web/css/app.css +1420 -0
- neuralui-0.12.2/neuralui/web/docs/actions.md +137 -0
- neuralui-0.12.2/neuralui/web/docs/admin.md +45 -0
- neuralui-0.12.2/neuralui/web/docs/automation.md +48 -0
- neuralui-0.12.2/neuralui/web/docs/console.md +25 -0
- neuralui-0.12.2/neuralui/web/docs/details.md +55 -0
- neuralui-0.12.2/neuralui/web/docs/explore.md +11 -0
- neuralui-0.12.2/neuralui/web/docs/graph.md +71 -0
- neuralui-0.12.2/neuralui/web/docs/history.md +83 -0
- neuralui-0.12.2/neuralui/web/docs/img/about.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/action-palette.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/add-page.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/admin-actions.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/applicator.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/automation.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/bulkedit.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/command-dialog.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/completion.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/drill.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/enrich-menu.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/error-state.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/graph-large.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/graph-okabe-ito.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/graph.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/guide.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/history.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/import.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/jobs-panel.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/lint.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/menu.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/note.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/open-in.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/output-view.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/palette.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/scrape-results.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/scrape-review.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/script.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/table.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/tagnode.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/tags.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/theme-high-contrast.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/theme-light.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/update-banner.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/img/viewer-hex.png +0 -0
- neuralui-0.12.2/neuralui/web/docs/import.md +57 -0
- neuralui-0.12.2/neuralui/web/docs/keyboard.md +45 -0
- neuralui-0.12.2/neuralui/web/docs/library.md +49 -0
- neuralui-0.12.2/neuralui/web/docs/overview.md +76 -0
- neuralui-0.12.2/neuralui/web/docs/queries.md +103 -0
- neuralui-0.12.2/neuralui/web/docs/results.md +103 -0
- neuralui-0.12.2/neuralui/web/docs/scrape.md +39 -0
- neuralui-0.12.2/neuralui/web/docs/settings.md +94 -0
- neuralui-0.12.2/neuralui/web/docs/tags.md +88 -0
- neuralui-0.12.2/neuralui/web/index.html +21 -0
- neuralui-0.12.2/neuralui/web/js/actionpalette.js +99 -0
- neuralui-0.12.2/neuralui/web/js/actions.js +165 -0
- neuralui-0.12.2/neuralui/web/js/addpage.js +62 -0
- neuralui-0.12.2/neuralui/web/js/admin.js +548 -0
- neuralui-0.12.2/neuralui/web/js/api.js +316 -0
- neuralui-0.12.2/neuralui/web/js/app.js +529 -0
- neuralui-0.12.2/neuralui/web/js/automation.js +723 -0
- neuralui-0.12.2/neuralui/web/js/boot.js +7 -0
- neuralui-0.12.2/neuralui/web/js/complete.js +272 -0
- neuralui-0.12.2/neuralui/web/js/console.js +204 -0
- neuralui-0.12.2/neuralui/web/js/dialog.js +267 -0
- neuralui-0.12.2/neuralui/web/js/drill.js +301 -0
- neuralui-0.12.2/neuralui/web/js/editor.js +725 -0
- neuralui-0.12.2/neuralui/web/js/editors.js +640 -0
- neuralui-0.12.2/neuralui/web/js/explore.js +161 -0
- neuralui-0.12.2/neuralui/web/js/export.js +222 -0
- neuralui-0.12.2/neuralui/web/js/files.js +85 -0
- neuralui-0.12.2/neuralui/web/js/graph.js +530 -0
- neuralui-0.12.2/neuralui/web/js/help.js +35 -0
- neuralui-0.12.2/neuralui/web/js/helppage.js +304 -0
- neuralui-0.12.2/neuralui/web/js/import.js +408 -0
- neuralui-0.12.2/neuralui/web/js/importfiles.js +262 -0
- neuralui-0.12.2/neuralui/web/js/jobs.js +211 -0
- neuralui-0.12.2/neuralui/web/js/layout-worker.js +107 -0
- neuralui-0.12.2/neuralui/web/js/livemd.js +509 -0
- neuralui-0.12.2/neuralui/web/js/lookups.js +81 -0
- neuralui-0.12.2/neuralui/web/js/menu.js +752 -0
- neuralui-0.12.2/neuralui/web/js/model.js +267 -0
- neuralui-0.12.2/neuralui/web/js/notepage.js +111 -0
- neuralui-0.12.2/neuralui/web/js/notes.js +363 -0
- neuralui-0.12.2/neuralui/web/js/page.js +91 -0
- neuralui-0.12.2/neuralui/web/js/palette.js +128 -0
- neuralui-0.12.2/neuralui/web/js/queries.js +163 -0
- neuralui-0.12.2/neuralui/web/js/results.js +1006 -0
- neuralui-0.12.2/neuralui/web/js/saved.js +250 -0
- neuralui-0.12.2/neuralui/web/js/scrape.js +328 -0
- neuralui-0.12.2/neuralui/web/js/script.js +373 -0
- neuralui-0.12.2/neuralui/web/js/state.js +268 -0
- neuralui-0.12.2/neuralui/web/js/tabs.js +1118 -0
- neuralui-0.12.2/neuralui/web/js/tags.js +114 -0
- neuralui-0.12.2/neuralui/web/js/undo.js +104 -0
- neuralui-0.12.2/neuralui/web/js/util.js +195 -0
- neuralui-0.12.2/neuralui/web/js/viewer.js +437 -0
- neuralui-0.12.2/neuralui/web/vendor/cytoscape.LICENSE +19 -0
- neuralui-0.12.2/neuralui/web/vendor/cytoscape.min.js +32 -0
- neuralui-0.12.2/neuralui/web/vendor/dompurify.LICENSE +202 -0
- neuralui-0.12.2/neuralui/web/vendor/inter/OFL.txt +93 -0
- neuralui-0.12.2/neuralui/web/vendor/inter/inter-latin-ext-wght-normal.woff2 +0 -0
- neuralui-0.12.2/neuralui/web/vendor/inter/inter-latin-wght-normal.woff2 +0 -0
- neuralui-0.12.2/neuralui/web/vendor/marked.LICENSE +44 -0
- neuralui-0.12.2/neuralui/web/vendor/marked.min.js +6 -0
- neuralui-0.12.2/neuralui/web/vendor/purify.min.js +7 -0
- neuralui-0.12.2/neuralui.egg-info/PKG-INFO +679 -0
- neuralui-0.12.2/neuralui.egg-info/SOURCES.txt +140 -0
- neuralui-0.12.2/neuralui.egg-info/dependency_links.txt +1 -0
- neuralui-0.12.2/neuralui.egg-info/requires.txt +17 -0
- neuralui-0.12.2/neuralui.egg-info/top_level.txt +1 -0
- neuralui-0.12.2/pyproject.toml +36 -0
- neuralui-0.12.2/setup.cfg +4 -0
- neuralui-0.12.2/tests/test_neuralui.py +2627 -0
neuralui-0.12.2/PKG-INFO
ADDED
|
@@ -0,0 +1,679 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: neuralui
|
|
3
|
+
Version: 0.12.2
|
|
4
|
+
Summary: A lightweight web UI for the open source Synapse Cortex, built as a Synapse service.
|
|
5
|
+
Requires-Python: <3.12,>=3.11
|
|
6
|
+
Description-Content-Type: text/markdown
|
|
7
|
+
Requires-Dist: synapse<3.0.0,>=2.256.0
|
|
8
|
+
Requires-Dist: stormlibpp<2,>=1.1.1
|
|
9
|
+
Provides-Extra: dev
|
|
10
|
+
Requires-Dist: pytest; extra == "dev"
|
|
11
|
+
Requires-Dist: aiohttp; extra == "dev"
|
|
12
|
+
Requires-Dist: build; extra == "dev"
|
|
13
|
+
Provides-Extra: docs
|
|
14
|
+
Requires-Dist: sphinx<10,>=9; extra == "docs"
|
|
15
|
+
Requires-Dist: myst-parser<6,>=5; extra == "docs"
|
|
16
|
+
Requires-Dist: shibuya; extra == "docs"
|
|
17
|
+
Provides-Extra: brand
|
|
18
|
+
Requires-Dist: fonttools; extra == "brand"
|
|
19
|
+
Requires-Dist: brotli; extra == "brand"
|
|
20
|
+
Requires-Dist: uharfbuzz; extra == "brand"
|
|
21
|
+
|
|
22
|
+
<p align="center">
|
|
23
|
+
<picture>
|
|
24
|
+
<source media="(prefers-color-scheme: dark)" srcset="assets/brand/v1.0.0/neuralui-lockup-color.svg">
|
|
25
|
+
<img src="assets/brand/v1.0.0/neuralui-lockup-dark.svg" width="320" alt="NeuralUI. See the connections.">
|
|
26
|
+
</picture>
|
|
27
|
+
</p>
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
A small, focused web UI for the open source [Synapse](https://github.com/vertexproject/synapse) Cortex.
|
|
31
|
+
It is a Synapse service (a `Cell` and a Storm service) rather than a separate web app, and it uses the
|
|
32
|
+
Cortex for everything it can: authentication, permissions, Storm, the Axon and user state.
|
|
33
|
+
|
|
34
|
+
It is not meant to replace Optic. It covers one analyst's daily workflow: query, review, tag, pivot,
|
|
35
|
+
scrape and move files.
|
|
36
|
+
|
|
37
|
+
**Documentation: [docs.gormo.co/neuralui](https://docs.gormo.co/neuralui/)**: install and upgrade, the
|
|
38
|
+
user guide (also in NeuralUI under **Help**), and the reference (settings, HTTP API, Storm packages).
|
|
39
|
+
`make docs` builds the same site locally into `build/docs/`.
|
|
40
|
+
|
|
41
|
+

|
|
42
|
+
|
|
43
|
+
## Features
|
|
44
|
+
|
|
45
|
+
| Requirement | How NeuralUI does it |
|
|
46
|
+
| --- | --- |
|
|
47
|
+
| Streaming query results | One WebSocket carries any number of concurrent Storm queries. Nodes are rendered as the Cortex emits them, and a query can be cancelled. |
|
|
48
|
+
| Smart results table | Results are grouped into one section per form. Each section shows that form's secondary props (only the ones that appear in the results, in model order) between fixed columns: primary value, `tags`, any pinned tag columns, `.seen` and `.created`. Columns can be sorted, hidden and filtered. |
|
|
49
|
+
| Multi-line, tabbed queries | Storm editor with syntax highlighting, line numbers, auto-indent, Tab/Shift+Tab indenting, `Ctrl+/` comments, line wrapping (`Alt+Z`) and `Shift+Enter` to run. Tabs (with rename and duplicate) are saved to your Cortex user profile. |
|
|
50
|
+
| Storm output | Each tab keeps a transcript of what Storm said, run by run, like the Storm CLI: prints, warnings, errors, lookup misses, fired events and CSV rows (never nodes), each run headed by its query and ended with its node count and time. It sits under the results as **Messages**, or fills the tab in the **Output** view, where a run only counts its nodes: use it for commands and admin work. |
|
|
51
|
+
| Editor help | Completion for forms, props, tags, commands and `$lib` (with signatures and docs), and docs when you hover a name. See [Writing queries](#writing-queries). |
|
|
52
|
+
| History, sharing, export | Query history kept by the service (not in the graph), short share links, and export as CSV, JSON lines, Synapse `.nodes` (optionally with related nodes) or STIX. See [History, links and export](#history-links-and-export). |
|
|
53
|
+
| Storm lint | Problems show up as you type, before you run anything: squiggles, gutter markers and a problems list. Unknown forms, props, commands, libraries and command args are checked along with syntax. See [Storm lint and highlighting](#storm-lint-and-highlighting). |
|
|
54
|
+
| Auth in the Cortex | You log in with Cortex credentials. The service keeps no users of its own; every permission check is made by the Cortex. |
|
|
55
|
+
| User state in the graph | No model extensions. Saved queries are `it:exec:query` nodes tagged `#meta.neuralui.query`; scrape sessions are `meta:note` nodes linked `-(about)>` to the nodes they accepted. Display preferences (open tabs, hidden columns, tag colors) are kept in the Cortex user profile. |
|
|
56
|
+
| A Synapse service | `NeuralUI` is a `s_cell.Cell` with a `StormSvc` API. `service.add` installs the `neuralui` Storm package (Storm module, command, default node actions and note types). |
|
|
57
|
+
| Bulk node editing | **Edit properties** works on any selection. Each field is model-driven and type-aware (text, time, interval, bool, arrays, `data` as JSON). Fields where the selected nodes disagree show "(mixed values)". Only the fields you change are applied, to every selected node, and props can be deleted. **New node** creates a node of any form with its props and tags. |
|
|
58
|
+
| Notes | A `meta:note` editor with live rendered markdown: the line you edit is plain text, the rest is rendered, like Obsidian (or raw / read views). The note's `:type` taxonomy picks how it is edited (markdown or plain text), and Storm packages can register more note types. The drill pane shows the notes about any node. |
|
|
59
|
+
| File viewer | **View file...** on a `file:bytes` node (right-click, the details pane, or after an upload). Nothing is read from the Axon until you open it. Every file opens in the **hex** view (paged, jump to offset). The **strings** view (ASCII and UTF-16LE, minimum length, filter) reads the file a MiB at a time. Images and PDFs can be **rendered**, but only when asked, each time. See [File viewer safety](#file-viewer-safety). |
|
|
60
|
+
| `colorize` | Colors from Storm's `colorize` command (`... \| colorize red`) mark rows in the table and override node colors in the graph. |
|
|
61
|
+
| Axon upload and download | Drag-and-drop upload creates `file:bytes` nodes (with hashes, name, mime and optional tags). Download from any `file:bytes` node's right-click menu. |
|
|
62
|
+
| Right-click actions per form | Actions are declared by Storm packages and matched to forms by glob: in a `neuralui` key, or the Optic actions of Rapid Power-Ups (`optic: actions:`). There are five types: `query` (new query), `newtab`, `filter` (appends to the current query), `storm` (runs on the Cortex with the selected nodes as input) and `link` (opens a web page for the node, as in **Open in**). Works on multiple selected nodes. |
|
|
63
|
+
| Tagging interface | Three editors, all bulk-capable. **Add tag** takes just a name. **Apply tag** sets a tag with an interval and any tag props declared in the model (`data` tag props take nested JSON). The **tag node editor** creates and edits `syn:tag` nodes (`:title`, `:doc`, ...). **Manage tags** (press `t`) lists the tags, intervals and tag props across the selection, with remove and apply-to-all. Tag chips have their own menu. There is a tag tree in the sidebar. |
|
|
64
|
+
| Scrape | A scrape is its own tab. Paste text, then review the candidates `$lib.scrape` found (marked new or existing). Edit their forms, values and tags, then accept them. The accepted nodes, new and existing, appear as normal results in the same tab, so you can bulk-tag, edit and pivot before leaving. Each session is recorded as a `neuralui.scrape` note linked to its nodes, so it can be reopened from Library → Scrapes. |
|
|
65
|
+
| Graph view | Cytoscape graph of the current results. Prop references and light edges are drawn between nodes. Nodes can be colored by form, by tag (honoring tag colors) or by any property value, with a legend. Right-click to expand neighbors. |
|
|
66
|
+
| Console | A **Console** tab is for developers and admins: the Storm editor and its streamed output side by side or stacked, with no result tables. Nodes are counted (or listed one per line), with the edits each run made. Runs can be read-only, take variables, and run in the current view, a **throwaway fork** (deleted when the run ends) or a **kept fork** you review and then merge or delete. |
|
|
67
|
+
| Import | An **Import** tab adds files to a console tab: drop Storm scripts and data files, a folder or a `.zip`, and they import in folder order the way [stormlibpp's import tool](https://github.com/gormaniac/stormlibpp/blob/main/src/stormlibpp/import.py) does (`.storm` scripts with their `.csv`, `.json` and `.txt` data files in `$rows`, `.storml` scripts line by line). |
|
|
68
|
+
| Admin | Users (admin, locked, password, email, roles, rules everywhere or per view, API keys), roles (rules and members) and Storm packages (add from a JSON file, remove). For admins. |
|
|
69
|
+
| Automation | Cron jobs, triggers, macros and dmons: list, add, edit their Storm in the editor, enable or disable, delete; dmon logs. |
|
|
70
|
+
| Help | A guide to NeuralUI itself (markdown in `neuralui/web/docs/`), every command's help as the Cortex prints it, the `$lib` reference, a data model explorer (forms, props, types, edges, node counts) and links to the Synapse docs. |
|
|
71
|
+
| Explore | The sidebar's lower panel switches between the tag tree and lists of the threats, tools and techniques (`risk:threat`, `risk:tool:software`, `ou:technique`) in the view. |
|
|
72
|
+
| Drill-down pane | Clicking a node shows all of its props and tags (with intervals and tag props), plus its neighbors in four groups: props out, props in, light edges out and light edges in. Each group shows the prop or verb that links it. You can navigate between nodes with back history. |
|
|
73
|
+
|
|
74
|
+
| Graph | Drill-down + menu | Bulk edit |
|
|
75
|
+
| --- | --- | --- |
|
|
76
|
+
|  |  |  |
|
|
77
|
+
|
|
78
|
+
| Scrape review | Scrape results (editable in place) | Markdown note |
|
|
79
|
+
| --- | --- | --- |
|
|
80
|
+
|  |  |  |
|
|
81
|
+
|
|
82
|
+
| Console (stacked) | Import | Automation |
|
|
83
|
+
| --- | --- | --- |
|
|
84
|
+
|  |  |  |
|
|
85
|
+
|
|
86
|
+
| Tag with interval + tag props | Tag node editor | Tag manager |
|
|
87
|
+
| --- | --- | --- |
|
|
88
|
+
|  |  |  |
|
|
89
|
+
|
|
90
|
+
| File viewer: hex | File viewer: strings | Render a PDF (opt in) |
|
|
91
|
+
| --- | --- | --- |
|
|
92
|
+
|  |  |  |
|
|
93
|
+
|
|
94
|
+
## Brand and themes
|
|
95
|
+
|
|
96
|
+
The brand (v1.0.0, the A2 Refined Network Compass) lives in [`assets/brand/v1.0.0/`](assets/brand/v1.0.0/):
|
|
97
|
+
the canonical SVG marks, derived single color marks and lockups, PNG exports, `tokens.json` and the
|
|
98
|
+
usage rules. The UI uses the
|
|
99
|
+
small mark below 64 px (top bar, favicon) and the primary mark above it (login, touch icon).
|
|
100
|
+
|
|
101
|
+
Themes are chosen per user under **user menu → Theme** and saved in the Cortex user profile:
|
|
102
|
+
|
|
103
|
+
| Theme | Description |
|
|
104
|
+
| --- | --- |
|
|
105
|
+
| System | The default: Dark or Light, following the OS setting |
|
|
106
|
+
| Dark / Light | The defaults, ignoring the OS |
|
|
107
|
+
| NeuralUI | The brand: Deep Navy surfaces, Ice text, Electric Cyan accent, Indigo / Violet highlights, Inter, and a brand-led graph palette |
|
|
108
|
+
| NeuralUI light | The brand reversed for light mode. Surfaces are Ice and white, text is Deep Navy, and the accents are deepened enough to pass contrast on white |
|
|
109
|
+
| High contrast / High contrast light | Black and white with bold accents, strong borders and thicker focus rings |
|
|
110
|
+
| Solarized dark / light, Nord, Dracula | The familiar editor palettes. Hues are kept; colors that failed contrast were lightened or darkened (lightness only) until they pass |
|
|
111
|
+
| Terminal / Terminal amber | Black, with phosphor green or amber text and borders, and monospace throughout. The logo is the single color mark tinted to match |
|
|
112
|
+
|
|
113
|
+
**Graph colors** (user menu) can be switched to a colorblind safe palette (Okabe-Ito) under any theme.
|
|
114
|
+
|
|
115
|
+
Every theme sets the same CSS variables (see the top of `neuralui/web/css/app.css`), so adding one is a
|
|
116
|
+
single block. The smoke test switches through every theme and checks WCAG contrast (4.5:1 for text,
|
|
117
|
+
3:1 for the large wordmark) for each text color on each surface it is used on, including the primary
|
|
118
|
+
button's gradient. Inter is bundled with the UI (SIL OFL), so no fonts are fetched from a CDN.
|
|
119
|
+
|
|
120
|
+

|
|
121
|
+
|
|
122
|
+
| Login (NeuralUI) | Colorblind safe graph |
|
|
123
|
+
| --- | --- |
|
|
124
|
+
|  |  |
|
|
125
|
+
|
|
126
|
+
## Architecture
|
|
127
|
+
|
|
128
|
+
```
|
|
129
|
+
browser ──HTTPS/WSS──▶ NeuralUI (s_cell.Cell) ──telepath──▶ Cortex ──▶ Axon
|
|
130
|
+
static UI /api/neuralui/v1/* storm, auth, model,
|
|
131
|
+
(no build step) getAuthCell() -> Cortex profiles, packages
|
|
132
|
+
StormSvc ◀──service.add──────────
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
* **Delegated auth.** The handlers subclass `synapse.lib.httpapi` handlers and override `getAuthCell()`
|
|
136
|
+
to return the Cortex telepath client. This is the hook Synapse provides for remote auth. Synapse's
|
|
137
|
+
own `LoginV1` handler checks passwords against the Cortex. The login form also takes a Cortex API key
|
|
138
|
+
(made with `$lib.auth.users.byname(<user>).genApiKey(<name>)`): the Cortex checks it, and the session
|
|
139
|
+
acts as the key's user until the key expires or is deleted (checked at most once a minute), or the
|
|
140
|
+
user is locked. Failed key logins are throttled per address. Scripts can send an API key in
|
|
141
|
+
`X-API-KEY` instead of logging in. HTTP Basic auth is turned off: browsers resend saved Basic credentials on their own, which
|
|
142
|
+
would survive logging out and skip the login throttle. Sessions are stored by NeuralUI, but the
|
|
143
|
+
user in each session is a Cortex user.
|
|
144
|
+
* **Storm as the user.** Every query runs with `opts['user']` set to the session user. The browser
|
|
145
|
+
can't override `user`: only `view`, `vars`, `idens`, `graph`, `limit` and `mode` are passed through.
|
|
146
|
+
* **Graph projection.** The drill pane and "add neighbors" use Storm's subgraph projection
|
|
147
|
+
(`opts['graph']`), so a neighbor comes back with the prop or verb that links it.
|
|
148
|
+
* **Storm service.** `NeuralUIApi` mixes in `StormSvc`, so `service.add` installs the `neuralui` package. The
|
|
149
|
+
package does not touch the data model:
|
|
150
|
+
* module `neuralui`: `saveQuery()`, `delQuery()`, `saveScrape()`, `addAbout()`, `delNote()`,
|
|
151
|
+
`getProfile()`/`setProfile()`. These run as the calling user, so they work from the CLI as well.
|
|
152
|
+
* commands `neuralui.query.save` and `neuralui.query.migrate` (see below).
|
|
153
|
+
* a `neuralui` key containing the default node actions and note types (pkgdefs allow extra keys).
|
|
154
|
+
* **State in the graph.**
|
|
155
|
+
* Saved queries are `it:exec:query` nodes tagged `#meta.neuralui.query`:
|
|
156
|
+
* `:text` is the query, runnable as is (`it:exec:query#meta.neuralui.query` lifts them all);
|
|
157
|
+
* `:language` is `storm`;
|
|
158
|
+
* `:opts` holds the Storm options to run it with (`mode`, `limit`) and NeuralUI's metadata under
|
|
159
|
+
`neuralui` (`{"v": 1, "name": ..., "desc": ...}`), so `:opts` minus `neuralui` can be passed to Storm;
|
|
160
|
+
* `:synuser` is who saved it and `:time` when it was last saved.
|
|
161
|
+
|
|
162
|
+
The query string is also an `it:query` node, shared with anything else that ran it. Saved queries from
|
|
163
|
+
NeuralUI 0.4 and earlier were `meta:note` nodes; `neuralui.query.migrate` lists them, and
|
|
164
|
+
`neuralui.query.migrate --apply [--delete]` converts them (keeping who saved them and when).
|
|
165
|
+
* Scrape sessions are `meta:note` nodes with `:type=neuralui.scrape` whose `:text` is the scraped text.
|
|
166
|
+
They are linked to the accepted nodes with `-(about)>`.
|
|
167
|
+
* **Flow control.** Queries on the WebSocket use credit based flow control: the service sends at most
|
|
168
|
+
a window of messages the browser has not acknowledged, then stops pulling from the Cortex. Telepath
|
|
169
|
+
applies backpressure from there, so a paused query really pauses. Result streams can also ask the
|
|
170
|
+
service to shorten long prop values (`truncate`); see [Large results](#large-results).
|
|
171
|
+
* **Versioned assets.** Static files are served under `/v/<build>/`, where the build id is the release
|
|
172
|
+
version plus a hash of the UI files. They are cached for good, and `index.html` (always revalidated)
|
|
173
|
+
points at the current build. Relative imports stay under the versioned path, so this needs no build
|
|
174
|
+
step. A changed build reported when the WebSocket reconnects shows a "Reload" banner.
|
|
175
|
+
* **Security.** Cross-origin POSTs are refused to prevent CSRF. The WebSocket uses Synapse's
|
|
176
|
+
same-origin check. Uploads require `axon.upload` and downloads require `axon.get` (both checked
|
|
177
|
+
against the Cortex user).
|
|
178
|
+
|
|
179
|
+
```
|
|
180
|
+
neuralui/
|
|
181
|
+
service.py the Cell, its config and the Cortex connection
|
|
182
|
+
handlers.py login/logout/me, model, extensions, call, scrape, storm websocket, axon up/down,
|
|
183
|
+
lint, storm docs, history, share links, export
|
|
184
|
+
store.py query history and share links (an LMDB slab owned by the service)
|
|
185
|
+
export.py .nodes and STIX exports, including the related node traversal
|
|
186
|
+
lint.py the Storm linter (cached Cortex metadata)
|
|
187
|
+
pkg.py the neuralui Storm package (default actions and note types)
|
|
188
|
+
storm/ Storm source for the package module
|
|
189
|
+
web/ the UI: plain ES modules, no build step
|
|
190
|
+
web/vendor/ cytoscape, marked (markdown), DOMPurify (sanitizing rendered notes), Inter
|
|
191
|
+
web/brand/ the marks and icons served by the UI (copied from assets/brand)
|
|
192
|
+
assets/brand/ versioned brand kit: canonical SVGs, PNG exports, tokens, rules
|
|
193
|
+
tests/
|
|
194
|
+
test_neuralui.py service tests against a real Cortex (pytest)
|
|
195
|
+
ui/smoke.js end to end browser test (Playwright)
|
|
196
|
+
Makefile every build, test, run and release step (make help); CI calls these targets
|
|
197
|
+
scripts/
|
|
198
|
+
devstack.py local Cortex + NeuralUI with demo data (make demo)
|
|
199
|
+
devstack_stormsvcs.py the same, with the gormaniac/stormsvcs Storm services in Docker (make demo-svcs)
|
|
200
|
+
smoke.sh the UI tests against a fresh dev stack (make test-smoke)
|
|
201
|
+
changelog.py roll the CHANGELOG's Unreleased section into a release, print release notes
|
|
202
|
+
version.py read, bump and set the version
|
|
203
|
+
brand_variants.py derive the single color marks and lockups from the canonical marks
|
|
204
|
+
brand_exports.js regenerate the PNG exports from the brand SVGs
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
## Running it
|
|
208
|
+
|
|
209
|
+
Requires Python 3.11 (the version Synapse supports; NeuralUI is only tested on it) and Synapse 2.256.0 or a
|
|
210
|
+
later 2.x release. NeuralUI 1.x is for Synapse 2 only.
|
|
211
|
+
|
|
212
|
+
Every build, test and run step is a make target (`make help` lists them); CI calls the same targets.
|
|
213
|
+
|
|
214
|
+
```bash
|
|
215
|
+
make setup # pip install -e '.[dev]' (PYTHON=... for another interpreter or a virtualenv)
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
### Local dev stack
|
|
219
|
+
|
|
220
|
+
```bash
|
|
221
|
+
make demo # python scripts/devstack.py --demo (ARGS='--https 8443' for more options)
|
|
222
|
+
# open https://localhost:4443/ (self-signed cert) - log in as analyst/analyst or root/root
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
This starts a Cortex and NeuralUI in one process under `./dev-data`, creates the users, registers the
|
|
226
|
+
Storm service and loads a few demo nodes (including threats, tools and techniques for the Explore panel).
|
|
227
|
+
|
|
228
|
+
To work with real Storm services too, run the same stack with the services from
|
|
229
|
+
[gormaniac/stormsvcs](https://github.com/gormaniac/stormsvcs) (it needs Docker):
|
|
230
|
+
|
|
231
|
+
```bash
|
|
232
|
+
make demo-svcs # python scripts/devstack_stormsvcs.py --demo (ARGS='--svcs shodan,ip_api' and so on)
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
It reads the services listed in that repository's `services.toml`, pulls each one's latest image
|
|
236
|
+
(`ghcr.io/gormaniac/stormsvcs/<name>:latest`), runs it in a container (`neuralui-dev-<name>`, its data in
|
|
237
|
+
`dev-data/stormsvcs/<name>/`, telepath on a free port of 127.0.0.1) and registers it with the Cortex as
|
|
238
|
+
`slib.<name>`. The containers are removed when the stack stops. `--svcs shodan,ip_api` picks some,
|
|
239
|
+
`--no-pull` uses images already on the machine (a failed pull also falls back to a local copy), `--keep`
|
|
240
|
+
leaves the containers running, and `--ref` reads `services.toml` from another branch or tag. The other
|
|
241
|
+
options are `devstack.py`'s.
|
|
242
|
+
|
|
243
|
+
### Against an existing Cortex
|
|
244
|
+
|
|
245
|
+
**For production, follow [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md):** both telepath connections (NeuralUI
|
|
246
|
+
to the Cortex, and the Cortex to NeuralUI for `service.add`) use `ssl://` with mutual TLS. NeuralUI refuses
|
|
247
|
+
to start with a plain `tcp://` address that leaves the machine, unless `neuralui:insecure` says this is an
|
|
248
|
+
isolated development setup. In short:
|
|
249
|
+
|
|
250
|
+
1. Make a CA, a server certificate for each side and client certificates (`neuralui` for NeuralUI,
|
|
251
|
+
`cortex` for the Cortex), and put them in each cell's `certs/` directory.
|
|
252
|
+
2. Have the Cortex listen with `dmon:listen: ssl://0.0.0.0:27492/?hostname=cortex.internal&ca=nuica` and create
|
|
253
|
+
the service account. It has to be an admin (but not `root`), because the telepath APIs for checking
|
|
254
|
+
passwords and API keys are admin-only (the same requirement Optic has). It logs in with its certificate,
|
|
255
|
+
so it needs no password:
|
|
256
|
+
|
|
257
|
+
```
|
|
258
|
+
storm> $user = $lib.auth.users.add(neuralui) $user.setAdmin($lib.true)
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
3. Create a service directory with a `cell.yaml`:
|
|
262
|
+
|
|
263
|
+
```yaml
|
|
264
|
+
cortex: ssl://cortex.internal:27492/?hostname=cortex.internal&certname=neuralui # or an aha:// url
|
|
265
|
+
dmon:listen: ssl://0.0.0.0:27495/?hostname=neuralui.internal&ca=nuica
|
|
266
|
+
https:port: 4443
|
|
267
|
+
neuralui:origin: https://neuralui.example.com
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
Then run it with `python -m neuralui /path/to/svcdir`. Any config value can also be set as an
|
|
271
|
+
environment variable, e.g. `SYN_NEURALUI_CORTEX`. Put your HTTPS certificate in the service directory as
|
|
272
|
+
`sslcert.pem`/`sslkey.pem`, as with any Synapse cell.
|
|
273
|
+
|
|
274
|
+
4. Create a plain (non-admin) user `cortex` on NeuralUI, and register the Storm service so the package
|
|
275
|
+
(module, command, actions, note types) loads. The Cortex only reads the package, so it needs no
|
|
276
|
+
permissions there:
|
|
277
|
+
|
|
278
|
+
```bash
|
|
279
|
+
python -m synapse.tools.service.moduser --svcurl cell:///path/to/svcdir --add cortex
|
|
280
|
+
```
|
|
281
|
+
```
|
|
282
|
+
storm> service.add neuralui ssl://neuralui.internal:27495/?hostname=neuralui.internal&certname=cortex
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
Without this step the UI still works with its built-in actions, but other packages can't be
|
|
286
|
+
relied on to find the `neuralui` Storm module.
|
|
287
|
+
|
|
288
|
+
For a quick look on one machine, plain TCP on the loopback interface is accepted
|
|
289
|
+
(`cortex: tcp://neuralui:<secret>@127.0.0.1:27492/`); across a network it needs `neuralui:insecure: true`,
|
|
290
|
+
which is only for isolated development.
|
|
291
|
+
|
|
292
|
+
### In a container
|
|
293
|
+
|
|
294
|
+
```bash
|
|
295
|
+
docker build -t neuralui .
|
|
296
|
+
docker run -d --name neuralui -v neuralui-data:/vertex/storage -p 4443:4443 neuralui
|
|
297
|
+
```
|
|
298
|
+
|
|
299
|
+
Released images are on `ghcr.io/<owner>/neuralui:<version>`. The service directory is the
|
|
300
|
+
`/vertex/storage` volume: put `cell.yaml`, `certs/` and `sslcert.pem`/`sslkey.pem` there (see
|
|
301
|
+
[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) for a compose file with a private backend network). The image
|
|
302
|
+
runs as an unprivileged user, has a health check and sets no telepath listener by default.
|
|
303
|
+
|
|
304
|
+
### Settings
|
|
305
|
+
|
|
306
|
+
Besides the usual Synapse cell settings, in `cell.yaml` or as `SYN_NEURALUI_*` variables:
|
|
307
|
+
|
|
308
|
+
| Setting | Default | |
|
|
309
|
+
| --- | --- | --- |
|
|
310
|
+
| `cortex` | | The Cortex telepath URL (the service user must be a Cortex admin). `ssl://` in production. |
|
|
311
|
+
| `neuralui:origin` | the request's | The public origin(s) of the UI (`https://neuralui.example.com`). Requests and WebSockets from other origins are refused. Set it behind a reverse proxy. |
|
|
312
|
+
| `neuralui:insecure` | `false` | Allow plain `tcp://` telepath across the network. Isolated development only. |
|
|
313
|
+
| `neuralui:share:days` | `365` | Share links older than this stop working and are deleted (`null`: kept). |
|
|
314
|
+
| `neuralui:session:idle` | `480` | Log sessions out after this many minutes without a request (`null`: never). |
|
|
315
|
+
| `neuralui:login:maxfails` | `5` | Failed logins for a user name (in 15 minutes) before further attempts must wait. |
|
|
316
|
+
| `neuralui:csp` | `true` | Send the Content-Security-Policy. |
|
|
317
|
+
| `neuralui:hsts` | none | Send `Strict-Transport-Security` with this max-age (seconds). Only for a host always served over HTTPS. |
|
|
318
|
+
| `neuralui:stix:config` | Synapse's default | The `$lib.stix.export` config for STIX exports. |
|
|
319
|
+
| `neuralui:upload:maxsize` | 1 GiB | The largest file upload. |
|
|
320
|
+
| `neuralui:proxy:trusted` | `[]` | Reverse proxies (addresses or networks) whose `X-Forwarded-For` gives the client's address (see [DEPLOYMENT.md](docs/DEPLOYMENT.md)). |
|
|
321
|
+
| `neuralui:limit:user:socks` | `8` | WebSocket connections one user may have open at once. |
|
|
322
|
+
| `neuralui:limit:user:runs` | `32` | Storm queries one user may run at once (background job batches included). |
|
|
323
|
+
| `neuralui:limit:user:jobs` | `4` | Background jobs one user may have running at once. |
|
|
324
|
+
| `neuralui:limit:jobs` | `16` | Background jobs the service runs at once, for everyone. |
|
|
325
|
+
| `neuralui:limit:lint` | `32` | Storm lint requests the service runs at once. |
|
|
326
|
+
| `neuralui:webroot` | the packaged UI | Serve the UI from another directory. |
|
|
327
|
+
|
|
328
|
+
### Security
|
|
329
|
+
|
|
330
|
+
* **Headers.** Every response has a strict Content-Security-Policy (same-origin scripts, styles, fonts and
|
|
331
|
+
connections; no inline scripts; no framing by other sites), `X-Frame-Options: SAMEORIGIN`, `X-Content-Type-Options: nosniff`,
|
|
332
|
+
`Referrer-Policy: same-origin`, `Cross-Origin-Opener-Policy` and a `Permissions-Policy`. HSTS is
|
|
333
|
+
available but off by default (it would stick to `localhost` during development).
|
|
334
|
+
* **Logins.** An unknown user, a wrong password and a locked account all get the same answer. After
|
|
335
|
+
5 failures for a user name (or 20 from one address) in 15 minutes, further attempts wait 30 seconds,
|
|
336
|
+
doubling with each failure up to 15 minutes. The wait is temporary, so it can delay a user's logins
|
|
337
|
+
but never lock the account. (The Cortex's own password policy, if set, still applies.)
|
|
338
|
+
* **Sessions** end after 8 hours without a request (`neuralui:session:idle`); the UI then asks to log in
|
|
339
|
+
again without losing the page.
|
|
340
|
+
* **Sessions and sockets.** Every login gets a new session and cookie (`SameSite=Lax`), and ends the
|
|
341
|
+
browser's previous session. Logging out (a POST), the idle timeout, an expired or deleted API key, and a
|
|
342
|
+
locked, archived or deleted Cortex account (checked every 10 seconds) end the session, close its
|
|
343
|
+
WebSockets and stop their queries. Every WebSocket message checks the session is still the one the
|
|
344
|
+
socket was opened with, and open sockets check again every 15 seconds while a query runs.
|
|
345
|
+
* **Origins.** Requests which change something, logging out and WebSockets must come from the UI's own
|
|
346
|
+
origin: scheme, host and port (`neuralui:origin`, or the origin the request was made to). No CORS
|
|
347
|
+
headers are sent. Give NeuralUI its own host name: cookies are not isolated by port.
|
|
348
|
+
* **Logins** are counted when they start, so concurrent guesses get no more tries than sequential ones,
|
|
349
|
+
and at most 8 password checks run at once.
|
|
350
|
+
* **Limits.** JSON request bodies are objects of at most 16 MB; Storm options are type-checked; a socket
|
|
351
|
+
runs at most 32 queries at once. Share links have titles of at most 200 characters, at most 1000 per
|
|
352
|
+
user and 60 an hour, expire after `neuralui:share:days`, and can be deleted by their owner.
|
|
353
|
+
* **Notes** are sanitized, and their images only load from `data:` URLs: an image pointing at this site
|
|
354
|
+
would be a request with the reader's session.
|
|
355
|
+
* **Only NeuralUI's own API** is served. Synapse's cell APIs (`/api/v1/login`, `/api/v1/auth/...`, which
|
|
356
|
+
manage the service's local users) are not, apart from the read-only `/api/v1/active` status.
|
|
357
|
+
* **Backend connections** are refused if they would be unencrypted across a network (see
|
|
358
|
+
[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)).
|
|
359
|
+
* As before: Storm runs as the logged in user.
|
|
360
|
+
|
|
361
|
+
### Permissions for analysts
|
|
362
|
+
|
|
363
|
+
These are normal Cortex permissions, for example:
|
|
364
|
+
|
|
365
|
+
* querying and editing: `node`, or narrower rules such as `node.add.inet:fqdn`, `node.tag.add.rep`
|
|
366
|
+
* saved queries: `node.add.it:exec:query`, `node.prop.set.it:exec:query`, `node.del.it:exec:query`,
|
|
367
|
+
`node.tag.add.meta.neuralui.query`, and `node.add.it:query` (the query string node)
|
|
368
|
+
* notes and scrape sessions: `node.add.meta:note`, `node.prop.set.meta:note`, `node.del.meta:note`,
|
|
369
|
+
`node.edge.add.about` (notes) and `node.edge.add.refs` (scrape sessions), and
|
|
370
|
+
`node.add.meta:note:type:taxonomy` the first time a new note type is used
|
|
371
|
+
* new tag properties from the tag dialog: `model.tagprop.add` (they change the Cortex's model)
|
|
372
|
+
* files: `axon.upload`, `axon.get`
|
|
373
|
+
* scripts and imports in a fork: `view.add` and `view.fork` (a fork's creator may edit, merge and
|
|
374
|
+
delete it; merging needs write access to the parent view's layer)
|
|
375
|
+
* the Admin and Automation pages use the Cortex's own permissions (`auth.*`, `pkg.*`, `cron.*`,
|
|
376
|
+
`trigger.*`, `storm.macro.*`, `dmon.*`): the pages show what the Cortex refuses
|
|
377
|
+
|
|
378
|
+
## Extending the UI from a Storm package
|
|
379
|
+
|
|
380
|
+
Any Storm package can add right-click actions and note types with a `neuralui:` key in its package
|
|
381
|
+
definition. **[docs/PACKAGES.md](docs/PACKAGES.md)** is the reference (package format 1, stable for 1.x);
|
|
382
|
+
[`neuralui/schema/neuralui.schema.json`](neuralui/schema/neuralui.schema.json) is its JSON Schema, and
|
|
383
|
+
`python -m neuralui.checkpkg <package.yaml>` checks a package against it.
|
|
384
|
+
|
|
385
|
+
```yaml
|
|
386
|
+
neuralui:
|
|
387
|
+
version: 1
|
|
388
|
+
actions:
|
|
389
|
+
- id: pdns
|
|
390
|
+
name: Passive DNS
|
|
391
|
+
forms: [inet:fqdn, inet:ipv4]
|
|
392
|
+
type: storm # runs on the Cortex with the selected nodes as its inbound nodes
|
|
393
|
+
storm: acme.pdns
|
|
394
|
+
- id: open.acme
|
|
395
|
+
name: Acme Intel
|
|
396
|
+
group: Open in # next to VirusTotal, urlscan.io and Shodan
|
|
397
|
+
type: link
|
|
398
|
+
url:
|
|
399
|
+
inet:fqdn: https://intel.acme.example/domain/{{valu}}
|
|
400
|
+
notetypes:
|
|
401
|
+
- {type: acme.report, title: Acme report, render: markdown}
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
Without any `neuralui:` key, a package's **commands** that say what forms they take are offered under
|
|
405
|
+
**Enrich › <package>** with a dialog built from their arguments, and a Rapid Power-Up's **Optic actions**
|
|
406
|
+
under **Actions › <package>**. Storm and command actions open a run dialog: on the selected nodes or all
|
|
407
|
+
the results, in the background as a job (the default, see **Jobs** in the top bar), in a new tab or in this
|
|
408
|
+
tab. **Search actions…** (or `/` in the menu) finds any action, ☆ pins one, and **Admin › Actions** lets a
|
|
409
|
+
Cortex admin hide actions or packages from everyone or from roles, set default pins and see what could not
|
|
410
|
+
be used. `scripts/devstack.py --demo` loads an example package, [`tests/pkgs/neuralui-demo`](tests/pkgs/neuralui-demo).
|
|
411
|
+
|
|
412
|
+
## Using the UI
|
|
413
|
+
|
|
414
|
+
* `Shift+Enter` runs the query (`Ctrl+Enter` and `Cmd+Enter` work too) and `Esc` stops it. `Tab`/`Shift+Tab`
|
|
415
|
+
indent, and `Ctrl+/` comments lines.
|
|
416
|
+
* **Table**, **Graph** and **Output** switch the tab's view. The Output view shows only what Storm prints,
|
|
417
|
+
like the Storm CLI, so it suits commands such as `auth.user.list` and queries that edit many nodes. Its
|
|
418
|
+
nodes are still kept (up to the results cap, without pausing the query), so switching to **Table** or
|
|
419
|
+
**Graph** shows them without running the query again: a query that called an outside service, or used
|
|
420
|
+
API credits, is not repeated. The transcript is kept until the page closes;
|
|
421
|
+
**Copy** and **Clear** are on the Messages bar. The Messages pane scrolls at a fixed height: drag its
|
|
422
|
+
top edge to resize it (double-click to reset).
|
|
423
|
+
* Click a row to show its details. `Ctrl`/`Shift`-click to select several rows, or use the header
|
|
424
|
+
checkbox to select a whole form.
|
|
425
|
+
* In the table, `Ctrl+A` selects everything, `t` opens the tag manager and `Esc` clears the selection.
|
|
426
|
+
* Right-click a selection for **Add tag**, **Manage tags**, **Edit properties** and, under Edit,
|
|
427
|
+
**Apply tag with interval / props** and **Add a note about these nodes**.
|
|
428
|
+
* Add `| colorize <color>` to a query to color its rows and graph nodes.
|
|
429
|
+
* Right-click anything: rows, prop cells (lift, filter or exclude by that value), tag chips, column
|
|
430
|
+
headers (hide a column, filter by it), tabs, saved queries and the tag tree.
|
|
431
|
+
* **Open in** (right-click one node) looks it up on VirusTotal, urlscan.io or Shodan in a new browser tab:
|
|
432
|
+
a `file:bytes` by its hash, hashes, domains, IPs and URLs. A prop cell that points at one of those (the
|
|
433
|
+
`:ipv4` of an `inet:dns:a`) offers the same. The value goes to that site, nowhere else.
|
|
434
|
+
* Clicking a tag chip selects its row like any other cell; right-click it for the tag's actions
|
|
435
|
+
(filter by it, lift it, edit it, pin it as a column).
|
|
436
|
+
* Storm problems are underlined as you type. Click the problem count under the editor to list them,
|
|
437
|
+
and click a problem to jump to it.
|
|
438
|
+
* The mode selector switches the Storm parser between `storm`, `lookup`, `autoadd` and `search`.
|
|
439
|
+
|
|
440
|
+
## Pages beyond queries
|
|
441
|
+
|
|
442
|
+
The top bar opens pages which are tabs like any other (they are saved, renamed and reopened the same
|
|
443
|
+
way). Admin, Automation and Help open once; opening them again selects the tab.
|
|
444
|
+
|
|
445
|
+
* **Console**: the editor on one side, Storm's output on the other (**Side by side** or **Stacked**,
|
|
446
|
+
and the divider drags). `Shift+Enter` runs. **run in** picks the current view, a **throwaway fork**
|
|
447
|
+
(forked for the run, deleted after it, so nothing is kept) or a **new fork (kept)**: the page then
|
|
448
|
+
offers to switch to the fork, merge it into its parent, or delete it. **read-only** runs Storm with
|
|
449
|
+
`readonly`, **debug** with `debug`, and **list nodes** prints each node as a line. **Variables** takes
|
|
450
|
+
a JSON object of `$variables`.
|
|
451
|
+
* **Import**: a Console tab with files. Drop files, a folder or a `.zip` (or use **Add files** /
|
|
452
|
+
**Add folder**). The plan lists the scripts in run order (a folder's files, then its sub-folders,
|
|
453
|
+
alphabetically: number names to set the order) with their data files and row counts. A data file
|
|
454
|
+
belongs to the script with its name (`orgs.storm` takes `orgs.csv` and `orgs_2.json`); it runs the
|
|
455
|
+
script with its rows in `$rows`, 1000 at a time. CSV rows are lists of strings, TXT rows are lines,
|
|
456
|
+
JSON lists give `[index, item]` rows and JSON objects `[key, value]` rows. **data files have a
|
|
457
|
+
header row** skips the first line of CSV and TXT files. `.storml` scripts run each line on its own.
|
|
458
|
+
Click a script to see or change it (for this import only). Data files no script claims can be
|
|
459
|
+
ticked and run with the editor's own script, like csvtool. **Import all** runs everything in the
|
|
460
|
+
chosen view or fork. Files stay in the page until it closes; the editor's script is kept.
|
|
461
|
+
(Unlike stormlibpp, a data file must match its script's name exactly: `orgs_extra.csv` is not
|
|
462
|
+
`orgs.storm`'s.)
|
|
463
|
+
* **Automation**: **Cron** jobs (add from a starting point or the `cron.add` fields, edit the query,
|
|
464
|
+
name and description, enable or disable), **Triggers** (when a node is added or deleted, a property
|
|
465
|
+
set, a tag added or removed, or an edge added or removed), **Macros** (edit, run in a console tab)
|
|
466
|
+
and **Dmons** (start, stop, restart, read the log).
|
|
467
|
+
* **Admin** (admins): users, roles, rules and API keys; Storm packages from a JSON file built with
|
|
468
|
+
`python -m synapse.tools.genpkg --save pkg.json`.
|
|
469
|
+
* **Help**: the command list and each command's `--help`, the `$lib` reference, the data model and the
|
|
470
|
+
Synapse docs. Forms link to the forms their props point at, and **Lift in a new tab** shows them.
|
|
471
|
+
|
|
472
|
+
## Writing queries
|
|
473
|
+
|
|
474
|
+
* **Completion** pops up as you type: forms (`inet:fq`), full props (`inet:fqdn:`), relative props of the
|
|
475
|
+
forms in the query (`:zo`), universal props (`.`), tags (`#cno.`), and `$lib` libraries and their
|
|
476
|
+
members with signatures (`$lib.axon.`). After a pipe, `Ctrl+Space` lists the commands. `Enter` or `Tab`
|
|
477
|
+
accepts, `Esc` closes.
|
|
478
|
+
* **Hover** a form, prop, command or `$lib` name for its docs.
|
|
479
|
+
* The model comes from the UI's cached copy and command and `$lib` docs from the service (one request),
|
|
480
|
+
so completion does not call the Cortex. Only tags are looked up in the Cortex.
|
|
481
|
+
* `Ctrl+K` opens the **command palette**: run, stop, share, export, switch view, mode, theme or time zone,
|
|
482
|
+
go to a tab or a saved query, and more. `?` lists the keyboard shortcuts.
|
|
483
|
+
|
|
484
|
+
## History, links and export
|
|
485
|
+
|
|
486
|
+
* **Query history.** Every query you run from the editor is kept, newest first, in the **History** tab of
|
|
487
|
+
the Notes sidebar (with search), and `Ctrl+Up`/`Ctrl+Down` in the editor walk through it. Running a
|
|
488
|
+
query again moves it to the top. The history is stored by the NeuralUI service in its own LMDB slab,
|
|
489
|
+
per user, so it adds no writes to the graph; the newest 1000 queries per user are kept.
|
|
490
|
+
* **Share links.** **Share** (next to Save) gives a short link, `https://<host>/#s=<11 characters>`. The
|
|
491
|
+
query is stored by the service under that random id; the URL never holds the query and does not change
|
|
492
|
+
as you work. Opening a link adds a tab with the query and who shared it, **without running it** (a
|
|
493
|
+
shared query could contain edits). A node's **link** (in its details, or its menu) opens the node.
|
|
494
|
+
* **Export** (results bar, or **Export...** on a selection) writes:
|
|
495
|
+
* **CSV**: one row per node, with a column for every property any of the nodes has. Cells a spreadsheet
|
|
496
|
+
would treat as formulas are escaped.
|
|
497
|
+
* **JSON lines**: the packed nodes.
|
|
498
|
+
* **Synapse `.nodes`**: the Cortex's own export (`exportStorm`): packed nodes plus the light edges
|
|
499
|
+
between them. Load it into another Cortex with `feedFromAxon` / the `syn.nodes` feed.
|
|
500
|
+
* **STIX 2.1 (experimental)**: a bundle made with `$lib.stix.export`. This is scaffolding: the mapping is
|
|
501
|
+
Synapse's default unless `neuralui:stix:config` is set, and `StixExporter.getConfig()` in
|
|
502
|
+
`neuralui/export.py` is the hook for a custom mapping. Forms with no mapping are reported.
|
|
503
|
+
* `.nodes` and STIX can **include related nodes**: nodes the props point to, nodes pointing at them, and
|
|
504
|
+
light edges in either direction, 1 to 3 steps out, up to a node limit. **Check size** counts first.
|
|
505
|
+
They can also re-run the tab's query in the Cortex instead of using the results in the page.
|
|
506
|
+
* **Copy** (node menu): primary values (one per line), idens, a Storm lift, CSV or JSON lines.
|
|
507
|
+
* CSV, JSON lines and copies use the full nodes, even when the results show shortened values.
|
|
508
|
+
|
|
509
|
+
## Results
|
|
510
|
+
|
|
511
|
+
* **Filter** the table with words (anywhere: value, props, tags, form), `:prop=text` (the prop contains
|
|
512
|
+
text), `:prop` (has the prop), `#tag` (the tag or one under it) and `-term` (does not match).
|
|
513
|
+
* **Keyboard**: click the table (or tab to it), then `Up`/`Down`, `PageUp`/`PageDown`, `Home`/`End` move
|
|
514
|
+
between rows, `Shift` extends the selection, `Space` toggles a row and `Enter` shows its details.
|
|
515
|
+
* The results bar shows how many nodes are selected, with **Select all** and **Clear**.
|
|
516
|
+
* **Columns** can be resized by dragging the right edge of a header (double-click to reset). Widths are
|
|
517
|
+
saved per form. Form headers and column headers stay in view while scrolling.
|
|
518
|
+
* **Times** show in UTC (as Synapse stores them) or in local time (user menu, or the palette). Time
|
|
519
|
+
tooltips show both. Editors always take UTC, because that is how Storm reads typed times.
|
|
520
|
+
* A tab whose query finishes while you are on another tab gets a dot (red if it failed), and the page
|
|
521
|
+
title counts them. Turn on **Notify me when a long query finishes** in the user menu to get a browser
|
|
522
|
+
notification when a query that took more than 10 seconds finishes while the page is in the background.
|
|
523
|
+
|
|
524
|
+
## Keyboard and screen readers
|
|
525
|
+
|
|
526
|
+
* **Everything works from the keyboard.** `Shift+F10` (or the menu key) opens the menu of whatever has
|
|
527
|
+
focus: a result row, a tab, a tag in the tag tree, a saved query or a history entry. In menus, the arrow
|
|
528
|
+
keys move, `Right`/`Enter` open a submenu, `Left`/`Esc` go back, `Home`/`End` jump, a letter jumps to
|
|
529
|
+
the next item starting with it, and closing a menu puts the focus back where it was.
|
|
530
|
+
* **Dialogs** keep `Tab` inside them, start in their first field and give the focus back when they close.
|
|
531
|
+
* **Tabs** are one stop: the arrow keys move between them, `Enter` opens one and `Delete` closes it.
|
|
532
|
+
* **Skip links** ("Skip to the query editor", "Skip to the results") are the first stops on the page.
|
|
533
|
+
* **Screen readers** hear when a query finishes, fails or pauses, the row as you move through the table,
|
|
534
|
+
and changes in the number of Storm problems while typing. Rows say whether they are selected. The graph
|
|
535
|
+
is described as a picture of the same nodes the table lists (the table is the accessible view).
|
|
536
|
+
* **Reduced motion.** When the operating system asks for less motion, transitions are turned off.
|
|
537
|
+
* The default theme follows the operating system's light or dark setting until you pick one.
|
|
538
|
+
|
|
539
|
+
## When there is nothing to show
|
|
540
|
+
|
|
541
|
+
* A new tab suggests example queries (click one to run it) and the shortcuts for completion and the
|
|
542
|
+
command palette. A tab restored from a previous visit says its query has not been run yet, with a
|
|
543
|
+
**Run it** button (results are not kept when the page closes).
|
|
544
|
+
* A query with no results says which view it ran in and how long it took, with hints (lookup mode,
|
|
545
|
+
printed messages, a stopped query, other views).
|
|
546
|
+
* A failed query explains the error and what to do next (syntax errors, unknown names, missing
|
|
547
|
+
permissions, a lost connection, timeouts, ...), with the original message under it.
|
|
548
|
+
* The notes, history and tag panels explain how to fill them. A lost connection to NeuralUI shows a
|
|
549
|
+
banner after a moment and clears itself when it reconnects.
|
|
550
|
+
|
|
551
|
+
## Tabs remember how you were looking
|
|
552
|
+
|
|
553
|
+
* Each tab keeps its filter, collapsed form sections and sort columns. Re-running the query keeps them.
|
|
554
|
+
* Switching tabs keeps each one's scroll position.
|
|
555
|
+
* After a reload, the first run of a tab's query puts back its scroll position, its selection (up to 200
|
|
556
|
+
nodes) and, for graphs of up to 200 nodes, the node positions and viewport.
|
|
557
|
+
|
|
558
|
+
## Large results
|
|
559
|
+
|
|
560
|
+
* **The table is virtualized.** Only the rows near the viewport are in the page, so tens of thousands
|
|
561
|
+
of results scroll, sort, filter and select (including `Shift`-click ranges) without slowing down.
|
|
562
|
+
Rows are one line high; a cell's tooltip shows its full value or tag list.
|
|
563
|
+
* **Queries pause at a cap.** After 50,000 nodes (the `results:cap` preference) a query pauses and a
|
|
564
|
+
bar offers **Load more** or **Stop here**. Paused means the query is waiting in the Cortex, not
|
|
565
|
+
finished.
|
|
566
|
+
* **Long values are shortened in results.** Prop values over 2,048 characters (or large arrays and
|
|
567
|
+
`data` values) are cut short in result sets. The details pane, the property editor, the note editor
|
|
568
|
+
and **Copy value** always fetch the full node first, so an edit never starts from a shortened value,
|
|
569
|
+
and value filters are not offered for shortened values.
|
|
570
|
+
* **Large graphs lay out in the background.** Above 300 nodes, `cose` is replaced by `force`, which runs
|
|
571
|
+
in a web worker so the page stays responsive. Streamed nodes are placed next to a neighbor and the
|
|
572
|
+
full layout runs once the query is done. Labels are hidden when zoomed out.
|
|
573
|
+
|
|
574
|
+
## Not losing work
|
|
575
|
+
|
|
576
|
+
* **Undo.** Bulk edits (properties, adding, applying and removing tags, the tag manager) show an
|
|
577
|
+
**Undo** button for 20 seconds. It restores the props, tags, tag intervals and tag props the edit
|
|
578
|
+
changed.
|
|
579
|
+
* **Confirmations.** Deleting props from, or removing a tag from, several nodes asks first and says
|
|
580
|
+
how many. Package actions that ask for confirmation say how many nodes they apply to.
|
|
581
|
+
* **Closed tabs** can be reopened from the toast, the tab menu or `Alt+Shift+T`.
|
|
582
|
+
* **Unsaved notes.** Closing a note with changes asks before discarding them.
|
|
583
|
+
* **Two windows.** Tabs are saved with a revision. If another window or browser saved its tabs in the
|
|
584
|
+
meantime, saving pauses and you choose whose tabs to keep, instead of one silently replacing the other.
|
|
585
|
+
* **Expired sessions.** If the session expires, you log in again over the page: tabs, results and open
|
|
586
|
+
dialogs stay, and the request that hit the expired session is retried. Logging in as a different
|
|
587
|
+
user reloads.
|
|
588
|
+
|
|
589
|
+
## Tests
|
|
590
|
+
|
|
591
|
+
```bash
|
|
592
|
+
make setup setup-ui # once: NeuralUI with its test tools, then Playwright + Chromium for the UI tests
|
|
593
|
+
make test # everything below
|
|
594
|
+
make test-svc # service tests against a real Cortex (pytest)
|
|
595
|
+
make test-js # JS syntax check and unit tests (tokenizer, completion, undo, export, filters)
|
|
596
|
+
make test-smoke # UI tests: a fresh dev stack in build/smoke/, the end to end smoke test, then
|
|
597
|
+
# per-theme screenshots compared with tests/ui/baseline/ (SMOKE_FROM='<step>'
|
|
598
|
+
# starts at a step, SMOKE_VISUAL=0 skips the screenshots)
|
|
599
|
+
make build # the Python packages (make build-py) and the container image (make build-docker)
|
|
600
|
+
```
|
|
601
|
+
|
|
602
|
+
The smoke test also fails on any Content-Security-Policy violation. Its dev stack's log and output are kept
|
|
603
|
+
in `build/smoke/`. CI (`.github/workflows/ci.yml`) runs these same targets on every pull request
|
|
604
|
+
and every push to main, plus a container image build; it can also be run by hand from the Actions tab.
|
|
605
|
+
To keep the runs down, pushes to other branches are not tested (a branch is tested through its pull
|
|
606
|
+
request), a new push to a pull request or to main cancels the run before it, and the release commit
|
|
607
|
+
Prepare Release pushes to main is skipped, since releases run their own tests (see
|
|
608
|
+
[docs/RELEASING.md](docs/RELEASING.md)).
|
|
609
|
+
The screenshots are kept as the `visual` artifact of each run; to start comparing them, copy the PNGs of a
|
|
610
|
+
good run on main into `tests/ui/baseline/` (baselines depend on the fonts of the machine that made them).
|
|
611
|
+
|
|
612
|
+
## Releases
|
|
613
|
+
|
|
614
|
+
The version is set in one place, `neuralui/version.py`. Changes are listed in [CHANGELOG.md](CHANGELOG.md)
|
|
615
|
+
(new ones under `## Unreleased`), and stored formats and their versions in [docs/FORMATS.md](docs/FORMATS.md).
|
|
616
|
+
A release is cut from GitHub with **Actions → Prepare Release**, which tests, rolls the changelog, sets the
|
|
617
|
+
version, then tags and pushes; the tag publishes Python packages, a container image on `ghcr.io` and a
|
|
618
|
+
GitHub release. See [docs/RELEASING.md](docs/RELEASING.md). **About NeuralUI** (user menu) shows the versions in use.
|
|
619
|
+
|
|
620
|
+
## Storm lint and highlighting
|
|
621
|
+
|
|
622
|
+
Highlighting happens in the browser, as plain JavaScript, on every keystroke. Linting goes to
|
|
623
|
+
`POST /api/neuralui/v1/storm/lint` 250ms after you stop typing, and an edit cancels any request still
|
|
624
|
+
in flight. **The Cortex is not called per keystroke.** The NeuralUI service:
|
|
625
|
+
|
|
626
|
+
* parses the query with Synapse's own Storm parser, locally;
|
|
627
|
+
* checks the parsed query against a copy of the Cortex's model, Storm commands, libraries and package
|
|
628
|
+
command definitions. The copy is refreshed in the background every 60s and when the service
|
|
629
|
+
reconnects to the Cortex;
|
|
630
|
+
* caches results by query text.
|
|
631
|
+
|
|
632
|
+
A lint takes a few milliseconds on the server.
|
|
633
|
+
|
|
634
|
+
| Rule | Severity | Finds |
|
|
635
|
+
| --- | --- | --- |
|
|
636
|
+
| `E001` | error | Syntax errors from the Storm parser, at the exact line and column |
|
|
637
|
+
| `E002`, `E003`, `R*`, `P*`, `S*`, `M*` | varies | Synapse's own `StormLinter` rules |
|
|
638
|
+
| `N001` | error | A form or full property name (`inet:fqdnx`, `inet:fqdn:newp`) that is not in the data model |
|
|
639
|
+
| `N002` | error | Unknown relative property (`:newp`) for the forms in the pipeline |
|
|
640
|
+
| `N003` | error | Unknown Storm command |
|
|
641
|
+
| `N004` | error | Bad command arguments: unknown options, missing or wrongly typed values. This is checked by the command's own argument parser; arguments only known at runtime (`$vars`) are skipped. |
|
|
642
|
+
| `N005` | error | Unknown `$lib` library or library member |
|
|
643
|
+
| `N006` | warning | A variable that is used but never set |
|
|
644
|
+
|
|
645
|
+
Lint only runs in `storm` mode. Running a query that has errors asks first ("Run anyway?"), and a lint
|
|
646
|
+
still in flight never holds up a run. The Storm editor in the saved query dialog is linted too.
|
|
647
|
+
|
|
648
|
+
## File viewer safety
|
|
649
|
+
|
|
650
|
+
* Bytes come from `GET /api/neuralui/v1/axon/files/<sha256>/range`, which runs `$lib.axon.read()` as the
|
|
651
|
+
session user (at most 1 MiB per request), so the Cortex enforces `axon.get`.
|
|
652
|
+
* The hex and strings views only format bytes as text. Nothing in the file is interpreted.
|
|
653
|
+
* Rendering is detected from the file's magic bytes, never from `:mime` or the file name, and is
|
|
654
|
+
limited to PNG, JPEG, GIF, WebP, BMP, ICO and PDF. SVG and other active formats are never
|
|
655
|
+
rendered, and files over 64 MiB are not rendered.
|
|
656
|
+
* NeuralUI never parses PDFs itself. On request, and after a confirmation, the bytes are wrapped in a
|
|
657
|
+
`Blob` typed `application/pdf` and handed to the browser's built-in viewer (PDFium in Chrome,
|
|
658
|
+
pdf.js in Firefox), either in a frame or in a new browser tab. The risk then rests with the
|
|
659
|
+
browser's PDF viewer, which is kept up to date with the browser, so keep your browser current.
|
|
660
|
+
* Images are decoded by the browser into an `<img>` from a `Blob` typed by the detected format.
|
|
661
|
+
|
|
662
|
+
## Notes and limits
|
|
663
|
+
|
|
664
|
+
* **Colors.** Node colors come from Storm's `colorize` command. Tag colors are a per-user display
|
|
665
|
+
preference stored in the Cortex user profile. Coloring by tag falls back to a stable hash color.
|
|
666
|
+
* **Tag props** must be declared in the Cortex model (`$lib.model.ext.addTagProp`). The applicator
|
|
667
|
+
edits whatever tag props the Cortex has, including `data` tag props for arbitrary nested values.
|
|
668
|
+
The dev stack declares `score` (int) and `meta` (data) only so there is something to demo.
|
|
669
|
+
* **Intervals merge.** Setting an interval prop (e.g. `.seen`) or a tag interval merges it with the
|
|
670
|
+
existing interval, which is how Synapse handles `ival` values.
|
|
671
|
+
* The live markdown editor edits one unit at a time (a line, or a whole list item, code block, table or
|
|
672
|
+
quote), so a text selection can not span units there; use the Raw view for that.
|
|
673
|
+
* The graph view shows at most 2000 nodes. See [Large results](#large-results) for the table and the
|
|
674
|
+
result cap.
|
|
675
|
+
* Light edges between results are only fetched in graph mode. Switching to the graph fetches them for the
|
|
676
|
+
nodes already in the tab, by iden, instead of running the query again; so does turning **property
|
|
677
|
+
nodes** on or off. Prop references are always drawn.
|
|
678
|
+
* Targets Synapse 2.x (`synapse_version: >=2.256.0,<3.0.0`). Synapse 3.x changes the model and some
|
|
679
|
+
APIs and has not been tested.
|