firewatcher 1.57__tar.gz → 1.59__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (31) hide show
  1. {firewatcher-1.57 → firewatcher-1.59}/MANIFEST.in +1 -1
  2. firewatcher-1.59/PKG-INFO +179 -0
  3. firewatcher-1.59/README.md +154 -0
  4. firewatcher-1.59/examples/firewatcher.conf +52 -0
  5. firewatcher-1.59/firewatcher.egg-info/PKG-INFO +179 -0
  6. firewatcher-1.59/firewatcher.egg-info/SOURCES.txt +23 -0
  7. firewatcher-1.59/firewatcher.py +1470 -0
  8. firewatcher-1.59/tests/conftest.py +133 -0
  9. firewatcher-1.59/tests/test_capture.py +347 -0
  10. firewatcher-1.59/tests/test_cli.py +154 -0
  11. firewatcher-1.59/tests/test_compressor.py +102 -0
  12. firewatcher-1.59/tests/test_config.py +320 -0
  13. firewatcher-1.59/tests/test_log_source.py +54 -0
  14. firewatcher-1.59/tests/test_matching.py +129 -0
  15. firewatcher-1.59/tests/test_notify.py +400 -0
  16. firewatcher-1.59/tests/test_patterns.py +161 -0
  17. firewatcher-1.59/tests/test_slugify.py +25 -0
  18. firewatcher-1.59/tests/test_systemd.py +432 -0
  19. firewatcher-1.57/PKG-INFO +0 -114
  20. firewatcher-1.57/README.md +0 -89
  21. firewatcher-1.57/firewatcher.egg-info/PKG-INFO +0 -114
  22. firewatcher-1.57/firewatcher.egg-info/SOURCES.txt +0 -12
  23. firewatcher-1.57/firewatcher.py +0 -625
  24. firewatcher-1.57/test_firewatcher.py +0 -866
  25. {firewatcher-1.57 → firewatcher-1.59}/examples/nvme_failure.regex +0 -0
  26. {firewatcher-1.57 → firewatcher-1.59}/examples/sys_msg.txt +0 -0
  27. {firewatcher-1.57 → firewatcher-1.59}/firewatcher.egg-info/dependency_links.txt +0 -0
  28. {firewatcher-1.57 → firewatcher-1.59}/firewatcher.egg-info/entry_points.txt +0 -0
  29. {firewatcher-1.57 → firewatcher-1.59}/firewatcher.egg-info/top_level.txt +0 -0
  30. {firewatcher-1.57 → firewatcher-1.59}/setup.cfg +0 -0
  31. {firewatcher-1.57 → firewatcher-1.59}/setup.py +0 -0
@@ -1,3 +1,3 @@
1
1
  include README.md
2
2
  include examples/*
3
- include test_firewatcher.py
3
+ recursive-include tests *.py
@@ -0,0 +1,179 @@
1
+ Metadata-Version: 2.4
2
+ Name: firewatcher
3
+ Version: 1.59
4
+ Summary: Watch logs for incident patterns and persist surrounding context off-box
5
+ Home-page: https://github.com/yufei-pan/firewatcher
6
+ Author: Yufei Pan
7
+ Author-email: pan@zopyr.us
8
+ License: GPLv3+
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
11
+ Classifier: Operating System :: POSIX :: Linux
12
+ Classifier: Topic :: System :: Logging
13
+ Classifier: Topic :: System :: Monitoring
14
+ Requires-Python: >=3.6
15
+ Description-Content-Type: text/markdown
16
+ Dynamic: author
17
+ Dynamic: author-email
18
+ Dynamic: classifier
19
+ Dynamic: description
20
+ Dynamic: description-content-type
21
+ Dynamic: home-page
22
+ Dynamic: license
23
+ Dynamic: requires-python
24
+ Dynamic: summary
25
+
26
+ # firewatcher
27
+
28
+ Watch a log stream (journald, syslog, or any file) for incident patterns, then persist a window of surrounding messages to a durable location — including a network filesystem — so the record survives even if the machine later dies.
29
+
30
+ The watcher prefers `journalctl --follow` when `journalctl` is on PATH. On hosts without journald it falls back to `/var/log/syslog`, then `/var/log/messages`, or `--log-file`. Running the daemon does not require systemd; only `--install-service` / `--uninstall-service` do.
31
+
32
+ ## Install
33
+
34
+ ```bash
35
+ pip install firewatcher
36
+ ```
37
+
38
+ From a clone of this repository:
39
+
40
+ ```bash
41
+ pip install .
42
+ ```
43
+
44
+ Requires **Python 3.6+**. No third-party runtime dependencies.
45
+
46
+ ## Quick start
47
+
48
+ ```bash
49
+ firewatcher /etc/firewatcher/patterns.d
50
+ firewatcher -t 300 -o /var/log/captured_messages/ /etc/firewatcher/patterns.d
51
+ ```
52
+
53
+ A directory argument loads every non-hidden pattern file inside it. Files ending in `.regex` are compiled as regular expressions. Any other pattern file is treated as fnmatch (bash-like) substrings, with `*` added on both ends.
54
+
55
+ `--install-service` seeds `/etc/firewatcher/patterns.d` with example patterns when that directory is empty. The `examples/` directory in the source tree is not installed onto `PATH` by `pip install`; copy the pattern files from a clone, or let `--install-service` write them.
56
+
57
+ On a match, `firewatcher` writes:
58
+
59
+ - a per-incident capture under `{output-folder}/{YYYY-MM}/…`
60
+ - a line in `{output-folder}/journal.log`
61
+
62
+ Live capture is written as lines arrive, so a copy can already be on another filesystem if the host then disappears. Each captured line is flushed as it is written, and the capture file is fsynced at most once a second and when the capture closes, so a host crash loses at most about the last second of it.
63
+
64
+ ## systemd service
65
+
66
+ ```bash
67
+ sudo firewatcher --install-service
68
+ sudo firewatcher --print-unit
69
+ sudo firewatcher --uninstall-service
70
+ ```
71
+
72
+ `--install-service` writes `/etc/systemd/system/firewatcher.service`, creates the output directory, seeds `/etc/firewatcher/patterns.d/` with example patterns **only if that directory is empty**, writes `/etc/firewatcher/firewatcher.conf` **only if that file is missing**, then `systemctl daemon-reload && systemctl enable --now firewatcher`.
73
+
74
+ If systemd is not available, `--install-service` and `--uninstall-service` print a warning and exit. `--print-unit` still works so you can copy the unit elsewhere.
75
+
76
+ Useful flags:
77
+
78
+ | Flag | Description |
79
+ |------|-------------|
80
+ | `--unit-name NAME` | Unit name (default: `firewatcher`) |
81
+ | `--requires-mounts-for PATH` | Add `RequiresMountsFor=` (repeat for NFS/remote output) |
82
+ | `--no-enable` | Write the unit and reload, but do not enable or start it |
83
+
84
+ ```bash
85
+ sudo firewatcher --install-service -o /mnt/logs/captured_messages \
86
+ --requires-mounts-for /mnt/logs/captured_messages \
87
+ /etc/firewatcher/patterns.d
88
+ ```
89
+
90
+ ## Options
91
+
92
+ | Flag | Description |
93
+ |------|-------------|
94
+ | `pattern_file …` | Pattern files or directories |
95
+ | `--log-file` | Log file to follow (default: journalctl / syslog / messages) |
96
+ | `-t`, `--capture-time` | Seconds of context after a match (default: 300) |
97
+ | `-o`, `--output-folder` | Where to write captures (default: `/var/log/captured_messages/`) |
98
+ | `--tail_lines` | Lines to start with (default: 20; `+N` from line N, not with journalctl) |
99
+ | `--filter_only` | Filter existing logs only (`cat` instead of `tail -F`, or `journalctl` without `--follow`) |
100
+ | `--compress-after-months` | Tar.xz monthly dirs after N months (default: 3) |
101
+ | `--delete-after-months` | Delete `YYYY-MM` dirs after N months (default: 0 = never) |
102
+ | `--capture_line_count_max` | Max lines before/after a match (default: 10000). Also caps how many recent lines are held in memory |
103
+ | `--config PATH` | INI config file. Default: `/etc/firewatcher/firewatcher.conf` when that file exists. An explicit path must exist |
104
+ | `--llm` / `--no-llm` | Enable or disable LLM analysis of each finished capture |
105
+ | `--llm-base-url URL` | OpenAI-compatible API root, such as `http://127.0.0.1:11434/v1` |
106
+ | `--llm-model NAME` | Model name |
107
+ | `--llm-timeout SEC` | LLM HTTP timeout (default: 60) |
108
+ | `--llm-max-bytes N` | Capture bytes sent to the model (default: 48000) |
109
+ | `--notify` / `--no-notify` | Enable or disable ntfy pushes |
110
+ | `--ntfy-url URL` | ntfy topic URL, such as `https://ntfy.sh/mytopic` |
111
+ | `--ntfy-priority N` | ntfy priority 1–5 (default: 4) |
112
+ | `--ntfy-timeout SEC` | Worker ntfy timeout (default: 10). A full analysis queue uses 5 seconds |
113
+ | `--install-service` | Install and enable a systemd unit |
114
+ | `--print-unit` | Print that unit and exit |
115
+ | `--uninstall-service` | Disable and remove the unit |
116
+ | `-V`, `--version` | Show version and exit |
117
+
118
+ ## Config file
119
+
120
+ Runtime settings live in an INI file. Explicit command-line flags override the file. Built-in defaults apply when neither sets a value. A missing default file is ignored. `--uninstall-service` does not read the file.
121
+
122
+ A commented example with all supported settings is in [`examples/firewatcher.conf`](examples/firewatcher.conf). Copy it to `/etc/firewatcher/firewatcher.conf`, or use `firewatcher --config /path/to/firewatcher.conf`.
123
+
124
+ ```ini
125
+ [watch]
126
+ patterns = /etc/firewatcher/patterns.d
127
+ capture_time = 300
128
+ output_folder = /var/log/captured_messages/
129
+ filter_only = false
130
+
131
+ [llm]
132
+ enabled = false
133
+ base_url = http://127.0.0.1:11434/v1
134
+ model = llama3.1
135
+ timeout = 60
136
+ max_bytes = 48000
137
+
138
+ [notify]
139
+ enabled = false
140
+ ntfy_url = https://ntfy.sh/firewatcher-myhost-REPLACE_ME
141
+ priority = 4
142
+ timeout = 10
143
+ ```
144
+
145
+ `patterns` is one path per line, or comma-separated. Positional pattern arguments replace that list.
146
+
147
+ `api_key` under `[llm]` and `token` under `[notify]` are not command-line flags. `FIREWATCHER_LLM_API_KEY` and `FIREWATCHER_NTFY_TOKEN` override them when set. `--install-service` creates `/etc/firewatcher/firewatcher.conf` with mode `0600` only when it is missing, and the unit runs `firewatcher --config` that path. An existing file is left as it is. Flags passed to a later install are appended on `ExecStart` so they override the file.
148
+
149
+ Generated configs include explanatory comments above each setting. When no ntfy URL is supplied, a new config gets `https://ntfy.sh/firewatcher-<hostname>-<random>`, with a 12-character URL-safe base64 suffix (72 random bits, no padding). The hostname is sanitized and shortened as needed to keep the topic within 64 characters. Explicit URLs are preserved, and reinstalling leaves an existing config and topic unchanged. Notifications still default to off; subscribe to the generated topic in ntfy, then set `[notify] enabled = true`. When copying the example above, replace its topic with your own.
150
+
151
+ ## LLM analysis and notifications
152
+
153
+ Both features default off.
154
+
155
+ - Notifications on, LLM off: every finished capture is pushed. The title is `{hostname}: {pattern}`.
156
+ - LLM on, notifications off: the model writes a title, summary, and notify decision. The report is appended to the capture file and logged. Nothing is pushed.
157
+ - Both on: a push is sent only when the model sets `notify` to true.
158
+ - If the LLM request times out, returns an HTTP error, or is not the expected JSON, and notifications are on, firewatcher still pushes, using the generated title.
159
+
160
+ The capture window closes even when no further log lines arrive. Analysis runs on a background worker so log following stays live. The queue holds four captures. Notify-only jobs run ahead of jobs that still need the model. When the queue is full, that incident skips the model and, if notifications are on, the capture thread sends the generated-title push itself.
161
+
162
+ Lines firewatcher writes back to the journal are a single physical line containing `firewatcher-self`, so a summary that quotes `panic` or `I/O error` does not start another capture. The capture files themselves stay in the log stream's original format.
163
+
164
+
165
+ ## Development
166
+
167
+ ```bash
168
+ pip install -r requirements-dev.txt
169
+ pytest
170
+ pytest --cov=firewatcher --cov-fail-under=95 --cov-report=term-missing
171
+ ```
172
+
173
+ ## Author
174
+
175
+ Yufei Pan (pan@zopyr.us)
176
+
177
+ ## License
178
+
179
+ GPL-3.0-or-later
@@ -0,0 +1,154 @@
1
+ # firewatcher
2
+
3
+ Watch a log stream (journald, syslog, or any file) for incident patterns, then persist a window of surrounding messages to a durable location — including a network filesystem — so the record survives even if the machine later dies.
4
+
5
+ The watcher prefers `journalctl --follow` when `journalctl` is on PATH. On hosts without journald it falls back to `/var/log/syslog`, then `/var/log/messages`, or `--log-file`. Running the daemon does not require systemd; only `--install-service` / `--uninstall-service` do.
6
+
7
+ ## Install
8
+
9
+ ```bash
10
+ pip install firewatcher
11
+ ```
12
+
13
+ From a clone of this repository:
14
+
15
+ ```bash
16
+ pip install .
17
+ ```
18
+
19
+ Requires **Python 3.6+**. No third-party runtime dependencies.
20
+
21
+ ## Quick start
22
+
23
+ ```bash
24
+ firewatcher /etc/firewatcher/patterns.d
25
+ firewatcher -t 300 -o /var/log/captured_messages/ /etc/firewatcher/patterns.d
26
+ ```
27
+
28
+ A directory argument loads every non-hidden pattern file inside it. Files ending in `.regex` are compiled as regular expressions. Any other pattern file is treated as fnmatch (bash-like) substrings, with `*` added on both ends.
29
+
30
+ `--install-service` seeds `/etc/firewatcher/patterns.d` with example patterns when that directory is empty. The `examples/` directory in the source tree is not installed onto `PATH` by `pip install`; copy the pattern files from a clone, or let `--install-service` write them.
31
+
32
+ On a match, `firewatcher` writes:
33
+
34
+ - a per-incident capture under `{output-folder}/{YYYY-MM}/…`
35
+ - a line in `{output-folder}/journal.log`
36
+
37
+ Live capture is written as lines arrive, so a copy can already be on another filesystem if the host then disappears. Each captured line is flushed as it is written, and the capture file is fsynced at most once a second and when the capture closes, so a host crash loses at most about the last second of it.
38
+
39
+ ## systemd service
40
+
41
+ ```bash
42
+ sudo firewatcher --install-service
43
+ sudo firewatcher --print-unit
44
+ sudo firewatcher --uninstall-service
45
+ ```
46
+
47
+ `--install-service` writes `/etc/systemd/system/firewatcher.service`, creates the output directory, seeds `/etc/firewatcher/patterns.d/` with example patterns **only if that directory is empty**, writes `/etc/firewatcher/firewatcher.conf` **only if that file is missing**, then `systemctl daemon-reload && systemctl enable --now firewatcher`.
48
+
49
+ If systemd is not available, `--install-service` and `--uninstall-service` print a warning and exit. `--print-unit` still works so you can copy the unit elsewhere.
50
+
51
+ Useful flags:
52
+
53
+ | Flag | Description |
54
+ |------|-------------|
55
+ | `--unit-name NAME` | Unit name (default: `firewatcher`) |
56
+ | `--requires-mounts-for PATH` | Add `RequiresMountsFor=` (repeat for NFS/remote output) |
57
+ | `--no-enable` | Write the unit and reload, but do not enable or start it |
58
+
59
+ ```bash
60
+ sudo firewatcher --install-service -o /mnt/logs/captured_messages \
61
+ --requires-mounts-for /mnt/logs/captured_messages \
62
+ /etc/firewatcher/patterns.d
63
+ ```
64
+
65
+ ## Options
66
+
67
+ | Flag | Description |
68
+ |------|-------------|
69
+ | `pattern_file …` | Pattern files or directories |
70
+ | `--log-file` | Log file to follow (default: journalctl / syslog / messages) |
71
+ | `-t`, `--capture-time` | Seconds of context after a match (default: 300) |
72
+ | `-o`, `--output-folder` | Where to write captures (default: `/var/log/captured_messages/`) |
73
+ | `--tail_lines` | Lines to start with (default: 20; `+N` from line N, not with journalctl) |
74
+ | `--filter_only` | Filter existing logs only (`cat` instead of `tail -F`, or `journalctl` without `--follow`) |
75
+ | `--compress-after-months` | Tar.xz monthly dirs after N months (default: 3) |
76
+ | `--delete-after-months` | Delete `YYYY-MM` dirs after N months (default: 0 = never) |
77
+ | `--capture_line_count_max` | Max lines before/after a match (default: 10000). Also caps how many recent lines are held in memory |
78
+ | `--config PATH` | INI config file. Default: `/etc/firewatcher/firewatcher.conf` when that file exists. An explicit path must exist |
79
+ | `--llm` / `--no-llm` | Enable or disable LLM analysis of each finished capture |
80
+ | `--llm-base-url URL` | OpenAI-compatible API root, such as `http://127.0.0.1:11434/v1` |
81
+ | `--llm-model NAME` | Model name |
82
+ | `--llm-timeout SEC` | LLM HTTP timeout (default: 60) |
83
+ | `--llm-max-bytes N` | Capture bytes sent to the model (default: 48000) |
84
+ | `--notify` / `--no-notify` | Enable or disable ntfy pushes |
85
+ | `--ntfy-url URL` | ntfy topic URL, such as `https://ntfy.sh/mytopic` |
86
+ | `--ntfy-priority N` | ntfy priority 1–5 (default: 4) |
87
+ | `--ntfy-timeout SEC` | Worker ntfy timeout (default: 10). A full analysis queue uses 5 seconds |
88
+ | `--install-service` | Install and enable a systemd unit |
89
+ | `--print-unit` | Print that unit and exit |
90
+ | `--uninstall-service` | Disable and remove the unit |
91
+ | `-V`, `--version` | Show version and exit |
92
+
93
+ ## Config file
94
+
95
+ Runtime settings live in an INI file. Explicit command-line flags override the file. Built-in defaults apply when neither sets a value. A missing default file is ignored. `--uninstall-service` does not read the file.
96
+
97
+ A commented example with all supported settings is in [`examples/firewatcher.conf`](examples/firewatcher.conf). Copy it to `/etc/firewatcher/firewatcher.conf`, or use `firewatcher --config /path/to/firewatcher.conf`.
98
+
99
+ ```ini
100
+ [watch]
101
+ patterns = /etc/firewatcher/patterns.d
102
+ capture_time = 300
103
+ output_folder = /var/log/captured_messages/
104
+ filter_only = false
105
+
106
+ [llm]
107
+ enabled = false
108
+ base_url = http://127.0.0.1:11434/v1
109
+ model = llama3.1
110
+ timeout = 60
111
+ max_bytes = 48000
112
+
113
+ [notify]
114
+ enabled = false
115
+ ntfy_url = https://ntfy.sh/firewatcher-myhost-REPLACE_ME
116
+ priority = 4
117
+ timeout = 10
118
+ ```
119
+
120
+ `patterns` is one path per line, or comma-separated. Positional pattern arguments replace that list.
121
+
122
+ `api_key` under `[llm]` and `token` under `[notify]` are not command-line flags. `FIREWATCHER_LLM_API_KEY` and `FIREWATCHER_NTFY_TOKEN` override them when set. `--install-service` creates `/etc/firewatcher/firewatcher.conf` with mode `0600` only when it is missing, and the unit runs `firewatcher --config` that path. An existing file is left as it is. Flags passed to a later install are appended on `ExecStart` so they override the file.
123
+
124
+ Generated configs include explanatory comments above each setting. When no ntfy URL is supplied, a new config gets `https://ntfy.sh/firewatcher-<hostname>-<random>`, with a 12-character URL-safe base64 suffix (72 random bits, no padding). The hostname is sanitized and shortened as needed to keep the topic within 64 characters. Explicit URLs are preserved, and reinstalling leaves an existing config and topic unchanged. Notifications still default to off; subscribe to the generated topic in ntfy, then set `[notify] enabled = true`. When copying the example above, replace its topic with your own.
125
+
126
+ ## LLM analysis and notifications
127
+
128
+ Both features default off.
129
+
130
+ - Notifications on, LLM off: every finished capture is pushed. The title is `{hostname}: {pattern}`.
131
+ - LLM on, notifications off: the model writes a title, summary, and notify decision. The report is appended to the capture file and logged. Nothing is pushed.
132
+ - Both on: a push is sent only when the model sets `notify` to true.
133
+ - If the LLM request times out, returns an HTTP error, or is not the expected JSON, and notifications are on, firewatcher still pushes, using the generated title.
134
+
135
+ The capture window closes even when no further log lines arrive. Analysis runs on a background worker so log following stays live. The queue holds four captures. Notify-only jobs run ahead of jobs that still need the model. When the queue is full, that incident skips the model and, if notifications are on, the capture thread sends the generated-title push itself.
136
+
137
+ Lines firewatcher writes back to the journal are a single physical line containing `firewatcher-self`, so a summary that quotes `panic` or `I/O error` does not start another capture. The capture files themselves stay in the log stream's original format.
138
+
139
+
140
+ ## Development
141
+
142
+ ```bash
143
+ pip install -r requirements-dev.txt
144
+ pytest
145
+ pytest --cov=firewatcher --cov-fail-under=95 --cov-report=term-missing
146
+ ```
147
+
148
+ ## Author
149
+
150
+ Yufei Pan (pan@zopyr.us)
151
+
152
+ ## License
153
+
154
+ GPL-3.0-or-later
@@ -0,0 +1,52 @@
1
+ # Example firewatcher configuration.
2
+ # Copy to /etc/firewatcher/firewatcher.conf or use --config /path/to/file.conf.
3
+ # Explicit command-line flags override these settings.
4
+ # Keep this file outside your pattern directory.
5
+
6
+ [watch]
7
+ # Pattern files or directories, comma-separated or one per indented line.
8
+ patterns = /etc/firewatcher/patterns.d
9
+ # Leave blank to use journald, then /var/log/syslog or /var/log/messages.
10
+ log_file =
11
+ # Seconds of context to capture after a match.
12
+ capture_time = 300
13
+ # Directory for incident captures and journal.log.
14
+ output_folder = /var/log/captured_messages/
15
+ # Initial lines to read; +N starts at line N when reading a file.
16
+ tail_lines = 20
17
+ # Process existing logs only instead of following new messages.
18
+ filter_only = false
19
+ # Compress monthly capture directories after this many months; 0 disables.
20
+ compress_after_months = 3
21
+ # Delete monthly capture directories after this many months; 0 disables.
22
+ delete_after_months = 0
23
+ # Maximum context lines before and after a match.
24
+ capture_line_count_max = 10000
25
+
26
+ [llm]
27
+ # Set true after configuring your OpenAI-compatible API and model.
28
+ enabled = false
29
+ # API root; do not add /chat/completions.
30
+ base_url = http://127.0.0.1:11434/v1
31
+ # Model name accepted by the API.
32
+ model = llama3.1
33
+ # Optional secret; FIREWATCHER_LLM_API_KEY overrides this value.
34
+ api_key =
35
+ # HTTP timeout in seconds.
36
+ timeout = 60
37
+ # Maximum capture bytes sent to the model.
38
+ max_bytes = 48000
39
+
40
+ [notify]
41
+ # Enable pushes; when LLM analysis is on, the model decides whether to notify.
42
+ enabled = false
43
+ # Example only: replace with your own topic and subscribe to it in ntfy.
44
+ # --install-service generates firewatcher-<hostname>-<12 random characters>
45
+ # when creating a new config without a supplied URL. Keep unprotected URLs private.
46
+ ntfy_url = https://ntfy.sh/firewatcher-myhost-REPLACE_ME
47
+ # Optional secret; FIREWATCHER_NTFY_TOKEN overrides this value.
48
+ token =
49
+ # ntfy priority: 1 (minimum) through 5 (maximum).
50
+ priority = 4
51
+ # HTTP timeout in seconds.
52
+ timeout = 10
@@ -0,0 +1,179 @@
1
+ Metadata-Version: 2.4
2
+ Name: firewatcher
3
+ Version: 1.59
4
+ Summary: Watch logs for incident patterns and persist surrounding context off-box
5
+ Home-page: https://github.com/yufei-pan/firewatcher
6
+ Author: Yufei Pan
7
+ Author-email: pan@zopyr.us
8
+ License: GPLv3+
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
11
+ Classifier: Operating System :: POSIX :: Linux
12
+ Classifier: Topic :: System :: Logging
13
+ Classifier: Topic :: System :: Monitoring
14
+ Requires-Python: >=3.6
15
+ Description-Content-Type: text/markdown
16
+ Dynamic: author
17
+ Dynamic: author-email
18
+ Dynamic: classifier
19
+ Dynamic: description
20
+ Dynamic: description-content-type
21
+ Dynamic: home-page
22
+ Dynamic: license
23
+ Dynamic: requires-python
24
+ Dynamic: summary
25
+
26
+ # firewatcher
27
+
28
+ Watch a log stream (journald, syslog, or any file) for incident patterns, then persist a window of surrounding messages to a durable location — including a network filesystem — so the record survives even if the machine later dies.
29
+
30
+ The watcher prefers `journalctl --follow` when `journalctl` is on PATH. On hosts without journald it falls back to `/var/log/syslog`, then `/var/log/messages`, or `--log-file`. Running the daemon does not require systemd; only `--install-service` / `--uninstall-service` do.
31
+
32
+ ## Install
33
+
34
+ ```bash
35
+ pip install firewatcher
36
+ ```
37
+
38
+ From a clone of this repository:
39
+
40
+ ```bash
41
+ pip install .
42
+ ```
43
+
44
+ Requires **Python 3.6+**. No third-party runtime dependencies.
45
+
46
+ ## Quick start
47
+
48
+ ```bash
49
+ firewatcher /etc/firewatcher/patterns.d
50
+ firewatcher -t 300 -o /var/log/captured_messages/ /etc/firewatcher/patterns.d
51
+ ```
52
+
53
+ A directory argument loads every non-hidden pattern file inside it. Files ending in `.regex` are compiled as regular expressions. Any other pattern file is treated as fnmatch (bash-like) substrings, with `*` added on both ends.
54
+
55
+ `--install-service` seeds `/etc/firewatcher/patterns.d` with example patterns when that directory is empty. The `examples/` directory in the source tree is not installed onto `PATH` by `pip install`; copy the pattern files from a clone, or let `--install-service` write them.
56
+
57
+ On a match, `firewatcher` writes:
58
+
59
+ - a per-incident capture under `{output-folder}/{YYYY-MM}/…`
60
+ - a line in `{output-folder}/journal.log`
61
+
62
+ Live capture is written as lines arrive, so a copy can already be on another filesystem if the host then disappears. Each captured line is flushed as it is written, and the capture file is fsynced at most once a second and when the capture closes, so a host crash loses at most about the last second of it.
63
+
64
+ ## systemd service
65
+
66
+ ```bash
67
+ sudo firewatcher --install-service
68
+ sudo firewatcher --print-unit
69
+ sudo firewatcher --uninstall-service
70
+ ```
71
+
72
+ `--install-service` writes `/etc/systemd/system/firewatcher.service`, creates the output directory, seeds `/etc/firewatcher/patterns.d/` with example patterns **only if that directory is empty**, writes `/etc/firewatcher/firewatcher.conf` **only if that file is missing**, then `systemctl daemon-reload && systemctl enable --now firewatcher`.
73
+
74
+ If systemd is not available, `--install-service` and `--uninstall-service` print a warning and exit. `--print-unit` still works so you can copy the unit elsewhere.
75
+
76
+ Useful flags:
77
+
78
+ | Flag | Description |
79
+ |------|-------------|
80
+ | `--unit-name NAME` | Unit name (default: `firewatcher`) |
81
+ | `--requires-mounts-for PATH` | Add `RequiresMountsFor=` (repeat for NFS/remote output) |
82
+ | `--no-enable` | Write the unit and reload, but do not enable or start it |
83
+
84
+ ```bash
85
+ sudo firewatcher --install-service -o /mnt/logs/captured_messages \
86
+ --requires-mounts-for /mnt/logs/captured_messages \
87
+ /etc/firewatcher/patterns.d
88
+ ```
89
+
90
+ ## Options
91
+
92
+ | Flag | Description |
93
+ |------|-------------|
94
+ | `pattern_file …` | Pattern files or directories |
95
+ | `--log-file` | Log file to follow (default: journalctl / syslog / messages) |
96
+ | `-t`, `--capture-time` | Seconds of context after a match (default: 300) |
97
+ | `-o`, `--output-folder` | Where to write captures (default: `/var/log/captured_messages/`) |
98
+ | `--tail_lines` | Lines to start with (default: 20; `+N` from line N, not with journalctl) |
99
+ | `--filter_only` | Filter existing logs only (`cat` instead of `tail -F`, or `journalctl` without `--follow`) |
100
+ | `--compress-after-months` | Tar.xz monthly dirs after N months (default: 3) |
101
+ | `--delete-after-months` | Delete `YYYY-MM` dirs after N months (default: 0 = never) |
102
+ | `--capture_line_count_max` | Max lines before/after a match (default: 10000). Also caps how many recent lines are held in memory |
103
+ | `--config PATH` | INI config file. Default: `/etc/firewatcher/firewatcher.conf` when that file exists. An explicit path must exist |
104
+ | `--llm` / `--no-llm` | Enable or disable LLM analysis of each finished capture |
105
+ | `--llm-base-url URL` | OpenAI-compatible API root, such as `http://127.0.0.1:11434/v1` |
106
+ | `--llm-model NAME` | Model name |
107
+ | `--llm-timeout SEC` | LLM HTTP timeout (default: 60) |
108
+ | `--llm-max-bytes N` | Capture bytes sent to the model (default: 48000) |
109
+ | `--notify` / `--no-notify` | Enable or disable ntfy pushes |
110
+ | `--ntfy-url URL` | ntfy topic URL, such as `https://ntfy.sh/mytopic` |
111
+ | `--ntfy-priority N` | ntfy priority 1–5 (default: 4) |
112
+ | `--ntfy-timeout SEC` | Worker ntfy timeout (default: 10). A full analysis queue uses 5 seconds |
113
+ | `--install-service` | Install and enable a systemd unit |
114
+ | `--print-unit` | Print that unit and exit |
115
+ | `--uninstall-service` | Disable and remove the unit |
116
+ | `-V`, `--version` | Show version and exit |
117
+
118
+ ## Config file
119
+
120
+ Runtime settings live in an INI file. Explicit command-line flags override the file. Built-in defaults apply when neither sets a value. A missing default file is ignored. `--uninstall-service` does not read the file.
121
+
122
+ A commented example with all supported settings is in [`examples/firewatcher.conf`](examples/firewatcher.conf). Copy it to `/etc/firewatcher/firewatcher.conf`, or use `firewatcher --config /path/to/firewatcher.conf`.
123
+
124
+ ```ini
125
+ [watch]
126
+ patterns = /etc/firewatcher/patterns.d
127
+ capture_time = 300
128
+ output_folder = /var/log/captured_messages/
129
+ filter_only = false
130
+
131
+ [llm]
132
+ enabled = false
133
+ base_url = http://127.0.0.1:11434/v1
134
+ model = llama3.1
135
+ timeout = 60
136
+ max_bytes = 48000
137
+
138
+ [notify]
139
+ enabled = false
140
+ ntfy_url = https://ntfy.sh/firewatcher-myhost-REPLACE_ME
141
+ priority = 4
142
+ timeout = 10
143
+ ```
144
+
145
+ `patterns` is one path per line, or comma-separated. Positional pattern arguments replace that list.
146
+
147
+ `api_key` under `[llm]` and `token` under `[notify]` are not command-line flags. `FIREWATCHER_LLM_API_KEY` and `FIREWATCHER_NTFY_TOKEN` override them when set. `--install-service` creates `/etc/firewatcher/firewatcher.conf` with mode `0600` only when it is missing, and the unit runs `firewatcher --config` that path. An existing file is left as it is. Flags passed to a later install are appended on `ExecStart` so they override the file.
148
+
149
+ Generated configs include explanatory comments above each setting. When no ntfy URL is supplied, a new config gets `https://ntfy.sh/firewatcher-<hostname>-<random>`, with a 12-character URL-safe base64 suffix (72 random bits, no padding). The hostname is sanitized and shortened as needed to keep the topic within 64 characters. Explicit URLs are preserved, and reinstalling leaves an existing config and topic unchanged. Notifications still default to off; subscribe to the generated topic in ntfy, then set `[notify] enabled = true`. When copying the example above, replace its topic with your own.
150
+
151
+ ## LLM analysis and notifications
152
+
153
+ Both features default off.
154
+
155
+ - Notifications on, LLM off: every finished capture is pushed. The title is `{hostname}: {pattern}`.
156
+ - LLM on, notifications off: the model writes a title, summary, and notify decision. The report is appended to the capture file and logged. Nothing is pushed.
157
+ - Both on: a push is sent only when the model sets `notify` to true.
158
+ - If the LLM request times out, returns an HTTP error, or is not the expected JSON, and notifications are on, firewatcher still pushes, using the generated title.
159
+
160
+ The capture window closes even when no further log lines arrive. Analysis runs on a background worker so log following stays live. The queue holds four captures. Notify-only jobs run ahead of jobs that still need the model. When the queue is full, that incident skips the model and, if notifications are on, the capture thread sends the generated-title push itself.
161
+
162
+ Lines firewatcher writes back to the journal are a single physical line containing `firewatcher-self`, so a summary that quotes `panic` or `I/O error` does not start another capture. The capture files themselves stay in the log stream's original format.
163
+
164
+
165
+ ## Development
166
+
167
+ ```bash
168
+ pip install -r requirements-dev.txt
169
+ pytest
170
+ pytest --cov=firewatcher --cov-fail-under=95 --cov-report=term-missing
171
+ ```
172
+
173
+ ## Author
174
+
175
+ Yufei Pan (pan@zopyr.us)
176
+
177
+ ## License
178
+
179
+ GPL-3.0-or-later
@@ -0,0 +1,23 @@
1
+ MANIFEST.in
2
+ README.md
3
+ firewatcher.py
4
+ setup.py
5
+ examples/firewatcher.conf
6
+ examples/nvme_failure.regex
7
+ examples/sys_msg.txt
8
+ firewatcher.egg-info/PKG-INFO
9
+ firewatcher.egg-info/SOURCES.txt
10
+ firewatcher.egg-info/dependency_links.txt
11
+ firewatcher.egg-info/entry_points.txt
12
+ firewatcher.egg-info/top_level.txt
13
+ tests/conftest.py
14
+ tests/test_capture.py
15
+ tests/test_cli.py
16
+ tests/test_compressor.py
17
+ tests/test_config.py
18
+ tests/test_log_source.py
19
+ tests/test_matching.py
20
+ tests/test_notify.py
21
+ tests/test_patterns.py
22
+ tests/test_slugify.py
23
+ tests/test_systemd.py