onlylogs 0.11.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +251 -77
- data/app/assets/images/onlylogs/favicon/apple-touch-icon.png +0 -0
- data/app/assets/images/onlylogs/favicon/favicon-96x96.png +0 -0
- data/app/assets/images/onlylogs/favicon/favicon.ico +0 -0
- data/app/assets/images/onlylogs/favicon/favicon.svg +1 -3
- data/app/assets/images/onlylogs/favicon/web-app-manifest-192x192.png +0 -0
- data/app/assets/images/onlylogs/favicon/web-app-manifest-512x512.png +0 -0
- data/app/assets/images/onlylogs/logo.png +0 -0
- data/app/assets/javascripts/onlylogs/clusterize.js +26 -1
- data/app/channels/onlylogs/logs_channel.rb +14 -9
- data/app/controllers/concerns/onlylogs/authentication.rb +26 -0
- data/app/controllers/onlylogs/application_controller.rb +1 -18
- data/app/controllers/onlylogs/queries_controller.rb +72 -0
- data/app/controllers/onlylogs/viewer_controller.rb +7 -0
- data/app/javascript/onlylogs/controllers/keyboard_shortcuts_controller.js +131 -21
- data/app/javascript/onlylogs/controllers/log_cursor_controller.js +286 -0
- data/app/javascript/onlylogs/controllers/log_streamer_controller.js +42 -2
- data/app/javascript/onlylogs/controllers/match_highlight_controller.js +82 -0
- data/app/javascript/onlylogs/controllers/queries_controller.js +470 -0
- data/app/javascript/onlylogs/controllers/range_slider_controller.js +5 -4
- data/app/javascript/onlylogs/controllers/text_selection_controller.js +136 -15
- data/app/models/onlylogs/file.rb +14 -5
- data/app/models/onlylogs/grep.rb +31 -12
- data/app/models/onlylogs/queries_record.rb +30 -0
- data/app/models/onlylogs/query.rb +29 -0
- data/app/views/onlylogs/logs/index.html.erb +1 -1
- data/app/views/onlylogs/shared/_log_container.html.erb +165 -27
- data/app/views/onlylogs/shared/_log_container_styles.html.erb +729 -44
- data/app/views/onlylogs/shared/_shortcuts_dialog.html.erb +62 -0
- data/bin/super_grep +7 -0
- data/bin/super_ripgrep +14 -0
- data/config/routes.rb +1 -0
- data/lib/onlylogs/configuration.rb +14 -1
- data/lib/onlylogs/engine.rb +9 -0
- data/lib/onlylogs/http_device.rb +28 -13
- data/lib/onlylogs/version.rb +1 -1
- data/lib/onlylogs.rb +3 -0
- metadata +38 -2
- data/app/jobs/onlylogs/application_job.rb +0 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: df2635aae242c5b3d3d1881c3dcb4c9297662200134c403b6f722036ef8f2357
|
|
4
|
+
data.tar.gz: f6faaccddcbd4042a0d6cb4cee8f087ce0ff7e9c75e66f0e403ee5c7de2edbc3
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e649a51a9841c162ad4c418eea5047462505054a040f5314fec37f171109fb39bf4198e12d107b04f52d77cc19d84ab873e5435ca28bb77c1b4aad17677d02b6
|
|
7
|
+
data.tar.gz: 4e6b8ab9a92d4317c04de55e4f560642e77baa8d8b73c7f583fdb181383eed1965c6e731cbb871ede1aa2ce291a0f3a4dd07759eada1e0d4fc1f5fb069ab479d
|
data/README.md
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
<img alt="w:100px" src="app/assets/images/onlylogs/logo.png" width="400px"/>
|
|
2
2
|
|
|
3
|
-
We believe logs are enough.
|
|
3
|
+
We believe logs are enough.
|
|
4
4
|
|
|
5
5
|
We believe logs in human-readable format are enough.
|
|
6
6
|
|
|
7
7
|
Stop streaming your logs to very expensive external services: just store your logs on disk.
|
|
8
8
|
|
|
9
|
-
When your application grows and you don't want to self-host your log files anymore, you can
|
|
10
|
-
|
|
9
|
+
When your application grows and you don't want to self-host your log files anymore, you can stream them
|
|
10
|
+
to https://onlylogs.io and continue enjoying the same features.
|
|
11
11
|
|
|
12
12
|
> [!IMPORTANT]
|
|
13
13
|
> https://onlylogs.io is still in beta. Send us an email to a@renuo.ch if you want access to the platform.
|
|
@@ -18,7 +18,11 @@ If you already have a disk, you can just keep there also your log files (as well
|
|
|
18
18
|
|
|
19
19
|
This section explains how to setup onlylogs to self host your logs and access them directly from your Rails app.
|
|
20
20
|
|
|
21
|
-
If instead you want to stream your logs to https://onlylogs.io, head
|
|
21
|
+
If instead you want to stream your logs to https://onlylogs.io, head
|
|
22
|
+
to [the onlylogs.io instructions page](https://onlylogs.io/instructions).
|
|
23
|
+
|
|
24
|
+
Onlylogs needs Rails 8 or newer. It works with Importmap, Sprockets and Propshaft (see
|
|
25
|
+
[INTEGRATION.md](INTEGRATION.md) if you have to register the Stimulus controller by hand).
|
|
22
26
|
|
|
23
27
|
Add this line to your application's Gemfile:
|
|
24
28
|
|
|
@@ -38,6 +42,7 @@ mount the engine in your `routes.rb`
|
|
|
38
42
|
Rails.application.routes.draw do
|
|
39
43
|
# ...
|
|
40
44
|
mount Onlylogs::Engine, at: "/onlylogs"
|
|
45
|
+
end
|
|
41
46
|
```
|
|
42
47
|
|
|
43
48
|
Finally, you **must secure the engine**. Read the section dedicated to the [Authentication](#authentication).
|
|
@@ -51,17 +56,50 @@ Finally, you **must secure the engine**. Read the section dedicated to the [Auth
|
|
|
51
56
|
|
|
52
57
|
Head to `/onlylogs` and enjoy your logs streamed right into your face!
|
|
53
58
|
|
|
54
|
-
|
|
59
|
+
The viewer has two modes:
|
|
60
|
+
|
|
61
|
+
- **Live**: the end of the file is streamed to you as new lines are written.
|
|
62
|
+
- **Search**: you look for something in the whole file.
|
|
63
|
+
|
|
64
|
+
Next to the query field you find two checkboxes:
|
|
65
|
+
|
|
66
|
+
- **Regexp**: the query is a regular expression. Without it, the query is searched as plain text.
|
|
67
|
+
- **Ignore case**: `error` also finds `ERROR`.
|
|
68
|
+
|
|
69
|
+
What the query matched is highlighted on every line. On any line you can also ask for the lines around it
|
|
70
|
+
(press `Enter` on it), and you can download the whole file.
|
|
55
71
|
|
|
56
72
|
> [!TIP]
|
|
57
|
-
> Onlylogs automatically detects and uses [ripgrep (rg)](https://github.com/BurntSushi/ripgrep) if available, which
|
|
58
|
-
|
|
73
|
+
> Onlylogs automatically detects and uses [ripgrep (rg)](https://github.com/BurntSushi/ripgrep) if available, which
|
|
74
|
+
provides significantly faster search experience.
|
|
75
|
+
> If ripgrep is not installed, onlylogs falls back to `grep`.
|
|
59
76
|
> A warning icon (⚠️) will be displayed in the toolbar when using `grep` to indicate slower search performance.
|
|
60
77
|
|
|
78
|
+
### Saved searches
|
|
79
|
+
|
|
80
|
+
Click the star in the query field to save the current query. The list next to it shows all saved searches: pick one to
|
|
81
|
+
apply it, rename it or remove it. A saved search is shared by everybody who opens the same log file.
|
|
82
|
+
|
|
83
|
+
Saved searches are stored in a small SQLite database, in a hidden `.onlylogs` folder next to the log file
|
|
84
|
+
(for example `log/.onlylogs/queries.sqlite3`). The folder must be writable. In Docker, keep it on a volume, like the
|
|
85
|
+
logs (see [Notes about Docker](#notes-about-docker)). You can change the folder name with `config.database_dir`.
|
|
86
|
+
|
|
87
|
+
### Keyboard shortcuts
|
|
88
|
+
|
|
89
|
+
Hold **⌃ Control** on macOS or **Alt** on Windows and Linux: a letter appears on every control of the toolbar, and
|
|
90
|
+
modifier + letter triggers it from anywhere, also while typing a query. Press `?` (or the ⌨ button) for the full list.
|
|
91
|
+
|
|
92
|
+
Inside the log lines, `↑` `↓` (or `j` `k`) move a line cursor, `←` `→` select words of that line (`Shift` grows the
|
|
93
|
+
selection) and `Enter` searches the selection, or shows the lines around the current one. `/` jumps to the query,
|
|
94
|
+
`Enter` or `Esc` in the query jumps back into the lines.
|
|
95
|
+
|
|
96
|
+
A host app can give its own controls a letter with the same attribute, e.g.
|
|
97
|
+
`<button data-onlylogs-shortcut="f">Fullscreen</button>`.
|
|
98
|
+
|
|
61
99
|
## Logging in production
|
|
62
100
|
|
|
63
|
-
Rails apps log to `STDOUT` in production and let the platform collect the stream.
|
|
64
|
-
|
|
101
|
+
Rails apps log to `STDOUT` in production and let the platform collect the stream. onlylogs keeps that default:
|
|
102
|
+
`Onlylogs::HttpLogger.new` writes every line to `$stdout` and, when
|
|
65
103
|
`ONLYLOGS_DRAIN_URL` is set, ships it to https://onlylogs.io as well. Nothing is written to disk.
|
|
66
104
|
|
|
67
105
|
```ruby
|
|
@@ -69,13 +107,19 @@ onlylogs keeps that default: `Onlylogs::HttpLogger.new` writes every line to `$s
|
|
|
69
107
|
config.logger = Onlylogs::HttpLogger.new
|
|
70
108
|
```
|
|
71
109
|
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
110
|
+
Shipping is fire and forget. Lines are batched in memory and sent in the background; when onlylogs.io is down or does
|
|
111
|
+
not answer within the timeouts, the sender backs off for a while and the lines of that period are lost. Your `$stdout`
|
|
112
|
+
stream still has them. If you would rather keep them, set `ONLYLOGS_SPOOL_ENABLED=true` (`1`, `yes` and `on` work
|
|
113
|
+
too): batches that could not be delivered are then written to `ONLYLOGS_SPOOL_DIR` (default `tmp/onlylogs/spool`, up
|
|
114
|
+
to `ONLYLOGS_SPOOL_MAX_BYTES`, 128 MB) and replayed once the drain is back, also after a restart when the directory is
|
|
115
|
+
on a persistent disk.
|
|
116
|
+
|
|
117
|
+
Keep this default on an **ephemeral filesystem** (Heroku, Deploio, Docker, Kubernetes). A log file there is lost on
|
|
118
|
+
every deploy and, while it grows, its page cache counts towards the container's memory limit: it shows up as a memory
|
|
119
|
+
leak that resets on each release.
|
|
75
120
|
|
|
76
|
-
If you have a **persistent disk** and want the files, for instance to browse them with the engine,
|
|
77
|
-
|
|
78
|
-
good size here too:
|
|
121
|
+
If you have a **persistent disk** and want the files, for instance to browse them with the engine, pass a rotating log
|
|
122
|
+
device as the local fallback. Rails rotates at 100 MB in development, which is a good size here too:
|
|
79
123
|
|
|
80
124
|
```ruby
|
|
81
125
|
# config/environments/production.rb
|
|
@@ -85,11 +129,82 @@ config.logger = Onlylogs::HttpLogger.new(local_fallback: log_file)
|
|
|
85
129
|
|
|
86
130
|
Any IO or log device works and is used as is. `Onlylogs::SocketLogger` accepts the same argument.
|
|
87
131
|
|
|
132
|
+
### Settings of `HttpLogger`
|
|
133
|
+
|
|
134
|
+
`HttpLogger` is configured with environment variables. Only `ONLYLOGS_DRAIN_URL` is needed. The others have sensible
|
|
135
|
+
defaults.
|
|
136
|
+
|
|
137
|
+
| Variable | Default | What it does |
|
|
138
|
+
|-------------------------------|--------------------------|-----------------------------------------------------------|
|
|
139
|
+
| `ONLYLOGS_DRAIN_URL` | none | Where the logs are sent. Without it, logs stay local. |
|
|
140
|
+
| `ONLYLOGS_BATCH_SIZE` | `100` | Lines sent together in one request. |
|
|
141
|
+
| `ONLYLOGS_FLUSH_INTERVAL` | `0.5` | Seconds to wait before sending an incomplete batch. |
|
|
142
|
+
| `ONLYLOGS_MAX_QUEUE_SIZE` | `10000` | Lines kept in memory while waiting to be sent. |
|
|
143
|
+
| `ONLYLOGS_MAX_BATCH_BYTES` | `1048576` (1 MB) | Maximum size of a batch. Longer lines are truncated. |
|
|
144
|
+
| `ONLYLOGS_OPEN_TIMEOUT` | `0.5` | Seconds to wait when connecting to the drain. |
|
|
145
|
+
| `ONLYLOGS_READ_TIMEOUT` | `0.5` | Seconds to wait for the answer of the drain. |
|
|
146
|
+
| `ONLYLOGS_KEEP_ALIVE_TIMEOUT` | `30` | Seconds an idle connection is kept open for reuse. |
|
|
147
|
+
| `ONLYLOGS_CIRCUIT_COOLDOWN` | `30` | Seconds to stop sending after repeated failures. |
|
|
148
|
+
| `ONLYLOGS_SPOOL_DIR` | `tmp/onlylogs/spool` | Folder where undelivered batches are stored on disk. |
|
|
149
|
+
| `ONLYLOGS_SPOOL_MAX_BYTES` | `134217728` (128 MB) | Maximum size of that folder. |
|
|
150
|
+
|
|
151
|
+
A wrong value (for example a negative number) is ignored with a warning and the default is used. A wrong
|
|
152
|
+
`ONLYLOGS_DRAIN_URL` never stops your app from booting: it just logs locally.
|
|
153
|
+
|
|
154
|
+
### What happens when the drain is down
|
|
155
|
+
|
|
156
|
+
Your app must never become slow or crash because of logging. `HttpLogger` sends the lines from a background thread,
|
|
157
|
+
never from the request thread, and it protects your app like this:
|
|
158
|
+
|
|
159
|
+
1. Lines wait in a queue in memory and are sent in batches.
|
|
160
|
+
2. After a few failed requests, a **circuit breaker** stops trying for a while, so your app does not wait on a dead
|
|
161
|
+
server.
|
|
162
|
+
3. Batches that could not be delivered are saved in the **spool** folder on disk. They are sent again when the drain
|
|
163
|
+
is back, even after a restart. Because of this, a line can very rarely arrive twice, but it is not lost.
|
|
164
|
+
4. If the queue is full, new lines are dropped. If the spool is full, its oldest batches are deleted. Your app keeps
|
|
165
|
+
running.
|
|
166
|
+
|
|
167
|
+
To see this in action, run `bin/fake_drain` and `bin/fake_app` (see [Development](#development--contributing)).
|
|
168
|
+
|
|
169
|
+
### Sidecar (alternative to `HttpLogger`)
|
|
170
|
+
|
|
171
|
+
Instead of sending logs from inside the app, you can let a small separate process do it. The app writes the lines to
|
|
172
|
+
a Unix socket with `Onlylogs::SocketLogger`, and the sidecar sends them to the drain.
|
|
173
|
+
|
|
174
|
+
```ruby
|
|
175
|
+
# config/environments/production.rb
|
|
176
|
+
config.logger = Onlylogs::SocketLogger.new
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
```ruby
|
|
180
|
+
# config/puma.rb
|
|
181
|
+
plugin :onlylogs_sidecar # starts and stops the sidecar together with Puma
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
You can also start it yourself with `bin/onlylogs_sidecar`. It reads `ONLYLOGS_DRAIN_URL` and, optionally,
|
|
185
|
+
`ONLYLOGS_SIDECAR_SOCKET`, `ONLYLOGS_BATCH_SIZE` and `ONLYLOGS_FLUSH_INTERVAL`.
|
|
186
|
+
|
|
187
|
+
Prefer `HttpLogger`: it works in every process (Puma, Sidekiq, rake tasks) and has the spool. Use the sidecar only if
|
|
188
|
+
you want the sending to happen outside of your app process.
|
|
189
|
+
|
|
190
|
+
### Filtering Log Lines with a Denylist
|
|
191
|
+
|
|
192
|
+
The `Onlylogs::Formatter` supports a denylist: an array of regular expressions that prevents matching lines from being
|
|
193
|
+
logged. This is useful for filtering out noisy or irrelevant entries like health checks or asset requests.
|
|
194
|
+
|
|
195
|
+
```ruby
|
|
196
|
+
# config/environments/production.rb
|
|
197
|
+
config.logger = Onlylogs::HttpLogger.new
|
|
198
|
+
config.logger.formatter.denylist = [/health_check/, /ping/, /\.css\z/]
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Any log message matching at least one pattern in the denylist will be silently dropped.
|
|
202
|
+
|
|
88
203
|
## Authentication
|
|
89
204
|
|
|
90
205
|
Yes, we should do this right away, because this engine gives access to your log files, so you want to be sure.
|
|
91
206
|
|
|
92
|
-
The engine has
|
|
207
|
+
The engine has controllers and one ActionCable channel that **must be protected**.
|
|
93
208
|
|
|
94
209
|
Please be sure to secure them properly.
|
|
95
210
|
|
|
@@ -103,15 +218,17 @@ If your app is running in a Docker container, for example with Kamal, remember t
|
|
|
103
218
|
```yaml
|
|
104
219
|
# config/deploy.yml
|
|
105
220
|
volumes:
|
|
106
|
-
- "storage:/rails/storage"
|
|
107
|
-
- "cache:/rails/tmp/cache"
|
|
108
|
-
- "logs:/rails/log"
|
|
221
|
+
- "storage:/rails/storage"
|
|
222
|
+
- "cache:/rails/tmp/cache"
|
|
223
|
+
- "logs:/rails/log"
|
|
109
224
|
```
|
|
110
225
|
|
|
226
|
+
The saved searches live in `log/.onlylogs`, so they are kept too.
|
|
227
|
+
|
|
111
228
|
### Basic Authentication Setup
|
|
112
229
|
|
|
113
|
-
Credentials can be configured using environment variables, Rails credentials, or programmatically.
|
|
114
|
-
|
|
230
|
+
Credentials can be configured using environment variables, Rails credentials, or programmatically. Environment variables
|
|
231
|
+
take precedence over Rails credentials.
|
|
115
232
|
|
|
116
233
|
```bash
|
|
117
234
|
# env variables
|
|
@@ -134,11 +251,10 @@ Onlylogs.configure do |config|
|
|
|
134
251
|
end
|
|
135
252
|
```
|
|
136
253
|
|
|
137
|
-
|
|
138
254
|
### Custom Authentication
|
|
139
255
|
|
|
140
|
-
When you need custom authentication logic beyond basic auth,
|
|
141
|
-
|
|
256
|
+
When you need custom authentication logic beyond basic auth, you can override the default authentication by configuring
|
|
257
|
+
a parent controller that defines the `authenticate_onlylogs_user!` method.
|
|
142
258
|
|
|
143
259
|
Configure a custom parent controller in your initializer:
|
|
144
260
|
|
|
@@ -158,7 +274,7 @@ class ApplicationController < ActionController::Base
|
|
|
158
274
|
private
|
|
159
275
|
|
|
160
276
|
def authenticate_onlylogs_user!
|
|
161
|
-
raise unless current_user.can_access_logs?
|
|
277
|
+
raise unless current_user.can_access_logs?
|
|
162
278
|
end
|
|
163
279
|
end
|
|
164
280
|
```
|
|
@@ -174,25 +290,36 @@ Onlylogs.configure do |config|
|
|
|
174
290
|
end
|
|
175
291
|
```
|
|
176
292
|
|
|
177
|
-
###
|
|
293
|
+
### Allowing only some users to see some files
|
|
294
|
+
|
|
295
|
+
By default everybody who gets through the authentication can open every file you allowed (see
|
|
296
|
+
[File Access Security](#file-access-security)). To decide per user and per file, set `authorize_log_file`. It runs
|
|
297
|
+
inside the controller, so you can use your own `current_user`. Return something falsy to refuse the file.
|
|
178
298
|
|
|
179
|
-
|
|
180
|
-
|
|
299
|
+
```ruby
|
|
300
|
+
# config/initializers/onlylogs.rb
|
|
301
|
+
Onlylogs.configure do |config|
|
|
302
|
+
config.authorize_log_file = ->(file_path) { current_user.admin? || file_path.to_s.end_with?("public.log") }
|
|
303
|
+
end
|
|
304
|
+
```
|
|
181
305
|
|
|
306
|
+
If you show the log viewer in your own pages to more people than the rest of the engine, you can give its endpoints a
|
|
307
|
+
different parent controller with `config.viewer_parent_controller`.
|
|
182
308
|
|
|
183
|
-
|
|
309
|
+
### WebSocket Authentication
|
|
184
310
|
|
|
185
|
-
|
|
186
|
-
Check the file [_log_container_styles.html.erb](app/views/onlylogs/shared/_log_container_styles.html.erb) for the complete list of CSS variables.
|
|
311
|
+
Logs are streamed through a WebSocket. To stream a file, two things must be true:
|
|
187
312
|
|
|
188
|
-
|
|
313
|
+
- the file path is allowed (see [File Access Security](#file-access-security));
|
|
314
|
+
- the file path is sent encrypted, with `Onlylogs::SecureFilePath.encrypt`. The viewer does this for you. Nobody can
|
|
315
|
+
ask for a path they were not given.
|
|
189
316
|
|
|
190
|
-
|
|
317
|
+
An empty or unknown path is refused.
|
|
191
318
|
|
|
192
319
|
### File Access Security
|
|
193
320
|
|
|
194
|
-
Onlylogs includes a secure file access system that prevents unauthorized access to files on your server.
|
|
195
|
-
|
|
321
|
+
Onlylogs includes a secure file access system that prevents unauthorized access to files on your server. By default,
|
|
322
|
+
onlylogs can access your Rails environment-specific log files (e.g., `log/development.log`, `log/production.log`).
|
|
196
323
|
|
|
197
324
|
#### Configuring Allowed Files
|
|
198
325
|
|
|
@@ -206,15 +333,15 @@ Onlylogs.configure do |config|
|
|
|
206
333
|
Rails.root.join("log/development.log"),
|
|
207
334
|
Rails.root.join("log/production.log"),
|
|
208
335
|
Rails.root.join("log/test.log"),
|
|
209
|
-
|
|
336
|
+
|
|
210
337
|
# Custom log files
|
|
211
338
|
Rails.root.join("log/custom.log"),
|
|
212
339
|
Rails.root.join("log/api.log"),
|
|
213
|
-
|
|
340
|
+
|
|
214
341
|
# Application-specific logs
|
|
215
342
|
Rails.root.join("log/background_jobs.log"),
|
|
216
343
|
Rails.root.join("log/imports.log"),
|
|
217
|
-
|
|
344
|
+
|
|
218
345
|
# Allow all .log files in a directory using glob patterns
|
|
219
346
|
Rails.root.join("log/*.log"),
|
|
220
347
|
Rails.root.join("tmp/logs/*.log")
|
|
@@ -223,6 +350,7 @@ end
|
|
|
223
350
|
```
|
|
224
351
|
|
|
225
352
|
**Default Behavior:**
|
|
353
|
+
|
|
226
354
|
- If not configured, onlylogs defaults to `Rails.root.join("log/#{Rails.env}.log").to_s`
|
|
227
355
|
- This means it will use `log/development.log` in development, `log/production.log` in production, etc.
|
|
228
356
|
|
|
@@ -236,11 +364,11 @@ Onlylogs.configure do |config|
|
|
|
236
364
|
config.log_file_patterns = [
|
|
237
365
|
# Allow all .log files in the log directory
|
|
238
366
|
Rails.root.join("log/*.log"),
|
|
239
|
-
|
|
367
|
+
|
|
240
368
|
# Allow specific pattern matches
|
|
241
369
|
Rails.root.join("log/*production*.log"),
|
|
242
370
|
Rails.root.join("log/*development*.log"),
|
|
243
|
-
|
|
371
|
+
|
|
244
372
|
# Allow files in subdirectories
|
|
245
373
|
Rails.root.join("log/**/*.log"),
|
|
246
374
|
Rails.root.join("tmp/**/*.log")
|
|
@@ -249,20 +377,43 @@ end
|
|
|
249
377
|
```
|
|
250
378
|
|
|
251
379
|
**Supported Glob Patterns:**
|
|
380
|
+
|
|
252
381
|
- `*.log` - Matches all files ending with `.log` in the specified directory
|
|
253
382
|
- `*production*.log` - Matches files containing "production" and ending with `.log`
|
|
254
383
|
- `**/*.log` - Matches all `.log` files in the directory and all subdirectories
|
|
255
384
|
|
|
256
385
|
**Important Notes:**
|
|
386
|
+
|
|
257
387
|
- Patterns are directory-specific - `log/*.log` only matches files in the `log/` directory
|
|
258
388
|
- Multiple patterns can be combined in the same configuration
|
|
259
389
|
|
|
390
|
+
## Configuration
|
|
391
|
+
|
|
392
|
+
All options go into `Onlylogs.configure`. The most useful ones:
|
|
393
|
+
|
|
394
|
+
| Option | Default | What it does |
|
|
395
|
+
|--------------------------------|-------------------------------|-----------------------------------------------------|
|
|
396
|
+
| `log_file_patterns` | `log/<Rails.env>.log` | Files (or globs) the viewer may open. |
|
|
397
|
+
| `basic_auth_user`, `basic_auth_password` | env or credentials | Credentials of the basic authentication. |
|
|
398
|
+
| `disable_basic_authentication` | `false` | Turn off basic auth (use it with your own auth). |
|
|
399
|
+
| `parent_controller` | none | Controller the engine inherits from. |
|
|
400
|
+
| `viewer_parent_controller` | `parent_controller` | Same, but only for the viewer endpoints. |
|
|
401
|
+
| `authorize_log_file` | none | Per-user, per-file permission check. |
|
|
402
|
+
| `ripgrep_enabled` | `true` if `rg` exists | Use ripgrep instead of grep. |
|
|
403
|
+
| `editor` | from env | Editor used for file path links. |
|
|
404
|
+
| `max_line_matches` | `100000` | Maximum lines returned by a search. |
|
|
405
|
+
| `search_timeout` | `120` | Seconds after which a search is stopped. |
|
|
406
|
+
| `database_dir` | `".onlylogs"` | Folder (next to the logs) with the saved searche. |
|
|
407
|
+
|
|
408
|
+
See `lib/onlylogs/configuration.rb` for the details.
|
|
260
409
|
|
|
261
410
|
### Configuring Code Editor for File Path Links
|
|
262
411
|
|
|
263
|
-
Onlylogs automatically detects file paths in log messages and converts them into clickable links that open in your
|
|
412
|
+
Onlylogs automatically detects file paths in log messages and converts them into clickable links that open in your
|
|
413
|
+
preferred code editor.
|
|
264
414
|
|
|
265
|
-
For a complete list of supported editors,
|
|
415
|
+
For a complete list of supported editors,
|
|
416
|
+
see [app/models/onlylogs/file_path_parser.rb](app/models/onlylogs/file_path_parser.rb).
|
|
266
417
|
|
|
267
418
|
```bash
|
|
268
419
|
# env variables
|
|
@@ -284,25 +435,26 @@ Onlylogs.configure do |config|
|
|
|
284
435
|
end
|
|
285
436
|
```
|
|
286
437
|
|
|
287
|
-
|
|
438
|
+
### Configuring Maximum Search Results
|
|
288
439
|
|
|
289
|
-
By default, onlylogs limits search results to 100,000 lines to prevent memory issues and ensure responsive performance.
|
|
440
|
+
By default, onlylogs limits search results to 100,000 lines to prevent memory issues and ensure responsive performance.
|
|
441
|
+
You can configure this limit based on your needs:
|
|
290
442
|
|
|
291
443
|
```ruby
|
|
292
444
|
# config/initializers/onlylogs.rb
|
|
293
445
|
Onlylogs.configure do |config|
|
|
294
446
|
# Set a custom limit (e.g., 50,000 lines)
|
|
295
447
|
config.max_line_matches = 50_000
|
|
296
|
-
|
|
448
|
+
|
|
297
449
|
# Or remove the limit entirely (use with caution)
|
|
298
450
|
config.max_line_matches = nil
|
|
299
451
|
end
|
|
300
452
|
```
|
|
301
453
|
|
|
302
|
-
|
|
454
|
+
### Bounding How Long a Viewer Search Can Run
|
|
303
455
|
|
|
304
|
-
Searches started from the log viewer stop after `search_timeout` seconds, keeping whatever they
|
|
305
|
-
|
|
456
|
+
Searches started from the log viewer stop after `search_timeout` seconds, keeping whatever they found and reporting that
|
|
457
|
+
they did not reach the end of the file. The default is 120 seconds:
|
|
306
458
|
|
|
307
459
|
```ruby
|
|
308
460
|
# config/initializers/onlylogs.rb
|
|
@@ -315,21 +467,30 @@ Onlylogs.configure do |config|
|
|
|
315
467
|
end
|
|
316
468
|
```
|
|
317
469
|
|
|
318
|
-
|
|
470
|
+
## Customization
|
|
471
|
+
|
|
472
|
+
Onlylogs provides two ways to customize the appearance of the log viewer: CSS Variables and a complete style override.
|
|
473
|
+
Check the file [_log_container_styles.html.erb](app/views/onlylogs/shared/_log_container_styles.html.erb) for the
|
|
474
|
+
complete list of CSS variables.
|
|
475
|
+
|
|
476
|
+
## Instrumentation
|
|
319
477
|
|
|
320
|
-
|
|
478
|
+
Every search sends an `ActiveSupport::Notifications` event called `search.onlylogs`. You can use it to measure how long
|
|
479
|
+
searches take, for example:
|
|
321
480
|
|
|
322
481
|
```ruby
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
482
|
+
ActiveSupport::Notifications.subscribe("search.onlylogs") do |event|
|
|
483
|
+
Rails.logger.info "Search #{event.payload[:query]} took #{event.duration.round}ms, " \
|
|
484
|
+
"#{event.payload[:matches]} matches, timed out: #{event.payload[:timed_out]}"
|
|
485
|
+
end
|
|
326
486
|
```
|
|
327
487
|
|
|
328
|
-
|
|
488
|
+
The payload contains `file_path`, `query`, `regexp`, `case_sensitive`, `start_position`, `end_position`,
|
|
489
|
+
`max_matches`, `matches` and `timed_out`.
|
|
329
490
|
|
|
330
491
|
## Development & Contributing
|
|
331
492
|
|
|
332
|
-
You are more than welcome to help and contribute to this package.
|
|
493
|
+
You are more than welcome to help and contribute to this package.
|
|
333
494
|
|
|
334
495
|
The app uses minitest and includes a dummy app, so getting started should be straightforward.
|
|
335
496
|
|
|
@@ -348,11 +509,13 @@ cd test/dummy
|
|
|
348
509
|
bundle exec rails s
|
|
349
510
|
```
|
|
350
511
|
|
|
351
|
-
The dummy app will be available at `http://localhost:3000` and onlylogs at
|
|
512
|
+
The dummy app will be available at `http://onlylogs.localhost:3000` and onlylogs at
|
|
513
|
+
`http://onlylogs.localhost:3000/onlylogs`.
|
|
352
514
|
|
|
353
515
|
### Generating Test Logs
|
|
354
516
|
|
|
355
|
-
To test onlylogs with live log data, use the continuous log writer script. This is especially useful for testing
|
|
517
|
+
To test onlylogs with live log data, use the continuous log writer script. This is especially useful for testing
|
|
518
|
+
real-time log streaming and UI behavior.
|
|
356
519
|
|
|
357
520
|
```bash
|
|
358
521
|
# Generate 1 log entry every 2 seconds (default)
|
|
@@ -365,9 +528,12 @@ bin/continuous_log_writer 5 1
|
|
|
365
528
|
bin/continuous_log_writer 10 3
|
|
366
529
|
```
|
|
367
530
|
|
|
368
|
-
The script will write logs to `test/dummy/log/development.log`, which will appear in real-time in the onlylogs UI at
|
|
531
|
+
The script will write logs to `test/dummy/log/development.log`, which will appear in real-time in the onlylogs UI at
|
|
532
|
+
`http://onlylogs.localhost:3000/onlylogs`.
|
|
369
533
|
|
|
370
|
-
The lines are generated by `Onlylogs::ContinuousLogWriter`, in the same format a Rails app logging through
|
|
534
|
+
The lines are generated by `Onlylogs::ContinuousLogWriter`, in the same format a Rails app logging through
|
|
535
|
+
`Onlylogs::Logger` produces. The script is a thin wrapper, so applications embedding onlylogs can reuse the writer
|
|
536
|
+
against any file of their own:
|
|
371
537
|
|
|
372
538
|
```ruby
|
|
373
539
|
require "onlylogs/continuous_log_writer"
|
|
@@ -375,7 +541,8 @@ require "onlylogs/continuous_log_writer"
|
|
|
375
541
|
Onlylogs::ContinuousLogWriter.new(some_log_path, logs_per_batch: 3, interval: 1).call
|
|
376
542
|
```
|
|
377
543
|
|
|
378
|
-
It is plain Ruby (it does not need Rails to be booted) and is not loaded by `require "onlylogs"` — require it explicitly
|
|
544
|
+
It is plain Ruby (it does not need Rails to be booted) and is not loaded by `require "onlylogs"` — require it explicitly
|
|
545
|
+
where you need it.
|
|
379
546
|
|
|
380
547
|
**Example workflow:**
|
|
381
548
|
|
|
@@ -387,15 +554,18 @@ cd test/dummy && bundle exec rails s
|
|
|
387
554
|
bin/continuous_log_writer 3 1
|
|
388
555
|
```
|
|
389
556
|
|
|
390
|
-
Open `http://localhost:3000/onlylogs` in your browser and watch logs appear as the script writes them.
|
|
557
|
+
Open `http://onlylogs.localhost:3000/onlylogs` in your browser and watch logs appear as the script writes them.
|
|
391
558
|
|
|
392
559
|
### Latency Simulation
|
|
393
560
|
|
|
394
|
-
For testing how onlylogs behaves under production-like network conditions, you can simulate latency for HTTP requests
|
|
561
|
+
For testing how onlylogs behaves under production-like network conditions, you can simulate latency for HTTP requests
|
|
562
|
+
and WebSocket connections using the included latency simulation tool.
|
|
395
563
|
|
|
396
564
|
**Parameters:**
|
|
565
|
+
|
|
397
566
|
- **Latency** (default: 120ms): The base network delay added to all traffic
|
|
398
|
-
- **Jitter** (default: 30ms): Random variation (±) applied to the latency (changes every 2 seconds to simulate real
|
|
567
|
+
- **Jitter** (default: 30ms): Random variation (±) applied to the latency (changes every 2 seconds to simulate real
|
|
568
|
+
network conditions)
|
|
399
569
|
- **Port** (default: 3000): The port to apply latency simulation to
|
|
400
570
|
|
|
401
571
|
**Common scenarios:**
|
|
@@ -414,7 +584,6 @@ bin/simulate_latency enable 300/100
|
|
|
414
584
|
bin/simulate_latency enable -p 8080
|
|
415
585
|
```
|
|
416
586
|
|
|
417
|
-
|
|
418
587
|
**Testing and monitoring:**
|
|
419
588
|
|
|
420
589
|
```bash
|
|
@@ -428,11 +597,25 @@ bin/simulate_latency status
|
|
|
428
597
|
bin/simulate_latency disable
|
|
429
598
|
```
|
|
430
599
|
|
|
431
|
-
The `test` command will run 10 HTTP requests and 10 WebSocket connections, showing you the actual round-trip times and
|
|
600
|
+
The `test` command will run 10 HTTP requests and 10 WebSocket connections, showing you the actual round-trip times and
|
|
601
|
+
helping you verify the latency is working as expected.
|
|
602
|
+
|
|
603
|
+
### Simulating a drain outage
|
|
604
|
+
|
|
605
|
+
To see how `HttpLogger` behaves when the drain is slow or broken, run a fake drain and a fake app in two terminals:
|
|
606
|
+
|
|
607
|
+
```bash
|
|
608
|
+
bin/fake_drain # press a key to change its state: 2=200 5=500 3=503 4=413 h=hang s=slow r=reset d=down
|
|
609
|
+
bin/fake_app # 5 "Puma threads" logging like Rails requests through HttpLogger
|
|
610
|
+
```
|
|
611
|
+
|
|
612
|
+
The dashboard shows request latency, CPU, memory, the queue, the circuit breaker, the spool and the drain counters.
|
|
613
|
+
A healthy app has a few % of CPU and no HTTP calls on request threads.
|
|
432
614
|
|
|
433
615
|
### Performance Testing
|
|
434
616
|
|
|
435
|
-
Performance tests require large log files that are not included in the repository. You can download them using the
|
|
617
|
+
Performance tests require large log files that are not included in the repository. You can download them using the
|
|
618
|
+
provided script:
|
|
436
619
|
|
|
437
620
|
```bash
|
|
438
621
|
bin/download_performance_fixtures
|
|
@@ -449,17 +632,8 @@ bin/rails test test/models/onlylogs/grep_performance_test.rb
|
|
|
449
632
|
|
|
450
633
|
### Plans for the future
|
|
451
634
|
|
|
452
|
-
We believe that by simply analysing your logs you can also
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
And you know what? You can get also performance reports.
|
|
456
|
-
|
|
457
|
-
All of a sudden you are 100% free from external services for three more things:
|
|
458
|
-
|
|
459
|
-
* logs
|
|
460
|
-
* errors
|
|
461
|
-
* performance
|
|
462
|
-
|
|
635
|
+
We believe that by simply analysing your logs you can also get error reports and performance reports. The goal is to
|
|
636
|
+
need no external service for logs, errors and performance.
|
|
463
637
|
|
|
464
638
|
## License
|
|
465
639
|
|
|
Binary file
|
|
Binary file
|
|
Binary file
|