ez_logs_agent 0.2.0 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +50 -0
- data/README.md +72 -59
- data/lib/ez_logs_agent/buffer.rb +14 -0
- data/lib/ez_logs_agent/bulk_sql_parser.rb +11 -1
- data/lib/ez_logs_agent/capturers/active_job_capturer.rb +28 -3
- data/lib/ez_logs_agent/capturers/bulk_database_capturer.rb +229 -19
- data/lib/ez_logs_agent/capturers/database_capturer.rb +36 -10
- data/lib/ez_logs_agent/configuration_validator.rb +2 -2
- data/lib/ez_logs_agent/flush_scheduler.rb +21 -3
- data/lib/ez_logs_agent/logger.rb +8 -2
- data/lib/ez_logs_agent/railtie.rb +39 -0
- data/lib/ez_logs_agent/sensitive_patterns.rb +25 -7
- data/lib/ez_logs_agent/transport.rb +9 -7
- data/lib/ez_logs_agent/version.rb +1 -1
- data/lib/generators/ez_logs_agent/install/templates/ez_logs_agent.rb.tt +2 -0
- metadata +4 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 60873dfcf28408365019a70880b7a28aa868142df967df60c5b4c2c32c3f2608
|
|
4
|
+
data.tar.gz: 36daaebd2e181f7ca987b630a3c64102212ef3466598c1b41dae84abe183d89c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2ae1777f5b2bdab3eaa688d6a543a5c796d32582f4fc269074eee4914403d06c36598497126707e64ba018340ca6f56d798310a358c187121b65414ccdb9a974
|
|
7
|
+
data.tar.gz: b39b3add6cb40cd7f061892302fe379879c25d3bc2d561c2a453ef59ebf96ab7e70e86e08a5ce25534ef3615c8c24b7119d0e8aa1068b8d4860cb2d4730592e7
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,56 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.2.2] — 2026-11-17
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
- **Startup line.** Each process logs one line, whatever the agent's `log_level` (visible unless Rails' own log level is above `info`), so you can check the install: `[EzLogsAgent] capturing → app.ezlogs.io (key ezl_a1b2c3d4…)`. The key is shown as Settings → API Keys shows it (`ezl_` + first 8 characters); other token shapes get no hint. With config missing it logs `[EzLogsAgent] not capturing: project_token missing`. Silent in the test env. In development it is in `log/development.log`.
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
- **Events lost at process exit.** `FlushScheduler.stop` (run from `at_exit`) now sends what is still buffered: one attempt, no retries, 1 s HTTP timeouts and a 3 s hard cap. Before, the last up-to-3 s of events (the `send_interval` default) were dropped whenever a process exited: always for `rails runner` / rake tasks, and on every Puma/Sidekiq restart. It also runs when the flush thread had crashed. Shutdown may take up to ~3–5 s longer when events are pending and the server is slow or unreachable.
|
|
12
|
+
- **Forked children no longer resend the parent's events.** The buffer inherited across `fork` (Puma cluster workers, etc.) is cleared in the child; the parent still sends it.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
- **`project_token` is required.** A missing token is now a configuration error and the agent stays off. Before, it was a warning: the agent ran and every batch was rejected with 401. Set the token in `config/initializers/ez_logs_agent.rb`: it is checked once at boot, so a token set later in `after_initialize` or `to_prepare` is not seen.
|
|
16
|
+
|
|
17
|
+
## [0.2.1] — 2026-06-05
|
|
18
|
+
|
|
19
|
+
### Fixed
|
|
20
|
+
- **Bulk-op row counts on Postgres.** The PG adapter does not populate
|
|
21
|
+
`payload[:row_count]` for plain `DELETE`/`UPDATE` notifications, so
|
|
22
|
+
the first 0.2.0 builds shipped `row_count: 0` for the cases that
|
|
23
|
+
matter most. `BulkDatabaseCapturer` now prepends a tiny shim onto
|
|
24
|
+
`ActiveRecord::Relation#delete_all` / `#update_all` that stashes the
|
|
25
|
+
returned row count and back-fills it onto the just-pushed event via
|
|
26
|
+
`Buffer.peek_last`. No change to the wire shape — `row_count` now
|
|
27
|
+
carries the real number.
|
|
28
|
+
- **Model resolution in Rails dev mode.** `ActiveRecord::Base.descendants`
|
|
29
|
+
only sees eager-loaded models. The resolver now falls through to
|
|
30
|
+
`safe_constantize` of the classified table name and verifies the
|
|
31
|
+
reconstructed class actually owns that table, so bulk ops on
|
|
32
|
+
lazy-autoloaded models in dev / test no longer silently drop.
|
|
33
|
+
- **Rails Query Log Tags noise in `where_template`.** The parser now
|
|
34
|
+
strips `/*application='X',action='Y'*/` comments before extracting
|
|
35
|
+
the WHERE clause, so the humanized filter line on the server reads
|
|
36
|
+
cleanly instead of leaking the framework's instrumentation tags.
|
|
37
|
+
|
|
38
|
+
### Changed
|
|
39
|
+
- **Framework-rewrite filter narrowed.** The 0.2.0 filter that swallowed
|
|
40
|
+
Rails-generated `SET col = NULL` writes was too aggressive — it also
|
|
41
|
+
hid deliberate "null this column out" operations the customer wrote.
|
|
42
|
+
The filter now only drops `COALESCE`-shaped counter bumps and
|
|
43
|
+
empty-`SET` shells; honest `SET col = NULL` writes are captured and
|
|
44
|
+
render as "Cleared col" on the timeline.
|
|
45
|
+
- **Hot-path performance.** `capture_jobs`, `capture_database`, the
|
|
46
|
+
excluded-tables / excluded-job-classes / display-name maps, and the
|
|
47
|
+
user-extended sensitive-key patterns are now memoized at install
|
|
48
|
+
time across `ActiveJobCapturer`, `DatabaseCapturer`,
|
|
49
|
+
`BulkDatabaseCapturer`, and `SensitivePatterns`. The
|
|
50
|
+
`sql.active_record` subscriber is also tuned: 5-arity block to avoid
|
|
51
|
+
splat allocation, cheap `end_with?` name-prefilter ahead of any
|
|
52
|
+
parsing, and an early bail before `safe_constantize`. Bulk capture
|
|
53
|
+
overhead measured below the noise floor on a 10 ms reference query.
|
|
54
|
+
|
|
5
55
|
## [0.2.0] — 2026-06-05
|
|
6
56
|
|
|
7
57
|
### Added
|
data/README.md
CHANGED
|
@@ -134,12 +134,13 @@ Test your configuration:
|
|
|
134
134
|
rails ez_logs_agent:test_connection
|
|
135
135
|
```
|
|
136
136
|
|
|
137
|
-
**Successful output
|
|
137
|
+
**Successful output** (excerpt):
|
|
138
138
|
```
|
|
139
139
|
✅ Configuration is valid
|
|
140
140
|
✅ Connection successful (HTTP 200)
|
|
141
|
-
|
|
142
|
-
|
|
141
|
+
Test event accepted by server
|
|
142
|
+
|
|
143
|
+
✅ All checks passed! EzLogs Agent is configured correctly.
|
|
143
144
|
```
|
|
144
145
|
|
|
145
146
|
If the test fails, you'll see exactly what's wrong and how to fix it.
|
|
@@ -154,15 +155,20 @@ rails server
|
|
|
154
155
|
# Restart your application using your deployment process
|
|
155
156
|
```
|
|
156
157
|
|
|
157
|
-
Check your Rails logs for the startup
|
|
158
|
+
Check your Rails logs for the startup line. It is printed once per process (web, Sidekiq, rake), whatever the agent's `log_level`, and is visible unless Rails' own log level is above `info`. In development it lands in `log/development.log` (`rails s` copies the log to the terminal only after boot). Silent in the test env.
|
|
159
|
+
|
|
160
|
+
```
|
|
161
|
+
[EzLogsAgent] capturing → app.ezlogs.io (key ezl_a1b2c3d4…)
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
The key is shown the way Settings → API Keys shows it (`ezl_` + first 8 characters), so you can tell which key the app uses. If something is missing, the line says what:
|
|
158
165
|
|
|
159
166
|
```
|
|
160
|
-
[EzLogsAgent]
|
|
161
|
-
[EzLogsAgent] ✓ HTTP capture enabled
|
|
162
|
-
[EzLogsAgent] ✓ Sidekiq capture enabled
|
|
163
|
-
[EzLogsAgent] ✓ Database capture enabled
|
|
167
|
+
[EzLogsAgent] not capturing: project_token missing
|
|
164
168
|
```
|
|
165
169
|
|
|
170
|
+
Set `project_token` in `config/initializers/ez_logs_agent.rb`. The agent checks it once at boot, so a token set later (in `after_initialize` or `to_prepare`) is not seen.
|
|
171
|
+
|
|
166
172
|
### 5. See Your Activity
|
|
167
173
|
|
|
168
174
|
Visit your EZLogs dashboard and interact with your application. Within seconds, you'll see activity appearing in real-time.
|
|
@@ -604,34 +610,37 @@ rails ez_logs_agent:test_connection
|
|
|
604
610
|
### Successful Output
|
|
605
611
|
|
|
606
612
|
```
|
|
607
|
-
|
|
613
|
+
🔍 EzLogs Agent Connection Test
|
|
614
|
+
|
|
615
|
+
Step 1: Validating configuration...
|
|
608
616
|
✅ Configuration is valid
|
|
617
|
+
|
|
618
|
+
Configuration Summary:
|
|
619
|
+
Server URL: https://app.ezlogs.io
|
|
620
|
+
Project Token: ***1a2b
|
|
621
|
+
Capture HTTP: true
|
|
622
|
+
Capture Jobs: true
|
|
623
|
+
Capture Database: true
|
|
624
|
+
|
|
625
|
+
Step 2: Testing connection to server...
|
|
609
626
|
✅ Connection successful (HTTP 200)
|
|
610
|
-
|
|
611
|
-
✅ All checks passed! EZLogs Agent is configured correctly.
|
|
627
|
+
Test event accepted by server
|
|
612
628
|
|
|
613
|
-
|
|
614
|
-
1. Restart your Rails application
|
|
615
|
-
2. Visit your EZLogs dashboard
|
|
616
|
-
3. Interact with your application to generate events
|
|
629
|
+
✅ All checks passed! EzLogs Agent is configured correctly.
|
|
617
630
|
```
|
|
618
631
|
|
|
619
632
|
### Failed Output
|
|
620
633
|
|
|
621
|
-
If the test fails, you'll see exactly what's wrong:
|
|
634
|
+
If the test fails, you'll see exactly what's wrong (excerpt, after the same Step 1 and summary):
|
|
622
635
|
|
|
623
636
|
```
|
|
624
|
-
|
|
625
|
-
❌
|
|
626
|
-
|
|
627
|
-
Possible causes:
|
|
628
|
-
- Invalid API key (project_token)
|
|
629
|
-
- API key has been revoked
|
|
630
|
-
- Check your project_token in config/initializers/ez_logs_agent.rb
|
|
631
|
-
|
|
632
|
-
Please fix the error and run this command again.
|
|
637
|
+
Step 2: Testing connection to server...
|
|
638
|
+
❌ Authentication failed (HTTP 401)
|
|
639
|
+
Check your project_token in config/initializers/ez_logs_agent.rb
|
|
633
640
|
```
|
|
634
641
|
|
|
642
|
+
Other failures print one of: `❌ Server endpoint not found (HTTP 404)`, `❌ Could not resolve host: …`, `❌ Connection refused: …`, `❌ Connection timeout: …`, or `❌ Configuration validation failed:` followed by the errors.
|
|
643
|
+
|
|
635
644
|
---
|
|
636
645
|
|
|
637
646
|
## Troubleshooting
|
|
@@ -657,19 +666,22 @@ rails ez_logs_agent:test_connection
|
|
|
657
666
|
rails ez_logs_agent:test_connection
|
|
658
667
|
```
|
|
659
668
|
|
|
660
|
-
2. **Check Rails logs** for agent
|
|
669
|
+
2. **Check Rails logs** for the startup line and for agent errors, e.g. a rejected key:
|
|
661
670
|
```
|
|
662
|
-
[EzLogsAgent]
|
|
663
|
-
[EzLogsAgent]
|
|
664
|
-
[EzLogsAgent] Sending batch of 3 events...
|
|
665
|
-
[EzLogsAgent] Batch sent successfully (HTTP 200)
|
|
671
|
+
[EzLogsAgent] capturing → app.ezlogs.io (key ezl_a1b2c3d4…)
|
|
672
|
+
[EzLogsAgent] [Transport] send failed (401)
|
|
666
673
|
```
|
|
667
674
|
|
|
668
675
|
3. **Enable debug logging** in `config/initializers/ez_logs_agent.rb`:
|
|
669
676
|
```ruby
|
|
670
677
|
config.log_level = :debug
|
|
671
678
|
```
|
|
672
|
-
Then restart Rails
|
|
679
|
+
Then restart Rails. A working agent logs, among others:
|
|
680
|
+
```
|
|
681
|
+
[EzLogsAgent] [Railtie] HTTP capture enabled
|
|
682
|
+
[EzLogsAgent] [Railtie] Agent initialized successfully
|
|
683
|
+
[EzLogsAgent] [Transport] send succeeded (200)
|
|
684
|
+
```
|
|
673
685
|
|
|
674
686
|
4. **Verify network connectivity:**
|
|
675
687
|
```bash
|
|
@@ -686,23 +698,23 @@ rails ez_logs_agent:test_connection
|
|
|
686
698
|
|
|
687
699
|
**Example output:**
|
|
688
700
|
```
|
|
689
|
-
[Railtie] Configuration validation failed:
|
|
690
|
-
|
|
691
|
-
[Railtie] Agent initialization skipped. Please fix configuration errors.
|
|
701
|
+
[EzLogsAgent] [Railtie] Configuration validation failed:
|
|
702
|
+
[EzLogsAgent] [Railtie] - server_url is required. Set it in config/initializers/ez_logs_agent.rb
|
|
703
|
+
[EzLogsAgent] [Railtie] Agent initialization skipped. Please fix configuration errors.
|
|
692
704
|
```
|
|
693
705
|
|
|
694
706
|
**Solution:** Fix the errors listed in the warning message and restart Rails.
|
|
695
707
|
|
|
696
708
|
Common validation errors:
|
|
697
|
-
- `server_url is required` → Set `config.server_url`
|
|
698
|
-
- `
|
|
699
|
-
- `
|
|
709
|
+
- `server_url is required. Set it in config/initializers/ez_logs_agent.rb` → Set `config.server_url`
|
|
710
|
+
- `server_url must be a valid URL (got: …)` → Fix the URL format
|
|
711
|
+
- `project_token is required. Set it in config/initializers/ez_logs_agent.rb` → Set `config.project_token`
|
|
700
712
|
|
|
701
713
|
---
|
|
702
714
|
|
|
703
715
|
### Authentication Errors (HTTP 401)
|
|
704
716
|
|
|
705
|
-
**Symptoms:** Connection test or agent logs show `HTTP 401
|
|
717
|
+
**Symptoms:** Connection test or agent logs show `HTTP 401`.
|
|
706
718
|
|
|
707
719
|
**Debug steps:**
|
|
708
720
|
|
|
@@ -747,7 +759,7 @@ Common validation errors:
|
|
|
747
759
|
|
|
748
760
|
3. **Look for Sidekiq registration in logs:**
|
|
749
761
|
```
|
|
750
|
-
[Railtie] Sidekiq server middleware registered
|
|
762
|
+
[EzLogsAgent] [Railtie] Sidekiq server middleware registered
|
|
751
763
|
```
|
|
752
764
|
**Note:** This message only appears in Sidekiq worker processes, not web processes.
|
|
753
765
|
|
|
@@ -789,7 +801,7 @@ Common validation errors:
|
|
|
789
801
|
|
|
790
802
|
3. **Look for database capture registration in logs:**
|
|
791
803
|
```
|
|
792
|
-
[Railtie] Database capture installed (per-row + bulk)
|
|
804
|
+
[EzLogsAgent] [Railtie] Database capture installed (per-row + bulk)
|
|
793
805
|
```
|
|
794
806
|
|
|
795
807
|
4. **Verify models aren't excluded:**
|
|
@@ -839,8 +851,7 @@ config.send_interval = 1
|
|
|
839
851
|
|
|
840
852
|
**If correlation is missing when it should be present:**
|
|
841
853
|
1. Enable debug logging: `config.log_level = :debug`
|
|
842
|
-
2.
|
|
843
|
-
3. Check that jobs are enqueued within the same request context
|
|
854
|
+
2. Check that jobs are enqueued within the same request context
|
|
844
855
|
|
|
845
856
|
---
|
|
846
857
|
|
|
@@ -877,30 +888,32 @@ config.send_interval = 1
|
|
|
877
888
|
|
|
878
889
|
### Understanding Agent Log Messages
|
|
879
890
|
|
|
880
|
-
**Normal operation
|
|
891
|
+
**Normal operation** (the startup line always shows; the other two only at `config.log_level = :debug`):
|
|
881
892
|
```
|
|
882
|
-
[EzLogsAgent]
|
|
883
|
-
[EzLogsAgent]
|
|
884
|
-
[EzLogsAgent]
|
|
893
|
+
[EzLogsAgent] capturing → app.ezlogs.io (key ezl_a1b2c3d4…)
|
|
894
|
+
[EzLogsAgent] [Railtie] Agent initialized successfully
|
|
895
|
+
[EzLogsAgent] [Transport] send succeeded (200)
|
|
885
896
|
```
|
|
886
897
|
|
|
887
|
-
**Warnings
|
|
898
|
+
**Warnings:**
|
|
888
899
|
```
|
|
889
|
-
[EzLogsAgent] Buffer full,
|
|
890
|
-
→ Your app
|
|
891
|
-
→ Increase buffer_size or send_interval
|
|
900
|
+
[EzLogsAgent] [Buffer] Buffer full (10000), dropped oldest event
|
|
901
|
+
→ Your app produces events faster than they are sent
|
|
902
|
+
→ Increase buffer_size or lower send_interval
|
|
892
903
|
```
|
|
893
904
|
|
|
894
905
|
**Errors (need attention):**
|
|
895
906
|
```
|
|
896
|
-
[EzLogsAgent]
|
|
897
|
-
→ Invalid API key, check config.project_token
|
|
907
|
+
[EzLogsAgent] [Transport] send failed (401)
|
|
908
|
+
→ Invalid or revoked API key, check config.project_token
|
|
909
|
+
|
|
910
|
+
[EzLogsAgent] [Transport] send failed: Net::OpenTimeout - execution expired
|
|
911
|
+
→ Network issue or the server is down; the batch is retried (retry_attempts)
|
|
898
912
|
|
|
899
|
-
[EzLogsAgent] Failed
|
|
900
|
-
→
|
|
901
|
-
→ Events will be retried automatically
|
|
913
|
+
[EzLogsAgent] [RetrySender] Failed after 4 attempts, dropping 12 events
|
|
914
|
+
→ All retries failed; those events are lost
|
|
902
915
|
|
|
903
|
-
[EzLogsAgent] Configuration validation failed
|
|
916
|
+
[EzLogsAgent] [Railtie] Configuration validation failed:
|
|
904
917
|
→ Fix configuration errors and restart Rails
|
|
905
918
|
```
|
|
906
919
|
|
|
@@ -910,7 +923,7 @@ config.send_interval = 1
|
|
|
910
923
|
|
|
911
924
|
If you're still stuck after trying the above:
|
|
912
925
|
|
|
913
|
-
1. **Check GitHub Issues:** [github.com/dezsirazvan/
|
|
926
|
+
1. **Check GitHub Issues:** [github.com/dezsirazvan/ez_logs_agent/issues](https://github.com/dezsirazvan/ez_logs_agent/issues)
|
|
914
927
|
2. **Open a new issue** with:
|
|
915
928
|
- Rails version and Ruby version
|
|
916
929
|
- Relevant logs (set `config.log_level = :debug`)
|
|
@@ -1028,13 +1041,13 @@ We welcome bug reports and pull requests!
|
|
|
1028
1041
|
|
|
1029
1042
|
- **786 tests**, all green
|
|
1030
1043
|
- **Wire-format parity-tested** against the Next.js agent's fixtures (every event shape byte-for-byte identical)
|
|
1031
|
-
- **Production-shipping** in multiple Rails apps including
|
|
1044
|
+
- **Production-shipping** in multiple Rails apps including our Bookhouse demo app (Rails 8 + Sidekiq + Devise, exercises every capture path)
|
|
1032
1045
|
|
|
1033
1046
|
---
|
|
1034
1047
|
|
|
1035
1048
|
## Support
|
|
1036
1049
|
|
|
1037
|
-
- **GitHub Issues:** [Report a bug](https://github.com/dezsirazvan/
|
|
1050
|
+
- **GitHub Issues:** [Report a bug](https://github.com/dezsirazvan/ez_logs_agent/issues)
|
|
1038
1051
|
- **Email:** support@ezlogs.io
|
|
1039
1052
|
|
|
1040
1053
|
---
|
data/lib/ez_logs_agent/buffer.rb
CHANGED
|
@@ -59,6 +59,20 @@ module EzLogsAgent
|
|
|
59
59
|
# Best effort, ignore failures
|
|
60
60
|
end
|
|
61
61
|
|
|
62
|
+
# Returns the most recently pushed event WITHOUT removing it.
|
|
63
|
+
# Used by BulkDatabaseCapturer to backfill the affected-row count
|
|
64
|
+
# after the relation method's return value is known (Rails'
|
|
65
|
+
# payload[:row_count] is unreliable for plain DELETE on PG).
|
|
66
|
+
# Mutating the returned hash in place is intentional and safe —
|
|
67
|
+
# the event hasn't been flushed yet.
|
|
68
|
+
# @return [Hash, nil] The last event, or nil if empty.
|
|
69
|
+
def peek_last
|
|
70
|
+
@monitor.synchronize { @queue.last }
|
|
71
|
+
rescue => error
|
|
72
|
+
log_error("[Buffer] peek_last failed: #{error.message}")
|
|
73
|
+
nil
|
|
74
|
+
end
|
|
75
|
+
|
|
62
76
|
private
|
|
63
77
|
|
|
64
78
|
def max_size
|
|
@@ -63,7 +63,7 @@ module EzLogsAgent
|
|
|
63
63
|
def parse(sql:, type_casted_binds:)
|
|
64
64
|
return { unparseable: true } if sql.nil? || sql.empty?
|
|
65
65
|
|
|
66
|
-
sql_stripped = sql.strip
|
|
66
|
+
sql_stripped = strip_query_log_tags(sql.strip)
|
|
67
67
|
|
|
68
68
|
case sql_stripped
|
|
69
69
|
when /\ADELETE FROM /i
|
|
@@ -84,6 +84,16 @@ module EzLogsAgent
|
|
|
84
84
|
{ unparseable: true }
|
|
85
85
|
end
|
|
86
86
|
|
|
87
|
+
# Strip Rails 7+ Query Log Tags (`/*application='X',action='Y'*/`)
|
|
88
|
+
# AND any other trailing SQL comments. They land at the end of every
|
|
89
|
+
# statement when `config.active_record.query_log_tags_enabled = true`
|
|
90
|
+
# and are pure noise on the timeline — they leak the host app's name
|
|
91
|
+
# and controller into the user-visible filter line. Removing them at
|
|
92
|
+
# parse time means we never ship them on the wire.
|
|
93
|
+
def strip_query_log_tags(sql)
|
|
94
|
+
sql.gsub(%r{/\*.*?\*/}m, "").rstrip
|
|
95
|
+
end
|
|
96
|
+
|
|
87
97
|
# Returns the symbolic operation name we expect downstream. Detected
|
|
88
98
|
# from SQL shape, independent of the `payload[:name]` Rails version
|
|
89
99
|
# variance (see plan §"insert_all/upsert_all payload :name varies").
|
|
@@ -51,6 +51,25 @@ module EzLogsAgent
|
|
|
51
51
|
def install
|
|
52
52
|
return unless defined?(ActiveJob)
|
|
53
53
|
|
|
54
|
+
# Memoize config values that the per-job hot path reads on
|
|
55
|
+
# every execute. Without this, capture_execution dispatches
|
|
56
|
+
# into EzLogsAgent.configuration twice per job (once for
|
|
57
|
+
# capture_jobs, once for all_excluded_job_classes). On
|
|
58
|
+
# job-heavy apps that's measurable. Runtime mutations need
|
|
59
|
+
# uninstall! + install.
|
|
60
|
+
@capture_enabled =
|
|
61
|
+
begin
|
|
62
|
+
EzLogsAgent.configuration.capture_jobs
|
|
63
|
+
rescue StandardError
|
|
64
|
+
false
|
|
65
|
+
end
|
|
66
|
+
@excluded_job_classes =
|
|
67
|
+
begin
|
|
68
|
+
EzLogsAgent.configuration.all_excluded_job_classes.dup.freeze
|
|
69
|
+
rescue StandardError
|
|
70
|
+
[].freeze
|
|
71
|
+
end
|
|
72
|
+
|
|
54
73
|
install_serialization_hooks unless @serialization_installed
|
|
55
74
|
|
|
56
75
|
ActiveJob::Base.before_enqueue do |job|
|
|
@@ -101,7 +120,11 @@ module EzLogsAgent
|
|
|
101
120
|
# @param block [Proc] The job execution block
|
|
102
121
|
# @return [Object] The result of the job execution
|
|
103
122
|
def capture_execution(job, block)
|
|
104
|
-
|
|
123
|
+
# Memoized at install time for hot-path perf. If we haven't
|
|
124
|
+
# installed yet (specs that test capture_execution directly),
|
|
125
|
+
# fall back to a live config read so the behavior matches.
|
|
126
|
+
enabled = defined?(@capture_enabled) ? @capture_enabled : EzLogsAgent.configuration.capture_jobs
|
|
127
|
+
return block.call unless enabled
|
|
105
128
|
|
|
106
129
|
if sidekiq_adapter?(job)
|
|
107
130
|
EzLogsAgent::Logger.debug("[ActiveJobCapturer] Skipping capture (Sidekiq adapter)")
|
|
@@ -196,8 +219,10 @@ module EzLogsAgent
|
|
|
196
219
|
# @param job [ActiveJob::Base] The job instance
|
|
197
220
|
# @return [Boolean] true if excluded, false otherwise
|
|
198
221
|
def excluded_job_class?(job)
|
|
199
|
-
|
|
200
|
-
|
|
222
|
+
# Memoized at install time for perf. Fall back to a live read
|
|
223
|
+
# for specs that don't call install (see capture_execution).
|
|
224
|
+
excluded = defined?(@excluded_job_classes) ? @excluded_job_classes : EzLogsAgent.configuration.all_excluded_job_classes
|
|
225
|
+
excluded.include?(job.class.name)
|
|
201
226
|
rescue StandardError
|
|
202
227
|
false
|
|
203
228
|
end
|
|
@@ -108,16 +108,107 @@ module EzLogsAgent
|
|
|
108
108
|
def install
|
|
109
109
|
return if @installed
|
|
110
110
|
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
111
|
+
# Cache configuration values that the hot path checks per
|
|
112
|
+
# notification. Re-read on every install (specs reinstall
|
|
113
|
+
# after toggling config). Runtime mutations to these settings
|
|
114
|
+
# require uninstall! + install to take effect — acceptable
|
|
115
|
+
# because nobody flips this at runtime in production.
|
|
116
|
+
@capture_enabled =
|
|
117
|
+
begin
|
|
118
|
+
::EzLogsAgent.configuration.capture_database
|
|
119
|
+
rescue StandardError
|
|
120
|
+
false
|
|
121
|
+
end
|
|
122
|
+
@excluded_tables =
|
|
123
|
+
begin
|
|
124
|
+
::EzLogsAgent.configuration.all_excluded_tables.dup.freeze
|
|
125
|
+
rescue StandardError
|
|
126
|
+
[].freeze
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
install_row_count_capture!
|
|
130
|
+
|
|
131
|
+
# 5-arity block bypasses the `*args` splat allocation per
|
|
132
|
+
# notification — measurable on hot paths where we ignore
|
|
133
|
+
# ~99% of events. Block accepts positional args matching the
|
|
134
|
+
# AS::N convention: (name, start, finish, id, payload).
|
|
135
|
+
@subscriber = ::ActiveSupport::Notifications.subscribe("sql.active_record") do |name, started, finished, _id, payload|
|
|
136
|
+
handle_notification(name, started, finished, payload)
|
|
117
137
|
end
|
|
118
138
|
@installed = true
|
|
119
139
|
end
|
|
120
140
|
|
|
141
|
+
# Patches ActiveRecord::Relation's bulk methods to backfill the
|
|
142
|
+
# affected-row count on the most recently captured bulk_database
|
|
143
|
+
# event. See the comment on RelationRowCountStash below for why
|
|
144
|
+
# this is necessary (Rails' payload[:row_count] is unreliable
|
|
145
|
+
# for plain DELETE/UPDATE on PG).
|
|
146
|
+
def install_row_count_capture!
|
|
147
|
+
return if @relation_patched
|
|
148
|
+
return unless defined?(::ActiveRecord::Relation)
|
|
149
|
+
|
|
150
|
+
::ActiveRecord::Relation.prepend(RelationRowCountStash)
|
|
151
|
+
@relation_patched = true
|
|
152
|
+
rescue StandardError => e
|
|
153
|
+
# Patching is best-effort — if AR's Relation isn't there or the
|
|
154
|
+
# prepend raises, the capturer still works with payload[:row_count].
|
|
155
|
+
::EzLogsAgent::Logger.debug("[BulkDatabaseCapturer] could not patch Relation: #{e.class}: #{e.message}")
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
# Patches the row_count on the most recently buffered bulk_database
|
|
159
|
+
# event when its model matches `model_class`. Called from the
|
|
160
|
+
# RelationRowCountStash module immediately after `super` returns
|
|
161
|
+
# from delete_all / update_all. The buffer's `peek_last` API is
|
|
162
|
+
# the lightweight read path; we mutate in place because the
|
|
163
|
+
# event hash hasn't been serialized yet.
|
|
164
|
+
def backfill_row_count(real_count, model_class)
|
|
165
|
+
return if real_count.nil?
|
|
166
|
+
|
|
167
|
+
tail = ::EzLogsAgent::Buffer.peek_last
|
|
168
|
+
return unless tail.is_a?(Hash)
|
|
169
|
+
return unless tail[:source_type] == "bulk_database"
|
|
170
|
+
return unless tail.dig(:source_data, :model_class) == model_class.name
|
|
171
|
+
|
|
172
|
+
tail[:source_data][:row_count] = real_count
|
|
173
|
+
if (rids = tail[:resource_ids]).is_a?(Array) && rids.first.is_a?(Hash)
|
|
174
|
+
rids.first[:resource_id] = "bulk:#{real_count}"
|
|
175
|
+
end
|
|
176
|
+
rescue StandardError => e
|
|
177
|
+
::EzLogsAgent::Logger.debug("[BulkDatabaseCapturer] backfill_row_count failed: #{e.class}: #{e.message}")
|
|
178
|
+
end
|
|
179
|
+
end
|
|
180
|
+
|
|
181
|
+
# Backfills the affected-row count on the most recently captured
|
|
182
|
+
# bulk_database event. Order of operations:
|
|
183
|
+
#
|
|
184
|
+
# 1. Customer calls Relation#delete_all (or update_all).
|
|
185
|
+
# 2. AR runs the SQL → sql.active_record fires → our AS::N
|
|
186
|
+
# handler captures the event with row_count=payload[:row_count]
|
|
187
|
+
# (often 0 on PG for plain DELETE).
|
|
188
|
+
# 3. delete_all's super returns the affected count to us.
|
|
189
|
+
# 4. We patch the most-recently-buffered bulk_database event's
|
|
190
|
+
# source_data[:row_count] AND its sentinel resource_id so the
|
|
191
|
+
# timeline shows the real number.
|
|
192
|
+
#
|
|
193
|
+
# This is a thin shim — only the count is touched, never the SQL,
|
|
194
|
+
# context, or wire shape. If the buffer's tail isn't a bulk_database
|
|
195
|
+
# event for our model (e.g. AS::N skipped it), we leave it alone.
|
|
196
|
+
module RelationRowCountStash
|
|
197
|
+
def delete_all
|
|
198
|
+
result = super
|
|
199
|
+
::EzLogsAgent::Capturers::BulkDatabaseCapturer.backfill_row_count(result, klass) if result.is_a?(Integer)
|
|
200
|
+
result
|
|
201
|
+
end
|
|
202
|
+
|
|
203
|
+
def update_all(*args, **kwargs, &block)
|
|
204
|
+
result = super
|
|
205
|
+
::EzLogsAgent::Capturers::BulkDatabaseCapturer.backfill_row_count(result, klass) if result.is_a?(Integer)
|
|
206
|
+
result
|
|
207
|
+
end
|
|
208
|
+
end
|
|
209
|
+
|
|
210
|
+
class << self
|
|
211
|
+
|
|
121
212
|
# Removes the subscription. Specs use this between examples to
|
|
122
213
|
# avoid leaked subscribers; production never calls it.
|
|
123
214
|
def uninstall!
|
|
@@ -132,8 +223,22 @@ module EzLogsAgent
|
|
|
132
223
|
# tools listening on the same channel. Hard rule: bulk capture failures
|
|
133
224
|
# never propagate.
|
|
134
225
|
def handle_notification(_event_name, started, finished, payload)
|
|
135
|
-
|
|
136
|
-
|
|
226
|
+
# Hot path: this runs once per SQL statement in the host app —
|
|
227
|
+
# often thousands of times per second on a busy tenant. EVERY
|
|
228
|
+
# branch above the cheapest filter has to be fast. Order is:
|
|
229
|
+
#
|
|
230
|
+
# 1. Cheapest possible name check (string suffix, no regex,
|
|
231
|
+
# no method dispatch on the configuration object). This
|
|
232
|
+
# rejects ~99% of notifications in <1 µs.
|
|
233
|
+
# 2. Then capture_enabled? (config check).
|
|
234
|
+
# 3. Then everything else.
|
|
235
|
+
# Cheapest possible early-exit: a single instance-variable
|
|
236
|
+
# read. This branch fires on EVERY SQL statement in the host
|
|
237
|
+
# app and must not allocate or call into configuration.
|
|
238
|
+
return unless @capture_enabled
|
|
239
|
+
return unless payload.is_a?(Hash)
|
|
240
|
+
name = payload[:name]
|
|
241
|
+
return unless name && name_eligible?(name)
|
|
137
242
|
|
|
138
243
|
operation = ::EzLogsAgent::BulkSqlParser.detect_operation(payload[:sql])
|
|
139
244
|
return unless operation
|
|
@@ -147,6 +252,27 @@ module EzLogsAgent
|
|
|
147
252
|
type_casted_binds: payload[:type_casted_binds]
|
|
148
253
|
)
|
|
149
254
|
|
|
255
|
+
# Drop Rails framework rewrites that aren't real business activity:
|
|
256
|
+
#
|
|
257
|
+
# * Foreign-key nullification — when a parent has `dependent:
|
|
258
|
+
# :restrict_with_error` AND the child's `belongs_to :parent`
|
|
259
|
+
# is `optional: true`, Rails rewrites `child.delete_all`
|
|
260
|
+
# into `UPDATE children SET parent_id = NULL WHERE ...`
|
|
261
|
+
# before the destroy. There is no business meaning here;
|
|
262
|
+
# it's framework cleanup before the parent goes away.
|
|
263
|
+
#
|
|
264
|
+
# * Counter-cache / increment! — `Model.increment_counter(:x)`
|
|
265
|
+
# and `record.increment!(:x)` compile to
|
|
266
|
+
# `UPDATE ... SET x = COALESCE(x, 0) + N WHERE id = ?`,
|
|
267
|
+
# which is Rails plumbing for a numeric counter bump, not
|
|
268
|
+
# a user-visible change.
|
|
269
|
+
#
|
|
270
|
+
# Both produce high-volume noise on the timeline (see EZLogs's
|
|
271
|
+
# own dogfood where Company#increment_actions_count! fires
|
|
272
|
+
# per ingest batch). Filtering at the capturer means they
|
|
273
|
+
# don't ride the wire at all.
|
|
274
|
+
return if framework_rewrite?(operation, parse_result)
|
|
275
|
+
|
|
150
276
|
source_data = build_source_data(
|
|
151
277
|
operation: operation,
|
|
152
278
|
model_class: model_class,
|
|
@@ -183,11 +309,21 @@ module EzLogsAgent
|
|
|
183
309
|
# @return [Boolean]
|
|
184
310
|
def eligible_payload?(payload)
|
|
185
311
|
return false unless payload.is_a?(Hash)
|
|
312
|
+
name = payload[:name]
|
|
313
|
+
name.is_a?(String) && name_eligible?(name)
|
|
314
|
+
end
|
|
186
315
|
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
316
|
+
# Cheapest possible bulk-name check — string `end_with?` calls,
|
|
317
|
+
# no regex compilation, no method dispatch. Runs on the hot path.
|
|
318
|
+
# The four shapes we care about (per AR convention):
|
|
319
|
+
# "<Model> Delete All", "<Model> Update All",
|
|
320
|
+
# "<Model> Insert", "<Model> Upsert"
|
|
321
|
+
# (Older Rails 7 also used " Bulk Insert" / " Bulk Upsert" — those
|
|
322
|
+
# still end with "Insert"/"Upsert" so end_with? catches them.)
|
|
323
|
+
# Per-row CRUD names like "<Model> Create", "<Model> Update",
|
|
324
|
+
# "<Model> Destroy" fail all four suffix checks and bail in <1 µs.
|
|
325
|
+
def name_eligible?(name)
|
|
326
|
+
name.end_with?(" All") || name.end_with?(" Insert") || name.end_with?(" Upsert")
|
|
191
327
|
end
|
|
192
328
|
|
|
193
329
|
# Looks up the model class from the SQL's table name. Returns nil
|
|
@@ -200,9 +336,26 @@ module EzLogsAgent
|
|
|
200
336
|
table = extract_table_name(sql)
|
|
201
337
|
return nil if table.nil?
|
|
202
338
|
|
|
203
|
-
|
|
339
|
+
# Try the descendants list first — cheap and works in environments
|
|
340
|
+
# where models are already eager-loaded (production, Sidekiq workers).
|
|
341
|
+
loaded = ::ActiveRecord::Base.descendants.find do |klass|
|
|
204
342
|
klass.respond_to?(:table_name) && klass.table_name == table && !klass.abstract_class?
|
|
205
343
|
end
|
|
344
|
+
return loaded if loaded
|
|
345
|
+
|
|
346
|
+
# Fallback for development mode and any lazy-autoload path: derive
|
|
347
|
+
# the model class name from the table name via Rails' inflector
|
|
348
|
+
# and try to safe_constantize it. ActiveRecord's reflection class
|
|
349
|
+
# caches it on first call, so subsequent bulk ops on the same
|
|
350
|
+
# table hit the descendants path above.
|
|
351
|
+
constant_name = table.to_s.classify
|
|
352
|
+
klass = constant_name.safe_constantize
|
|
353
|
+
return klass if klass.is_a?(Class) &&
|
|
354
|
+
klass < ::ActiveRecord::Base &&
|
|
355
|
+
klass.respond_to?(:table_name) &&
|
|
356
|
+
klass.table_name == table
|
|
357
|
+
|
|
358
|
+
nil
|
|
206
359
|
rescue StandardError
|
|
207
360
|
nil
|
|
208
361
|
end
|
|
@@ -310,11 +463,12 @@ module EzLogsAgent
|
|
|
310
463
|
end
|
|
311
464
|
|
|
312
465
|
# Builds the sentinel resource entry. row_count may be nil (Rails
|
|
313
|
-
# < 7 didn't ship it; some adapters still don't)
|
|
314
|
-
#
|
|
315
|
-
#
|
|
466
|
+
# < 7 didn't ship it; some adapters still don't), and 0 is also
|
|
467
|
+
# not informative (PG's update_all returns 0 in some paths) —
|
|
468
|
+
# fall back to "many" so the display reads naturally and the
|
|
469
|
+
# server-side ResourceAggregationStage doesn't drop it.
|
|
316
470
|
def build_resource_ids(model_class, row_count)
|
|
317
|
-
count_str = row_count.is_a?(Integer) ? row_count.to_s : "
|
|
471
|
+
count_str = (row_count.is_a?(Integer) && row_count > 0) ? row_count.to_s : "many"
|
|
318
472
|
[{ resource_type: model_class.name, resource_id: "bulk:#{count_str}" }]
|
|
319
473
|
end
|
|
320
474
|
|
|
@@ -327,15 +481,71 @@ module EzLogsAgent
|
|
|
327
481
|
end
|
|
328
482
|
|
|
329
483
|
# Uses DatabaseCapturer's existing all_excluded_tables list — one
|
|
330
|
-
# config knob, both capturers obey it.
|
|
484
|
+
# config knob, both capturers obey it. The list is memoized at
|
|
485
|
+
# install time (`@excluded_tables`) so we don't pay a Hash
|
|
486
|
+
# method dispatch on every captured event. Customers who change
|
|
487
|
+
# config at runtime need to call uninstall! + install — the same
|
|
488
|
+
# constraint that already applies to `capture_database`.
|
|
331
489
|
def table_excluded?(model_class)
|
|
332
490
|
return false unless model_class.respond_to?(:table_name)
|
|
333
491
|
|
|
334
|
-
|
|
492
|
+
@excluded_tables.include?(model_class.table_name)
|
|
335
493
|
rescue StandardError
|
|
336
494
|
false
|
|
337
495
|
end
|
|
338
496
|
|
|
497
|
+
# Detects Rails-generated update_all SQL that we can't render
|
|
498
|
+
# meaningfully OR that is pure framework noise. We are deliberately
|
|
499
|
+
# narrow here — anything that COULD be a real change a customer
|
|
500
|
+
# cares about (even SET col = NULL on N rows) we keep.
|
|
501
|
+
#
|
|
502
|
+
# Filtered shapes:
|
|
503
|
+
#
|
|
504
|
+
# 1. Empty SET hash — the parser couldn't extract any column →
|
|
505
|
+
# value pairs (e.g. unquoted column names in raw SQL). The
|
|
506
|
+
# captured event would render with no "Set X to Y" detail
|
|
507
|
+
# and no humanized filter, just "Updated 26 things" with an
|
|
508
|
+
# empty Operation block. That's misleading.
|
|
509
|
+
#
|
|
510
|
+
# 2. Counter cache / increment! / decrement! — any SET value is
|
|
511
|
+
# a `COALESCE(<col>, 0) + N` expression. Rails uses this
|
|
512
|
+
# shape for `increment_counter`, `increment!`,
|
|
513
|
+
# `update_counters`. The semantics are "bump a number by N",
|
|
514
|
+
# which is plumbing-grade noise: high volume (per-request
|
|
515
|
+
# counter bumps fire dozens of times per minute on a busy
|
|
516
|
+
# tenant) and zero business meaning to a non-technical
|
|
517
|
+
# reader. On the EZLogs server alone these would dominate
|
|
518
|
+
# the timeline.
|
|
519
|
+
#
|
|
520
|
+
# SET <fk> = NULL on N rows IS captured — even though Rails sometimes
|
|
521
|
+
# generates it implicitly as cleanup before a destroy, it's also
|
|
522
|
+
# the shape of a real customer-issued nullification (soft-orphaning,
|
|
523
|
+
# disassociating tags from an item, etc.), and from the SQL alone
|
|
524
|
+
# we can't tell the two apart. Showing it honestly is the right call:
|
|
525
|
+
# the reader sees that N rows had their column X set to NULL.
|
|
526
|
+
#
|
|
527
|
+
# @return [Boolean]
|
|
528
|
+
def framework_rewrite?(operation, parse_result)
|
|
529
|
+
return false unless operation == :update_all
|
|
530
|
+
return false unless parse_result.is_a?(Hash)
|
|
531
|
+
|
|
532
|
+
set = parse_result[:set]
|
|
533
|
+
|
|
534
|
+
# Empty SET hash: parser bailed. Drop — would render as an
|
|
535
|
+
# empty Operation block.
|
|
536
|
+
return true if set.is_a?(Hash) && set.empty? && !parse_result[:unparseable]
|
|
537
|
+
|
|
538
|
+
return false unless set.is_a?(Hash) && set.any?
|
|
539
|
+
|
|
540
|
+
# Counter-cache / increment! — any value contains COALESCE().
|
|
541
|
+
# Distinct from a deliberate UPDATE because the SET expression
|
|
542
|
+
# references the column on its own RHS, which no business
|
|
543
|
+
# update_all ever does.
|
|
544
|
+
return true if set.values.any? { |v| v.to_s.include?("COALESCE(") }
|
|
545
|
+
|
|
546
|
+
false
|
|
547
|
+
end
|
|
548
|
+
|
|
339
549
|
# Same formatter as DatabaseCapturer. Keeps Date / Time / BigDecimal
|
|
340
550
|
# from collapsing to "[Object]" when they reach Sanitizer / wire.
|
|
341
551
|
def format_value_for_json(value)
|
|
@@ -78,6 +78,33 @@ module EzLogsAgent
|
|
|
78
78
|
return unless defined?(ActiveRecord::Base)
|
|
79
79
|
return if @installed
|
|
80
80
|
|
|
81
|
+
# Memoize config values that the per-row hot path reads on
|
|
82
|
+
# every callback. Without this, each captured create / update
|
|
83
|
+
# / destroy pays a method dispatch into the configuration
|
|
84
|
+
# object (`EzLogsAgent.configuration.capture_database`) plus
|
|
85
|
+
# an `all_excluded_tables.include?` Hash dispatch. On a request
|
|
86
|
+
# that touches dozens of rows the overhead is real.
|
|
87
|
+
# Runtime mutations require uninstall! + install to take effect
|
|
88
|
+
# (acceptable — nobody flips capture_database at runtime).
|
|
89
|
+
@capture_enabled =
|
|
90
|
+
begin
|
|
91
|
+
EzLogsAgent.configuration.capture_database
|
|
92
|
+
rescue StandardError
|
|
93
|
+
false
|
|
94
|
+
end
|
|
95
|
+
@excluded_tables =
|
|
96
|
+
begin
|
|
97
|
+
EzLogsAgent.configuration.all_excluded_tables.dup.freeze
|
|
98
|
+
rescue StandardError
|
|
99
|
+
[].freeze
|
|
100
|
+
end
|
|
101
|
+
@display_name_for =
|
|
102
|
+
begin
|
|
103
|
+
(EzLogsAgent.configuration.display_name_for || {}).dup.freeze
|
|
104
|
+
rescue StandardError
|
|
105
|
+
{}.freeze
|
|
106
|
+
end
|
|
107
|
+
|
|
81
108
|
# Only register callbacks once per Ruby process
|
|
82
109
|
unless @callbacks_registered
|
|
83
110
|
ActiveRecord::Base.class_eval do
|
|
@@ -137,24 +164,24 @@ module EzLogsAgent
|
|
|
137
164
|
|
|
138
165
|
private
|
|
139
166
|
|
|
140
|
-
# Checks if database capture is enabled
|
|
167
|
+
# Checks if database capture is enabled. Reads the memoized
|
|
168
|
+
# value set at install time (no config-object dispatch on the
|
|
169
|
+
# hot path).
|
|
141
170
|
#
|
|
142
171
|
# @return [Boolean]
|
|
143
172
|
def capture_enabled?
|
|
144
|
-
|
|
145
|
-
rescue StandardError
|
|
146
|
-
false
|
|
173
|
+
@capture_enabled
|
|
147
174
|
end
|
|
148
175
|
|
|
149
|
-
# Checks if the model's table is in the excluded_tables list
|
|
150
|
-
#
|
|
176
|
+
# Checks if the model's table is in the excluded_tables list.
|
|
177
|
+
# Reads the memoized list set at install time.
|
|
151
178
|
#
|
|
152
179
|
# @param model [ActiveRecord::Base] The model instance
|
|
153
180
|
# @return [Boolean]
|
|
154
181
|
def table_excluded?(model)
|
|
155
182
|
return false unless model.class.respond_to?(:table_name)
|
|
156
183
|
|
|
157
|
-
|
|
184
|
+
@excluded_tables.include?(model.class.table_name)
|
|
158
185
|
rescue StandardError
|
|
159
186
|
false
|
|
160
187
|
end
|
|
@@ -219,9 +246,8 @@ module EzLogsAgent
|
|
|
219
246
|
# @param model [ActiveRecord::Base] The model instance
|
|
220
247
|
# @return [String, nil] The display name, or nil if no meaningful name found
|
|
221
248
|
def resolve_display_name(model)
|
|
222
|
-
# Check for configured custom field
|
|
223
|
-
|
|
224
|
-
custom_field = display_name_config[model.class.name]
|
|
249
|
+
# Check for configured custom field (memoized list, see install).
|
|
250
|
+
custom_field = @display_name_for[model.class.name]
|
|
225
251
|
|
|
226
252
|
if custom_field && model.respond_to?(custom_field)
|
|
227
253
|
value = model.public_send(custom_field)
|
|
@@ -42,9 +42,9 @@ module EzLogsAgent
|
|
|
42
42
|
errors << "server_url must be a valid URL (got: #{config.server_url})"
|
|
43
43
|
end
|
|
44
44
|
|
|
45
|
-
#
|
|
45
|
+
# Required: the server rejects every event sent without a token
|
|
46
46
|
if config.project_token.nil? || config.project_token.to_s.strip.empty?
|
|
47
|
-
|
|
47
|
+
errors << "project_token is required. Set it in config/initializers/ez_logs_agent.rb"
|
|
48
48
|
end
|
|
49
49
|
|
|
50
50
|
# Validate numeric fields
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "timeout"
|
|
4
|
+
|
|
3
5
|
module EzLogsAgent
|
|
4
6
|
# FlushScheduler orchestrates periodic flushing of buffered events to the server.
|
|
5
7
|
#
|
|
@@ -14,6 +16,10 @@ module EzLogsAgent
|
|
|
14
16
|
# - Does NOT manage Rails lifecycle (Railtie handles that)
|
|
15
17
|
# - Does NOT mutate or inspect events
|
|
16
18
|
class FlushScheduler
|
|
19
|
+
# Exit budget: wait this long for the loop thread, then cap the final send
|
|
20
|
+
STOP_JOIN_SECONDS = 2
|
|
21
|
+
FINAL_FLUSH_SECONDS = 3
|
|
22
|
+
|
|
17
23
|
class << self
|
|
18
24
|
# Start the background flush thread
|
|
19
25
|
# Idempotent: safe to call multiple times
|
|
@@ -28,14 +34,26 @@ module EzLogsAgent
|
|
|
28
34
|
Logger.error("[FlushScheduler] Failed to start: #{error.class} - #{error.message}")
|
|
29
35
|
end
|
|
30
36
|
|
|
31
|
-
# Stop the background flush thread
|
|
37
|
+
# Stop the background flush thread, then send what is still buffered
|
|
32
38
|
# Blocks until thread exits
|
|
33
39
|
# Idempotent: safe to call multiple times
|
|
34
40
|
def stop
|
|
35
|
-
return unless
|
|
41
|
+
return unless @thread
|
|
36
42
|
|
|
37
43
|
@stop_requested = true
|
|
38
|
-
|
|
44
|
+
begin
|
|
45
|
+
@thread.wakeup # cut the interval sleep short so exit isn't delayed
|
|
46
|
+
rescue ThreadError
|
|
47
|
+
nil # thread already dead (e.g. the loop crashed): still drain below
|
|
48
|
+
end
|
|
49
|
+
@thread.join(STOP_JOIN_SECONDS)
|
|
50
|
+
@thread = nil
|
|
51
|
+
|
|
52
|
+
# Events buffered since the last tick would otherwise be lost at
|
|
53
|
+
# exit (rails runner, rake, deploy restarts). One attempt, short
|
|
54
|
+
# HTTP timeouts, and a hard cap (DNS isn't covered by
|
|
55
|
+
# open_timeout), so a down server can't stall shutdown.
|
|
56
|
+
Timeout.timeout(FINAL_FLUSH_SECONDS) { Transport.send(Buffer.flush, timeout: 1) }
|
|
39
57
|
|
|
40
58
|
Logger.debug("[FlushScheduler] Stopped")
|
|
41
59
|
rescue => error
|
data/lib/ez_logs_agent/logger.rb
CHANGED
|
@@ -29,11 +29,17 @@ module EzLogsAgent
|
|
|
29
29
|
log(:error, message)
|
|
30
30
|
end
|
|
31
31
|
|
|
32
|
+
# Logs at :info regardless of the configured log_level. Reserved for
|
|
33
|
+
# the one startup line that tells users whether capture is on.
|
|
34
|
+
def notice(message)
|
|
35
|
+
log(:info, message, force: true)
|
|
36
|
+
end
|
|
37
|
+
|
|
32
38
|
private
|
|
33
39
|
|
|
34
|
-
def log(level, message)
|
|
40
|
+
def log(level, message, force: false)
|
|
35
41
|
# Early return if this log level should not be logged
|
|
36
|
-
return unless should_log?(level)
|
|
42
|
+
return unless force || should_log?(level)
|
|
37
43
|
|
|
38
44
|
prefixed_message = "[EzLogsAgent] #{message}"
|
|
39
45
|
|
|
@@ -32,6 +32,7 @@ module EzLogsAgent
|
|
|
32
32
|
|
|
33
33
|
# Reset registration state (for testing)
|
|
34
34
|
def reset_registration_state!
|
|
35
|
+
@startup_announced = false
|
|
35
36
|
@sidekiq_client_registered = false
|
|
36
37
|
@sidekiq_server_registered = false
|
|
37
38
|
@activejob_installed = false
|
|
@@ -78,6 +79,7 @@ module EzLogsAgent
|
|
|
78
79
|
config.after_initialize do
|
|
79
80
|
# Validate configuration before doing anything
|
|
80
81
|
validation_result = validate_configuration
|
|
82
|
+
announce_startup(validation_result)
|
|
81
83
|
next unless validation_result
|
|
82
84
|
|
|
83
85
|
# Register Sidekiq middleware
|
|
@@ -267,6 +269,8 @@ module EzLogsAgent
|
|
|
267
269
|
# ActiveSupport::ForkTracker (Rails 6.1+) is the most reliable way
|
|
268
270
|
if defined?(ActiveSupport::ForkTracker)
|
|
269
271
|
ActiveSupport::ForkTracker.after_fork do
|
|
272
|
+
# The parent still owns (and sends) what it buffered before the fork
|
|
273
|
+
EzLogsAgent::Buffer.clear
|
|
270
274
|
start_flush_scheduler
|
|
271
275
|
end
|
|
272
276
|
EzLogsAgent::Logger.debug("[Railtie] Registered ActiveSupport::ForkTracker hook for FlushScheduler")
|
|
@@ -349,6 +353,41 @@ module EzLogsAgent
|
|
|
349
353
|
EzLogsAgent::Logger.error("[Railtie] Failed to log configuration summary: #{e.class} - #{e.message}")
|
|
350
354
|
end
|
|
351
355
|
|
|
356
|
+
# One line per process at the default log level, so "is it on?" is
|
|
357
|
+
# answerable from the logs: where events go and which key, shown the
|
|
358
|
+
# way the dashboard's Settings → API Keys shows it (`ezl_` + first 8
|
|
359
|
+
# characters). Other token shapes get no hint. Silent in the test env.
|
|
360
|
+
#
|
|
361
|
+
# @param capturing [Boolean] result of validate_configuration
|
|
362
|
+
# @return [void]
|
|
363
|
+
def self.announce_startup(capturing)
|
|
364
|
+
return if @startup_announced
|
|
365
|
+
@startup_announced = true
|
|
366
|
+
return if Rails.env.test?
|
|
367
|
+
|
|
368
|
+
config = EzLogsAgent.configuration
|
|
369
|
+
if capturing
|
|
370
|
+
uri = URI.parse(config.server_url.to_s)
|
|
371
|
+
host = uri.port == uri.default_port ? uri.host : "#{uri.host}:#{uri.port}"
|
|
372
|
+
token = config.project_token.to_s
|
|
373
|
+
project = token.start_with?("ezl_") && token.length >= 24 ? " (key #{token[0, 12]}…)" : ""
|
|
374
|
+
EzLogsAgent::Logger.notice("capturing → #{host}#{project}")
|
|
375
|
+
else
|
|
376
|
+
missing = [
|
|
377
|
+
blank?(config.server_url) && "server_url",
|
|
378
|
+
blank?(config.project_token) && "project_token"
|
|
379
|
+
].select { |m| m }
|
|
380
|
+
reason = missing.any? ? "#{missing.join(' and ')} missing" : "invalid configuration (see errors above)"
|
|
381
|
+
EzLogsAgent::Logger.warn("not capturing: #{reason}")
|
|
382
|
+
end
|
|
383
|
+
rescue StandardError => e
|
|
384
|
+
EzLogsAgent::Logger.error("[Railtie] Failed to log startup line: #{e.class} - #{e.message}")
|
|
385
|
+
end
|
|
386
|
+
|
|
387
|
+
def self.blank?(value)
|
|
388
|
+
value.nil? || value.to_s.strip.empty?
|
|
389
|
+
end
|
|
390
|
+
|
|
352
391
|
# Load rake tasks
|
|
353
392
|
rake_tasks do
|
|
354
393
|
load "tasks/ez_logs_agent.rake"
|
|
@@ -48,17 +48,35 @@ module EzLogsAgent
|
|
|
48
48
|
key_lower = key.to_s.downcase
|
|
49
49
|
return true if PATTERNS.any? { |pattern| key_lower.include?(pattern) }
|
|
50
50
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
# rescue would silently fall back to "no extra patterns" on a
|
|
54
|
-
# config bug and the outer rescue would never fire, which means
|
|
55
|
-
# the broken-config path becomes "leak", not "mask".
|
|
56
|
-
user_patterns = EzLogsAgent.configuration.excluded_graphql_variable_keys || []
|
|
57
|
-
user_patterns.any? { |pattern| key_lower.include?(pattern.to_s.downcase) }
|
|
51
|
+
user_patterns = user_patterns_cache
|
|
52
|
+
user_patterns.any? { |pattern| key_lower.include?(pattern) }
|
|
58
53
|
rescue StandardError
|
|
59
54
|
# Defensive: if configuration access raises, treat as sensitive.
|
|
60
55
|
# Better to over-mask than to leak.
|
|
61
56
|
true
|
|
62
57
|
end
|
|
58
|
+
|
|
59
|
+
# Memoized lookup of user-configured patterns, already lowercased.
|
|
60
|
+
# Called per HTTP param, per DB attribute, per job arg key. Without
|
|
61
|
+
# the cache it would dispatch into EzLogsAgent.configuration on
|
|
62
|
+
# every check.
|
|
63
|
+
#
|
|
64
|
+
# The cache is keyed by `object_id` of the configured array, so
|
|
65
|
+
# `EzLogsAgent.configure { |c| c.excluded_graphql_variable_keys = [...] }`
|
|
66
|
+
# invalidates it naturally (assigning a new array changes its id).
|
|
67
|
+
# No explicit invalidation needed.
|
|
68
|
+
def user_patterns_cache
|
|
69
|
+
configured = EzLogsAgent.configuration.excluded_graphql_variable_keys
|
|
70
|
+
return @cached_user_patterns if configured.nil? && @cached_source_id.nil?
|
|
71
|
+
|
|
72
|
+
source_id = configured&.object_id
|
|
73
|
+
if source_id != @cached_source_id
|
|
74
|
+
@cached_user_patterns =
|
|
75
|
+
(configured || []).map { |p| p.to_s.downcase }.freeze
|
|
76
|
+
@cached_source_id = source_id
|
|
77
|
+
end
|
|
78
|
+
@cached_user_patterns
|
|
79
|
+
end
|
|
80
|
+
private_class_method :user_patterns_cache
|
|
63
81
|
end
|
|
64
82
|
end
|
|
@@ -26,11 +26,12 @@ module EzLogsAgent
|
|
|
26
26
|
# Send events to the server
|
|
27
27
|
#
|
|
28
28
|
# @param events [Array<Hash>] Array of event hashes
|
|
29
|
+
# @param timeout [Numeric, nil] open/read/write timeout override (seconds)
|
|
29
30
|
# @return [Symbol] :success if 2xx response, :failure otherwise
|
|
30
|
-
def send(events)
|
|
31
|
+
def send(events, timeout: nil)
|
|
31
32
|
return :success if events.nil? || events.empty?
|
|
32
33
|
|
|
33
|
-
response = post_events(events)
|
|
34
|
+
response = post_events(events, timeout)
|
|
34
35
|
classify_response(response)
|
|
35
36
|
rescue => error
|
|
36
37
|
Logger.error("[Transport] send failed: #{error.class} - #{error.message}")
|
|
@@ -39,9 +40,9 @@ module EzLogsAgent
|
|
|
39
40
|
|
|
40
41
|
private
|
|
41
42
|
|
|
42
|
-
def post_events(events)
|
|
43
|
+
def post_events(events, timeout)
|
|
43
44
|
uri = build_uri
|
|
44
|
-
http = build_http_client(uri)
|
|
45
|
+
http = build_http_client(uri, timeout)
|
|
45
46
|
request = build_request(uri, events)
|
|
46
47
|
|
|
47
48
|
http.request(request)
|
|
@@ -56,11 +57,12 @@ module EzLogsAgent
|
|
|
56
57
|
raise "Invalid server_url: #{error.message}"
|
|
57
58
|
end
|
|
58
59
|
|
|
59
|
-
def build_http_client(uri)
|
|
60
|
+
def build_http_client(uri, timeout = nil)
|
|
60
61
|
http = Net::HTTP.new(uri.host, uri.port)
|
|
61
62
|
http.use_ssl = (uri.scheme == "https")
|
|
62
|
-
http.open_timeout = 5
|
|
63
|
-
http.read_timeout = 10
|
|
63
|
+
http.open_timeout = timeout || 5
|
|
64
|
+
http.read_timeout = timeout || 10
|
|
65
|
+
http.write_timeout = timeout if timeout
|
|
64
66
|
http
|
|
65
67
|
end
|
|
66
68
|
|
|
@@ -16,6 +16,8 @@ EzLogsAgent.configure do |config|
|
|
|
16
16
|
config.server_url = ENV.fetch("EZLOGS_SERVER_URL", "https://app.ezlogs.io")
|
|
17
17
|
|
|
18
18
|
# API key for your tenant. Get one at https://app.ezlogs.io/settings/api-keys.
|
|
19
|
+
# On boot the log shows "[EzLogsAgent] capturing → <host> (key ezl_<first 8>…)",
|
|
20
|
+
# or "[EzLogsAgent] not capturing: project_token missing" when this is unset.
|
|
19
21
|
config.project_token = ENV["EZLOGS_API_KEY"]
|
|
20
22
|
|
|
21
23
|
# =============================================================================
|
metadata
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ez_logs_agent
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- dezsirazvan
|
|
8
8
|
bindir: bin
|
|
9
9
|
cert_chain: []
|
|
10
|
-
date: 2026-
|
|
10
|
+
date: 2026-11-17 00:00:00.000000000 Z
|
|
11
11
|
dependencies:
|
|
12
12
|
- !ruby/object:Gem::Dependency
|
|
13
13
|
name: request_store
|
|
@@ -99,14 +99,14 @@ dependencies:
|
|
|
99
99
|
requirements:
|
|
100
100
|
- - "~>"
|
|
101
101
|
- !ruby/object:Gem::Version
|
|
102
|
-
version: '
|
|
102
|
+
version: '2.9'
|
|
103
103
|
type: :development
|
|
104
104
|
prerelease: false
|
|
105
105
|
version_requirements: !ruby/object:Gem::Requirement
|
|
106
106
|
requirements:
|
|
107
107
|
- - "~>"
|
|
108
108
|
- !ruby/object:Gem::Version
|
|
109
|
-
version: '
|
|
109
|
+
version: '2.9'
|
|
110
110
|
description: Captures HTTP requests, background jobs, and database changes from Rails
|
|
111
111
|
applications. Events are automatically correlated and sent to EzLogs Server, where
|
|
112
112
|
they are transformed into human-readable activity stories.
|