whodunit-chronicles 0.3.0 β†’ 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (54) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +22 -226
  3. data/LICENSE +1 -1
  4. data/README.md +96 -599
  5. data/exe/whodunit-chronicles +6 -0
  6. data/lib/whodunit/chronicles/chronicler.rb +62 -0
  7. data/lib/whodunit/chronicles/cli.rb +131 -0
  8. data/lib/whodunit/chronicles/errors.rb +7 -33
  9. data/lib/whodunit/chronicles/ledger.rb +69 -0
  10. data/lib/whodunit/chronicles/ledger_entry.rb +143 -0
  11. data/lib/whodunit/chronicles/ledger_factory.rb +66 -0
  12. data/lib/whodunit/chronicles/ledgers/file_ledger.rb +56 -0
  13. data/lib/whodunit/chronicles/ledgers/memory_ledger.rb +29 -0
  14. data/lib/whodunit/chronicles/ledgers/sqlite_ledger.rb +172 -0
  15. data/lib/whodunit/chronicles/version.rb +2 -1
  16. data/lib/whodunit/chronicles.rb +12 -65
  17. data/lib/whodunit-chronicles.rb +0 -1
  18. data/sig/whodunit/chronicles/chronicler.rbs +14 -0
  19. data/sig/whodunit/chronicles/cli.rbs +17 -0
  20. data/sig/whodunit/chronicles/errors.rbs +15 -0
  21. data/sig/whodunit/chronicles/ledger.rbs +13 -0
  22. data/sig/whodunit/chronicles/ledger_entry.rbs +62 -0
  23. data/sig/whodunit/chronicles/ledger_factory.rbs +14 -0
  24. data/sig/whodunit/chronicles/ledgers/file_ledger.rbs +14 -0
  25. data/sig/whodunit/chronicles/ledgers/memory_ledger.rbs +12 -0
  26. data/sig/whodunit/chronicles/ledgers/sqlite_ledger.rbs +30 -0
  27. data/sig/whodunit/chronicles.rbs +5 -0
  28. metadata +40 -326
  29. data/.codeclimate.yml +0 -50
  30. data/.rubocop.yml +0 -93
  31. data/.yardopts +0 -14
  32. data/CODE_OF_CONDUCT.md +0 -132
  33. data/CONTRIBUTING.md +0 -186
  34. data/Rakefile +0 -18
  35. data/docker/mysql/init.sql +0 -33
  36. data/docker/postgres/init.sql +0 -40
  37. data/docker-compose.yml +0 -138
  38. data/examples/images/campaign-performance-analytics.png +0 -0
  39. data/examples/images/candidate-journey-analytics.png +0 -0
  40. data/examples/images/recruitment-funnel-analytics.png +0 -0
  41. data/lib/.gitkeep +0 -0
  42. data/lib/whodunit/chronicles/adapter_loader.rb +0 -69
  43. data/lib/whodunit/chronicles/adapters/mysql.rb +0 -261
  44. data/lib/whodunit/chronicles/adapters/postgresql.rb +0 -278
  45. data/lib/whodunit/chronicles/change_event.rb +0 -201
  46. data/lib/whodunit/chronicles/composite_processor.rb +0 -86
  47. data/lib/whodunit/chronicles/configuration.rb +0 -112
  48. data/lib/whodunit/chronicles/connection.rb +0 -88
  49. data/lib/whodunit/chronicles/persistence.rb +0 -129
  50. data/lib/whodunit/chronicles/processor.rb +0 -127
  51. data/lib/whodunit/chronicles/service.rb +0 -207
  52. data/lib/whodunit/chronicles/stream_adapter.rb +0 -91
  53. data/lib/whodunit/chronicles/table.rb +0 -120
  54. data/whodunit-chronicles.gemspec +0 -79
data/README.md CHANGED
@@ -1,667 +1,164 @@
1
- # πŸ“œ Whodunit Chronicles
1
+ # whodunit-chronicles
2
2
 
3
3
  [![Gem Version](https://badge.fury.io/rb/whodunit-chronicles.svg)](https://badge.fury.io/rb/whodunit-chronicles)
4
4
  [![CI](https://github.com/kanutocd/whodunit-chronicles/workflows/CI/badge.svg)](https://github.com/kanutocd/whodunit-chronicles/actions)
5
- [![Coverage Status](https://codecov.io/gh/kanutocd/whodunit-chronicles/branch/main/graph/badge.svg)](https://codecov.io/gh/kanutocd/whodunit-chronicles)
6
- [![Ruby Version](https://img.shields.io/badge/ruby-%3E%3D%203.1.0-ruby.svg)](https://www.ruby-lang.org/en/)
5
+ [![Ruby Version](https://img.shields.io/badge/ruby-%3E%3D%203.4-ruby.svg)](https://www.ruby-lang.org/en/)
7
6
  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
8
7
 
9
- > **The complete historical record of your _Whodunit Dun Wat?_ data**
8
+ `whodunit-chronicles` is the minimal audit sink for the Ruby CDC ecosystem. It has a runtime dependency on `cdc-core` and consumes `CDC::Core::ChangeEvent` directly.
10
9
 
11
- > **πŸ’‘ Origin Story:** Chronicles is inspired by the challenge of streaming database changes for real-time analytics without impacting application performance. The concept proved so effective in a previous project that it became the foundation for this Ruby implementation.
10
+ ## Release status
12
11
 
13
- While [Whodunit](https://github.com/kanutocd/whodunit) tracks _who_ made changes, **Chronicles** captures _what_ changed by streaming database events into comprehensive audit trails with **zero Rails application overhead**.
12
+ The currently published RubyGems release is `whodunit-chronicles` `0.3.0`, described as "The complete historical record of your data".
14
13
 
15
- ## ✨ Features
14
+ The next release is a complete rewrite of the gem. This README describes the rewritten design: a smaller CDC-native audit sink centered on `CDC::Core::ChangeEvent`, immutable ledger entries, lightweight built-in ledgers, and extension gems for heavier storage targets.
16
15
 
17
- - **πŸš„ Zero-Latency Streaming**: PostgreSQL logical replication + MySQL/MariaDB binary log streaming
18
- - **πŸ”„ Zero Application Overhead**: No Rails callbacks or Active Record hooks required
19
- - **πŸ—οΈ Database Agnostic**: Abstract adapter pattern supports PostgreSQL and MySQL/MariaDB
20
- - **⚑ Thread-Safe**: Concurrent processing with configurable thread pools
21
- - **πŸ›‘οΈ Resilient**: Built-in error handling, retry logic, and monitoring
22
- - **πŸ“Š Complete Audit Trail**: Captures INSERT, UPDATE, DELETE with full before/after data
23
- - **πŸ§ͺ Code Coverage**: 94%+ test coverage with comprehensive error scenarios
16
+ A `Chronicler` writes immutable `LedgerEntry` objects into any object that implements the `Ledger` contract.
24
17
 
25
- ## πŸš€ Quick Start
26
-
27
- ### 🎯 Usage Scenarios
28
-
29
- Chronicles excels at transforming database changes into business intelligence. Here are two common patterns:
30
-
31
- #### 1. Basic Audit Trail Integration
32
-
33
- ---
34
-
35
- Perfect for applications that need comprehensive change tracking alongside Whodunit's user attribution:
36
-
37
- ```ruby
38
- # Basic setup for user activity tracking
39
- class BasicProcessor < Whodunit::Chronicles::Processor
40
- def build_chronicles_record(change_event)
41
- super.tap do |record|
42
- # Add basic business context
43
- record[:change_category] = categorize_change(change_event)
44
- record[:business_impact] = assess_impact(change_event)
45
- end
46
- end
47
-
48
- private
49
-
50
- def categorize_change(change_event)
51
- case change_event.table_name
52
- when 'users' then 'user_management'
53
- when 'posts' then 'content'
54
- when 'comments' then 'engagement'
55
- else 'system'
56
- end
57
- end
58
- end
59
- ```
60
-
61
- **Use Case**: Blog platform tracking user posts and comments for community management and content moderation.
62
-
63
- #### 2. Advanced Recruitment Analytics
64
-
65
- Sophisticated business intelligence for talent acquisition platforms:
66
-
67
- ```ruby
68
- # Advanced processor for recruitment metrics
69
- class RecruitmentAnalyticsProcessor < Whodunit::Chronicles::Processor
70
- def build_chronicles_record(change_event)
71
- super.tap do |record|
72
- # Add recruitment-specific business metrics
73
- record[:recruitment_stage] = determine_stage(change_event)
74
- record[:funnel_position] = calculate_funnel_position(change_event)
75
- record[:time_to_hire_impact] = assess_time_impact(change_event)
76
- record[:cost_per_hire_impact] = calculate_cost_impact(change_event)
77
-
78
- # Campaign attribution
79
- record[:utm_source] = extract_utm_source(change_event)
80
- record[:campaign_id] = extract_campaign_id(change_event)
81
-
82
- # Quality metrics
83
- record[:candidate_quality_score] = assess_candidate_quality(change_event)
84
- end
85
- end
86
-
87
- def process(change_event)
88
- record = build_chronicles_record(change_event)
89
- store_audit_record(record)
90
-
91
- # Stream to analytics platforms
92
- stream_to_prometheus(record) if track_metrics?
93
- update_grafana_dashboard(record)
94
- trigger_real_time_alerts(record) if alert_worthy?(record)
95
- end
96
-
97
- private
98
-
99
- def determine_stage(change_event)
100
- return 'unknown' unless change_event.table_name == 'applications'
101
-
102
- case change_event.new_data&.dig('status')
103
- when 'submitted' then 'application'
104
- when 'screening', 'in_review' then 'screening'
105
- when 'interview_scheduled', 'interviewed' then 'interview'
106
- when 'offer_extended', 'offer_accepted' then 'offer'
107
- when 'hired' then 'hire'
108
- else 'unknown'
109
- end
110
- end
111
-
112
- def stream_to_prometheus(record)
113
- # Track key recruitment metrics
114
- RECRUITMENT_APPLICATIONS_TOTAL.increment(
115
- source: record[:utm_source],
116
- department: record.dig(:new_data, 'department')
117
- )
118
-
119
- if record[:action] == 'UPDATE' && status_changed_to_hired?(record)
120
- RECRUITMENT_HIRES_TOTAL.increment(
121
- source: record[:utm_source],
122
- time_to_hire: record[:time_to_hire_impact]
123
- )
124
- end
125
- end
126
-
127
- def update_grafana_dashboard(record)
128
- # Send time-series data for Grafana visualization
129
- InfluxDB.write_point('recruitment_events', {
130
- timestamp: record[:occurred_at],
131
- table: record[:table_name],
132
- action: record[:action],
133
- stage: record[:recruitment_stage],
134
- source: record[:utm_source],
135
- cost_impact: record[:cost_per_hire_impact],
136
- quality_score: record[:candidate_quality_score]
137
- })
138
- end
139
- end
18
+ ```text
19
+ CDC::Core::ChangeEvent
20
+ ↓
21
+ Whodunit::Chronicles::Chronicler
22
+ ↓
23
+ Whodunit::Chronicles::LedgerEntry
24
+ ↓
25
+ Ledger
140
26
  ```
141
27
 
142
- **Use Case**: Imagine a Spherical Cow Talent acquisition platform tracking candidate journey from application through hire, with real-time dashboards showing conversion rates, time-to-hire, cost-per-hire, and source effectiveness.
143
-
144
- #### πŸ“Š Visual Analytics Dashboard
145
-
146
- The recruitment analytics processor creates comprehensive Grafana dashboards for executive reporting and operational insights:
147
-
148
- <div align="center">
149
-
150
- **Campaign Performance Analytics**
151
- <a href="examples/images/campaign-performance-analytics.png" title="Click to view full size image">
152
- <img src="examples/images/campaign-performance-analytics.png" width="300" />
153
- </a>
154
- _Track campaign ROI, cost-per-hire by channel, and conversion rates across marketing sources_
155
-
156
- **Candidate Journey Analytics**
157
- <a href="examples/images/candidate-journey-analytics.png" title="Click to view full size image">
158
- <img src="examples/images/candidate-journey-analytics.png" width="300" />
159
- </a>
160
- _Monitor candidate engagement, funnel conversion rates, and application completion patterns_
28
+ ## Core boundary
161
29
 
162
- **Recruitment Funnel Analytics**
163
- <a href="examples/images/recruitment-funnel-analytics.png" title="Click to view full size image">
164
- <img src="examples/images/recruitment-funnel-analytics.png" width="300" />
165
- </a>
166
- _Analyze hiring pipeline progression, department performance, and time-series trends_
30
+ Core Chronicles owns:
167
31
 
168
- </div>
32
+ - `cdc-core` processor integration
169
33
 
170
- These dashboards are automatically populated by Chronicles as candidates move through your hiring funnel, providing real-time visibility into recruitment performance without any manual data entry.
34
+ - `Chronicler`
35
+ - immutable `LedgerEntry`
36
+ - `Ledger` contract
37
+ - `MemoryLedger`
38
+ - `FileLedger`
39
+ - `SQLiteLedger`
40
+ - ledger lifecycle CLI
171
41
 
172
- ### Installation
42
+ Core Chronicles does not own heavyweight storage integrations such as PostgreSQL, MySQL, Oracle, MongoDB, S3, Snowflake, Glacier, Mixpanel, or ClickHouse. Those belong in extension gems that implement the `Ledger` contract.
173
43
 
174
- Add to your Gemfile:
44
+ ## Runtime usage
175
45
 
176
46
  ```ruby
177
- gem 'whodunit-chronicles'
178
- ```
179
-
180
- Or install directly:
47
+ ledger = Whodunit::Chronicles::Ledgers::SQLiteLedger.new(path: "chronicles.db")
48
+ chronicler = Whodunit::Chronicles::Chronicler.new(ledger: ledger, prepare: false)
181
49
 
182
- ```bash
183
- gem install whodunit-chronicles
50
+ chronicler.process(change_event) # change_event is CDC::Core::ChangeEvent
184
51
  ```
185
52
 
186
- ### Basic Usage
53
+ The runtime hot path is intentionally small:
187
54
 
188
55
  ```ruby
189
- require 'whodunit/chronicles'
190
-
191
- # Database Configuration
192
-
193
- ## PostgreSQL Configuration
194
- Whodunit::Chronicles.configure do |config|
195
- config.adapter = :postgresql
196
- config.database_url = 'postgresql://localhost/myapp_production'
197
- config.audit_database_url = 'postgresql://localhost/myapp'
198
- config.publication_name = 'myapp_chronicles'
199
- config.replication_slot_name = 'myapp_chronicles_slot'
200
- end
201
-
202
- ## MySQL/MariaDB Configuration
203
- Whodunit::Chronicles.configure do |config|
204
- config.adapter = :mysql
205
- config.database_url = 'mysql://user:password@localhost/myapp_production'
206
- config.audit_database_url = 'mysql://user:password@localhost/myapp_audit'
207
- config.mysql_server_id = 1001 # Unique server ID for replication
208
- end
209
-
210
- # Create and start the service
211
- service = Whodunit::Chronicles.service
212
- service.setup! # Create publication/replication setup
213
- service.start # Begin streaming changes
214
-
215
- # Service runs in background threads
216
- sleep 10
217
-
218
- # Stop gracefully
219
- service.stop
220
- service.teardown! # Clean up database objects
56
+ entry = Whodunit::Chronicles::LedgerEntry.from_change_event(event)
57
+ ledger.append(entry)
221
58
  ```
222
59
 
223
- ## πŸ—οΈ Architecture
224
-
225
- Chronicles uses **PostgreSQL logical replication** and **MySQL/MariaDB binary log streaming** to capture database changes without impacting your application:
226
-
227
- ```
228
- β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
229
- β”‚ Application β”‚ β”‚ Chronicles β”‚ β”‚ Audit Store β”‚
230
- β”‚ Database │───▢│ Service │───▢│ Database β”‚
231
- β”‚ β”‚ β”‚ β”‚ β”‚ β”‚
232
- β”‚ β€’ Users β”‚ β”‚ β€’ Stream Adapter β”‚ β”‚ β€’ audit_records β”‚
233
- β”‚ β€’ Posts β”‚ β”‚ β€’ Event Parser β”‚ β”‚ β€’ Searchable β”‚
234
- β”‚ β€’ Comments β”‚ β”‚ β€’ Audit Builder β”‚ β”‚ β€’ Reportable β”‚
235
- β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
236
- β”‚ β”‚
237
- β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
238
- β”‚ β”‚ PostgreSQL β”‚
239
- └──────────────│ Logical β”‚
240
- β”‚ Replication β”‚
241
- β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
242
- ```
243
-
244
- ### Core Components
245
-
246
- - **StreamAdapter**: Database-specific change streaming (PostgreSQL logical replication, MySQL/MariaDB binary log streaming)
247
- - **ChangeEvent**: Unified change representation across adapters
248
- - **Processor**: Transforms changes into searchable audit records
249
- - **Service**: Orchestrates streaming with error handling and retry logic
60
+ ## Ledger contract
250
61
 
251
- ## βš™οΈ Configuration
62
+ Required:
252
63
 
253
64
  ```ruby
254
- Whodunit::Chronicles.configure do |config|
255
- # Database connections
256
- config.database_url = ENV['DATABASE_URL']
257
- config.audit_database_url = ENV['AUDIT_DATABASE_URL']
258
-
259
- # Database adapter (postgresql, mysql, mariadb)
260
- config.adapter = :postgresql
261
-
262
- # PostgreSQL-specific settings
263
- config.publication_name = 'whodunit_chronicles'
264
- config.replication_slot_name = 'whodunit_chronicles_slot'
265
-
266
- # Performance tuning
267
- config.batch_size = 1000
268
- config.max_retry_attempts = 5
269
- config.retry_delay = 10
270
-
271
- # Table filtering
272
- config.include_tables = %w[users posts comments]
273
- config.exclude_tables = %w[sessions temp_data]
274
- config.include_schemas = %w[public app]
275
- config.exclude_schemas = %w[information_schema pg_catalog]
276
- end
65
+ ledger.append(entry)
277
66
  ```
278
67
 
279
- ## πŸ“Š Audit Records
280
-
281
- Chronicles creates structured audit records for each database change:
68
+ Optional operational capabilities:
282
69
 
283
70
  ```ruby
284
- {
285
- id: 123,
286
- table_name: "users",
287
- schema_name: "public",
288
- record_id: { "id" => 456 },
289
- action: "UPDATE",
290
- old_data: { "id" => 456, "email" => "old@example.com", "name" => "Old Name" },
291
- new_data: { "id" => 456, "email" => "new@example.com", "name" => "New Name" },
292
- changes: { "email" => ["old@example.com", "new@example.com"] },
293
- user_id: 789, # From creator_id/updater_id/deleter_id columns
294
- user_type: "User",
295
- transaction_id: "tx_abc123",
296
- sequence_number: 42,
297
- occurred_at: 2025-01-21 10:30:00 UTC,
298
- created_at: 2025-01-21 10:30:01 UTC,
299
- metadata: {
300
- table_schema: "public",
301
- qualified_table_name: "public.users",
302
- changed_columns: ["email"],
303
- chronicles_version: "0.1.0"
304
- }
305
- }
71
+ ledger.prepare!
72
+ ledger.migrate!
73
+ ledger.ensure_indexes!
74
+ ledger.verify
75
+ ledger.status
76
+ ledger.partition_for(entry)
306
77
  ```
307
78
 
308
- ## πŸ”§ Advanced Usage
309
-
310
- ### Custom Processors for Analytics & Monitoring
311
-
312
- **The real power of Chronicles** comes from creating custom processors tailored for your specific analytics needs. While Whodunit captures basic "who changed what," Chronicles lets you build sophisticated data pipelines for tools like **Grafana**, **DataDog**, or **Elasticsearch**.
313
-
314
- Transform database changes into actionable business intelligence with features like:
79
+ ## Built-in ledgers
315
80
 
316
- - **25+ Custom Metrics**: Track business KPIs like conversion rates, time-to-hire, and cost-per-acquisition
317
- - **Real-time Dashboards**: Stream data to Grafana for executive reporting and operational monitoring
318
- - **Smart Alerting**: Trigger notifications based on business rules and thresholds
319
- - **Multi-destination Streaming**: Send data simultaneously to multiple analytics platforms
81
+ ### MemoryLedger
320
82
 
321
- #### Analytics-Focused Processor
83
+ For tests, examples, and short-lived scripts.
322
84
 
323
85
  ```ruby
324
- class AnalyticsProcessor < Whodunit::Chronicles::Processor
325
- def build_chronicles_record(change_event)
326
- super.tap do |record|
327
- # Add business metrics
328
- record[:business_impact] = calculate_business_impact(change_event)
329
- record[:user_segment] = determine_user_segment(change_event)
330
- record[:feature_flag] = current_feature_flags
331
-
332
- # Add performance metrics
333
- record[:change_size] = calculate_change_size(change_event)
334
- record[:peak_hours] = during_peak_hours?
335
- record[:geographic_region] = user_region(change_event)
336
-
337
- # Add time-series friendly fields for Grafana
338
- record[:hour_of_day] = Time.current.hour
339
- record[:day_of_week] = Time.current.wday
340
- record[:is_weekend] = weekend?
341
-
342
- # Custom tagging for filtering
343
- record[:tags] = generate_tags(change_event)
344
- end
345
- end
346
-
347
- private
348
-
349
- def calculate_business_impact(change_event)
350
- case change_event.table_name
351
- when 'orders' then 'revenue_critical'
352
- when 'users' then 'customer_critical'
353
- when 'products' then 'inventory_critical'
354
- else 'standard'
355
- end
356
- end
357
-
358
- def determine_user_segment(change_event)
359
- return 'anonymous' unless change_event.user_id
360
-
361
- # Look up user tier from your business logic
362
- User.find(change_event.user_id)&.tier || 'standard'
363
- end
364
-
365
- def generate_tags(change_event)
366
- tags = [change_event.action.downcase]
367
- tags << 'bulk_operation' if bulk_operation?(change_event)
368
- tags << 'api_driven' if api_request?
369
- tags << 'admin_action' if admin_user?(change_event.user_id)
370
- tags
371
- end
372
- end
86
+ ledger = Whodunit::Chronicles::Ledgers::MemoryLedger.new
373
87
  ```
374
88
 
375
- #### Grafana Dashboard Ready
89
+ ### FileLedger
376
90
 
377
- ```ruby
378
- class GrafanaProcessor < Whodunit::Chronicles::Processor
379
- def build_chronicles_record(change_event)
380
- {
381
- # Core metrics for Grafana time series
382
- timestamp: change_event.occurred_at,
383
- table_name: change_event.table_name,
384
- action: change_event.action,
385
-
386
- # Numerical metrics for graphs
387
- records_affected: calculate_records_affected(change_event),
388
- change_magnitude: calculate_change_magnitude(change_event),
389
- user_session_duration: calculate_session_duration(change_event),
390
-
391
- # Categorical dimensions for filtering
392
- environment: Rails.env,
393
- application_version: app_version,
394
- database_instance: database_identifier,
395
-
396
- # Business KPIs
397
- revenue_impact: calculate_revenue_impact(change_event),
398
- customer_satisfaction_risk: assess_satisfaction_risk(change_event),
399
-
400
- # Performance indicators
401
- query_duration_ms: extract_query_duration(change_event),
402
- concurrent_users: current_concurrent_users,
403
- system_load: current_system_load
404
- }
405
- end
406
- end
407
- ```
408
-
409
- #### Real-Time Alerts Processor
91
+ Append-only newline-delimited JSON file.
410
92
 
411
93
  ```ruby
412
- class AlertingProcessor < Whodunit::Chronicles::Processor
413
- def process(change_event)
414
- record = build_chronicles_record(change_event)
415
-
416
- # Store the audit record
417
- store_audit_record(record)
418
-
419
- # Real-time alerting logic
420
- send_alert(record) if alert_worthy?(record)
421
-
422
- # Stream to monitoring systems
423
- stream_to_datadog(record) if production?
424
- stream_to_grafana(record)
425
- end
426
-
427
- private
428
-
429
- def alert_worthy?(record)
430
- # Define your alerting criteria
431
- record[:business_impact] == 'revenue_critical' ||
432
- record[:records_affected] > 1000 ||
433
- record[:action] == 'DELETE' && record[:table_name] == 'orders'
434
- end
435
-
436
- def stream_to_grafana(record)
437
- # Send metrics to Grafana via InfluxDB/Prometheus
438
- InfluxDB.write_point("chronicles_events", record)
439
- end
440
- end
94
+ ledger = Whodunit::Chronicles::Ledgers::FileLedger.new(path: "chronicles.ndjson")
95
+ ledger.prepare!
441
96
  ```
442
97
 
443
- #### Multiple Processor Pipeline
98
+ ### SQLiteLedger
444
99
 
445
- ```ruby
446
- # Chain multiple processors for different purposes
447
- service = Whodunit::Chronicles::Service.new(
448
- adapter: Adapters::PostgreSQL.new,
449
- processor: Whodunit::Chronicles::CompositeProcessor.new([
450
- AnalyticsProcessor.new, # For business intelligence
451
- AlertingProcessor.new, # For real-time monitoring
452
- ComplianceProcessor.new, # For regulatory requirements
453
- ArchivalProcessor.new # For long-term storage
454
- ])
455
- )
456
- ```
457
-
458
- **Use Cases:**
459
-
460
- - **πŸ“Š Business Intelligence**: Track user behavior patterns, feature adoption, revenue impact
461
- - **🚨 Real-Time Monitoring**: Alert on suspicious activities, bulk operations, data anomalies
462
- - **πŸ“ˆ Performance Analytics**: Database performance metrics, query optimization insights
463
- - **πŸ” Compliance Auditing**: Regulatory compliance, data governance, access patterns
464
- - **πŸ’‘ Product Analytics**: Feature usage, A/B testing data, user journey tracking
465
-
466
- ### Service Monitoring
100
+ Embedded durable local ledger.
467
101
 
468
102
  ```ruby
469
- service = Whodunit::Chronicles.service
470
-
471
- # Check service status
472
- status = service.status
473
- puts "Running: #{status[:running]}"
474
- puts "Adapter Position: #{status[:adapter_position]}"
475
- puts "Retry Count: #{status[:retry_count]}"
476
- puts "Active Threads: #{status[:executor_status][:active_count]}"
477
-
478
- # Monitor in production
479
- Thread.new do
480
- loop do
481
- status = service.status
482
- Rails.logger.info "Chronicles Status: #{status}"
483
- sleep 60
484
- end
485
- end
103
+ ledger = Whodunit::Chronicles::Ledgers::SQLiteLedger.new(path: "chronicles.db")
104
+ ledger.prepare!
105
+ ledger.ensure_indexes!
486
106
  ```
487
107
 
488
- ## πŸ§ͺ Testing
108
+ `SQLiteLedger` depends on the `sqlite3` gem. Tests and advanced users may inject a compatible connection.
489
109
 
490
- ### Integration Testing
110
+ `ensure_indexes!` creates a unique index on `event_id`. Appending the same event more than once is not idempotent: SQLite rejects the duplicate row, and `SQLiteLedger#append` raises `Whodunit::Chronicles::AppendError`.
491
111
 
492
- Test Chronicles with your Rails application using these patterns:
112
+ ## CLI
493
113
 
494
- #### Basic Testing Pattern
114
+ Ledger evolution is operational work, similar to `db:migrate`. The gem consumer decides when to run it.
495
115
 
496
- ```ruby
497
- # Test basic Chronicles functionality
498
- class ChroniclesIntegrationTest < ActiveSupport::TestCase
499
- def setup
500
- @service = Whodunit::Chronicles.service
501
- @service.setup!
502
- @service.start
503
- end
504
-
505
- def teardown
506
- @service.stop
507
- @service.teardown!
508
- end
509
-
510
- def test_audit_record_creation
511
- # Create a user (triggers Whodunit)
512
- user = User.create!(name: "John", email: "john@example.com")
513
-
514
- # Wait for Chronicles to process
515
- sleep 1
516
-
517
- # Check Chronicles audit record
518
- audit_record = AuditRecord.find_by(
519
- table_name: 'users',
520
- action: 'INSERT',
521
- record_id: { 'id' => user.id }
522
- )
523
-
524
- assert audit_record
525
- assert_equal 'INSERT', audit_record.action
526
- assert_equal user.name, audit_record.new_data['name']
527
- end
528
- end
116
+ ```bash
117
+ whodunit-chronicles ledger prepare config.yml
118
+ whodunit-chronicles ledger migrate config.yml
119
+ whodunit-chronicles ledger ensure-indexes config.yml
120
+ whodunit-chronicles ledger verify config.yml
121
+ whodunit-chronicles ledger status config.yml
529
122
  ```
530
123
 
531
- #### Advanced Analytics Testing
124
+ Example configs:
532
125
 
533
- ```ruby
534
- # Test custom processor functionality
535
- class RecruitmentAnalyticsTest < ActiveSupport::TestCase
536
- def setup
537
- @processor = RecruitmentAnalyticsProcessor.new
538
- end
539
-
540
- def test_recruitment_stage_determination
541
- change_event = create_change_event(
542
- table_name: 'applications',
543
- action: 'UPDATE',
544
- new_data: { 'status' => 'hired' }
545
- )
546
-
547
- record = @processor.build_chronicles_record(change_event)
548
-
549
- assert_equal 'hire', record[:recruitment_stage]
550
- assert record[:cost_per_hire_impact]
551
- end
552
-
553
- def test_metrics_streaming
554
- # Mock Prometheus and Grafana integrations
555
- assert_difference 'RECRUITMENT_HIRES_TOTAL.get' do
556
- @processor.stream_to_prometheus(hired_record)
557
- end
558
- end
559
- end
126
+ ```yaml
127
+ ledger:
128
+ adapter: memory
560
129
  ```
561
130
 
562
- ### Unit Testing
563
-
564
- Chronicles includes comprehensive test coverage:
565
-
566
- ```bash
567
- # Run test suite
568
- bundle exec rake test
569
-
570
- # Run with coverage
571
- bundle exec rake test
572
- open coverage/index.html
573
-
574
- # Security scanning
575
- bundle exec bundler-audit check
576
- bundle exec brakeman
131
+ ```yaml
132
+ ledger:
133
+ adapter: file
134
+ path: chronicles.ndjson
577
135
  ```
578
136
 
579
- ## πŸ“ˆ Performance
580
-
581
- - **Minimal Overhead**: No Rails callback performance impact
582
- - **Efficient Streaming**: PostgreSQL logical replication is highly optimized
583
- - **Configurable Batching**: Process changes in configurable batch sizes
584
- - **Thread Pool**: Concurrent processing with bounded resource usage
585
- - **Memory Efficient**: Streaming processing without loading full datasets
586
-
587
- ## πŸ›‘οΈ Security
588
-
589
- - **Dependency Scanning**: Automated bundler-audit checks
590
- - **Code Analysis**: GitHub CodeQL integration
591
- - **Vulnerability Monitoring**: Weekly security scans
592
- - **Safe Defaults**: Secure configuration out of the box
593
-
594
- ## 🀝 Contributing
595
-
596
- We welcome contributions! Chronicles is designed to be extensible and work across different business domains.
597
-
598
- 1. Fork the repository
599
- 2. Set up your development environment:
600
- ```bash
601
- bundle install
602
- bundle exec rake test # Ensure tests pass
603
- ```
604
- 3. Create your feature branch (`git checkout -b feature/amazing-feature`)
605
- 4. Make your changes with comprehensive tests
606
- 5. Test your changes:
607
- - Unit tests: `bundle exec rake test`
608
- - Code style: `bundle exec rubocop`
609
- - Security: `bundle exec bundler-audit check`
610
- 6. Commit your changes (`git commit -m 'Add amazing feature'`)
611
- 7. Push to the branch (`git push origin feature/amazing-feature`)
612
- 8. Open a Pull Request with a detailed description
613
-
614
- ### Contributing Custom Processors
615
-
616
- We especially welcome custom processors for different business domains. Consider contributing processors for:
617
-
618
- - E-commerce analytics (order tracking, inventory management)
619
- - Financial services (transaction monitoring, compliance reporting)
620
- - Healthcare (patient data tracking, regulatory compliance)
621
- - Education (student progress, course analytics)
622
- - SaaS metrics (user engagement, feature adoption)
623
-
624
- ## πŸ“‹ Requirements
625
-
626
- - **Ruby**: 3.1.0 or higher
627
- - **PostgreSQL**: 10.0 or higher (with logical replication enabled)
628
- - **MySQL/MariaDB**: 5.6+ (with binary logging enabled)
629
-
630
- ## πŸ—ΊοΈ Roadmap
631
-
632
- - [ ] **Prometheus Metrics**: Production monitoring integration (with complete codebase included in examples/)
633
- - [ ] **Advanced Example Apps**: Real-world use cases with complete monitoring stack (with complete codebase included in examples/)
634
- - [ ] **Custom Analytics Processors**: Business intelligence and real-time monitoring (with complete codebase included in examples/)
635
- - [x] **MySQL/MariaDB Support**: MySQL/MariaDB databases binlog streaming adapter
636
- - [ ] **Redis Streams**: Alternative lightweight streaming backend
637
- - [ ] **Compression**: Optional audit record compression
638
- - [ ] **Retention Policies**: Automated audit record cleanup
639
- - [ ] **Web UI**: Management interface for monitoring and configuration
640
-
641
- ## πŸ“š Documentation
642
-
643
- - **[API Documentation](https://kanutocd.github.io/whodunit-chronicles/)**
644
- - [ ] TODO: **Configuration Guide** _(docs/configuration-todo.md)_
645
- - [ ] TODO: **Architecture Deep Dive** _(docs/architecture-todo.md)_
646
- - [ ] TODO: **PostgreSQL Setup** _(docs/postgresql-setup-todo.md)_
647
- - [ ] TODO: **MySQL/MariaDB Setup** _(docs/mysql-setup.md)_
648
- - [ ] TODO: **Production Deployment** _(docs/production-todo.md)_
649
-
650
- ## πŸ“„ License
651
-
652
- This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
653
-
654
- ## πŸ™ Acknowledgments
137
+ ```yaml
138
+ ledger:
139
+ adapter: sqlite
140
+ path: chronicles.db
141
+ table_name: whodunit_chronicles_entries
142
+ ```
655
143
 
656
- - **PostgreSQL Team**: For excellent logical replication functionality
657
- - **Ruby Community**: For amazing gems and tools that make this possible
144
+ Status can be printed as human-readable lines or JSON:
658
145
 
659
- ---
146
+ ```bash
147
+ whodunit-chronicles ledger status config.yml
148
+ whodunit-chronicles ledger status config.yml --json
149
+ ```
660
150
 
661
- <div align="center">
151
+ ## Extension direction
662
152
 
663
- **[⭐ Star us on GitHub](https://github.com/kanutocd/whodunit-chronicles)** β€’ **[πŸ› Report Bug](https://github.com/kanutocd/whodunit-chronicles/issues)** β€’ **[πŸ’‘ Request Feature](https://github.com/kanutocd/whodunit-chronicles/issues)**
153
+ Future storage targets and compatibility adapters should be separate gems:
664
154
 
665
- Made with ❀️ by a Spherical Cow
155
+ ```text
156
+ whodunit-chronicles-postgres
157
+ whodunit-chronicles-mysql
158
+ whodunit-chronicles-mongo
159
+ whodunit-chronicles-s3
160
+ whodunit-chronicles-snowflake
161
+ whodunit-chronicles-paper_trail
162
+ ```
666
163
 
667
- </div>
164
+ Storage extensions implement the same `Ledger` contract. Compatibility adapters may translate ledger entries into another ecosystem's expected shape.