whodunit-chronicles 0.1.0.pre
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.rubocop.yml +92 -0
- data/.yardopts +12 -0
- data/CHANGELOG.md +106 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/LICENSE +21 -0
- data/README.md +281 -0
- data/Rakefile +18 -0
- data/lib/.gitkeep +0 -0
- data/lib/whodunit/chronicles/adapters/postgresql.rb +278 -0
- data/lib/whodunit/chronicles/audit_processor.rb +270 -0
- data/lib/whodunit/chronicles/change_event.rb +201 -0
- data/lib/whodunit/chronicles/configuration.rb +101 -0
- data/lib/whodunit/chronicles/service.rb +205 -0
- data/lib/whodunit/chronicles/stream_adapter.rb +91 -0
- data/lib/whodunit/chronicles/version.rb +7 -0
- data/lib/whodunit/chronicles.rb +69 -0
- data/lib/whodunit-chronicles.rb +4 -0
- data/whodunit-chronicles.gemspec +61 -0
- metadata +290 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 527e3470b4156a28e1972948cc79cbe4ce2ee72ec7582692434e98611c5e0071
|
4
|
+
data.tar.gz: 51302c24757877422d8ec97f894aa4af06561d9d53fa60ac65f6bc94d4b89705
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 5ce050e09a6f0407c2d5c64b2196910f84834fc17408a3477fd082fbab04a346fb69dbc43d28fbd1d8868f25915206c6af6b9ac1ffe52700d7c0b2e9e46916b9
|
7
|
+
data.tar.gz: cea9a584c9fb99306a6cbcbdd8afc87f23f14db4ec6e90f96c8364006defb869ce3abd69fb72a02a4e4a79ce9144494e96c47dacca27e9ef669f4bb0ff44558a
|
data/.rubocop.yml
ADDED
@@ -0,0 +1,92 @@
|
|
1
|
+
# Ruby version
|
2
|
+
AllCops:
|
3
|
+
TargetRubyVersion: 3.1
|
4
|
+
NewCops: enable
|
5
|
+
Exclude:
|
6
|
+
- 'vendor/**/*'
|
7
|
+
- 'tmp/**/*'
|
8
|
+
- 'bin/**/*'
|
9
|
+
- 'node_modules/**/*'
|
10
|
+
|
11
|
+
# Use plugins for extensions
|
12
|
+
plugins:
|
13
|
+
- rubocop-performance
|
14
|
+
- rubocop-minitest
|
15
|
+
|
16
|
+
# Layout cops
|
17
|
+
Layout/LineLength:
|
18
|
+
Max: 120
|
19
|
+
AllowedPatterns: ['\A\s*#']
|
20
|
+
|
21
|
+
Layout/MultilineMethodCallBraceLayout:
|
22
|
+
Enabled: false
|
23
|
+
|
24
|
+
Layout/ArgumentAlignment:
|
25
|
+
EnforcedStyle: with_fixed_indentation
|
26
|
+
|
27
|
+
# Metrics cops - relax for complex business logic
|
28
|
+
Metrics/ClassLength:
|
29
|
+
Max: 250 # Allow larger classes for database adapters and processors
|
30
|
+
Exclude:
|
31
|
+
- 'test/**/*' # Test classes can be longer
|
32
|
+
|
33
|
+
Metrics/MethodLength:
|
34
|
+
Max: 50 # Allow longer methods for SQL generation and complex logic
|
35
|
+
|
36
|
+
Metrics/AbcSize:
|
37
|
+
Max: 40 # Allow higher complexity for database operations
|
38
|
+
|
39
|
+
Metrics/CyclomaticComplexity:
|
40
|
+
Max: 15 # Allow more complex conditional logic
|
41
|
+
|
42
|
+
Metrics/PerceivedComplexity:
|
43
|
+
Max: 15
|
44
|
+
|
45
|
+
Metrics/ParameterLists:
|
46
|
+
Max: 12 # Allow more params for configuration objects and event initialization
|
47
|
+
|
48
|
+
# Style cops
|
49
|
+
Style/Documentation:
|
50
|
+
Enabled: false # We have YARD docs
|
51
|
+
|
52
|
+
Style/FrozenStringLiteralComment:
|
53
|
+
Enabled: true
|
54
|
+
EnforcedStyle: always
|
55
|
+
|
56
|
+
Style/StringLiterals:
|
57
|
+
EnforcedStyle: single_quotes
|
58
|
+
|
59
|
+
Style/HashSyntax:
|
60
|
+
EnforcedStyle: ruby19
|
61
|
+
|
62
|
+
Style/TrailingCommaInArguments:
|
63
|
+
EnforcedStyleForMultiline: comma
|
64
|
+
|
65
|
+
Style/TrailingCommaInArrayLiteral:
|
66
|
+
EnforcedStyleForMultiline: comma
|
67
|
+
|
68
|
+
Style/TrailingCommaInHashLiteral:
|
69
|
+
EnforcedStyleForMultiline: comma
|
70
|
+
|
71
|
+
# Naming
|
72
|
+
Naming/FileName:
|
73
|
+
Exclude:
|
74
|
+
- 'lib/whodunit-chronicles.rb' # Main gem file
|
75
|
+
|
76
|
+
# Gemspec
|
77
|
+
Gemspec/DevelopmentDependencies:
|
78
|
+
Enabled: false # We're fine with dev dependencies in gemspec
|
79
|
+
|
80
|
+
# Minitest specific cops
|
81
|
+
Minitest/MultipleAssertions:
|
82
|
+
Max: 10 # Allow more assertions in comprehensive tests
|
83
|
+
|
84
|
+
Minitest/AssertNil:
|
85
|
+
Enabled: false # Prefer assert_nil over assert_equal(nil, ...)
|
86
|
+
|
87
|
+
# Performance cops
|
88
|
+
Performance/Casecmp:
|
89
|
+
Enabled: true
|
90
|
+
|
91
|
+
Performance/StringReplacement:
|
92
|
+
Enabled: true
|
data/.yardopts
ADDED
data/CHANGELOG.md
ADDED
@@ -0,0 +1,106 @@
|
|
1
|
+
# Changelog
|
2
|
+
|
3
|
+
All notable changes to this project will be documented in this file.
|
4
|
+
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
7
|
+
|
8
|
+
## [Unreleased]
|
9
|
+
|
10
|
+
### Added
|
11
|
+
|
12
|
+
- Comprehensive GitHub Actions CI/CD pipeline with multi-Ruby testing
|
13
|
+
- Automated security scanning with bundler-audit and CodeQL
|
14
|
+
- YARD documentation generation with GitHub Pages deployment
|
15
|
+
- Automated gem publishing workflow for tagged releases
|
16
|
+
- Structured issue and pull request templates for better contributor experience
|
17
|
+
- Dependabot configuration for automated dependency updates
|
18
|
+
- Security scanning dependencies (bundler-audit, brakeman)
|
19
|
+
- Comprehensive README with architecture diagrams and examples
|
20
|
+
- RuboCop configuration with relaxed metrics for test files
|
21
|
+
|
22
|
+
### Changed
|
23
|
+
|
24
|
+
- Updated dependencies to latest versions with security patches
|
25
|
+
- Improved RuboCop configuration to exclude test files from ClassLength limits
|
26
|
+
- Enhanced gemspec with proper metadata and security dependencies
|
27
|
+
|
28
|
+
## [0.1.0] - 2025-01-21
|
29
|
+
|
30
|
+
### Added
|
31
|
+
|
32
|
+
- **Core Architecture**: Complete zero-latency audit streaming implementation
|
33
|
+
- **PostgreSQL Adapter**: Logical replication streaming with WAL decoding
|
34
|
+
- **ChangeEvent System**: Unified change representation across database adapters
|
35
|
+
- **AuditProcessor**: Intelligent transformation of changes into audit records
|
36
|
+
- **Configuration Management**: Comprehensive settings with validation using dry-configurable
|
37
|
+
- **Service Orchestration**: Thread-safe service with error handling and retry logic
|
38
|
+
- **Abstract Adapter Pattern**: Extensible design supporting multiple database systems
|
39
|
+
- **User Attribution**: Automatic extraction of user information from creator/updater/deleter fields
|
40
|
+
- **Batch Processing**: Efficient bulk processing of audit records
|
41
|
+
- **Connection Management**: Robust database connection handling with retries
|
42
|
+
- **Error Handling**: Comprehensive error recovery and logging
|
43
|
+
- **Thread Safety**: Concurrent processing with thread pool management
|
44
|
+
|
45
|
+
### Technical Implementation
|
46
|
+
|
47
|
+
- **Modern Ruby Support**: Ruby 3.1+ with frozen string literals
|
48
|
+
- **Database Features**:
|
49
|
+
- PostgreSQL logical replication with pgoutput plugin
|
50
|
+
- Publication and replication slot management
|
51
|
+
- WAL position tracking and resumption
|
52
|
+
- Connection pooling and management
|
53
|
+
- **Event Processing**:
|
54
|
+
- Real-time change capture at database level
|
55
|
+
- Structured change events with metadata
|
56
|
+
- Configurable table and schema filtering
|
57
|
+
- Transaction ID and sequence number tracking
|
58
|
+
- **Audit Records**:
|
59
|
+
- Complete before/after data capture
|
60
|
+
- Calculated field-level changes
|
61
|
+
- User attribution from standard columns
|
62
|
+
- Timestamps and version tracking
|
63
|
+
- JSON metadata storage
|
64
|
+
|
65
|
+
### Development & Testing
|
66
|
+
|
67
|
+
- **Test Coverage**: 94.3% code coverage (447/474 lines)
|
68
|
+
- **Testing Framework**: Minitest with shoulda-style assertions and mocha mocking
|
69
|
+
- **Test Suite**: 129 tests covering all components with comprehensive error scenarios
|
70
|
+
- **Code Quality**: RuboCop compliance with modern Ruby standards
|
71
|
+
- **Security**: bundler-audit integration for vulnerability scanning
|
72
|
+
- **Documentation**: Inline YARD documentation for all public APIs
|
73
|
+
|
74
|
+
### Dependencies
|
75
|
+
|
76
|
+
- **Core**: concurrent-ruby, dry-configurable, dry-logger, pg
|
77
|
+
- **Development**: minitest, rubocop, simplecov, yard
|
78
|
+
- **Testing**: mocha for mocking, pry for debugging
|
79
|
+
- **Security**: bundler-audit for vulnerability scanning
|
80
|
+
|
81
|
+
### Configuration Options
|
82
|
+
|
83
|
+
- Database connection URLs (source and audit)
|
84
|
+
- PostgreSQL publication and replication slot names
|
85
|
+
- Batch processing sizes and retry policies
|
86
|
+
- Table and schema inclusion/exclusion filters
|
87
|
+
- Logging levels and output configuration
|
88
|
+
- Thread pool and concurrency settings
|
89
|
+
|
90
|
+
### Performance Features
|
91
|
+
|
92
|
+
- **Zero Application Overhead**: No Rails callbacks or Active Record hooks
|
93
|
+
- **Efficient Streaming**: PostgreSQL logical replication optimization
|
94
|
+
- **Memory Management**: Streaming processing without dataset loading
|
95
|
+
- **Configurable Batching**: Tunable batch sizes for optimal throughput
|
96
|
+
- **Connection Reuse**: Persistent connections with automatic recovery
|
97
|
+
- **Background Processing**: Non-blocking operation with thread pools
|
98
|
+
|
99
|
+
### Production Ready Features
|
100
|
+
|
101
|
+
- **Monitoring**: Service status and health checking
|
102
|
+
- **Resilience**: Automatic retry logic with exponential backoff
|
103
|
+
- **Graceful Shutdown**: Clean service termination
|
104
|
+
- **Error Recovery**: Robust error handling with detailed logging
|
105
|
+
- **Position Tracking**: WAL position persistence for reliable resumption
|
106
|
+
- **Resource Management**: Bounded thread pools and memory usage
|
data/CODE_OF_CONDUCT.md
ADDED
@@ -0,0 +1,132 @@
|
|
1
|
+
# Contributor Covenant Code of Conduct
|
2
|
+
|
3
|
+
## Our Pledge
|
4
|
+
|
5
|
+
We as members, contributors, and leaders pledge to make participation in our
|
6
|
+
community a harassment-free experience for everyone, regardless of age, body
|
7
|
+
size, visible or invisible disability, ethnicity, sex characteristics, gender
|
8
|
+
identity and expression, level of experience, education, socio-economic status,
|
9
|
+
nationality, personal appearance, race, caste, color, religion, or sexual
|
10
|
+
identity and orientation.
|
11
|
+
|
12
|
+
We pledge to act and interact in ways that contribute to an open, welcoming,
|
13
|
+
diverse, inclusive, and healthy community.
|
14
|
+
|
15
|
+
## Our Standards
|
16
|
+
|
17
|
+
Examples of behavior that contributes to a positive environment for our
|
18
|
+
community include:
|
19
|
+
|
20
|
+
* Demonstrating empathy and kindness toward other people
|
21
|
+
* Being respectful of differing opinions, viewpoints, and experiences
|
22
|
+
* Giving and gracefully accepting constructive feedback
|
23
|
+
* Accepting responsibility and apologizing to those affected by our mistakes,
|
24
|
+
and learning from the experience
|
25
|
+
* Focusing on what is best not just for us as individuals, but for the overall
|
26
|
+
community
|
27
|
+
|
28
|
+
Examples of unacceptable behavior include:
|
29
|
+
|
30
|
+
* The use of sexualized language or imagery, and sexual attention or advances of
|
31
|
+
any kind
|
32
|
+
* Trolling, insulting or derogatory comments, and personal or political attacks
|
33
|
+
* Public or private harassment
|
34
|
+
* Publishing others' private information, such as a physical or email address,
|
35
|
+
without their explicit permission
|
36
|
+
* Other conduct which could reasonably be considered inappropriate in a
|
37
|
+
professional setting
|
38
|
+
|
39
|
+
## Enforcement Responsibilities
|
40
|
+
|
41
|
+
Community leaders are responsible for clarifying and enforcing our standards of
|
42
|
+
acceptable behavior and will take appropriate and fair corrective action in
|
43
|
+
response to any behavior that they deem inappropriate, threatening, offensive,
|
44
|
+
or harmful.
|
45
|
+
|
46
|
+
Community leaders have the right and responsibility to remove, edit, or reject
|
47
|
+
comments, commits, code, wiki edits, issues, and other contributions that are
|
48
|
+
not aligned to this Code of Conduct, and will communicate reasons for moderation
|
49
|
+
decisions when appropriate.
|
50
|
+
|
51
|
+
## Scope
|
52
|
+
|
53
|
+
This Code of Conduct applies within all community spaces, and also applies when
|
54
|
+
an individual is officially representing the community in public spaces.
|
55
|
+
Examples of representing our community include using an official email address,
|
56
|
+
posting via an official social media account, or acting as an appointed
|
57
|
+
representative at an online or offline event.
|
58
|
+
|
59
|
+
## Enforcement
|
60
|
+
|
61
|
+
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
62
|
+
reported to the community leaders responsible for enforcement at
|
63
|
+
[INSERT CONTACT METHOD].
|
64
|
+
All complaints will be reviewed and investigated promptly and fairly.
|
65
|
+
|
66
|
+
All community leaders are obligated to respect the privacy and security of the
|
67
|
+
reporter of any incident.
|
68
|
+
|
69
|
+
## Enforcement Guidelines
|
70
|
+
|
71
|
+
Community leaders will follow these Community Impact Guidelines in determining
|
72
|
+
the consequences for any action they deem in violation of this Code of Conduct:
|
73
|
+
|
74
|
+
### 1. Correction
|
75
|
+
|
76
|
+
**Community Impact**: Use of inappropriate language or other behavior deemed
|
77
|
+
unprofessional or unwelcome in the community.
|
78
|
+
|
79
|
+
**Consequence**: A private, written warning from community leaders, providing
|
80
|
+
clarity around the nature of the violation and an explanation of why the
|
81
|
+
behavior was inappropriate. A public apology may be requested.
|
82
|
+
|
83
|
+
### 2. Warning
|
84
|
+
|
85
|
+
**Community Impact**: A violation through a single incident or series of
|
86
|
+
actions.
|
87
|
+
|
88
|
+
**Consequence**: A warning with consequences for continued behavior. No
|
89
|
+
interaction with the people involved, including unsolicited interaction with
|
90
|
+
those enforcing the Code of Conduct, for a specified period of time. This
|
91
|
+
includes avoiding interactions in community spaces as well as external channels
|
92
|
+
like social media. Violating these terms may lead to a temporary or permanent
|
93
|
+
ban.
|
94
|
+
|
95
|
+
### 3. Temporary Ban
|
96
|
+
|
97
|
+
**Community Impact**: A serious violation of community standards, including
|
98
|
+
sustained inappropriate behavior.
|
99
|
+
|
100
|
+
**Consequence**: A temporary ban from any sort of interaction or public
|
101
|
+
communication with the community for a specified period of time. No public or
|
102
|
+
private interaction with the people involved, including unsolicited interaction
|
103
|
+
with those enforcing the Code of Conduct, is allowed during this period.
|
104
|
+
Violating these terms may lead to a permanent ban.
|
105
|
+
|
106
|
+
### 4. Permanent Ban
|
107
|
+
|
108
|
+
**Community Impact**: Demonstrating a pattern of violation of community
|
109
|
+
standards, including sustained inappropriate behavior, harassment of an
|
110
|
+
individual, or aggression toward or disparagement of classes of individuals.
|
111
|
+
|
112
|
+
**Consequence**: A permanent ban from any sort of public interaction within the
|
113
|
+
community.
|
114
|
+
|
115
|
+
## Attribution
|
116
|
+
|
117
|
+
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
|
118
|
+
version 2.1, available at
|
119
|
+
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
|
120
|
+
|
121
|
+
Community Impact Guidelines were inspired by
|
122
|
+
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
|
123
|
+
|
124
|
+
For answers to common questions about this code of conduct, see the FAQ at
|
125
|
+
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
|
126
|
+
[https://www.contributor-covenant.org/translations][translations].
|
127
|
+
|
128
|
+
[homepage]: https://www.contributor-covenant.org
|
129
|
+
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
|
130
|
+
[Mozilla CoC]: https://github.com/mozilla/diversity
|
131
|
+
[FAQ]: https://www.contributor-covenant.org/faq
|
132
|
+
[translations]: https://www.contributor-covenant.org/translations
|
data/LICENSE
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
MIT License
|
2
|
+
|
3
|
+
Copyright (c) 2025 Ken C. Demanawa
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
13
|
+
copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,281 @@
|
|
1
|
+
# π Whodunit Chronicles
|
2
|
+
|
3
|
+
[](https://badge.fury.io/rb/whodunit-chronicles)
|
4
|
+
[](https://github.com/kanutocd/whodunit-chronicles/actions)
|
5
|
+
[](https://codecov.io/gh/kanutocd/whodunit-chronicles)
|
6
|
+
[](https://www.ruby-lang.org/en/)
|
7
|
+
[](https://opensource.org/licenses/MIT)
|
8
|
+
|
9
|
+
> **The complete historical record of your _Whodunit Dun Wat?_ data**
|
10
|
+
|
11
|
+
While [Whodunit](https://github.com/kanutocd/whodunit) tracks _who_ made changes, **Chronicles** captures _what_ changed by streaming database events into comprehensive audit trails with **zero Rails application overhead**.
|
12
|
+
|
13
|
+
## β¨ Features
|
14
|
+
|
15
|
+
- **π Zero-Latency Streaming**: PostgreSQL logical replication
|
16
|
+
- **π Zero Application Overhead**: No Rails callbacks or Active Record hooks required
|
17
|
+
- **ποΈ Database Agnostic**: Abstract adapter pattern supports PostgreSQL (TODO: MySQL/MariaDB support)
|
18
|
+
- **β‘ Thread-Safe**: Concurrent processing with configurable thread pools
|
19
|
+
- **π‘οΈ Resilient**: Built-in error handling, retry logic, and monitoring
|
20
|
+
- **π Complete Audit Trail**: Captures INSERT, UPDATE, DELETE with full before/after data
|
21
|
+
- **π§ͺ VERY Soon to be Production Ready**: 94%+ test coverage with comprehensive error scenarios
|
22
|
+
|
23
|
+
## π Quick Start
|
24
|
+
|
25
|
+
### Installation
|
26
|
+
|
27
|
+
Add to your Gemfile:
|
28
|
+
|
29
|
+
```ruby
|
30
|
+
gem 'whodunit-chronicles'
|
31
|
+
```
|
32
|
+
|
33
|
+
Or install directly:
|
34
|
+
|
35
|
+
```bash
|
36
|
+
gem install whodunit-chronicles
|
37
|
+
```
|
38
|
+
|
39
|
+
### Basic Usage
|
40
|
+
|
41
|
+
```ruby
|
42
|
+
require 'whodunit/chronicles'
|
43
|
+
|
44
|
+
# PostgreSQL Configuration
|
45
|
+
Whodunit::Chronicles.configure do |config|
|
46
|
+
config.adapter = :postgresql
|
47
|
+
config.database_url = 'postgresql://localhost/myapp_production'
|
48
|
+
config.audit_database_url = 'postgresql://localhost/myapp'
|
49
|
+
config.publication_name = 'myapp_chronicles'
|
50
|
+
config.replication_slot_name = 'myapp_chronicles_slot'
|
51
|
+
end
|
52
|
+
|
53
|
+
# Create and start the service
|
54
|
+
service = Whodunit::Chronicles.service
|
55
|
+
service.setup! # Create publication/replication setup
|
56
|
+
service.start # Begin streaming changes
|
57
|
+
|
58
|
+
# Service runs in background threads
|
59
|
+
sleep 10
|
60
|
+
|
61
|
+
# Stop gracefully
|
62
|
+
service.stop
|
63
|
+
service.teardown! # Clean up database objects
|
64
|
+
```
|
65
|
+
|
66
|
+
## ποΈ Architecture
|
67
|
+
|
68
|
+
Chronicles uses **PostgreSQL logical replication** (TODO: **MySQL/MariaDB binary log streaming**) to capture database changes without impacting your application:
|
69
|
+
|
70
|
+
```
|
71
|
+
βββββββββββββββββββ ββββββββββββββββββββ βββββββββββββββββββ
|
72
|
+
β Application β β Chronicles β β Audit Store β
|
73
|
+
β Database βββββΆβ Service βββββΆβ Database β
|
74
|
+
β β β β β β
|
75
|
+
β β’ Users β β β’ Stream Adapter β β β’ audit_records β
|
76
|
+
β β’ Posts β β β’ Event Parser β β β’ Searchable β
|
77
|
+
β β’ Comments β β β’ Audit Builder β β β’ Reportable β
|
78
|
+
βββββββββββββββββββ ββββββββββββββββββββ βββββββββββββββββββ
|
79
|
+
β β
|
80
|
+
β ββββββββββΌβββββββββ
|
81
|
+
β β PostgreSQL β
|
82
|
+
ββββββββββββββββ Logical β
|
83
|
+
β Replication β
|
84
|
+
βββββββββββββββββββ
|
85
|
+
```
|
86
|
+
|
87
|
+
### Core Components
|
88
|
+
|
89
|
+
- **StreamAdapter**: Database-specific change streaming (PostgreSQL, MySQL/MariaDB)
|
90
|
+
- **ChangeEvent**: Unified change representation across adapters
|
91
|
+
- **AuditProcessor**: Transforms changes into searchable audit records
|
92
|
+
- **Service**: Orchestrates streaming with error handling and retry logic
|
93
|
+
|
94
|
+
## βοΈ Configuration
|
95
|
+
|
96
|
+
```ruby
|
97
|
+
Whodunit::Chronicles.configure do |config|
|
98
|
+
# Database connections
|
99
|
+
config.database_url = ENV['DATABASE_URL']
|
100
|
+
config.audit_database_url = ENV['AUDIT_DATABASE_URL']
|
101
|
+
|
102
|
+
# Database adapter (postgresql, mysql, mariadb)
|
103
|
+
config.adapter = :postgresql
|
104
|
+
|
105
|
+
# PostgreSQL-specific settings
|
106
|
+
config.publication_name = 'whodunit_chronicles'
|
107
|
+
config.replication_slot_name = 'whodunit_chronicles_slot'
|
108
|
+
|
109
|
+
# Performance tuning
|
110
|
+
config.batch_size = 1000
|
111
|
+
config.max_retry_attempts = 5
|
112
|
+
config.retry_delay = 10
|
113
|
+
|
114
|
+
# Table filtering
|
115
|
+
config.include_tables = %w[users posts comments]
|
116
|
+
config.exclude_tables = %w[sessions temp_data]
|
117
|
+
config.include_schemas = %w[public app]
|
118
|
+
config.exclude_schemas = %w[information_schema pg_catalog]
|
119
|
+
end
|
120
|
+
```
|
121
|
+
|
122
|
+
## π Audit Records
|
123
|
+
|
124
|
+
Chronicles creates structured audit records for each database change:
|
125
|
+
|
126
|
+
```ruby
|
127
|
+
{
|
128
|
+
id: 123,
|
129
|
+
table_name: "users",
|
130
|
+
schema_name: "public",
|
131
|
+
record_id: { "id" => 456 },
|
132
|
+
action: "UPDATE",
|
133
|
+
old_data: { "id" => 456, "email" => "old@example.com", "name" => "Old Name" },
|
134
|
+
new_data: { "id" => 456, "email" => "new@example.com", "name" => "New Name" },
|
135
|
+
changes: { "email" => ["old@example.com", "new@example.com"] },
|
136
|
+
user_id: 789, # From creator_id/updater_id/deleter_id columns
|
137
|
+
user_type: "User",
|
138
|
+
transaction_id: "tx_abc123",
|
139
|
+
sequence_number: 42,
|
140
|
+
occurred_at: 2025-01-21 10:30:00 UTC,
|
141
|
+
created_at: 2025-01-21 10:30:01 UTC,
|
142
|
+
metadata: {
|
143
|
+
table_schema: "public",
|
144
|
+
qualified_table_name: "public.users",
|
145
|
+
changed_columns: ["email"],
|
146
|
+
chronicles_version: "0.1.0"
|
147
|
+
}
|
148
|
+
}
|
149
|
+
```
|
150
|
+
|
151
|
+
## π§ Advanced Usage
|
152
|
+
|
153
|
+
### Custom Audit Processing
|
154
|
+
|
155
|
+
```ruby
|
156
|
+
class MyCustomProcessor < Whodunit::Chronicles::AuditProcessor
|
157
|
+
def build_chronicles_record(change_event)
|
158
|
+
super.tap do |record|
|
159
|
+
record[:custom_field] = extract_custom_data(change_event)
|
160
|
+
record[:environment] = Rails.env
|
161
|
+
end
|
162
|
+
end
|
163
|
+
|
164
|
+
private
|
165
|
+
|
166
|
+
def extract_custom_data(change_event)
|
167
|
+
# Your custom logic here
|
168
|
+
end
|
169
|
+
end
|
170
|
+
|
171
|
+
# Use custom processor
|
172
|
+
service = Whodunit::Chronicles::Service.new(
|
173
|
+
processor: MyCustomProcessor.new
|
174
|
+
)
|
175
|
+
```
|
176
|
+
|
177
|
+
### Service Monitoring
|
178
|
+
|
179
|
+
```ruby
|
180
|
+
service = Whodunit::Chronicles.service
|
181
|
+
|
182
|
+
# Check service status
|
183
|
+
status = service.status
|
184
|
+
puts "Running: #{status[:running]}"
|
185
|
+
puts "Adapter Position: #{status[:adapter_position]}"
|
186
|
+
puts "Retry Count: #{status[:retry_count]}"
|
187
|
+
puts "Active Threads: #{status[:executor_status][:active_count]}"
|
188
|
+
|
189
|
+
# Monitor in production
|
190
|
+
Thread.new do
|
191
|
+
loop do
|
192
|
+
status = service.status
|
193
|
+
Rails.logger.info "Chronicles Status: #{status}"
|
194
|
+
sleep 60
|
195
|
+
end
|
196
|
+
end
|
197
|
+
```
|
198
|
+
|
199
|
+
## π§ͺ Testing
|
200
|
+
|
201
|
+
Chronicles includes comprehensive test coverage:
|
202
|
+
|
203
|
+
```bash
|
204
|
+
# Run test suite
|
205
|
+
bundle exec rake test
|
206
|
+
|
207
|
+
# Run with coverage
|
208
|
+
bundle exec rake test
|
209
|
+
open coverage/index.html
|
210
|
+
|
211
|
+
# Security scanning
|
212
|
+
bundle exec bundler-audit check
|
213
|
+
bundle exec brakeman
|
214
|
+
```
|
215
|
+
|
216
|
+
## π Performance
|
217
|
+
|
218
|
+
- **Minimal Overhead**: No Rails callback performance impact
|
219
|
+
- **Efficient Streaming**: PostgreSQL logical replication is highly optimized
|
220
|
+
- **Configurable Batching**: Process changes in configurable batch sizes
|
221
|
+
- **Thread Pool**: Concurrent processing with bounded resource usage
|
222
|
+
- **Memory Efficient**: Streaming processing without loading full datasets
|
223
|
+
|
224
|
+
## π‘οΈ Security
|
225
|
+
|
226
|
+
- **Dependency Scanning**: Automated bundler-audit checks
|
227
|
+
- **Code Analysis**: GitHub CodeQL integration
|
228
|
+
- **Vulnerability Monitoring**: Weekly security scans
|
229
|
+
- **Safe Defaults**: Secure configuration out of the box
|
230
|
+
|
231
|
+
## π€ Contributing
|
232
|
+
|
233
|
+
1. Fork the repository
|
234
|
+
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
|
235
|
+
3. Make your changes with tests
|
236
|
+
4. Ensure tests pass (`bundle exec rake test`)
|
237
|
+
5. Ensure RuboCop passes (`bundle exec rubocop`)
|
238
|
+
6. Commit your changes (`git commit -m 'Add amazing feature'`)
|
239
|
+
7. Push to the branch (`git push origin feature/amazing-feature`)
|
240
|
+
8. Open a Pull Request
|
241
|
+
|
242
|
+
## π Requirements
|
243
|
+
|
244
|
+
- **Ruby**: 3.1.0 or higher
|
245
|
+
- **PostgreSQL**: 10.0 or higher (with logical replication enabled)
|
246
|
+
|
247
|
+
## πΊοΈ Roadmap
|
248
|
+
|
249
|
+
- [ ] **MySQL/MariaDB Support**: MySQL/MariaDB databases binlog streaming adapter
|
250
|
+
- [ ] **Redis Streams**: Alternative lightweight streaming backend
|
251
|
+
- [ ] **Compression**: Optional audit record compression
|
252
|
+
- [ ] **Retention Policies**: Automated audit record cleanup
|
253
|
+
- [ ] **Web UI**: Management interface for monitoring and configuration
|
254
|
+
- [ ] **Prometheus Metrics**: Production monitoring integration
|
255
|
+
|
256
|
+
## π Documentation
|
257
|
+
|
258
|
+
- **[API Documentation](https://kanutocd.github.io/whodunit-chronicles/)**
|
259
|
+
- **[Configuration Guide](docs/configuration-todo.md)**
|
260
|
+
- **[Architecture Deep Dive](docs/architecture-todo.md)**
|
261
|
+
- **[PostgreSQL Setup](docs/postgresql-setup-todo.md)**
|
262
|
+
- **[Production Deployment](docs/production-todo.md)**
|
263
|
+
|
264
|
+
## π License
|
265
|
+
|
266
|
+
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
267
|
+
|
268
|
+
## π Acknowledgments
|
269
|
+
|
270
|
+
- **PostgreSQL Team**: For excellent logical replication functionality
|
271
|
+
- **Ruby Community**: For amazing gems and tools that make this possible
|
272
|
+
|
273
|
+
---
|
274
|
+
|
275
|
+
<div align="center">
|
276
|
+
|
277
|
+
**[β Star us on GitHub](https://github.com/kanutocd/whodunit-chronicles)** β’ **[π Report Bug](https://github.com/kanutocd/whodunit-chronicles/issues)** β’ **[π‘ Request Feature](https://github.com/kanutocd/whodunit-chronicles/issues)**
|
278
|
+
|
279
|
+
Made with β€οΈ by a Spherical Cow
|
280
|
+
|
281
|
+
</div>
|
data/Rakefile
ADDED
@@ -0,0 +1,18 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'bundler/gem_tasks'
|
4
|
+
require 'rake/testtask'
|
5
|
+
require 'rubocop/rake_task'
|
6
|
+
|
7
|
+
Rake::TestTask.new(:test) do |t|
|
8
|
+
t.libs << 'test'
|
9
|
+
t.libs << 'lib'
|
10
|
+
t.test_files = FileList['test/**/*_test.rb']
|
11
|
+
end
|
12
|
+
|
13
|
+
RuboCop::RakeTask.new
|
14
|
+
|
15
|
+
desc 'Run all checks (rubocop, tests)'
|
16
|
+
task check: %i[rubocop test]
|
17
|
+
|
18
|
+
task default: :check
|
data/lib/.gitkeep
ADDED
File without changes
|