compare_logs 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +11 -0
- data/Gemfile +3 -0
- data/IMPLEMENTATION.md +114 -0
- data/LICENSE.txt +21 -0
- data/README.md +135 -0
- data/Rakefile +137 -0
- data/compare_logs.gemspec +33 -0
- data/exe/compare_logs +6 -0
- data/lib/compare_logs/version.rb +3 -0
- data/lib/compare_logs.rb +222 -0
- data/tasks.md +4 -0
- metadata +115 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 1231dd6a6387dab1b7f230036742c38c0315296d2f278679b0c6d973590bea37
|
|
4
|
+
data.tar.gz: '06117930de01a60aa13ae67ea7e735eafe50e8bfe5bf488af6dfb64a6896db5d'
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: c0830e159fc5516a68ed8fc73ef787b5b800c4d4b50a896ac4de0f5faea40abdc8962d5e18a3704e06a84a99581a65d1fec884e464fd4b1ddda2ab8c19f917ea
|
|
7
|
+
data.tar.gz: da1a0521ac0e78dc9b8592a052de95ffcda02f947d2d2ac1aa99de18dba3c5cd3630db80d50783735dd22c0daea6adc29d4b239f1d18293dd62ccc80b30b2d66
|
data/CHANGELOG.md
ADDED
data/Gemfile
ADDED
data/IMPLEMENTATION.md
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# Implementation Summary
|
|
2
|
+
|
|
3
|
+
All tasks from `tasks.md` have been completed:
|
|
4
|
+
|
|
5
|
+
## ✅ 1. Move substitutions to .env
|
|
6
|
+
|
|
7
|
+
- Added `dotenv` gem dependency to `compare_logs.gemspec`
|
|
8
|
+
- Refactored `lib/compare_logs.rb` to:
|
|
9
|
+
- Define `DEFAULT_SUBSTITUTIONS` constant with all hardcoded patterns
|
|
10
|
+
- Load custom substitutions from `.env` file via `COMPARE_LOGS_SUBSTITUTIONS`
|
|
11
|
+
- Support multiple custom patterns (comma-separated)
|
|
12
|
+
- Format: `pattern|||replacement`
|
|
13
|
+
- Created `.env.example` with documentation
|
|
14
|
+
|
|
15
|
+
**Usage:**
|
|
16
|
+
```bash
|
|
17
|
+
# In .env file
|
|
18
|
+
COMPARE_LOGS_SUBSTITUTIONS=C:\\Users\\buildkite-agent.*source.wxs|||BUILDKITEPATH,custom_pattern|||REPLACEMENT
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## ✅ 2. Add auto version bumping to Rakefile
|
|
22
|
+
|
|
23
|
+
Added rake tasks to `Rakefile`:
|
|
24
|
+
- `rake version:bump_patch` - Bumps x.y.Z version
|
|
25
|
+
- `rake version:bump_minor` - Bumps x.Y.0 version
|
|
26
|
+
- `rake version:bump_major` - Bumps X.0.0 version
|
|
27
|
+
|
|
28
|
+
Each task updates `lib/compare_logs/version.rb` automatically.
|
|
29
|
+
|
|
30
|
+
## ✅ 3. Add changelog generation to Rakefile
|
|
31
|
+
|
|
32
|
+
Added `rake changelog:generate` task to `Rakefile`:
|
|
33
|
+
- Reads git commits since last tag
|
|
34
|
+
- Categorizes commits by type (Added, Changed, Fixed)
|
|
35
|
+
- Updates `CHANGELOG.md` with new version entry
|
|
36
|
+
- Preserves existing changelog entries
|
|
37
|
+
|
|
38
|
+
## ✅ 4. Add GitHub Actions for auto-release
|
|
39
|
+
|
|
40
|
+
Created two workflow files:
|
|
41
|
+
|
|
42
|
+
### `.github/workflows/ci.yml`
|
|
43
|
+
- Runs on PRs and pushes to main
|
|
44
|
+
- Tests on Ruby 2.7, 3.0, 3.1, 3.2
|
|
45
|
+
- Builds gem artifact
|
|
46
|
+
|
|
47
|
+
### `.github/workflows/release.yml`
|
|
48
|
+
- Triggers on push to main branch
|
|
49
|
+
- Reads PR labels to determine version bump:
|
|
50
|
+
- `major` label → major version bump
|
|
51
|
+
- `minor` label → minor version bump
|
|
52
|
+
- No label → patch version bump (default)
|
|
53
|
+
- Automatically:
|
|
54
|
+
- Bumps version using rake task
|
|
55
|
+
- Generates changelog
|
|
56
|
+
- Commits changes with `[skip ci]`
|
|
57
|
+
- Creates GitHub release with gem file
|
|
58
|
+
- Publishes to RubyGems (requires `RUBYGEMS_API_KEY` secret)
|
|
59
|
+
|
|
60
|
+
## Next Steps
|
|
61
|
+
|
|
62
|
+
To use the GitHub Actions workflows:
|
|
63
|
+
|
|
64
|
+
1. Initialize git repository if not already done:
|
|
65
|
+
```bash
|
|
66
|
+
git init
|
|
67
|
+
git add .
|
|
68
|
+
git commit -m "Initial commit"
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
2. Create GitHub repository and push:
|
|
72
|
+
```bash
|
|
73
|
+
git remote add origin https://github.com/stringsn88keys/compare_logs.git
|
|
74
|
+
git push -u origin main
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
3. Add RubyGems API key as GitHub secret:
|
|
78
|
+
- Go to https://rubygems.org/profile/edit
|
|
79
|
+
- Create API key
|
|
80
|
+
- Add to GitHub repository secrets as `RUBYGEMS_API_KEY`
|
|
81
|
+
|
|
82
|
+
4. Create PRs with labels `minor` or `major` to control version bumps
|
|
83
|
+
|
|
84
|
+
## Testing Locally
|
|
85
|
+
|
|
86
|
+
1. Install dependencies:
|
|
87
|
+
```bash
|
|
88
|
+
bundle install
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
2. Test version bumping:
|
|
92
|
+
```bash
|
|
93
|
+
rake version:bump_patch
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
3. Test changelog generation (requires git repository):
|
|
97
|
+
```bash
|
|
98
|
+
git init
|
|
99
|
+
git add .
|
|
100
|
+
git commit -m "feat: initial version"
|
|
101
|
+
rake changelog:generate
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
4. Create `.env` file to test custom substitutions:
|
|
105
|
+
```bash
|
|
106
|
+
cp .env.example .env
|
|
107
|
+
# Edit .env and add custom patterns
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
5. Rebuild and reinstall gem:
|
|
111
|
+
```bash
|
|
112
|
+
gem build compare_logs.gemspec
|
|
113
|
+
gem install compare_logs-0.1.0.gem
|
|
114
|
+
```
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,135 @@
|
|
|
1
|
+
# CompareLogs
|
|
2
|
+
|
|
3
|
+
A Ruby gem to compare log files with intelligent normalization of timestamps, GUIDs, and other variable data.
|
|
4
|
+
|
|
5
|
+
## Installation
|
|
6
|
+
|
|
7
|
+
Install the gem:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
gem install compare_logs
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Or add it to your Gemfile:
|
|
14
|
+
|
|
15
|
+
```ruby
|
|
16
|
+
gem 'compare_logs'
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Then run:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
bundle install
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Usage
|
|
26
|
+
|
|
27
|
+
### Command Line
|
|
28
|
+
|
|
29
|
+
Compare the two most recent log files in your Downloads folder:
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
compare_logs
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Compare action logs for a specific platform:
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
compare_logs actions ubuntu-2004
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Compare specific files:
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
compare_logs file1.log file2.log
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Show help:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
compare_logs help
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
### As a Library
|
|
54
|
+
|
|
55
|
+
```ruby
|
|
56
|
+
require 'compare_logs'
|
|
57
|
+
|
|
58
|
+
comparator = CompareLogs::Comparator.new(['file1.log', 'file2.log'])
|
|
59
|
+
comparator.run
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
## What it Does
|
|
63
|
+
|
|
64
|
+
CompareLogs normalizes variable data in log files to make meaningful comparisons easier:
|
|
65
|
+
|
|
66
|
+
- Timestamps (various formats)
|
|
67
|
+
- GUIDs and GUID-like identifiers
|
|
68
|
+
- Build paths
|
|
69
|
+
- Timing measurements
|
|
70
|
+
- Directory names with timestamps
|
|
71
|
+
|
|
72
|
+
After normalization, it opens the files in `gvimdiff` for visual comparison.
|
|
73
|
+
|
|
74
|
+
## Custom Substitutions
|
|
75
|
+
|
|
76
|
+
You can add your own custom substitutions using a `.env` file:
|
|
77
|
+
|
|
78
|
+
1. Create a `.env` file in your current directory or the gem directory
|
|
79
|
+
2. Add custom patterns using the `COMPARE_LOGS_SUBSTITUTIONS` variable
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
# Format: pattern|||replacement (use ||| as separator)
|
|
83
|
+
COMPARE_LOGS_SUBSTITUTIONS=C:\\Users\\buildkite-agent.*source.wxs|||BUILDKITEPATH
|
|
84
|
+
|
|
85
|
+
# Multiple substitutions (comma-separated)
|
|
86
|
+
COMPARE_LOGS_SUBSTITUTIONS=pattern1|||replacement1,pattern2|||replacement2
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Custom substitutions are applied in addition to the default patterns.
|
|
90
|
+
|
|
91
|
+
## Development
|
|
92
|
+
|
|
93
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests.
|
|
94
|
+
|
|
95
|
+
To install this gem onto your local machine, run `bundle exec rake install`.
|
|
96
|
+
|
|
97
|
+
### Version Management
|
|
98
|
+
|
|
99
|
+
Bump versions using rake tasks:
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
rake version:bump_patch # x.y.Z
|
|
103
|
+
rake version:bump_minor # x.Y.0
|
|
104
|
+
rake version:bump_major # X.0.0
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Generate changelog from git commits:
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
rake changelog:generate
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
### GitHub Actions
|
|
114
|
+
|
|
115
|
+
This gem uses GitHub Actions for:
|
|
116
|
+
|
|
117
|
+
- **CI**: Tests on multiple Ruby versions (2.7, 3.0, 3.1, 3.2)
|
|
118
|
+
- **Auto Release**: On merge to `main`, automatically:
|
|
119
|
+
- Bumps version (patch by default, minor/major via PR labels)
|
|
120
|
+
- Updates changelog
|
|
121
|
+
- Creates GitHub release
|
|
122
|
+
- Publishes to RubyGems
|
|
123
|
+
|
|
124
|
+
To control version bumps, add labels to your PRs:
|
|
125
|
+
- `minor` - Bump minor version
|
|
126
|
+
- `major` - Bump major version
|
|
127
|
+
- No label - Bump patch version (default)
|
|
128
|
+
|
|
129
|
+
## Contributing
|
|
130
|
+
|
|
131
|
+
Bug reports and pull requests are welcome on GitHub.
|
|
132
|
+
|
|
133
|
+
## License
|
|
134
|
+
|
|
135
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
data/Rakefile
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
require "bundler/gem_tasks"
|
|
2
|
+
require "rspec/core/rake_task"
|
|
3
|
+
|
|
4
|
+
RSpec::Core::RakeTask.new(:spec)
|
|
5
|
+
|
|
6
|
+
task default: :spec
|
|
7
|
+
|
|
8
|
+
namespace :version do
|
|
9
|
+
desc "Bump patch version (x.y.Z)"
|
|
10
|
+
task :bump_patch do
|
|
11
|
+
bump_version(:patch)
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
desc "Bump minor version (x.Y.0)"
|
|
15
|
+
task :bump_minor do
|
|
16
|
+
bump_version(:minor)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
desc "Bump major version (X.0.0)"
|
|
20
|
+
task :bump_major do
|
|
21
|
+
bump_version(:major)
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
def bump_version(level)
|
|
25
|
+
version_file = "lib/compare_logs/version.rb"
|
|
26
|
+
content = File.read(version_file)
|
|
27
|
+
|
|
28
|
+
current_version = content.match(/VERSION = "(\d+)\.(\d+)\.(\d+)"/)[1..3].map(&:to_i)
|
|
29
|
+
major, minor, patch = current_version
|
|
30
|
+
|
|
31
|
+
new_version = case level
|
|
32
|
+
when :major
|
|
33
|
+
"#{major + 1}.0.0"
|
|
34
|
+
when :minor
|
|
35
|
+
"#{major}.#{minor + 1}.0"
|
|
36
|
+
when :patch
|
|
37
|
+
"#{major}.#{minor}.#{patch + 1}"
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
new_content = content.gsub(/VERSION = "\d+\.\d+\.\d+"/, "VERSION = \"#{new_version}\"")
|
|
41
|
+
File.write(version_file, new_content)
|
|
42
|
+
|
|
43
|
+
puts "Version bumped from #{current_version.join('.')} to #{new_version}"
|
|
44
|
+
puts "Don't forget to update CHANGELOG.md and commit the changes!"
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
namespace :changelog do
|
|
49
|
+
desc "Generate changelog from git commits"
|
|
50
|
+
task :generate do
|
|
51
|
+
generate_changelog
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
def generate_changelog
|
|
55
|
+
require 'time'
|
|
56
|
+
require_relative 'lib/compare_logs/version'
|
|
57
|
+
|
|
58
|
+
version = CompareLogs::VERSION
|
|
59
|
+
date = Time.now.strftime("%Y-%m-%d")
|
|
60
|
+
|
|
61
|
+
changelog_file = "CHANGELOG.md"
|
|
62
|
+
|
|
63
|
+
# Get commits since last tag (or all commits if no tags)
|
|
64
|
+
last_tag = `git describe --tags --abbrev=0 2>/dev/null`.strip
|
|
65
|
+
commit_range = last_tag.empty? ? "" : "#{last_tag}..HEAD"
|
|
66
|
+
|
|
67
|
+
commits = `git log #{commit_range} --pretty=format:"%s" --reverse`.split("\n")
|
|
68
|
+
|
|
69
|
+
if commits.empty?
|
|
70
|
+
puts "No commits found to add to changelog"
|
|
71
|
+
return
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# Categorize commits
|
|
75
|
+
added = []
|
|
76
|
+
changed = []
|
|
77
|
+
fixed = []
|
|
78
|
+
other = []
|
|
79
|
+
|
|
80
|
+
commits.each do |commit|
|
|
81
|
+
case commit
|
|
82
|
+
when /^(feat|add)/i
|
|
83
|
+
added << commit.gsub(/^(feat|add):?\s*/i, '')
|
|
84
|
+
when /^(fix|bugfix)/i
|
|
85
|
+
fixed << commit.gsub(/^(fix|bugfix):?\s*/i, '')
|
|
86
|
+
when /^(change|update|refactor)/i
|
|
87
|
+
changed << commit.gsub(/^(change|update|refactor):?\s*/i, '')
|
|
88
|
+
else
|
|
89
|
+
other << commit
|
|
90
|
+
end
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
# Build new entry
|
|
94
|
+
new_entry = "\n## [#{version}] - #{date}\n\n"
|
|
95
|
+
|
|
96
|
+
if added.any?
|
|
97
|
+
new_entry += "### Added\n"
|
|
98
|
+
added.each { |item| new_entry += "- #{item}\n" }
|
|
99
|
+
new_entry += "\n"
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
if changed.any?
|
|
103
|
+
new_entry += "### Changed\n"
|
|
104
|
+
changed.each { |item| new_entry += "- #{item}\n" }
|
|
105
|
+
new_entry += "\n"
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
if fixed.any?
|
|
109
|
+
new_entry += "### Fixed\n"
|
|
110
|
+
fixed.each { |item| new_entry += "- #{item}\n" }
|
|
111
|
+
new_entry += "\n"
|
|
112
|
+
end
|
|
113
|
+
|
|
114
|
+
if other.any?
|
|
115
|
+
new_entry += "### Other\n"
|
|
116
|
+
other.each { |item| new_entry += "- #{item}\n" }
|
|
117
|
+
new_entry += "\n"
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
# Read existing changelog
|
|
121
|
+
if File.exist?(changelog_file)
|
|
122
|
+
existing = File.read(changelog_file)
|
|
123
|
+
# Insert new entry after the # Changelog header
|
|
124
|
+
if existing =~ /^# Changelog\s*\n/
|
|
125
|
+
updated = existing.sub(/^# Changelog\s*\n/, "# Changelog\n#{new_entry}")
|
|
126
|
+
else
|
|
127
|
+
updated = "# Changelog\n#{new_entry}#{existing}"
|
|
128
|
+
end
|
|
129
|
+
else
|
|
130
|
+
updated = "# Changelog\n#{new_entry}"
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
File.write(changelog_file, updated)
|
|
134
|
+
puts "Changelog updated with version #{version}"
|
|
135
|
+
puts "Review CHANGELOG.md and commit the changes"
|
|
136
|
+
end
|
|
137
|
+
end
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
require_relative 'lib/compare_logs/version'
|
|
2
|
+
|
|
3
|
+
Gem::Specification.new do |spec|
|
|
4
|
+
spec.name = "compare_logs"
|
|
5
|
+
spec.version = CompareLogs::VERSION
|
|
6
|
+
spec.authors = ["Thomas Powell"]
|
|
7
|
+
spec.email = ["twilliampowell@gmail.com"]
|
|
8
|
+
|
|
9
|
+
spec.summary = "A tool to compare log files with timestamp normalization"
|
|
10
|
+
spec.description = "CompareLogs helps you compare log files by normalizing timestamps, GUIDs, and other variable data, then opening them in a diff viewer."
|
|
11
|
+
spec.homepage = "https://github.com/stringsn88keys/compare_logs"
|
|
12
|
+
spec.license = "MIT"
|
|
13
|
+
spec.required_ruby_version = ">= 2.6.0"
|
|
14
|
+
|
|
15
|
+
spec.metadata["homepage_uri"] = spec.homepage
|
|
16
|
+
spec.metadata["source_code_uri"] = "https://github.com/yourusername/compare_logs"
|
|
17
|
+
spec.metadata["changelog_uri"] = "https://github.com/yourusername/compare_logs/blob/main/CHANGELOG.md"
|
|
18
|
+
|
|
19
|
+
# Specify which files should be added to the gem when it is released.
|
|
20
|
+
spec.files = Dir["{lib,exe}/**/*", "*.md", "*.txt", "*.gemspec", "Gemfile", "Rakefile"]
|
|
21
|
+
|
|
22
|
+
spec.bindir = "exe"
|
|
23
|
+
spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
|
|
24
|
+
spec.require_paths = ["lib"]
|
|
25
|
+
|
|
26
|
+
# Runtime dependencies
|
|
27
|
+
spec.add_dependency "dotenv", "~> 2.8"
|
|
28
|
+
|
|
29
|
+
# Development dependencies
|
|
30
|
+
spec.add_development_dependency "bundler", "~> 2.0"
|
|
31
|
+
spec.add_development_dependency "rake", "~> 13.0"
|
|
32
|
+
spec.add_development_dependency "rspec", "~> 3.0"
|
|
33
|
+
end
|
data/exe/compare_logs
ADDED
data/lib/compare_logs.rb
ADDED
|
@@ -0,0 +1,222 @@
|
|
|
1
|
+
require_relative "compare_logs/version"
|
|
2
|
+
require "dotenv"
|
|
3
|
+
|
|
4
|
+
module CompareLogs
|
|
5
|
+
class Error < StandardError; end
|
|
6
|
+
|
|
7
|
+
class Comparator
|
|
8
|
+
attr_reader :files
|
|
9
|
+
|
|
10
|
+
# Default substitutions that are always applied
|
|
11
|
+
DEFAULT_SUBSTITUTIONS = [
|
|
12
|
+
[/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{7}/, 'BUILD_TIMESTAMP'],
|
|
13
|
+
[/\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{9} [-+]\d{4}/, 'INTERNAL_TIMESTAMP'],
|
|
14
|
+
[/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[-+]\d{2}:\d{2}/, 'INTERNAL_TIMESTAMP2'],
|
|
15
|
+
[/\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/, 'INTERNAL_TIMESTAMP3'],
|
|
16
|
+
[/\d{14}/, 'INTEGER_TIMESTAMP'],
|
|
17
|
+
[/\d{2}\/\d{2}\/\d{4} \d{1,2}:\d{2}:\d{2} [AP]M/, 'WINDOWS_TIMESTAMP'],
|
|
18
|
+
[/\d{14}\.\d{4}/, 'BUILDLOG_TIMESTAMP'],
|
|
19
|
+
[/ \d+\.\d+s/, ' SOME_TIMING'],
|
|
20
|
+
[/\h{8}-\h{4}-\h{4}-\h{4}-\h{12}/, 'GUID'],
|
|
21
|
+
[/\h{8}-\h{4}-\h{4}-\h{12}/, 'GUIDLIKE'],
|
|
22
|
+
[/\d{8}-\d{5}-[[:alnum:]]{5,6}\./, 'DIRNAME.']
|
|
23
|
+
].freeze
|
|
24
|
+
|
|
25
|
+
def initialize(args = [])
|
|
26
|
+
@args = args
|
|
27
|
+
@files = []
|
|
28
|
+
@aggressive = false
|
|
29
|
+
@ignore_versions = false
|
|
30
|
+
@ignore_timings = false
|
|
31
|
+
load_custom_substitutions
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def run
|
|
35
|
+
parse_arguments
|
|
36
|
+
normalize_files
|
|
37
|
+
if @aggressive
|
|
38
|
+
remove_common_lines
|
|
39
|
+
end
|
|
40
|
+
launch_diff_viewer
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
private
|
|
44
|
+
|
|
45
|
+
def parse_arguments
|
|
46
|
+
# Check for --aggressive flag
|
|
47
|
+
if @args.include?('--aggressive')
|
|
48
|
+
@aggressive = true
|
|
49
|
+
@args.delete('--aggressive')
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
# Check for --ignore-versions flag
|
|
53
|
+
if @args.include?('--ignore-versions')
|
|
54
|
+
@ignore_versions = true
|
|
55
|
+
@args.delete('--ignore-versions')
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
# Check for --ignore-timings flag
|
|
59
|
+
if @args.include?('--ignore-timings')
|
|
60
|
+
@ignore_timings = true
|
|
61
|
+
@args.delete('--ignore-timings')
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
if @args.empty? || @args.length > 2
|
|
65
|
+
print_usage
|
|
66
|
+
exit
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
if @args.first == "help"
|
|
70
|
+
print_usage
|
|
71
|
+
exit
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
@files = if @args.length <= 1
|
|
75
|
+
find_files_from_pattern
|
|
76
|
+
else
|
|
77
|
+
@args
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
if @files.empty?
|
|
81
|
+
puts "No files found to compare"
|
|
82
|
+
exit 1
|
|
83
|
+
end
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
def find_files_from_pattern
|
|
87
|
+
glob_string = case @args[0]
|
|
88
|
+
when /\*/
|
|
89
|
+
@args[0]
|
|
90
|
+
when /actions/
|
|
91
|
+
"#{ENV['HOME']}/Downloads/logs_*/*#{@args[1]}*.txt"
|
|
92
|
+
else
|
|
93
|
+
"#{ENV['HOME']}/Downloads/*.log"
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
Dir.glob(glob_string).sort { |a, b| File.ctime(a) <=> File.ctime(b) }.last(2).reverse
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
def normalize_files
|
|
100
|
+
@files.each do |file|
|
|
101
|
+
normalized_content = normalize_content(file)
|
|
102
|
+
File.open("#{file}.modified", 'wt') do |f|
|
|
103
|
+
f.puts normalized_content
|
|
104
|
+
end
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
def normalize_content(file)
|
|
109
|
+
# Try reading with UTF-8 first, fall back to UTF-16LE if that fails
|
|
110
|
+
content = begin
|
|
111
|
+
File.read(file, encoding: 'UTF-8')
|
|
112
|
+
rescue ArgumentError, Encoding::InvalidByteSequenceError, Encoding::UndefinedConversionError
|
|
113
|
+
begin
|
|
114
|
+
File.read(file, encoding: 'UTF-16LE:UTF-8')
|
|
115
|
+
rescue ArgumentError, Encoding::InvalidByteSequenceError, Encoding::UndefinedConversionError
|
|
116
|
+
# Last resort: read as binary and force UTF-8, replacing invalid bytes
|
|
117
|
+
File.read(file, encoding: 'binary').force_encoding('UTF-8').scrub('?')
|
|
118
|
+
end
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
# Ensure the content is valid UTF-8 before applying substitutions
|
|
122
|
+
content = content.encode('UTF-8', invalid: :replace, undef: :replace, replace: '?')
|
|
123
|
+
|
|
124
|
+
# Apply version substitution if --ignore-versions flag is set
|
|
125
|
+
if @ignore_versions
|
|
126
|
+
content = content.gsub(/\b\d+\.\d+\.\d+\b/, 'VERSION')
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
# Apply timing substitution if --ignore-timings flag is set
|
|
130
|
+
if @ignore_timings
|
|
131
|
+
# Match patterns like "1.234 seconds", "3.231322 minutes", "5 hours", etc.
|
|
132
|
+
content = content.gsub(/\b\d+\.?\d*\s+(seconds?|minutes?|hours?|milliseconds?|ms)\b/i, 'TIMING')
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
# Apply all substitutions (default + custom)
|
|
136
|
+
all_substitutions.each do |pattern, replacement|
|
|
137
|
+
content = content.gsub(pattern, replacement)
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
content
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
def all_substitutions
|
|
144
|
+
DEFAULT_SUBSTITUTIONS + @custom_substitutions
|
|
145
|
+
end
|
|
146
|
+
|
|
147
|
+
def load_custom_substitutions
|
|
148
|
+
@custom_substitutions = []
|
|
149
|
+
|
|
150
|
+
# Try to load .env file from current directory or gem root
|
|
151
|
+
[Dir.pwd, File.expand_path('../..', __dir__)].each do |dir|
|
|
152
|
+
env_file = File.join(dir, '.env')
|
|
153
|
+
if File.exist?(env_file)
|
|
154
|
+
Dotenv.load(env_file)
|
|
155
|
+
break
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
|
|
159
|
+
# Parse COMPARE_LOGS_SUBSTITUTIONS if present
|
|
160
|
+
if ENV['COMPARE_LOGS_SUBSTITUTIONS'] && !ENV['COMPARE_LOGS_SUBSTITUTIONS'].empty?
|
|
161
|
+
ENV['COMPARE_LOGS_SUBSTITUTIONS'].split(',').each do |sub|
|
|
162
|
+
pattern_str, replacement = sub.split('|||', 2)
|
|
163
|
+
next unless pattern_str && replacement
|
|
164
|
+
|
|
165
|
+
begin
|
|
166
|
+
pattern = Regexp.new(pattern_str.strip)
|
|
167
|
+
@custom_substitutions << [pattern, replacement.strip]
|
|
168
|
+
rescue RegexpError => e
|
|
169
|
+
warn "Invalid regex pattern '#{pattern_str}': #{e.message}"
|
|
170
|
+
end
|
|
171
|
+
end
|
|
172
|
+
end
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
def remove_common_lines
|
|
176
|
+
return unless @files.length == 2
|
|
177
|
+
|
|
178
|
+
# Read both modified files
|
|
179
|
+
file1_path = "#{@files[0]}.modified"
|
|
180
|
+
file2_path = "#{@files[1]}.modified"
|
|
181
|
+
|
|
182
|
+
lines1 = File.readlines(file1_path)
|
|
183
|
+
lines2 = File.readlines(file2_path)
|
|
184
|
+
|
|
185
|
+
# Find common lines (lines that exist in both files)
|
|
186
|
+
common_lines = lines1 & lines2
|
|
187
|
+
|
|
188
|
+
if common_lines.empty?
|
|
189
|
+
puts "No common lines found to remove"
|
|
190
|
+
return
|
|
191
|
+
end
|
|
192
|
+
|
|
193
|
+
# Remove common lines from both files
|
|
194
|
+
unique_lines1 = lines1 - common_lines
|
|
195
|
+
unique_lines2 = lines2 - common_lines
|
|
196
|
+
|
|
197
|
+
# Write back the filtered content
|
|
198
|
+
File.write(file1_path, unique_lines1.join)
|
|
199
|
+
File.write(file2_path, unique_lines2.join)
|
|
200
|
+
|
|
201
|
+
puts "Aggressive mode: Removed #{common_lines.length} common lines"
|
|
202
|
+
end
|
|
203
|
+
|
|
204
|
+
def launch_diff_viewer
|
|
205
|
+
modified_files = @files.map { |f| "\"#{f}.modified\"" }.join(' ')
|
|
206
|
+
system("gvimdiff #{modified_files}")
|
|
207
|
+
end
|
|
208
|
+
|
|
209
|
+
def print_usage
|
|
210
|
+
puts "Usage:"
|
|
211
|
+
puts " compare_logs [OPTIONS] actions ubuntu-2004"
|
|
212
|
+
puts " compare_logs [OPTIONS] file1.log file2.log"
|
|
213
|
+
puts " compare_logs [OPTIONS] # for Downloads/*.log"
|
|
214
|
+
puts " compare_logs help # show this help"
|
|
215
|
+
puts ""
|
|
216
|
+
puts "Options:"
|
|
217
|
+
puts " --aggressive Remove all lines that are identical in both files after normalization"
|
|
218
|
+
puts " --ignore-versions Normalize x.y.z version strings to VERSION"
|
|
219
|
+
puts " --ignore-timings Normalize timing strings (e.g., '1.234 seconds') to TIMING"
|
|
220
|
+
end
|
|
221
|
+
end
|
|
222
|
+
end
|
data/tasks.md
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
1
|
+
- [ ] move substitutions like .gsub(/C:\\Users\\buildkite-agent.*source.wxs/, 'BUILDKITEPATH') to COMPARE_LOGS_SUBSTITUTIONS in .env
|
|
2
|
+
- [ ] add auto version bumping to Rakefile
|
|
3
|
+
- [ ] add changelog generation to Rakefile
|
|
4
|
+
- [ ] add version bumping to github actions on merge to main, optional minor and major bumps based on labels
|
metadata
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: compare_logs
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.4.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Thomas Powell
|
|
8
|
+
autorequire:
|
|
9
|
+
bindir: exe
|
|
10
|
+
cert_chain: []
|
|
11
|
+
date: 2025-11-18 00:00:00.000000000 Z
|
|
12
|
+
dependencies:
|
|
13
|
+
- !ruby/object:Gem::Dependency
|
|
14
|
+
name: dotenv
|
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
|
16
|
+
requirements:
|
|
17
|
+
- - "~>"
|
|
18
|
+
- !ruby/object:Gem::Version
|
|
19
|
+
version: '2.8'
|
|
20
|
+
type: :runtime
|
|
21
|
+
prerelease: false
|
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
23
|
+
requirements:
|
|
24
|
+
- - "~>"
|
|
25
|
+
- !ruby/object:Gem::Version
|
|
26
|
+
version: '2.8'
|
|
27
|
+
- !ruby/object:Gem::Dependency
|
|
28
|
+
name: bundler
|
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
|
30
|
+
requirements:
|
|
31
|
+
- - "~>"
|
|
32
|
+
- !ruby/object:Gem::Version
|
|
33
|
+
version: '2.0'
|
|
34
|
+
type: :development
|
|
35
|
+
prerelease: false
|
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
37
|
+
requirements:
|
|
38
|
+
- - "~>"
|
|
39
|
+
- !ruby/object:Gem::Version
|
|
40
|
+
version: '2.0'
|
|
41
|
+
- !ruby/object:Gem::Dependency
|
|
42
|
+
name: rake
|
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
|
44
|
+
requirements:
|
|
45
|
+
- - "~>"
|
|
46
|
+
- !ruby/object:Gem::Version
|
|
47
|
+
version: '13.0'
|
|
48
|
+
type: :development
|
|
49
|
+
prerelease: false
|
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
51
|
+
requirements:
|
|
52
|
+
- - "~>"
|
|
53
|
+
- !ruby/object:Gem::Version
|
|
54
|
+
version: '13.0'
|
|
55
|
+
- !ruby/object:Gem::Dependency
|
|
56
|
+
name: rspec
|
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
|
58
|
+
requirements:
|
|
59
|
+
- - "~>"
|
|
60
|
+
- !ruby/object:Gem::Version
|
|
61
|
+
version: '3.0'
|
|
62
|
+
type: :development
|
|
63
|
+
prerelease: false
|
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
65
|
+
requirements:
|
|
66
|
+
- - "~>"
|
|
67
|
+
- !ruby/object:Gem::Version
|
|
68
|
+
version: '3.0'
|
|
69
|
+
description: CompareLogs helps you compare log files by normalizing timestamps, GUIDs,
|
|
70
|
+
and other variable data, then opening them in a diff viewer.
|
|
71
|
+
email:
|
|
72
|
+
- twilliampowell@gmail.com
|
|
73
|
+
executables:
|
|
74
|
+
- compare_logs
|
|
75
|
+
extensions: []
|
|
76
|
+
extra_rdoc_files: []
|
|
77
|
+
files:
|
|
78
|
+
- CHANGELOG.md
|
|
79
|
+
- Gemfile
|
|
80
|
+
- IMPLEMENTATION.md
|
|
81
|
+
- LICENSE.txt
|
|
82
|
+
- README.md
|
|
83
|
+
- Rakefile
|
|
84
|
+
- compare_logs.gemspec
|
|
85
|
+
- exe/compare_logs
|
|
86
|
+
- lib/compare_logs.rb
|
|
87
|
+
- lib/compare_logs/version.rb
|
|
88
|
+
- tasks.md
|
|
89
|
+
homepage: https://github.com/stringsn88keys/compare_logs
|
|
90
|
+
licenses:
|
|
91
|
+
- MIT
|
|
92
|
+
metadata:
|
|
93
|
+
homepage_uri: https://github.com/stringsn88keys/compare_logs
|
|
94
|
+
source_code_uri: https://github.com/yourusername/compare_logs
|
|
95
|
+
changelog_uri: https://github.com/yourusername/compare_logs/blob/main/CHANGELOG.md
|
|
96
|
+
post_install_message:
|
|
97
|
+
rdoc_options: []
|
|
98
|
+
require_paths:
|
|
99
|
+
- lib
|
|
100
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
101
|
+
requirements:
|
|
102
|
+
- - ">="
|
|
103
|
+
- !ruby/object:Gem::Version
|
|
104
|
+
version: 2.6.0
|
|
105
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
106
|
+
requirements:
|
|
107
|
+
- - ">="
|
|
108
|
+
- !ruby/object:Gem::Version
|
|
109
|
+
version: '0'
|
|
110
|
+
requirements: []
|
|
111
|
+
rubygems_version: 3.4.10
|
|
112
|
+
signing_key:
|
|
113
|
+
specification_version: 4
|
|
114
|
+
summary: A tool to compare log files with timestamp normalization
|
|
115
|
+
test_files: []
|