git-pkgs 0.1.0

data/README.md

@@ -0,0 +1,279 @@
+# git-pkgs
+
+A git subcommand for tracking package dependencies across git history. Analyzes your repository to show when dependencies were added, modified, or removed, who made those changes, and why.
+
+## Why this exists
+
+Your lockfile shows what dependencies you have. It doesn't show how you got here. `git log Gemfile.lock` is useless noise.
+
+git-pkgs indexes your dependency history into a queryable database. You can ask: when did we add this? who added it? what changed between these two releases? has anyone touched this in the last year?
+
+It works across ecosystems. Gemfile, package.json, Dockerfile, GitHub Actions workflows - one unified history instead of separate tools per ecosystem.
+
+## Installation
+
+```bash
+gem install git-pkgs
+```
+
+## Quick start
+
+```bash
+cd your-repo
+git pkgs init    # analyze history (one-time, ~300 commits/sec)
+git pkgs stats   # see overview
+git pkgs blame   # who added each dependency
+git pkgs history rails  # track a package over time
+```
+
+## Commands
+
+### Initialize the database
+
+```bash
+git pkgs init
+```
+
+Walks through git history and builds a SQLite database of dependency changes, stored in `.git/pkgs.sqlite3`.
+
+Options:
+- `--branch=NAME` - analyze a specific branch (default: default branch)
+- `--since=SHA` - start analysis from a specific commit
+- `--force` - rebuild the database from scratch
+- `--hooks` - install git hooks for auto-updating
+
+Example output:
+```
+Analyzing branch: main
+Processing commit 5191/5191...
+Done!
+Analyzed 5191 commits
+Found 2531 commits with dependency changes
+Stored 28239 snapshots (every 20 changes)
+Blob cache: 3141 unique blobs, 2349 had cache hits
+```
+
+### Database info
+
+```bash
+git pkgs info
+```
+
+Shows database size and row counts:
+
+```
+Database Info
+========================================
+
+Location: /path/to/repo/.git/pkgs.sqlite3
+Size: 8.3 MB
+
+Row Counts
+----------------------------------------
+  Branches                        1
+  Commits                      3988
+  Branch-Commits               3988
+  Manifests                       9
+  Dependency Changes           4732
+  Dependency Snapshots        28239
+  ----------------------------------
+  Total                       40957
+
+Snapshot Coverage
+----------------------------------------
+  Commits with dependency changes: 2531
+  Commits with snapshots: 127
+  Coverage: 5.0% (1 snapshot per ~20 changes)
+```
+
+### List dependencies
+
+```bash
+git pkgs list
+git pkgs list --commit=abc123
+git pkgs list --ecosystem=rubygems
+```
+
+Example output:
+```
+Gemfile (rubygems):
+  bootsnap >= 0 [runtime]
+  bootstrap = 4.6.2 [runtime]
+  bugsnag >= 0 [runtime]
+  rails = 8.0.1 [runtime]
+  sidekiq >= 0 [runtime]
+  ...
+```
+
+### View package history
+
+```bash
+git pkgs history rails
+```
+
+Shows when the package was added, version changes, and removal:
+
+```
+History for rails:
+
+2016-12-16 Added = 5.0.0.1
+  Commit: e323669 Hello World
+  Author: Andrew Nesbitt <andrew@example.com>
+  Manifest: Gemfile
+
+2016-12-21 Updated = 5.0.0.1 -> = 5.0.1
+  Commit: 0c70eee Update rails to 5.0.1
+  Author: Andrew Nesbitt <andrew@example.com>
+  Manifest: Gemfile
+
+2024-11-21 Updated = 7.2.2 -> = 8.0.0
+  Commit: 86a07f4 Upgrade to Rails 8
+  Author: Andrew Nesbitt <andrew@example.com>
+  Manifest: Gemfile
+```
+
+### Blame
+
+Show who added each current dependency:
+
+```bash
+git pkgs blame
+git pkgs blame --ecosystem=rubygems
+```
+
+Example output:
+```
+Gemfile (rubygems):
+  bootsnap                        Andrew Nesbitt     2018-04-10  7da4369
+  bootstrap                       Andrew Nesbitt     2018-08-02  0b39dc0
+  bugsnag                         Andrew Nesbitt     2016-12-23  a87f1bf
+  factory_bot                     Lewis Buckley      2017-12-25  f6cceb0
+  faraday                         Andrew Nesbitt     2021-11-25  98de229
+  jwt                             Andrew Nesbitt     2018-09-10  a39f0ea
+  octokit                         Andrew Nesbitt     2016-12-16  e323669
+  omniauth-rails_csrf_protection  dependabot[bot]    2021-11-02  02474ab
+  rails                           Andrew Nesbitt     2016-12-16  e323669
+  sidekiq                         Mark Tareshawty    2018-02-19  29a1c70
+```
+
+### Show statistics
+
+```bash
+git pkgs stats
+```
+
+Example output:
+```
+Dependency Statistics
+========================================
+
+Branch: main
+Commits analyzed: 3988
+Commits with changes: 2531
+
+Current Dependencies
+--------------------
+Total: 250
+  rubygems: 232
+  actions: 14
+  docker: 4
+
+Dependency Changes
+--------------------
+Total changes: 4732
+  added: 391
+  modified: 4200
+  removed: 141
+
+Most Changed Dependencies
+-------------------------
+  rails (rubygems): 135 changes
+  pagy (rubygems): 116 changes
+  nokogiri (rubygems): 85 changes
+  puma (rubygems): 73 changes
+
+Manifest Files
+--------------
+  Gemfile (rubygems): 294 changes
+  Gemfile.lock (rubygems): 4269 changes
+  .github/workflows/ci.yml (actions): 36 changes
+```
+
+### Explain why a dependency exists
+
+```bash
+git pkgs why rails
+```
+
+Shows the commit that added the dependency with author and message.
+
+### Dependency tree
+
+```bash
+git pkgs tree
+git pkgs tree --ecosystem=rubygems
+```
+
+Shows dependencies grouped by type (runtime, development, etc).
+
+### Diff between commits
+
+```bash
+git pkgs diff --from=abc123 --to=def456
+git pkgs diff --from=HEAD~10
+```
+
+Shows added, removed, and modified packages with version info.
+
+### Keep database updated
+
+```bash
+git pkgs update
+```
+
+Or install git hooks to update automatically after commits and merges:
+
+```bash
+git pkgs hooks --install
+```
+
+## Performance
+
+Benchmarked on a MacBook Pro analyzing [octobox](https://github.com/octobox/octobox) (5191 commits, 8 years of history): init takes about 18 seconds at roughly 300 commits/sec, producing an 8.3 MB database. About half the commits (2531) had dependency changes.
+
+Optimizations:
+- Bulk inserts with transaction batching (100 commits per transaction)
+- Blob SHA caching (75% cache hit rate for repeated manifest content)
+- Deferred index creation during bulk load
+- Sparse snapshots (every 20 dependency-changing commits) for storage efficiency
+- SQLite WAL mode for write performance
+
+## Supported ecosystems
+
+git-pkgs uses [ecosystems-bibliothecary](https://github.com/ecosyste-ms/bibliothecary) for parsing, supporting:
+
+Actions, Anaconda, BentoML, Bower, Cargo, CocoaPods, Cog, CPAN, CRAN, CycloneDX, Docker, Dub, DVC, Elm, Go, Haxelib, Homebrew, Julia, Maven, Meteor, MLflow, npm, NuGet, Ollama, Packagist, Pub, PyPI, RubyGems, Shards, SPDX, Vcpkg
+
+## How it works
+
+git-pkgs walks your git history, extracts dependency files at each commit, and diffs them to detect changes. Results are stored in a SQLite database for fast querying.
+
+The database schema stores:
+- Commits with dependency changes
+- Dependency changes (added/modified/removed) with before/after versions
+- Periodic snapshots of full dependency state for efficient point-in-time queries
+
+See [docs/schema.md](docs/schema.md) for full schema documentation.
+
+## Development
+
+```bash
+git clone https://github.com/andrew/git-pkgs
+cd git-pkgs
+bin/setup
+rake test
+```
+
+## License
+
+AGPL-3.0

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "minitest/test_task"
+
+Minitest::TestTask.create
+
+task default: :test

data/benchmark_bulk.rb

@@ -0,0 +1,167 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "git/pkgs"
+require "benchmark"
+
+repo_path = ARGV[0] || "/Users/andrew/code/octobox"
+sample_size = (ARGV[1] || 500).to_i
+
+# In-memory with WAL mode equivalent (journal_mode=memory for in-memory DB)
+Git::Pkgs::Database.connect_memory
+ActiveRecord::Base.connection.execute("PRAGMA synchronous = OFF")
+ActiveRecord::Base.connection.execute("PRAGMA journal_mode = MEMORY")
+
+repo = Git::Pkgs::Repository.new(repo_path)
+analyzer = Git::Pkgs::Analyzer.new(repo)
+
+walker = repo.walk(repo.default_branch)
+commits = walker.take(sample_size)
+
+puts "Bulk insert benchmark: #{commits.size} commits"
+puts "=" * 60
+
+# Pre-collect all data
+all_commits = []
+all_branch_commits = []
+all_changes = []
+all_snapshots = []
+
+snapshot = {}
+branch = Git::Pkgs::Models::Branch.find_or_create("main")
+position = 0
+manifests_cache = {}
+
+now = Time.now
+
+collect_time = Benchmark.realtime do
+  commits.each do |rugged_commit|
+    next if repo.merge_commit?(rugged_commit)
+    position += 1
+
+    result = analyzer.analyze_commit(rugged_commit, snapshot)
+
+    all_commits << {
+      sha: rugged_commit.oid,
+      message: rugged_commit.message,
+      author_name: rugged_commit.author[:name],
+      author_email: rugged_commit.author[:email],
+      committed_at: rugged_commit.time,
+      has_dependency_changes: result && result[:changes].any?,
+      created_at: now,
+      updated_at: now
+    }
+
+    all_branch_commits << {
+      branch_id: branch.id,
+      commit_position: position,  # placeholder, need to resolve after commit insert
+      commit_sha: rugged_commit.oid
+    }
+
+    next unless result && result[:changes].any?
+
+    result[:changes].each do |change|
+      manifest_key = change[:manifest_path]
+      unless manifests_cache[manifest_key]
+        manifests_cache[manifest_key] = Git::Pkgs::Models::Manifest.find_or_create(
+          path: change[:manifest_path],
+          platform: change[:platform],
+          kind: change[:kind]
+        )
+      end
+
+      all_changes << {
+        commit_sha: rugged_commit.oid,
+        manifest_path: manifest_key,
+        name: change[:name],
+        platform: change[:platform],
+        change_type: change[:change_type],
+        requirement: change[:requirement],
+        previous_requirement: change[:previous_requirement],
+        dependency_type: change[:dependency_type],
+        created_at: now,
+        updated_at: now
+      }
+    end
+
+    snapshot = result[:snapshot]
+
+    snapshot.each do |(manifest_path, name), dep_info|
+      all_snapshots << {
+        commit_sha: rugged_commit.oid,
+        manifest_path: manifest_path,
+        name: name,
+        platform: dep_info[:platform],
+        requirement: dep_info[:requirement],
+        dependency_type: dep_info[:dependency_type],
+        created_at: now,
+        updated_at: now
+      }
+    end
+  end
+end
+
+puts "Collection time: #{collect_time.round(3)}s"
+puts "Data collected:"
+puts "  Commits: #{all_commits.size}"
+puts "  Changes: #{all_changes.size}"
+puts "  Snapshots: #{all_snapshots.size}"
+
+# Bulk insert
+insert_time = Benchmark.realtime do
+  # Insert commits
+  Git::Pkgs::Models::Commit.insert_all(all_commits) if all_commits.any?
+
+  # Build SHA -> ID map
+  commit_ids = Git::Pkgs::Models::Commit.where(sha: all_commits.map { |c| c[:sha] }).pluck(:sha, :id).to_h
+  manifest_ids = Git::Pkgs::Models::Manifest.pluck(:path, :id).to_h
+
+  # Insert branch_commits with resolved IDs
+  branch_commit_records = all_branch_commits.map do |bc|
+    {
+      branch_id: bc[:branch_id],
+      commit_id: commit_ids[bc[:commit_sha]],
+      position: bc[:commit_position]
+    }
+  end
+  Git::Pkgs::Models::BranchCommit.insert_all(branch_commit_records) if branch_commit_records.any?
+
+  # Insert changes with resolved IDs
+  change_records = all_changes.map do |c|
+    {
+      commit_id: commit_ids[c[:commit_sha]],
+      manifest_id: manifest_ids[c[:manifest_path]],
+      name: c[:name],
+      platform: c[:platform],
+      change_type: c[:change_type],
+      requirement: c[:requirement],
+      previous_requirement: c[:previous_requirement],
+      dependency_type: c[:dependency_type],
+      created_at: c[:created_at],
+      updated_at: c[:updated_at]
+    }
+  end
+  Git::Pkgs::Models::DependencyChange.insert_all(change_records) if change_records.any?
+
+  # Insert snapshots with resolved IDs
+  snapshot_records = all_snapshots.map do |s|
+    {
+      commit_id: commit_ids[s[:commit_sha]],
+      manifest_id: manifest_ids[s[:manifest_path]],
+      name: s[:name],
+      platform: s[:platform],
+      requirement: s[:requirement],
+      dependency_type: s[:dependency_type],
+      created_at: s[:created_at],
+      updated_at: s[:updated_at]
+    }
+  end
+  Git::Pkgs::Models::DependencySnapshot.insert_all(snapshot_records) if snapshot_records.any?
+end
+
+puts "Insert time: #{insert_time.round(3)}s"
+
+total = collect_time + insert_time
+puts "\nTotal: #{total.round(3)}s"
+puts "Throughput: #{(all_commits.size / total).round(1)} commits/sec"

data/benchmark_db.rb

@@ -0,0 +1,138 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "git/pkgs"
+require "benchmark"
+
+repo_path = ARGV[0] || "/Users/andrew/code/octobox"
+sample_size = (ARGV[1] || 200).to_i
+
+Git::Pkgs::Database.connect_memory
+
+repo = Git::Pkgs::Repository.new(repo_path)
+analyzer = Git::Pkgs::Analyzer.new(repo)
+
+walker = repo.walk(repo.default_branch)
+commits = walker.take(sample_size)
+
+puts "DB operation breakdown: #{commits.size} commits"
+puts "=" * 60
+
+timings = {
+  commit_create: 0.0,
+  branch_commit_create: 0.0,
+  commit_update: 0.0,
+  manifest_find_create: 0.0,
+  change_create: 0.0,
+  snapshot_create: 0.0
+}
+
+counts = {
+  commits: 0,
+  branch_commits: 0,
+  changes: 0,
+  snapshots: 0
+}
+
+snapshot = {}
+branch = Git::Pkgs::Models::Branch.find_or_create("main")
+position = 0
+
+commits.each do |rugged_commit|
+  next if repo.merge_commit?(rugged_commit)
+  position += 1
+
+  result = analyzer.analyze_commit(rugged_commit, snapshot)
+
+  commit = nil
+  timings[:commit_create] += Benchmark.realtime do
+    commit = Git::Pkgs::Models::Commit.find_or_create_from_rugged(rugged_commit)
+  end
+  counts[:commits] += 1
+
+  timings[:branch_commit_create] += Benchmark.realtime do
+    Git::Pkgs::Models::BranchCommit.find_or_create_by(
+      branch: branch,
+      commit: commit,
+      position: position
+    )
+  end
+  counts[:branch_commits] += 1
+
+  next unless result && result[:changes].any?
+
+  timings[:commit_update] += Benchmark.realtime do
+    commit.update(has_dependency_changes: true)
+  end
+
+  result[:changes].each do |change|
+    manifest = nil
+    timings[:manifest_find_create] += Benchmark.realtime do
+      manifest = Git::Pkgs::Models::Manifest.find_or_create(
+        path: change[:manifest_path],
+        platform: change[:platform],
+        kind: change[:kind]
+      )
+    end
+
+    timings[:change_create] += Benchmark.realtime do
+      Git::Pkgs::Models::DependencyChange.create!(
+        commit: commit,
+        manifest: manifest,
+        name: change[:name],
+        platform: change[:platform],
+        change_type: change[:change_type],
+        requirement: change[:requirement],
+        previous_requirement: change[:previous_requirement],
+        dependency_type: change[:dependency_type]
+      )
+    end
+    counts[:changes] += 1
+  end
+
+  snapshot = result[:snapshot]
+
+  snapshot.each do |(manifest_path, name), dep_info|
+    timings[:snapshot_create] += Benchmark.realtime do
+      manifest = Git::Pkgs::Models::Manifest.find_by(path: manifest_path)
+      Git::Pkgs::Models::DependencySnapshot.find_or_create_by(
+        commit: commit,
+        manifest: manifest,
+        name: name
+      ) do |s|
+        s.platform = dep_info[:platform]
+        s.requirement = dep_info[:requirement]
+        s.dependency_type = dep_info[:dependency_type]
+      end
+    end
+    counts[:snapshots] += 1
+  end
+end
+
+total = timings.values.sum
+
+puts "\nDB operation breakdown:"
+puts "-" * 60
+timings.each do |op, time|
+  pct = total > 0 ? (time / total * 100).round(1) : 0
+  puts "  #{op.to_s.ljust(22)} #{time.round(3).to_s.rjust(8)}s  (#{pct}%)"
+end
+puts "-" * 60
+puts "  #{'Total'.ljust(22)} #{total.round(3).to_s.rjust(8)}s"
+
+puts "\nRecord counts:"
+puts "  Commits:        #{counts[:commits]}"
+puts "  BranchCommits:  #{counts[:branch_commits]}"
+puts "  Changes:        #{counts[:changes]}"
+puts "  Snapshots:      #{counts[:snapshots]}"
+
+puts "\nPer-operation averages:"
+puts "  commit_create:        #{(timings[:commit_create] / counts[:commits] * 1000).round(3)}ms"
+puts "  branch_commit_create: #{(timings[:branch_commit_create] / counts[:branch_commits] * 1000).round(3)}ms"
+if counts[:changes] > 0
+  puts "  change_create:        #{(timings[:change_create] / counts[:changes] * 1000).round(3)}ms"
+end
+if counts[:snapshots] > 0
+  puts "  snapshot_create:      #{(timings[:snapshot_create] / counts[:snapshots] * 1000).round(3)}ms"
+end