gemvault 0.1.5 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.containerignore +10 -0
- data/.rubocop.yml +9 -1
- data/CHANGELOG.md +61 -0
- data/CLAUDE.md +70 -11
- data/Dockerfile.test +65 -10
- data/README.md +3 -11
- data/Rakefile +26 -2
- data/_claude/archives/issue-13-root-cause.md +197 -0
- data/docs/Rakefile.vault_example +76 -0
- data/issues.rec +614 -15
- data/lib/bundler/plugin/vault_source.rb +48 -73
- data/lib/bundler/plugin/vaulted_gem.rb +66 -0
- data/lib/gemvault/archive_entry.rb +6 -0
- data/lib/gemvault/cli/command.rb +9 -5
- data/lib/gemvault/cli/commands/extract.rb +11 -11
- data/lib/gemvault/cli/commands/new.rb +3 -2
- data/lib/gemvault/cli/commands/remove.rb +15 -5
- data/lib/gemvault/cli/commands/upgrade.rb +37 -13
- data/lib/gemvault/dbvault.rb +12 -9
- data/lib/gemvault/deprecation.rb +26 -24
- data/lib/gemvault/gem_entry.rb +20 -24
- data/lib/gemvault/gem_extraction.rb +19 -23
- data/lib/gemvault/gem_reference/any_version.rb +6 -2
- data/lib/gemvault/gem_reference/parser.rb +44 -0
- data/lib/gemvault/gem_reference/specific_version.rb +9 -25
- data/lib/gemvault/gem_reference.rb +13 -45
- data/lib/gemvault/manifest.rb +44 -42
- data/lib/gemvault/tarball.rb +53 -0
- data/lib/gemvault/tarvault.rb +47 -56
- data/lib/gemvault/vault.rb +17 -13
- data/lib/gemvault/vault_path.rb +33 -0
- data/lib/gemvault/vault_upgrade.rb +31 -24
- data/lib/gemvault/version.rb +1 -1
- data/lib/gemvault.rb +1 -0
- data/lib/rubygems/resolver/vault_set.rb +5 -7
- data/lib/rubygems/source/vault.rb +28 -32
- data/lib/rubygems_plugin.rb +8 -6
- metadata +23 -17
- data/LICENSE +0 -21
- data/lib/gemvault/tar_archive.rb +0 -66
- /data/{ASSESSMENT.md → _claude/archives/ASSESSMENT.md} +0 -0
- /data/{MACROPLAN.md → _claude/archives/MACROPLAN.md} +0 -0
- /data/{docs/superpowers → _claude/archives}/plans/2026-04-16-container-integration-tests.md +0 -0
- /data/{docs → _claude/archives}/tarvault-findings.md +0 -0
- /data/{docs → _claude/archives}/tarvault.md +0 -0
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 6a1d5d0c990e232ac54514f5837db4224d92b127175f370a3443c8d15d44099f
|
|
4
|
+
data.tar.gz: ab40029793c2dd05206b9812e27da8921eb7fc478164f71fa7485296d5455f15
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 916b9a114590c9eb5fa3a377d59e9639c33a28e948429b8bac18399473eba0dcacf6711a9808106e3f91348945d71ebd051c1f65fe05239f9804289f56fcf859
|
|
7
|
+
data.tar.gz: 9ea36ee67195ca7f7a8bafd7ebc8658e000b72bcbf72356e3a2266c2f8d2679e0d8f5212f0b977a36a36c95441152f2dabae8ada78c5831770c0cd3498c4253d
|
data/.containerignore
ADDED
data/.rubocop.yml
CHANGED
|
@@ -17,10 +17,11 @@ plugins:
|
|
|
17
17
|
|
|
18
18
|
AllCops:
|
|
19
19
|
NewCops: enable
|
|
20
|
-
TargetRubyVersion: 4.
|
|
20
|
+
TargetRubyVersion: 3.4.8
|
|
21
21
|
Exclude:
|
|
22
22
|
- bin/*
|
|
23
23
|
- vendor/**/*
|
|
24
|
+
- references/**/*
|
|
24
25
|
- lib/core_ext/**/*
|
|
25
26
|
- rakelib/project.rb
|
|
26
27
|
- rakelib/project_version.rb
|
|
@@ -137,6 +138,13 @@ Style/ArgumentsForwarding:
|
|
|
137
138
|
Style/RedundantBegin:
|
|
138
139
|
Enabled: false
|
|
139
140
|
|
|
141
|
+
# Data value objects that carry constants, nested value types, or any class
|
|
142
|
+
# body read best as `class X < Data.define(...)`. The assignment form the cop
|
|
143
|
+
# prefers leaks block-level constants to the enclosing scope and forces a
|
|
144
|
+
# `.freeze` (Style/MutableConstant) on the class. Permitted by David.
|
|
145
|
+
Style/DataInheritance:
|
|
146
|
+
Enabled: false
|
|
147
|
+
|
|
140
148
|
# Compact class names are fine for small files and tests.
|
|
141
149
|
Style/ClassAndModuleChildren:
|
|
142
150
|
Enabled: false
|
data/CHANGELOG.md
CHANGED
|
@@ -19,10 +19,71 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
19
19
|
preserving every gem and timestamp, writing a `.bak` backup by default, with
|
|
20
20
|
`--dry-run` and `--no-backup` flags. It is a no-op on an already-current vault.
|
|
21
21
|
|
|
22
|
+
- All CLI commands accept `vault://` and `file://` locators wherever they take
|
|
23
|
+
a vault path, e.g. `gemvault list vault:///path/to/myvault.gemv`; resolution
|
|
24
|
+
is shared with the RubyGems source via `Gemvault::VaultPath` (issue #9).
|
|
25
|
+
|
|
26
|
+
### Fixed
|
|
27
|
+
- `bundle plugin install bundler-source-vault` no longer dies with
|
|
28
|
+
`LoadError: cannot load such file -- bundler/plugin/vault_source`. The shim's
|
|
29
|
+
`plugins.rb` now derives the gem root from its own installed location (local
|
|
30
|
+
plugin root, global plugin root, or plain GEM_HOME), registers any spec dirs
|
|
31
|
+
RubyGems is not searching, and resets RubyGems' spec stub cache when it
|
|
32
|
+
predates the just-installed gems (issue #10).
|
|
33
|
+
- `bundle exec` no longer fails Gemfile parsing with `Could not find 'bundler'
|
|
34
|
+
(>= 2.0)` under `path: vendor` on rubies that ship bundler as a regular gem:
|
|
35
|
+
gemvault no longer declares `bundler` as a runtime dependency (issue #12).
|
|
36
|
+
- Vault gems whose versions carry a non-numeric suffix (e.g. `0.2.1.patch1`)
|
|
37
|
+
install from a vault with `gem install --pre`, matching RubyGems' prerelease
|
|
38
|
+
semantics (issue #6).
|
|
39
|
+
- Repeated `bundle install` no longer fails with `cannot load such file --
|
|
40
|
+
bundler/plugin/vault_source` on machines that have gemvault installed
|
|
41
|
+
system-wide. Bundler skips installing a plugin dependency already present on
|
|
42
|
+
the ambient GEM_PATH, so the plugin root holds only the shim; once the app
|
|
43
|
+
bundle is populated Bundler restricts GEM_PATH to it and the ambient copy
|
|
44
|
+
falls out of scope. The shim now locates gemvault's `lib` across every root
|
|
45
|
+
that can hold it — `Gem.default_path` above all, which is what RubyGems knows
|
|
46
|
+
about its own gem roots and therefore covers rubies from rbenv, asdf, chruby,
|
|
47
|
+
Homebrew and distros, none of which export `GEM_HOME` or `GEM_PATH` — and
|
|
48
|
+
loads it via `$LOAD_PATH` rather than gem activation (issue #13).
|
|
49
|
+
- The vault source registers correctly when Bundler evaluates the plugin more
|
|
50
|
+
than once in a process; gemvault is resolved and required only on the first
|
|
51
|
+
evaluation, so `Gemvault::GemEntry` can no longer be defined twice from two
|
|
52
|
+
different gem roots (issue #13).
|
|
53
|
+
- `gemvault new` no longer raises `cannot load such file -- json` on a stock
|
|
54
|
+
distro ruby. json backs the tarball vault's manifest, so every current-format
|
|
55
|
+
vault needs it; it is a default gem upstream but a separate package on
|
|
56
|
+
distros, where `dnf install ruby` leaves it absent. It is now a declared
|
|
57
|
+
runtime dependency.
|
|
58
|
+
- Reading a vault through the Bundler source no longer raises `cannot load such
|
|
59
|
+
file -- json` either. Loading gemvault off `$LOAD_PATH` skips activation, and
|
|
60
|
+
therefore skips its dependencies, so the declared dependency alone did not
|
|
61
|
+
reach the plugin path. The shim now resolves gemvault's declared runtime
|
|
62
|
+
dependencies the same way it resolves gemvault and puts their require paths —
|
|
63
|
+
extension directories included — on `$LOAD_PATH`. A dependency it cannot find
|
|
64
|
+
is skipped rather than fatal, which is what activation could not do.
|
|
65
|
+
- `bundle install` no longer fails with `Could not find 'command_kit' (~> 0.6)`
|
|
66
|
+
when gemvault is installed into the plugin root without its dependencies.
|
|
67
|
+
Loading the vault source no longer activates the gemvault gem, which would
|
|
68
|
+
demand the full runtime dependency set; `vault_source.rb` reaches the rest of
|
|
69
|
+
gemvault through `require_relative` alone (issue #13).
|
|
70
|
+
|
|
22
71
|
### Changed
|
|
23
72
|
- `sqlite3` is no longer a runtime dependency. Gemvault runs dependency-free on
|
|
24
73
|
the tarball path (including JRuby); `sqlite3` is loaded lazily only to read a
|
|
25
74
|
legacy SQLite vault, with a clear error if it is not installed.
|
|
75
|
+
- `bundler` is no longer a runtime dependency; the Bundler plugin always runs
|
|
76
|
+
inside an existing Bundler process, and the declared dependency broke gem
|
|
77
|
+
activation in restricted GEM_PATH contexts.
|
|
78
|
+
- CI is unblocked (issue #11): RuboCop's TargetRubyVersion now matches the
|
|
79
|
+
gemspec's `required_ruby_version` floor (3.4.8), the previously-missing
|
|
80
|
+
`spec:integration` rake task exists alongside a non-integration `spec:core`, the
|
|
81
|
+
unit job no longer needs podman, and the integration job installs podman
|
|
82
|
+
when the runner image lacks it.
|
|
83
|
+
- Reading a vault's `manifest.json` is now strict: an entry missing any required
|
|
84
|
+
field (`name`, `version`, `platform`, `created_at`, `sha256`, `encrypted`)
|
|
85
|
+
raises instead of silently loading `nil` fields, so a truncated or hand-edited
|
|
86
|
+
manifest fails fast rather than yielding a subtly broken vault.
|
|
26
87
|
|
|
27
88
|
### Deprecated
|
|
28
89
|
- The SQLite vault format is deprecated and now **read-only**: existing SQLite
|
data/CLAUDE.md
CHANGED
|
@@ -17,6 +17,7 @@ Do NOT modify `.rubocop.yml` or use inline `# rubocop:disable` tags without expl
|
|
|
17
17
|
9. ALWAYS write specs first. The workflow is: identify the domain concept (rule 5), write specs describing its behavior, then implement. No implementation without a failing spec.
|
|
18
18
|
10. Integration specs are the first line of defense for CLI-tool bugs. For any bug reported from using the CLI tool (not the gemvault lib / Ruby API), the FIRST spec you write is an integration spec that reproduces the user's exact invocation — real subprocess, real vault, real exit code. Stub-heavy unit specs are complementary, not sufficient: they prove internal logic produces the expected value assuming surrounding wiring works, but a user's bug report is evidence the wiring didn't work.
|
|
19
19
|
11. If an integration spec is not catching a reported CLI-tool bug, one of two things is true, and the fix starts by diagnosing which: (a) existing integration specs are not specific enough — extend them to cover the exact scenario before touching production code; or (b) the scenario is not spec'd at all, which means the work is not a bug fix but a new feature — write integration specs for the contract first (per rule 1), then implement.
|
|
20
|
+
12. NEVER write to /tmp. Use /workspace/tmp
|
|
20
21
|
|
|
21
22
|
## Additional rules
|
|
22
23
|
|
|
@@ -25,7 +26,7 @@ Do NOT modify `.rubocop.yml` or use inline `# rubocop:disable` tags without expl
|
|
|
25
26
|
|
|
26
27
|
## Project Overview
|
|
27
28
|
|
|
28
|
-
Multi-gem portable archives
|
|
29
|
+
Multi-gem portable archives. A single `.gemv` file is a tarball holding multiple `.gem` files plus a `manifest.json` index; legacy SQLite vaults are read-only.
|
|
29
30
|
|
|
30
31
|
Two gems, one repo:
|
|
31
32
|
|
|
@@ -56,7 +57,7 @@ gem install --source file:///path/to/myvault.gemv foo
|
|
|
56
57
|
## Architecture
|
|
57
58
|
|
|
58
59
|
- `gemvault.gemspec` — main gem spec (name: `gemvault`)
|
|
59
|
-
- `lib/gemvault/vault.rb` —
|
|
60
|
+
- `lib/gemvault/vault.rb` — Vault facade choosing a backend by file format (Tarvault current, legacy Dbvault read-only)
|
|
60
61
|
- `lib/gemvault/cli.rb` — CLI dispatcher (new/add/list/remove/extract)
|
|
61
62
|
- `lib/bundler/plugin/vault_source.rb` — Bundler `Plugin::API::Source` implementation
|
|
62
63
|
- `lib/rubygems_plugin.rb` — RubyGems plugin: monkey-patches for `--source myvault.gemv` support
|
|
@@ -69,13 +70,22 @@ gem install --source file:///path/to/myvault.gemv foo
|
|
|
69
70
|
|
|
70
71
|
## Key Design Decisions
|
|
71
72
|
|
|
72
|
-
-
|
|
73
|
+
- Tar storage — portable, dependency-free, single file, inspectable with `tar`; legacy SQLite vaults readable via lazily-loaded sqlite3
|
|
73
74
|
- Specs extracted from gem blobs at runtime (no separate spec storage)
|
|
74
75
|
- Vault opened/closed per operation in the source plugin (no persistent connection)
|
|
75
76
|
- `fetch_gemspec_files` checks installed state — Bundler computes `full_gem_path` as `dirname(loaded_from)`, so `loaded_from` must point inside the gem directory
|
|
76
77
|
- `bundler-source-vault` name exists because Bundler auto-infers plugin name from `type: :vault` → `bundler-source-vault`
|
|
77
78
|
- `file://` URIs stripped to plain paths in `Gem::Source::Vault#initialize`
|
|
78
79
|
- Verbose logging via `Gem::UserInteraction#verbose` for `--verbose` support
|
|
80
|
+
- `shim/plugins.rb` loads gemvault by putting its `lib` on `$LOAD_PATH`, never by
|
|
81
|
+
activating the gem. Bundler skips installing a plugin dependency already
|
|
82
|
+
present on the ambient GEM_PATH, so the plugin root often lacks gemvault, and
|
|
83
|
+
by load time GEM_PATH is restricted to the plugin root and the app bundle. The
|
|
84
|
+
shim therefore searches every root that can hold gemvault, including the ones
|
|
85
|
+
Bundler masked (`Bundler.original_env` GEM_HOME/GEM_PATH). Activation is
|
|
86
|
+
avoided because it demands gemvault's full runtime dependency set (command_kit),
|
|
87
|
+
which Bundler skips for the same reason — `vault_source.rb` reaches the rest of
|
|
88
|
+
gemvault through `require_relative` alone and needs none of it.
|
|
79
89
|
|
|
80
90
|
## Testing
|
|
81
91
|
|
|
@@ -83,17 +93,66 @@ gem install --source file:///path/to/myvault.gemv foo
|
|
|
83
93
|
bundle exec rake test
|
|
84
94
|
```
|
|
85
95
|
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
- `test/integration_test.rb` — 12 end-to-end bundle install tests
|
|
89
|
-
- `test/cli_test.rb` — 32 CLI tests
|
|
90
|
-
- `test/rubygems_plugin_test.rb` — 28 tests (source, resolver, monkey-patches, gem install integration, file:// URI, verbose logging)
|
|
96
|
+
Minitest covers the library; RSpec covers the CLI and the containerized
|
|
97
|
+
integration suite. `rake` (the default task) runs `test`, `spec` and `rubocop`.
|
|
91
98
|
|
|
92
|
-
|
|
99
|
+
```bash
|
|
100
|
+
bundle exec rake test # minitest only
|
|
101
|
+
bundle exec rake spec:core # rspec, no containers
|
|
102
|
+
bundle exec rake spec:integration # rspec, containers (builds the image first)
|
|
103
|
+
bundle exec rake spec:build # rebuild the container image
|
|
104
|
+
bundle exec rake spec:teardown # remove it
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
- `test/vault_test.rb` — unit tests for Vault class
|
|
108
|
+
- `test/vault_source_test.rb` — unit tests for Bundler source plugin
|
|
109
|
+
- `test/cli_test.rb` — CLI tests
|
|
110
|
+
- `test/rubygems_plugin_test.rb` — source, resolver, monkey-patches, gem install integration, file:// URI, verbose logging
|
|
111
|
+
- `spec/integration/` — end-to-end specs, each run inside a podman container
|
|
112
|
+
- `spec/support/` — script fragments the integration specs assemble into those containers
|
|
113
|
+
|
|
114
|
+
Integration specs serve the tree's own gems from a local gem index (`GemIndex`)
|
|
115
|
+
to avoid rubygems.org resolution during testing.
|
|
116
|
+
|
|
117
|
+
### Container fidelity — do not undo these
|
|
118
|
+
|
|
119
|
+
The container has to look like a machine a user actually has. The `ruby` base
|
|
120
|
+
image does not, and every way it differs has already hidden a real defect. These
|
|
121
|
+
four are load-bearing; reverting any of them silently makes the suite green
|
|
122
|
+
against something other than the code under test:
|
|
123
|
+
|
|
124
|
+
1. **`bundler-source-vault` is NOT installed system-wide in `Dockerfile.test`.**
|
|
125
|
+
An ambient copy satisfies Bundler's plugin resolution without ever populating
|
|
126
|
+
the plugin root. Specs resolve the shim from the local gem index (`GemIndex`)
|
|
127
|
+
so they exercise the tree's shim. `gemvault` IS installed system-wide on
|
|
128
|
+
purpose — that is a real user's machine and the trigger for issue #13.
|
|
129
|
+
2. **`BUNDLE_APP_CONFIG` is not set in the container.** The `ruby` image sets it
|
|
130
|
+
to `/usr/local/bundle`, which moves `Bundler::Plugin.root` out of the project
|
|
131
|
+
so no spec ever touches a project-local `.bundle/plugin`. Fedora sets it
|
|
132
|
+
nowhere; do not add it, and do not adopt a base image that does.
|
|
133
|
+
3. **`GEM_HOME` and `GEM_PATH` are not exported in the container, and nothing
|
|
134
|
+
pins them.** Gems land in RubyGems' own default dirs. The `ruby` image
|
|
135
|
+
exports `GEM_HOME`; rubies from rbenv, asdf, chruby, Homebrew and distros
|
|
136
|
+
export nothing. Anything that reads gem roots back out of the environment
|
|
137
|
+
finds them on the image and finds nothing on a user's machine — that is what
|
|
138
|
+
made the first fix for issue #13 pass its specs while still failing for the
|
|
139
|
+
reporter. Neither `Dockerfile.test` nor `ContainerHelper#podman_run` should
|
|
140
|
+
grow an `ENV`/`-e` for either variable.
|
|
141
|
+
4. **Bundler is pinned to the version users run, not the image's default gem.**
|
|
142
|
+
The plugin machinery under test is Bundler's own, so a stale default silently
|
|
143
|
+
tests different code.
|
|
144
|
+
|
|
145
|
+
Fidelity is a property of the image, not of a script fragment: integration specs
|
|
146
|
+
run only commands a user would actually type.
|
|
147
|
+
|
|
148
|
+
If a spec fails only after removing a system-installed gem or an exported
|
|
149
|
+
variable, the spec was passing for the wrong reason — fix the spec, not the
|
|
150
|
+
container.
|
|
93
151
|
|
|
94
152
|
## Dependencies
|
|
95
153
|
|
|
96
|
-
- `bundler`
|
|
154
|
+
- `bundler` — NOT a dependency; the plugin always runs inside an existing Bundler process, and declaring it breaks gem activation under `bundle exec`'s restricted GEM_PATH
|
|
97
155
|
- `command_kit` (~> 0.6) — runtime (CLI)
|
|
156
|
+
- `json` (~> 2.0) — runtime; backs the tarball vault's manifest. A default gem upstream, but distros package it separately, so it is declared. Because the shim loads gemvault off `$LOAD_PATH` without activating it, it has to put json's require paths on `$LOAD_PATH` itself — see `dependency_specs` and `entries` in `shim/gemvault_load_path.rb`.
|
|
98
157
|
- `sqlite3` (~> 2.0) — NOT a runtime dependency; loaded lazily only to read a legacy SQLite (Dbvault) vault. Declared in the Gemfile for development/test.
|
|
99
|
-
- `minitest`, `rake` — development
|
|
158
|
+
- `minitest`, `rspec`, `rake` — development
|
data/Dockerfile.test
CHANGED
|
@@ -1,15 +1,70 @@
|
|
|
1
|
-
|
|
1
|
+
# A dev machine, not a packaging convenience.
|
|
2
|
+
#
|
|
3
|
+
# The official ruby image is not shaped like anything a user runs: it redirects
|
|
4
|
+
# gems to /usr/local/bundle, exports GEM_HOME and BUNDLE_APP_CONFIG to match,
|
|
5
|
+
# and freezes bundler at whatever shipped. Each of those differences hid a real
|
|
6
|
+
# defect, and patching them one at a time only moved the blind spot. A distro
|
|
7
|
+
# ruby has none of them: gems land in RubyGems' own default dirs, nothing is
|
|
8
|
+
# exported, and bundler is a regular gem rather than a default one -- which is
|
|
9
|
+
# also the shape that broke `bundle exec` in issue #12.
|
|
10
|
+
FROM docker.io/library/fedora:44
|
|
2
11
|
|
|
3
|
-
|
|
12
|
+
# redhat-rpm-config carries the hardening specs Fedora's ruby-devel compiles
|
|
13
|
+
# extensions against; without it every native gem fails in extconf. Any Fedora
|
|
14
|
+
# machine that can build a gem has it.
|
|
15
|
+
RUN dnf -y install --setopt=install_weak_deps=False \
|
|
16
|
+
ruby ruby-devel rubygem-bundler \
|
|
17
|
+
gcc make redhat-rpm-config sqlite-devel tar \
|
|
18
|
+
&& dnf clean all
|
|
4
19
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
&& gem build bundler-source-vault.gemspec \
|
|
10
|
-
&& gem install --no-document bundler-source-vault-*.gem \
|
|
11
|
-
&& rm -rf /tmp/build
|
|
20
|
+
# Keep this in step with the `BUNDLED WITH` line the project's own lockfile
|
|
21
|
+
# carries. The plugin machinery these specs exercise is Bundler's own, so a
|
|
22
|
+
# stale version silently tests different code.
|
|
23
|
+
RUN gem install --no-document bundler -v 4.0.17
|
|
12
24
|
|
|
13
25
|
# sqlite3 is not a gemvault runtime dependency; install it here so the upgrade
|
|
14
|
-
# integration test can
|
|
26
|
+
# integration test can read the committed legacy SQLite (Dbvault) fixture it
|
|
27
|
+
# migrates.
|
|
15
28
|
RUN gem install --no-document sqlite3
|
|
29
|
+
|
|
30
|
+
# Gem files for integration specs that install the plugin from a local gem
|
|
31
|
+
# index at spec runtime, kept for offline use: command_kit and json are
|
|
32
|
+
# gemvault's runtime dependencies, and `bundle plugin install` resolves them
|
|
33
|
+
# against the Gemfile's sources with GEM_PATH already narrowed to the plugin
|
|
34
|
+
# root, so an ambient copy does not satisfy it -- every declared dependency has
|
|
35
|
+
# to be servable from the index. The bundler gem lets specs reinstall bundler
|
|
36
|
+
# in place.
|
|
37
|
+
RUN mkdir -p /opt/gems \
|
|
38
|
+
&& cd /opt/gems \
|
|
39
|
+
&& gem fetch command_kit -v 0.6.0 \
|
|
40
|
+
&& gem fetch json -v "~> 2.0" \
|
|
41
|
+
&& gem fetch bundler -v "$(ruby -rbundler -e 'print Bundler::VERSION')"
|
|
42
|
+
|
|
43
|
+
# The tree arrives as a bind mount at /src rather than a COPY: the build context
|
|
44
|
+
# lives on a filesystem whose extended-attribute listing podman's copier cannot
|
|
45
|
+
# read. `rake spec:build` supplies both the mount and SOURCE_DIGEST.
|
|
46
|
+
#
|
|
47
|
+
# A bind mount's contents are not part of podman's cache key, so without
|
|
48
|
+
# SOURCE_DIGEST in the command line this layer is reused forever and the image
|
|
49
|
+
# keeps whichever gemvault the tree held the first time it was built -- the
|
|
50
|
+
# suite would then run green against code that is not under test. The digest
|
|
51
|
+
# changes whenever the installed source does, which invalidates the layer. It is
|
|
52
|
+
# last so that editing lib/ does not re-fetch the layers above.
|
|
53
|
+
#
|
|
54
|
+
# gemvault is installed system-wide because integration specs need the CLI to
|
|
55
|
+
# build vault fixtures, and because that is what a real user's machine looks
|
|
56
|
+
# like: `gem install gemvault` puts it on the ambient GEM_PATH. That ambient
|
|
57
|
+
# copy is load-bearing for the plugin root specs -- Bundler skips installing a
|
|
58
|
+
# plugin dependency that is already installed, so the plugin root only gets a
|
|
59
|
+
# complete dependency set if gemvault puts it there itself.
|
|
60
|
+
#
|
|
61
|
+
# bundler-source-vault is deliberately NOT installed system-wide. An ambient
|
|
62
|
+
# copy satisfies Bundler's plugin resolution without ever populating the plugin
|
|
63
|
+
# root, which masks every plugin-root defect the integration specs exist to
|
|
64
|
+
# catch. Specs install it from the local gem index built by GemIndex instead.
|
|
65
|
+
ARG SOURCE_DIGEST=unset
|
|
66
|
+
RUN test "$SOURCE_DIGEST" != unset \
|
|
67
|
+
&& cd /src \
|
|
68
|
+
&& gem build gemvault.gemspec -o /tmp/gemvault.gem \
|
|
69
|
+
&& gem install --no-document /tmp/gemvault.gem \
|
|
70
|
+
&& rm -f /tmp/gemvault.gem
|
data/README.md
CHANGED
|
@@ -1,10 +1,8 @@
|
|
|
1
1
|
# Gemvault
|
|
2
2
|
|
|
3
|
-
A
|
|
3
|
+
A file that behaves like a private gem server.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
New vaults are portable tarballs (no runtime dependencies, works on JRuby); the original SQLite format is still read transparently. See [Vault format versioning](#vault-format-versioning).
|
|
5
|
+
You can commit it to your repo, drop it on S3, email it, put it on a USB drive, or whatever you feel like doing. Bundler and RubyGems read from it directly.
|
|
8
6
|
|
|
9
7
|
## Installation
|
|
10
8
|
|
|
@@ -29,7 +27,6 @@ end
|
|
|
29
27
|
bundle install
|
|
30
28
|
```
|
|
31
29
|
|
|
32
|
-
Bundler auto-discovers the `bundler-source-vault` plugin, installs it, and resolves gems from the vault alongside rubygems.org. No extra configuration.
|
|
33
30
|
|
|
34
31
|
### RubyGems CLI
|
|
35
32
|
|
|
@@ -67,12 +64,7 @@ gemvault upgrade myvault.gemv --no-backup # skip the default myvault.gemv.bak c
|
|
|
67
64
|
|
|
68
65
|
## How It Works
|
|
69
66
|
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
```bash
|
|
73
|
-
tar -tf myvault.gemv # tarball (v2) vault
|
|
74
|
-
sqlite3 myvault.gemv "SELECT name, version, platform FROM gems" # SQLite (v1) vault
|
|
75
|
-
```
|
|
67
|
+
The gemv file is a tarball containing your .gem files and a json manifest. It has no dependencies other than the tar utilities that rubygems provides.
|
|
76
68
|
|
|
77
69
|
When Bundler sees `type: :vault` in your Gemfile, it auto-installs the `bundler-source-vault` plugin from rubygems.org. The plugin implements the `Bundler::Plugin::API::Source` interface — it reads gemspecs from the vault, participates in dependency resolution, then extracts and installs gems from the vault's blob storage.
|
|
78
70
|
|
data/Rakefile
CHANGED
|
@@ -10,20 +10,43 @@ require "rspec/core/rake_task"
|
|
|
10
10
|
|
|
11
11
|
RSpec::Core::RakeTask.new(:spec)
|
|
12
12
|
|
|
13
|
+
RSpec::Core::RakeTask.new("spec:core") do |t|
|
|
14
|
+
t.rspec_opts = "--tag ~integration"
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
RSpec::Core::RakeTask.new("spec:integration") do |t|
|
|
18
|
+
t.rspec_opts = "--tag integration"
|
|
19
|
+
end
|
|
20
|
+
|
|
13
21
|
require "rubocop/rake_task"
|
|
14
22
|
RuboCop::RakeTask.new
|
|
15
23
|
|
|
16
24
|
require "gempilot/version_task"
|
|
17
25
|
Gempilot::VersionTask.new
|
|
18
26
|
|
|
19
|
-
|
|
27
|
+
require "digest"
|
|
28
|
+
require_relative "spec/support/container_helper"
|
|
29
|
+
|
|
30
|
+
CACHED_IMAGE = ContainerHelper::CACHED_IMAGE
|
|
31
|
+
|
|
32
|
+
# Dockerfile.test reads the tree through a bind mount, whose contents podman
|
|
33
|
+
# does not fold into its layer cache key. Feeding this digest in as a build arg
|
|
34
|
+
# is what makes `rake spec:build` notice that the source changed.
|
|
35
|
+
def image_source_digest
|
|
36
|
+
fingerprints = FileList["lib/**/*", "exe/*", "gemvault.gemspec"]
|
|
37
|
+
.select { |path| File.file?(path) }.sort
|
|
38
|
+
.map { |path| "#{path} #{Digest::SHA256.hexdigest(File.binread(path))}" }
|
|
39
|
+
Digest::SHA256.hexdigest(fingerprints.join("\n"))
|
|
40
|
+
end
|
|
20
41
|
|
|
21
42
|
def cached_image_exists?
|
|
22
43
|
system("podman", "image", "exists", CACHED_IMAGE, out: File::NULL, err: File::NULL)
|
|
23
44
|
end
|
|
24
45
|
|
|
25
46
|
def build_cached_image
|
|
26
|
-
sh "podman", "build", "--network=host", "-
|
|
47
|
+
sh "podman", "build", "--network=host", "-v", "#{__dir__}:/src:ro,z",
|
|
48
|
+
"--build-arg", "SOURCE_DIGEST=#{image_source_digest}",
|
|
49
|
+
"-t", CACHED_IMAGE, "-f", "Dockerfile.test", "."
|
|
27
50
|
end
|
|
28
51
|
|
|
29
52
|
def destroy_cached_image
|
|
@@ -59,6 +82,7 @@ namespace :shim do
|
|
|
59
82
|
end
|
|
60
83
|
|
|
61
84
|
Rake::Task[:spec].enhance ["spec:setup"]
|
|
85
|
+
Rake::Task["spec:integration"].enhance ["spec:setup"]
|
|
62
86
|
Rake::Task[:build].enhance ["shim:build"]
|
|
63
87
|
Rake::Task[:release].enhance ["shim:release"]
|
|
64
88
|
Rake::Task[:clobber].enhance ["spec:teardown"]
|
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
# Issue #13 — `cannot load such file -- bundler/plugin/vault_source`
|
|
2
|
+
|
|
3
|
+
Line references are to the vendored RubyGems/Bundler checkout in
|
|
4
|
+
`references/rubygems` (`bundler-v4.0.0-1207-gf912ac9b0b`).
|
|
5
|
+
|
|
6
|
+
## Symptom
|
|
7
|
+
|
|
8
|
+
```
|
|
9
|
+
❯ bundle install
|
|
10
|
+
Fetching gem metadata from https://rubygems.org/.......
|
|
11
|
+
Resolving dependencies...
|
|
12
|
+
Installing bundler-source-vault 0.2.0
|
|
13
|
+
|
|
14
|
+
[!] There was an error parsing `Gemfile`: cannot load such file -- bundler/plugin/vault_source. Bundler cannot continue.
|
|
15
|
+
|
|
16
|
+
# from /Users/davidgillis/repos/reversal-store/Gemfile:19
|
|
17
|
+
> source "vendor/vendored.gemv", type: :vault do
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
Reported as permanent. `gemvault doctor`, `rm -rf .bundle/plugin`, `rm -f
|
|
21
|
+
Gemfile.lock`, and the two combined all left it unchanged.
|
|
22
|
+
|
|
23
|
+
## Reproduction
|
|
24
|
+
|
|
25
|
+
The reporter's `.bundle` tarball showed a plugin root holding the shim and
|
|
26
|
+
nothing else:
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
.bundle/plugin/cache/bundler-source-vault-0.2.0.gem
|
|
30
|
+
.bundle/plugin/gems/bundler-source-vault-0.2.0/plugins.rb
|
|
31
|
+
.bundle/plugin/specifications/bundler-source-vault-0.2.0.gemspec
|
|
32
|
+
.bundle/plugin/index
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
No `gemvault` anywhere. Reproduced exactly with: gemvault installed on the
|
|
36
|
+
ambient `GEM_PATH`, the shim not installed, a project-local `.bundle` with a
|
|
37
|
+
configured bundle path, and `bundle install` run **twice**.
|
|
38
|
+
|
|
39
|
+
## Root cause
|
|
40
|
+
|
|
41
|
+
Three independent mechanisms compose. Each is fine alone.
|
|
42
|
+
|
|
43
|
+
### 1. Bundler never puts gemvault in the plugin root
|
|
44
|
+
|
|
45
|
+
`Source::Rubygems#install` (`lib/bundler/source/rubygems.rb:206`) short-circuits:
|
|
46
|
+
|
|
47
|
+
```ruby
|
|
48
|
+
if (spec.default_gem? && !cached_built_in_gem(...)) || (installed?(spec) && !options[:force])
|
|
49
|
+
print_using_message "Using #{version_message(spec, options[:previous_spec])}"
|
|
50
|
+
return nil
|
|
51
|
+
end
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
`installed?` (`:484`) consults `installed_specs`. On any machine where
|
|
55
|
+
`gem install gemvault` has run — that is, anyone who uses the CLI — the ambient
|
|
56
|
+
copy satisfies it. Bundler prints "Using gemvault" and the plugin root receives
|
|
57
|
+
the shim alone. Nothing is downloaded, so the plugin root's `cache/` has no
|
|
58
|
+
gemvault either, which is exactly what the reporter's tarball showed.
|
|
59
|
+
|
|
60
|
+
### 2. The ambient copy disappears precisely when it is needed
|
|
61
|
+
|
|
62
|
+
`Plugin::Installer#install_all_sources` calls
|
|
63
|
+
`Bundler.configure_gem_home_and_path(Plugin.root)`
|
|
64
|
+
(`lib/bundler/plugin/installer.rb:100`), and `configure_gem_path`
|
|
65
|
+
(`lib/bundler.rb:661`) does:
|
|
66
|
+
|
|
67
|
+
```ruby
|
|
68
|
+
unless use_system_gems?
|
|
69
|
+
Bundler::SharedHelpers.set_env "GEM_PATH", ""
|
|
70
|
+
end
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
With a configured bundle path, `use_system_gems?` is false, so `GEM_PATH`
|
|
74
|
+
becomes empty and `Gem.path` collapses to the bundle. The copy that made step 1
|
|
75
|
+
"safe" is now out of scope.
|
|
76
|
+
|
|
77
|
+
This is why the failure is **permanent and why it starts on the second run**.
|
|
78
|
+
The first `bundle install` succeeds because the ambient gemvault is still
|
|
79
|
+
visible. It populates the bundle path. From then on every `bundle install`
|
|
80
|
+
narrows `GEM_PATH`, and every recovery step the reporter tried rebuilds the same
|
|
81
|
+
incomplete plugin root.
|
|
82
|
+
|
|
83
|
+
### 3. Locating gemvault is not sufficient — activation cannot survive here
|
|
84
|
+
|
|
85
|
+
The obvious repair — find gemvault and load it — has a trap. Loading it via
|
|
86
|
+
`$LOAD_PATH` skips activation, and therefore skips its dependencies. gemvault
|
|
87
|
+
reads a vault manifest through `json` (`lib/gemvault/manifest.rb:1`), so the
|
|
88
|
+
plugin dies later, at first vault read, instead of at load.
|
|
89
|
+
|
|
90
|
+
Activating instead does not work, for two separate reasons:
|
|
91
|
+
|
|
92
|
+
- **The path would be stripped one line before it is used.** `Runtime#setup`
|
|
93
|
+
(`lib/bundler/runtime.rb:12-38`) runs `clean_load_path` at line 16 and
|
|
94
|
+
`@definition.specs_for(groups)` at line 18 — and resolution is what asks the
|
|
95
|
+
vault source for specs. `clean_load_path`
|
|
96
|
+
(`lib/bundler/shared_helpers.rb:371-380`) rejects a `$LOAD_PATH` entry only
|
|
97
|
+
when `loaded_gem_paths.delete(p)` is truthy, and `loaded_gem_paths`
|
|
98
|
+
(`lib/bundler/rubygems_integration.rb:132-135`) is built from
|
|
99
|
+
`Gem.loaded_specs` — i.e. **activated gems**. Activating is what would put the
|
|
100
|
+
path on the chopping block. A directory pushed without activation is absent
|
|
101
|
+
from that list, so `delete` returns nil and the entry survives.
|
|
102
|
+
|
|
103
|
+
- **`Gem::Specification` cannot see outside the bundle.** `stub_rubygems`
|
|
104
|
+
(`lib/bundler/rubygems_integration.rb:336-340`) sets
|
|
105
|
+
`Gem::Specification.all = specs` and reapplies it through `Gem.post_reset`, so
|
|
106
|
+
under `bundle exec` `find_by_name` cannot find a gem installed outside the
|
|
107
|
+
bundle, and `Gem::Specification.reset` will not restore it.
|
|
108
|
+
|
|
109
|
+
## Why the test suite could not catch any of this
|
|
110
|
+
|
|
111
|
+
The integration container was the official `ruby` image. It differs from a
|
|
112
|
+
machine anyone develops on in four ways, and each one hid one layer:
|
|
113
|
+
|
|
114
|
+
| Image behaviour | What it hid |
|
|
115
|
+
|---|---|
|
|
116
|
+
| `gemvault` **and** `bundler-source-vault` installed system-wide | `require "bundler/plugin/vault_source"` resolved from `/usr/local/bundle` no matter what the plugin root held |
|
|
117
|
+
| `BUNDLE_APP_CONFIG=/usr/local/bundle` | `Bundler::Plugin.root` was never project-local, so no spec touched a real `.bundle/plugin` |
|
|
118
|
+
| `GEM_HOME` exported | a first repair that read gem roots out of the environment appeared to work |
|
|
119
|
+
| `json` present as a default gem | the undeclared dependency was invisible |
|
|
120
|
+
|
|
121
|
+
The first repair for this issue reconstructed gem roots from
|
|
122
|
+
`Bundler.original_env`, which is a snapshot of environment variables.
|
|
123
|
+
`GEM_HOME`/`GEM_PATH` are only present there if something exported them —
|
|
124
|
+
rbenv, asdf, chruby, Homebrew and distro rubies export neither. The reporter is
|
|
125
|
+
on rbenv (`issues.rec`: `/Users/davidgillis/.rbenv/versions/4.0.1/`,
|
|
126
|
+
`rbenv 1.3.2`), so that repair passed its specs in Docker and still failed on
|
|
127
|
+
their machine.
|
|
128
|
+
|
|
129
|
+
The suite now runs on `fedora:44` with a distro ruby: gems in RubyGems' own
|
|
130
|
+
default dirs, nothing exported, bundler a regular gem. Moving to it immediately
|
|
131
|
+
surfaced two further real defects — the undeclared `json` dependency, and the
|
|
132
|
+
`json/ext/parser` extension path.
|
|
133
|
+
|
|
134
|
+
## The fix
|
|
135
|
+
|
|
136
|
+
`shim/gemvault_load_path.rb` locates gemvault and its declared runtime
|
|
137
|
+
dependencies across every root that could hold them, and pushes their require
|
|
138
|
+
paths onto `$LOAD_PATH`.
|
|
139
|
+
|
|
140
|
+
- **`Gem.default_path` is the load-bearing root.** It is what RubyGems knows
|
|
141
|
+
about its own gem roots, independent of the narrowing in step 2, so it covers
|
|
142
|
+
rubies that export nothing. `Bundler.original_env` is still consulted, for the
|
|
143
|
+
rubies that do export a root outside those defaults, such as RVM.
|
|
144
|
+
- **`$LOAD_PATH`, not activation**, for the reasons in step 3.
|
|
145
|
+
- **Dependencies resolved by hand**, using `full_require_paths`
|
|
146
|
+
(`lib/rubygems/basic_specification.rb:172-183`), which appends `extension_dir`
|
|
147
|
+
when `have_extensions?` — without it `json/ext/parser` is unfindable, since
|
|
148
|
+
json's compiled half lives outside its gem directory.
|
|
149
|
+
- **A dependency that cannot be found is skipped rather than fatal.** That is
|
|
150
|
+
the one thing activation could not do: it would abort on `command_kit`, which
|
|
151
|
+
only the CLI needs.
|
|
152
|
+
- **`json` is now a declared runtime dependency**, so `gemvault new` works on a
|
|
153
|
+
stock distro ruby.
|
|
154
|
+
|
|
155
|
+
## Does the suite actually protect this?
|
|
156
|
+
|
|
157
|
+
Green is not evidence on its own — the whole reason this bug shipped is that a
|
|
158
|
+
green suite was being satisfied by an ambient gem. So the two pieces of the fix
|
|
159
|
+
were checked by mutation:
|
|
160
|
+
|
|
161
|
+
| Mutation | Result |
|
|
162
|
+
|---|---|
|
|
163
|
+
| `Gem.default_path` dropped from the searched roots | 18 examples fail |
|
|
164
|
+
| the shim's pinned gemvault requirement ignored, so the newest wins | both examples of the upgrade-residue scenario fail |
|
|
165
|
+
|
|
166
|
+
Restoring each returns the suite to green.
|
|
167
|
+
|
|
168
|
+
## Platforms verified
|
|
169
|
+
|
|
170
|
+
| Platform | Status |
|
|
171
|
+
|---|---|
|
|
172
|
+
| CRuby 4.0.6, Fedora 44, aarch64 | Full suite, 265 examples |
|
|
173
|
+
| JRuby 10.1.1.0 (the reporter runs 10.1.0.0) | Verified once by hand, **not covered by the suite**: vault create/add/list, `bundle install` twice against a vault source, `bundle exec require`. `json 2.21.1 (java)` installed as a platform gem and resolved, which is the case `full_require_paths` has to get right |
|
|
174
|
+
| macOS `arm64-darwin-24` | **Not verified.** The reporter's own platform. Nothing here can run it |
|
|
175
|
+
|
|
176
|
+
JRuby has no permanent coverage because JRuby caps its own heap from the cgroup
|
|
177
|
+
and the development sandbox is small enough that the cap lands at 148 MB, where
|
|
178
|
+
even a single `gem fetch` dies. The one successful hand-run above happened in a
|
|
179
|
+
lucky memory window and did not reproduce. A JRuby image, rake task and spec
|
|
180
|
+
were written and then reverted rather than shipped unverified: harness code
|
|
181
|
+
nobody has watched pass is the same defect as a spec that cannot fail. On a
|
|
182
|
+
machine with ordinary memory this is worth adding, and it is the highest-value
|
|
183
|
+
gap left, since issue #13 was reported against JRuby as well as CRuby.
|
|
184
|
+
|
|
185
|
+
## Coverage
|
|
186
|
+
|
|
187
|
+
`spec/support/vault_sourced_gemfile_examples.rb` applies "a complete bundle" to
|
|
188
|
+
a vault-sourced Gemfile and then re-applies it after each user action: deleting
|
|
189
|
+
`Gemfile.lock`, deleting `.bundle`, deleting both, removing the vaulted gem,
|
|
190
|
+
adding a gem from rubygems, adding the project's own gemspec (the reporter's
|
|
191
|
+
Gemfile shape), adding another gem from the same vault, and adding one from a
|
|
192
|
+
second vault. `spec/integration/vaulted_project_spec.rb` runs that whole
|
|
193
|
+
group under four bundler configurations — stock, an install path chosen, that
|
|
194
|
+
path undone, and the project's gems cached — and
|
|
195
|
+
`spec/integration/no_ambient_gemvault_spec.rb` covers the machine that has no
|
|
196
|
+
gemvault of its own, where the plugin root has to carry the whole dependency
|
|
197
|
+
set.
|