gemkeeper 0.8.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +10 -1
- data/lib/gemkeeper/version.rb +1 -1
- metadata +17 -16
- data/specs/20260518-154733-gemkeeper-contractor-support/implementation-summary.md +0 -75
- data/specs/20260518-154733-gemkeeper-contractor-support/spec.md +0 -287
- data/specs/20260529-091429-replace-geminabox-compact-proxy/critique-consolidated-v-1.md +0 -168
- data/specs/20260529-091429-replace-geminabox-compact-proxy/critique-v-1-claude.md +0 -124
- data/specs/20260529-091429-replace-geminabox-compact-proxy/critique-v-1-codex.md +0 -125
- data/specs/20260529-091429-replace-geminabox-compact-proxy/critique-v-1-copilot.md +0 -261
- data/specs/20260529-091429-replace-geminabox-compact-proxy/spec.md +0 -360
- data/specs/20260529-131354-sync-serve-cache-contract/critique-consolidated-v-1.md +0 -95
- data/specs/20260529-131354-sync-serve-cache-contract/critique-v-1-claude.md +0 -47
- data/specs/20260529-131354-sync-serve-cache-contract/critique-v-1-codex.md +0 -112
- data/specs/20260529-131354-sync-serve-cache-contract/critique-v-1-copilot.md +0 -169
- data/specs/20260529-131354-sync-serve-cache-contract/implementation-summary.md +0 -59
- data/specs/20260529-131354-sync-serve-cache-contract/spec.md +0 -169
|
@@ -1,169 +0,0 @@
|
|
|
1
|
-
# Critique: Spec 20260529-131354 — Server-authoritative sync cache check
|
|
2
|
-
|
|
3
|
-
Reviewer: GitHub Copilot (claude-sonnet-4.6)
|
|
4
|
-
Date: 2026-05-29
|
|
5
|
-
|
|
6
|
-
## Summary
|
|
7
|
-
|
|
8
|
-
The spec is well-scoped and the core idea is sound.
|
|
9
|
-
The major gap is an unresolved contradiction between the Goals and AR-1.2 around `version: latest`, and several smaller gaps in error handling, format parsing, and implementation structure that need tightening before handoff to an implementer.
|
|
10
|
-
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
## Missing Requirements
|
|
14
|
-
|
|
15
|
-
### MR-1: The `version: latest` + clone contradiction is unresolved
|
|
16
|
-
|
|
17
|
-
The Goals promise "Recovery never re-clones or rebuilds a gem whose `.gem` artifact already exists locally."
|
|
18
|
-
AR-1.2 then acknowledges that for `version: latest`, version resolution happens *post-checkout*, meaning `clone_or_pull` still runs before any server check.
|
|
19
|
-
These two statements directly contradict each other for the case where a `latest` gem is already on the server.
|
|
20
|
-
|
|
21
|
-
The spec needs to explicitly choose one of:
|
|
22
|
-
(a) The no-reclone goal only applies to non-`latest` gems (add that scope qualifier to the Goals section).
|
|
23
|
-
(b) For `version: latest` with a local artifact, read the version from the `.gem` file via `Gem::Package.new(path).spec.version` to avoid cloning, then check the server — add this as a new code path.
|
|
24
|
-
|
|
25
|
-
Without a resolution, an implementer will produce inconsistent behavior and the Goals will be technically false for `latest` gems.
|
|
26
|
-
|
|
27
|
-
### MR-2: 503 response from `/info/<name>` is not addressed
|
|
28
|
-
|
|
29
|
-
When a private gem has never been uploaded to a running server, `GemIndex#[]` returns `nil` for that name.
|
|
30
|
-
The server then calls `serve_upstream_info`, which returns 503 (`upstream_unavailable`) when offline.
|
|
31
|
-
This is the exact recovery scenario the spec is designed to fix — the gem doesn't exist on the server — yet only 404 is explicitly listed as "not present."
|
|
32
|
-
A 503 from the server during the presence check should also be treated as "not present" (or at minimum documented as an expected response that triggers the upload path), but it is not mentioned in FR-1.2, AR-1.4, or the Assumptions & Risks.
|
|
33
|
-
|
|
34
|
-
### MR-3: No description of how `build_and_upload` is decomposed
|
|
35
|
-
|
|
36
|
-
FR-1.3 requires a new code path: upload an existing artifact without building.
|
|
37
|
-
`GemSyncer#build_and_upload` currently performs both steps unconditionally.
|
|
38
|
-
The spec never mentions that this method must be split (e.g., `upload_artifact(gem_path)` vs. `build_gem(local_path, gems_path)`) or what the resulting `GemSyncer` flow looks like.
|
|
39
|
-
An implementer must infer the structural change from the FR alone, which increases the chance of inventing different designs across teams or in agentic handoff.
|
|
40
|
-
|
|
41
|
-
### MR-4: Test file for `sync()` behavior does not yet exist in `test_gem_syncer.rb`
|
|
42
|
-
|
|
43
|
-
The Constraints point to `test_gem_syncer.rb` for `GemSyncer` test coverage, and FR-1.3's Verify describes stubs for `GitRepository` and `GemBuilder`.
|
|
44
|
-
The existing `test_gem_syncer.rb` tests only `resolve_repo` — it has no tests for `sync()` at all.
|
|
45
|
-
The spec should acknowledge that new test methods for `sync()` must be added (not just extended), and whether the integration-level Verify for FR-1.3 is better placed in the lifecycle integration test or in a new unit-level test for `GemSyncer#sync`.
|
|
46
|
-
|
|
47
|
-
---
|
|
48
|
-
|
|
49
|
-
## Ambiguous Language
|
|
50
|
-
|
|
51
|
-
### AM-1: "replacing the `list_gems` `NotImplementedError` stub" conflates two different methods
|
|
52
|
-
|
|
53
|
-
AR-1.1 says to add a presence method "e.g. `has_version?(name, version)` / replacing the `list_gems` `NotImplementedError` stub."
|
|
54
|
-
`list_gems` returns a list; `has_version?` checks existence by name+version.
|
|
55
|
-
They are not substitutes for each other.
|
|
56
|
-
The parenthetical implies `list_gems` would be removed and `has_version?` added in its place, but `list_gems` has its own contract (error message references `gemkeeper list`).
|
|
57
|
-
The spec should state explicitly whether `list_gems` is being removed, renamed, or left alone, and where `has_version?` fits in the public interface.
|
|
58
|
-
|
|
59
|
-
### AM-2: "version line for that exact version" underspecifies the compact-index format
|
|
60
|
-
|
|
61
|
-
FR-1.2 says the gem is present when "the info document contains a version line for that exact version."
|
|
62
|
-
The compact-index `/info` format emitted by `CompactIndex.info(versions)` includes a `---` separator and lines structured as `<version> <platform> <checksum>|<deps>`.
|
|
63
|
-
"Version line" is not defined.
|
|
64
|
-
The Assumptions & Risks note does say "parsing only the leading version token per line," which is a workable rule, but it belongs in FR-1.2 as a normative requirement, not only as a risk mitigation note.
|
|
65
|
-
The spec should state explicitly: skip lines starting with `---` or `created_at:`; split on whitespace; compare the first token against the bare semver.
|
|
66
|
-
|
|
67
|
-
### AM-3: "present only if the server reports that exact `<name>` and `<version>`" — case sensitivity unspecified
|
|
68
|
-
|
|
69
|
-
Gem names in the compact-index are case-sensitive by convention, but the spec does not state whether the comparison is case-sensitive.
|
|
70
|
-
This matters if a name is stored differently in the manifest vs. what the server indexes.
|
|
71
|
-
A one-line clarification ("comparison is case-sensitive, matching the gem name exactly as configured") would prevent a subtle bug.
|
|
72
|
-
|
|
73
|
-
---
|
|
74
|
-
|
|
75
|
-
## Codebase Assumptions That May Be Wrong
|
|
76
|
-
|
|
77
|
-
### CA-1: The `GemUploader` Faraday connection uses `multipart` middleware, which is unnecessary for a plain GET
|
|
78
|
-
|
|
79
|
-
The existing `connection` method in `GemUploader` builds a Faraday connection with `:multipart` and `:url_encoded` middleware.
|
|
80
|
-
Both are only relevant for `POST /upload`.
|
|
81
|
-
A `GET /info/<name>` on that same connection is harmless but wasteful (adds a `Content-Type: multipart/form-data` header that is ignored by the server).
|
|
82
|
-
The spec should either (a) note this is acceptable and leave it as-is, or (b) instruct the implementer to use a separate plain connection for read-only requests.
|
|
83
|
-
Ignoring this risks the presence check failing on edge-case Rack middleware that rejects multipart headers on GETs.
|
|
84
|
-
|
|
85
|
-
### CA-2: `latest_version!` raises `BuildError`, but the new flow may need a softer return
|
|
86
|
-
|
|
87
|
-
`latest_version!` in `GemSyncer` raises `BuildError` if `current_version` returns `nil`.
|
|
88
|
-
With the new server-check path for `version: latest`, there is a question of whether the existing exception should propagate before or after the server presence check.
|
|
89
|
-
The spec does not address what happens if `current_version` returns `nil` and the server already has the gem (build failure, but gem is present — should it be a skip or an error?).
|
|
90
|
-
|
|
91
|
-
### CA-3: `GemIndex#add` raises `Errno::EEXIST` on conflict — not HTTP 409
|
|
92
|
-
|
|
93
|
-
AR-1.4 says "an upload that races with an already-present gem (server returns the existing-gem conflict) is treated as success/skip."
|
|
94
|
-
`GemUploader#handle_response` already handles 409 as `{ success: true, skipped: true }`, so this is correct at the HTTP layer.
|
|
95
|
-
However, the spec says the *presence check* comes before the upload.
|
|
96
|
-
After a successful presence check shows the gem is absent, the gem could be uploaded by another process before this process's upload completes — the race actually produces a 409 at upload time, not a "missing" from the presence check.
|
|
97
|
-
The spec's idempotency claim in FR-1.4 is correct by accident (existing 409 handling covers it), but the causal chain described is subtly wrong and could confuse an implementer.
|
|
98
|
-
|
|
99
|
-
---
|
|
100
|
-
|
|
101
|
-
## Error Handling & Edge Cases
|
|
102
|
-
|
|
103
|
-
### EH-1: HTTP status codes other than 200, 404 are unspecified for the presence check
|
|
104
|
-
|
|
105
|
-
AR-1.4 addresses "unreachable or erroring server" and "malformed `/info` response" but not intermediate HTTP errors: 400 (invalid name), 500, 503.
|
|
106
|
-
A 503 is the expected response when the gem is absent and the server is offline (see MR-2 above).
|
|
107
|
-
A 400 would indicate a bug in name validation, which is a different class of problem.
|
|
108
|
-
The spec should enumerate what status codes are treated as "not present" vs. "raise `ServerNotReachableError`" vs. "raise a different error."
|
|
109
|
-
|
|
110
|
-
### EH-2: Connection failure during presence check timing vs. upload timing
|
|
111
|
-
|
|
112
|
-
The spec says a connection failure during the presence check raises `ServerNotReachableError`.
|
|
113
|
-
But the current `GemSyncer#sync` does no upfront reachability check; it discovers unreachability only when `upload` is called.
|
|
114
|
-
With the new flow, the error surface moves earlier (before any git work), which is an improvement.
|
|
115
|
-
However, the spec should note whether this earlier detection changes any user-visible behavior (e.g., error message wording, exit code).
|
|
116
|
-
Currently `GemUploader#upload` and the proposed `has_version?` would raise the same exception class, but the message may differ.
|
|
117
|
-
|
|
118
|
-
### EH-3: What if the local artifact is corrupt?
|
|
119
|
-
|
|
120
|
-
FR-1.3 says when the server is missing the gem and the local `.gem` exists, upload it directly.
|
|
121
|
-
The spec does not address what happens if the local artifact is a partial/corrupt file (e.g., interrupted build).
|
|
122
|
-
The server's `GemIndex#add` reads the file via `Gem::Package.new(source_path).spec` and will raise `StandardError` on a corrupt gem, which `UploadHandler` likely does not gracefully convert to a 4xx.
|
|
123
|
-
The spec should either require a pre-upload integrity check (`Gem::Package.new(path).spec` before upload) or explicitly acknowledge this as out of scope.
|
|
124
|
-
|
|
125
|
-
---
|
|
126
|
-
|
|
127
|
-
## Security Concerns
|
|
128
|
-
|
|
129
|
-
The spec's security review says "N/A beyond existing posture."
|
|
130
|
-
This is broadly correct given the loopback-only deployment, but there is one minor gap worth noting:
|
|
131
|
-
|
|
132
|
-
`GemUploader` currently does not validate the gem name before constructing the `/info/<name>` URL.
|
|
133
|
-
On the server side, `VALID_NAME` (line 17 of `compact_index_server.rb`) rejects names that don't match `[a-zA-Z0-9._-]+` and returns 400.
|
|
134
|
-
The client should treat a 400 response as a programming error (not a "not present" outcome) rather than silently proceeding.
|
|
135
|
-
This is a defense-in-depth note, not a new attack surface, but it is relevant to EH-1 above.
|
|
136
|
-
|
|
137
|
-
---
|
|
138
|
-
|
|
139
|
-
## Performance Implications
|
|
140
|
-
|
|
141
|
-
### PI-1: `version: latest` still always clones/pulls before the skip can fire
|
|
142
|
-
|
|
143
|
-
For `version: latest`, the server presence check can only fire after `clone_or_pull` and version resolution.
|
|
144
|
-
If the intent is fast recovery (re-run sync, get back to work quickly), `version: latest` gems will still incur a network round-trip to the git remote before they are skipped.
|
|
145
|
-
The spec acknowledges this implicitly in AR-1.2 but does not flag it as a known cost or consider mitigations (e.g., reading the version from the existing artifact).
|
|
146
|
-
This is worth calling out explicitly so the implementer does not optimize away the clone for `latest` and inadvertently break version freshness.
|
|
147
|
-
|
|
148
|
-
---
|
|
149
|
-
|
|
150
|
-
## Spec Completeness Checklist — Gaps
|
|
151
|
-
|
|
152
|
-
| Item | Status | Issue |
|
|
153
|
-
| ---- | ------ | ----- |
|
|
154
|
-
| Scope & acceptance criteria | Partial | Goals "never re-clones" is untrue for `version: latest` (MR-1) |
|
|
155
|
-
| Testing strategy | Partial | `test_gem_syncer.rb` has no `sync()` tests; spec implies extension, not creation (MR-4) |
|
|
156
|
-
| Error handling & failure modes | Partial | 503 and other non-200/404 statuses unspecified (MR-2, EH-1) |
|
|
157
|
-
| Assumptions & risks | Partial | 503/offline risk for never-uploaded private gems not listed; corrupt artifact not listed |
|
|
158
|
-
| Architecture & interfaces | Partial | `build_and_upload` decomposition not specified (MR-3); `list_gems` ambiguity (AM-1) |
|
|
159
|
-
|
|
160
|
-
---
|
|
161
|
-
|
|
162
|
-
## Recommended Changes Before Implementation
|
|
163
|
-
|
|
164
|
-
1. Clarify the `version: latest` goal: either scope "never re-clones" to non-`latest` gems, or add the `Gem::Package` version-read path as an explicit alternative to cloning (MR-1).
|
|
165
|
-
2. Add 503 to FR-1.2 as a "not present" signal, or explain why it should be an error (MR-2).
|
|
166
|
-
3. Replace the risk-mitigation note about `/info` parsing with a normative parsing rule in FR-1.2 (AM-2).
|
|
167
|
-
4. Specify the `build_and_upload` decomposition or name the resulting private methods (MR-3).
|
|
168
|
-
5. Clarify whether `list_gems` is removed or retained, and add `has_version?` as a standalone addition (AM-1).
|
|
169
|
-
6. Add EH-1's status-code table (200 = check version, 404/503 = not present, others = raise).
|
|
@@ -1,59 +0,0 @@
|
|
|
1
|
-
# Implementation Summary: 20260529-131354-sync-serve-cache-contract
|
|
2
|
-
|
|
3
|
-
**Status:** Completed
|
|
4
|
-
**Date:** 2026-05-29
|
|
5
|
-
|
|
6
|
-
## Overview
|
|
7
|
-
|
|
8
|
-
Made `gemkeeper sync`'s skip decision authoritative against the server's *private* store instead of a local build artifact, fixing the bug where a fresh or repointed server could never be repopulated (every gem skipped, 404s on `bundle install`). Added a private-store-only presence endpoint, server-authoritative `serves?` on the uploader, and reordered `GemSyncer` to defer repo work — skipping when the server already has a gem and re-uploading an existing artifact without rebuilding.
|
|
9
|
-
|
|
10
|
-
## Execution
|
|
11
|
-
|
|
12
|
-
Solo, sequential (the client depends on the server endpoint contract): server endpoint → client `serves?` → `GemSyncer` reorder → tests → quality-gate refactor.
|
|
13
|
-
|
|
14
|
-
## Files Created
|
|
15
|
-
- `lib/gemkeeper/repo_fetcher.rb` — `RepoFetcher`: resolves a gem's repo URL (manifest + override) and clones/pulls it, with git-auth error mapping. Extracted from `GemSyncer` to keep it under the rubycritic gate.
|
|
16
|
-
- `test/gemkeeper/test_repo_fetcher.rb` — repo-resolution tests (migrated from `test_gem_syncer.rb`).
|
|
17
|
-
|
|
18
|
-
## Files Modified
|
|
19
|
-
- `lib/gemkeeper/compact_index_server.rb` — new `GET /gemkeeper/has/<name>/<version>` route (`serve_presence`) + `present` responder; reads the private index only.
|
|
20
|
-
- `lib/gemkeeper/compact_index_server/gem_index.rb` — `serves?(name, version)` predicate over the in-memory index.
|
|
21
|
-
- `lib/gemkeeper/gem_uploader.rb` — `serves?(name, version)` (hits the private endpoint; 200/404/else→`ServerError`; connection failure→`ServerNotReachableError`); extracted `not_reachable!` helper.
|
|
22
|
-
- `lib/gemkeeper/gem_syncer.rb` — reordered: `sync_pinned` (server check → artifact reuse → build) / `sync_latest` (always fetch); `reusable_artifact?` identity check; delegates repo acquisition to `RepoFetcher`.
|
|
23
|
-
- `lib/gemkeeper.rb` — require `repo_fetcher`.
|
|
24
|
-
- `test/gemkeeper/test_compact_index_server.rb` — 4 presence-endpoint tests.
|
|
25
|
-
- `test/gemkeeper/test_gem_uploader.rb` — `serves?` connection-failure test.
|
|
26
|
-
- `test/gemkeeper/test_gem_syncer.rb` — `reusable_artifact?` (4) + `sync()` orchestration (2) tests; resolve tests moved out.
|
|
27
|
-
- `test/integration/test_server_lifecycle_integration.rb` — original-bug regression + skip-when-served tests.
|
|
28
|
-
- `test/integration/test_cli_integration.rb` — repurposed the obsolete local-cache-skip test to assert the new unreachable-server error.
|
|
29
|
-
- `CHANGELOG.md` — `[Unreleased]` Fixed entry.
|
|
30
|
-
|
|
31
|
-
## Test Results
|
|
32
|
-
`bundle exec rake test` → 230 runs, 512 assertions, 0 failures, 0 errors.
|
|
33
|
-
`bundle exec rubocop` → 76 files, no offenses.
|
|
34
|
-
`bundle exec rubycritic lib --no-browser` → 90.13 (gate ≥ 90); `gem_syncer` B, `repo_fetcher` A, no C/D/F.
|
|
35
|
-
|
|
36
|
-
## Spec Adherence
|
|
37
|
-
| Requirement | Status | Implementation | Test |
|
|
38
|
-
|-------------|--------|---------------|------|
|
|
39
|
-
| FR-1.1 | Done | `gem_syncer.rb` `sync_pinned`/`sync_latest` via `@uploader.serves?` | `test_sync_skips_when_server_already_serves` (unit + integration) |
|
|
40
|
-
| FR-1.2 | Done | `gem_uploader.rb` `serves?` hits `/gemkeeper/has` | `test_gem_uploader`, presence endpoint tests |
|
|
41
|
-
| FR-1.3 | Done | `reusable_artifact?` + `reupload` | `test_sync_reuploads_existing_artifact_without_rebuild`, regression |
|
|
42
|
-
| FR-1.4 | Done | `GemUploader#handle_response` 409→skip | `test_sync_skips_when_server_already_serves` (runs sync twice) |
|
|
43
|
-
| FR-1.5 | Done | `skip`/`reupload`/`build_gem` symbols + output text | syncer unit tests, integration `1 skipped` |
|
|
44
|
-
| FR-1.6 | Done | `sync_latest` always fetches | covered by `sync_latest` path |
|
|
45
|
-
| FR-2.1 | Done | `compact_index_server.rb` `serve_presence` + `GemIndex#serves?` | `test_presence_endpoint_*` (incl. no-upstream-probe) |
|
|
46
|
-
| FR-2.2 | Done | `serve_presence` `VALID_NAME` check → 400 | `test_presence_endpoint_rejects_invalid_name` |
|
|
47
|
-
| AR-1.1 | Done | `serves?` on `GemUploader`; `list_gems` untouched | `test_gem_uploader` |
|
|
48
|
-
| AR-1.2 | Done | reuse `resolve_version`; `<name>-<version>.gem` | syncer tests |
|
|
49
|
-
| AR-1.3 | Done | presence strictly over HTTP | integration (real server) |
|
|
50
|
-
| AR-1.4 | Done | status mapping in `serves?` | `test_serves_raises_server_not_reachable_on_connection_failure`, CLI unreachable test |
|
|
51
|
-
| AR-1.5 | Done | `RepoFetcher` deferral; build/upload split | unit stubs assert no fetch/build on skip/reuse |
|
|
52
|
-
| AR-2.1 | Done | endpoint reads `@index` only; gates green | rubycritic 90.13 |
|
|
53
|
-
|
|
54
|
-
## Deviations from Spec
|
|
55
|
-
- **`RepoFetcher` extraction (not in the spec's file list).** The `GemSyncer` reorder added methods and pushed it to a rubycritic C (120); reek flagged `resolve_repo` and the auth helpers as belonging elsewhere. Extracting `RepoFetcher` both satisfied the spec's quality-gate constraint and is the cleaner boundary (sync orchestrates; `RepoFetcher` acquires the repo). The 6 repo-resolution tests moved with it. No behavior change.
|
|
56
|
-
- **`test_cli_integration#test_sync_skips_already_cached_gem` repurposed.** It asserted the old local-cache skip (no server) that the spec deliberately removes; rewritten as `test_sync_errors_when_server_unreachable` to assert the new server-authoritative guard (AR-1.4).
|
|
57
|
-
|
|
58
|
-
## Living docs
|
|
59
|
-
`specs/docs/` does not exist — skipped. Run `/spec-docs --full` to bootstrap if desired.
|
|
@@ -1,169 +0,0 @@
|
|
|
1
|
-
# Spec 20260529-131354: Server-authoritative sync cache check
|
|
2
|
-
|
|
3
|
-
## Overview
|
|
4
|
-
|
|
5
|
-
`gemkeeper sync` decides a gem is "already synced" by checking for a local build artifact, but the running server serves from a separate store populated only by HTTP upload. These two states can diverge, so a fresh or repointed server can never be repopulated by re-running `sync` — every gem is skipped and nothing is uploaded, leaving the server returning 404 for gems that exist on disk. This spec makes `sync`'s skip decision authoritative against the server's *private* store via a new read-only endpoint, and lets `sync` recover by re-uploading an already-built artifact without rebuilding.
|
|
6
|
-
|
|
7
|
-
## Goals
|
|
8
|
-
|
|
9
|
-
- `sync` skips a gem only when the running server's private store actually serves that exact version.
|
|
10
|
-
- The presence check consults the private store only — never the upstream-proxied compact index — so it cannot be fooled by a public gem of the same name and never probes rubygems.org.
|
|
11
|
-
- A fresh, empty, or repointed server is fully repopulated by re-running `sync`, with no manual cache clearing.
|
|
12
|
-
- For pinned and `from_lockfile` versions, recovery never re-clones or rebuilds when the `.gem` artifact already exists locally. (`version: latest` keeps today's always-fetch behavior — see FR-1.6.)
|
|
13
|
-
- No regression to the shared Homebrew-service deployment, where `sync`'s `gems_path` legitimately differs from the server's.
|
|
14
|
-
|
|
15
|
-
---
|
|
16
|
-
|
|
17
|
-
## Feature 1: Server-authoritative sync skip and recovery
|
|
18
|
-
|
|
19
|
-
**Who & why:** A developer working offline runs `gemkeeper sync` to populate a local gem server, then `bundle install` against it. Today, if they restart the server against an empty store (or it was never uploaded to), `sync` silently skips every gem because the local `.gem` files still exist, and `bundle install` fails with 404s that give no hint the cache is stale. They need `sync` to detect that the *server* is missing a gem and fix it, so "run sync, then bundle" always works.
|
|
20
|
-
|
|
21
|
-
### Functional Requirements
|
|
22
|
-
|
|
23
|
-
#### FR-1.1: Skip decision queries the server's private store, not the local filesystem
|
|
24
|
-
`GemSyncer` determines whether a gem version is already available by asking the running server's private store, replacing the current `cached?` check that tests for a local file at `gems_path/<name>-<version>.gem`. A gem is considered present only if the server's private store reports that exact `<name>` and `<version>`. The version comparison uses bare semver (no `v` prefix), consistent with existing cache-key normalization.
|
|
25
|
-
**Verify:** With a `.gem` present in the local `gems_path` but absent from the server's store, `sync` reports the gem as synced (not skipped) and the server subsequently serves it; with the gem present on the server, `sync` reports it skipped.
|
|
26
|
-
|
|
27
|
-
#### FR-1.2: Presence is read from the private-store endpoint, not `/info`
|
|
28
|
-
The server-presence check calls the dedicated private-store endpoint defined in FR-2.1 (`GET /gemkeeper/has/<name>/<version>`), not the Bundler-facing `/info/<name>`. A `200` means present; a `404` means not present. `/info/<name>` is deliberately not used because it falls back to the upstream rubygems.org index (`serve_upstream_info`), which would cause public-name-collision false positives, offline failures, and leakage of private gem names to rubygems.org.
|
|
29
|
-
**Verify:** A unit test stubs `GET /gemkeeper/has/mimir/1.0.5` returning `200` → present; stubs `404` → not present; confirms `sync`'s presence path issues no request to `/info` or to rubygems.org.
|
|
30
|
-
|
|
31
|
-
#### FR-1.3: Re-upload an existing artifact instead of rebuilding
|
|
32
|
-
When the server reports a gem version missing but the corresponding `.gem` already exists in the local `gems_path`, `sync` uploads that existing artifact directly. Before uploading, it verifies the artifact's embedded gemspec name and version match the requested gem; on mismatch or an unreadable artifact it falls through to the build path rather than uploading the wrong file. It does not clone, pull, or run `gem build` when a valid artifact is reused. Only when no valid local artifact exists does `sync` fall back to the full clone/checkout/build path.
|
|
33
|
-
**Verify:** With a valid `mimir-1.0.5.gem` in `gems_path` and absent from the server, running `sync mimir` uploads the gem without invoking the git or build steps (assert via stubbed `GitRepository`/`GemBuilder` that they are not called), and the server then serves `mimir 1.0.5`; with a corrupt `mimir-1.0.5.gem`, `sync` does not upload it and proceeds to rebuild.
|
|
34
|
-
|
|
35
|
-
#### FR-1.4: Idempotent re-sync against any server state
|
|
36
|
-
Running `sync` twice in a row is safe and convergent: the first run uploads whatever the server is missing; the second run finds everything present and skips it. If an upload targets a gem the server already has, the server's existing-gem response (`409`, already handled by `GemUploader#handle_response` as success/skip) is treated as success, not failure.
|
|
37
|
-
**Verify:** Run `sync` against an empty server, then immediately again; first run uploads N gems, second run skips all N with no errors and a zero failure count.
|
|
38
|
-
|
|
39
|
-
#### FR-1.5: Output distinguishes the three outcomes
|
|
40
|
-
`sync` output distinguishes (a) skipped because the server already has it, (b) uploaded an existing local artifact without rebuilding, and (c) built and uploaded from source. Outcomes (b) and (c) both count as `:synced` in the run summary; only (a) counts as `:skipped`, so the existing `:synced`/`:skipped` tally and `report_results` are unchanged. The distinction between (b) and (c) is conveyed in the per-gem output text using the existing `Output.skip`/`Output.step`/`Output.success` vocabulary, with wording that makes clear when a rebuild was avoided.
|
|
41
|
-
**Verify:** Output for a server-present gem shows a "skipped" line and increments `:skipped`; output for an artifact re-upload names that the cached artifact was uploaded without rebuilding and increments `:synced`; output for a fresh gem shows the build step and increments `:synced`.
|
|
42
|
-
|
|
43
|
-
#### FR-1.6: `version: latest` keeps today's always-fetch behavior
|
|
44
|
-
For `version: latest`, the version is only known after clone/checkout (`current_version` post-checkout), so `latest` is exempt from the "skip before fetching" and "re-upload without rebuild" guarantees. `latest` continues to fetch the repo and resolve the gemspec version as it does today (the existing `!gem_def.latest?` cache bypass). Once the version is resolved, the server-presence check still applies to decide whether the resolved version needs uploading.
|
|
45
|
-
**Verify:** A `version: latest` gem always fetches the repo; after resolving its gemspec version, if the server already has that version the gem is reported skipped, otherwise it is built/uploaded.
|
|
46
|
-
|
|
47
|
-
### Architectural Requirements
|
|
48
|
-
|
|
49
|
-
#### AR-1.1: Presence check lives behind the uploader/server-client seam
|
|
50
|
-
The server-presence query belongs with the HTTP client that already owns server communication (`GemUploader`, `lib/gemkeeper/gem_uploader.rb`), which holds `server_url`, a Faraday connection, `reachable?`, and the `POST /upload` call. Add `has_version?(name, version)` there; `GemSyncer` calls it rather than holding HTTP concerns. The existing `list_gems` stub is left as-is (it has its own `gemkeeper list` contract). Reusing the existing Faraday connection (with its `:multipart`/`:url_encoded` middleware) for the GET is acceptable.
|
|
51
|
-
**Verify:** `GemSyncer` contains no direct HTTP/Faraday code; `has_version?` is covered by `test/gemkeeper/test_gem_uploader.rb`.
|
|
52
|
-
|
|
53
|
-
#### AR-1.2: Reuse existing version resolution and cache-key normalization
|
|
54
|
-
The presence check and artifact lookup use the version produced by the existing resolution path (`resolve_version`, `latest_version!`, `current_version`) and the bare-semver normalization already applied for cache keys and filenames (`checkout_tag`, the former `cached?`). Artifact filenames are derived one way, as `<name>-<version>.gem` (see the platform assumption in Assumptions & Risks).
|
|
55
|
-
**Verify:** A `from_lockfile` gem resolves its version, then the presence check and any upload use that bare version and the `<name>-<version>.gem` filename.
|
|
56
|
-
|
|
57
|
-
#### AR-1.3: Preserve both deployment modes
|
|
58
|
-
The design must not assume `sync`'s `gems_path` equals the server's `gems_path`. In the shared Homebrew-service mode they differ (project-local build dir vs. absolute service store), and the HTTP bridge between them is retained. The presence check is therefore strictly server-side over HTTP, never a local-path comparison against the server's store.
|
|
59
|
-
**Verify:** A test exercising a server whose store path differs from `sync`'s `gems_path` still skips/uploads correctly based solely on server responses.
|
|
60
|
-
|
|
61
|
-
#### AR-1.4: Presence-check status → outcome mapping
|
|
62
|
-
`has_version?` maps server responses as follows: `200` → present; `404` → not present; `400` (invalid name/version — should not occur for config-sourced names) → raise as a programming error, not silently "absent"; a connection failure or timeout → `ServerNotReachableError` with the existing "run 'gemkeeper server start'" guidance. Because the skip decision now depends on a server round-trip, an unreachable server surfaces this actionable error early (before any git work) rather than treating the gem as present or absent.
|
|
63
|
-
**Verify:** Server down → `sync` exits with the not-reachable error and guidance; `400` → raises a non-`ServerNotReachableError`; `404` → gem treated as not present and the run continues.
|
|
64
|
-
|
|
65
|
-
#### AR-1.5: `GemSyncer` flow defers source work until after the cheap checks
|
|
66
|
-
`GemSyncer#sync` is reordered so that, for pinned and `from_lockfile` versions, the server-presence check and the local-artifact check run before any repo or manifest resolution. Repo URL resolution (`resolve_repo`), `GitRepository` creation, and `clone_or_pull` happen only when a build is actually required. The upload-existing-artifact path and the build-then-upload path are separated (e.g. distinct private methods) rather than the current single `build_and_upload`.
|
|
67
|
-
**Verify:** For a pinned gem the server already has, `sync` makes no manifest/repo resolution and creates no `GitRepository` (assert via stubs); the build and upload-existing paths are independently testable.
|
|
68
|
-
|
|
69
|
-
---
|
|
70
|
-
|
|
71
|
-
## Feature 2: Private-store presence endpoint
|
|
72
|
-
|
|
73
|
-
**Who & why:** `sync` needs an authoritative yes/no on whether the server's *private* store holds a specific gem version, without the Bundler `/info/<name>` endpoint's upstream fallback that proxies rubygems.org. A public gem sharing a private gem's name, or an offline upstream probe, must never affect the answer.
|
|
74
|
-
|
|
75
|
-
### Functional Requirements
|
|
76
|
-
|
|
77
|
-
#### FR-2.1: Read-only private-store presence endpoint
|
|
78
|
-
`CompactIndexServer` exposes `GET /gemkeeper/has/<name>/<version>` that consults the private `GemIndex` only and never invokes the upstream cache. It returns `200` when the index holds `<name>` with a version whose recorded number equals `<version>` (bare semver), and `404` otherwise. The response body is minimal (status is the signal).
|
|
79
|
-
**Verify:** With `mimir 1.0.5` uploaded, `GET /gemkeeper/has/mimir/1.0.5` → `200` and `GET /gemkeeper/has/mimir/9.9.9` → `404`; with the server offline and `mimir` absent, the same request returns `404` immediately with no upstream request and no 503.
|
|
80
|
-
|
|
81
|
-
#### FR-2.2: Name and version validation
|
|
82
|
-
The endpoint validates `<name>` with the existing `VALID_NAME` pattern and rejects a malformed name or empty version with `400`, consistent with the server's other parameterized routes.
|
|
83
|
-
**Verify:** `GET /gemkeeper/has/..%2Fetc/1.0.0` (or any name failing `VALID_NAME`) → `400`; a well-formed-but-absent gem → `404`.
|
|
84
|
-
|
|
85
|
-
### Architectural Requirements
|
|
86
|
-
|
|
87
|
-
#### AR-2.1: Endpoint reads only the private index and follows existing server patterns
|
|
88
|
-
The endpoint is wired through the existing router and built with the existing response helpers (`ResponseBuilder`/the small `not_found`/`invalid_name` responders), reading exclusively from `GemIndex` (`@index`) and never touching `UpstreamCache`. It must not regress the server's quality gates (rubocop clean, rubycritic ≥ 90) — keep the routing addition consistent with the `RESOURCE_ROUTES` table approach so `CompactIndexServer` does not gain a new smell.
|
|
89
|
-
**Verify:** The endpoint handler references `@index` only (no `@cache`); rubocop and rubycritic remain green after the addition.
|
|
90
|
-
|
|
91
|
-
---
|
|
92
|
-
|
|
93
|
-
## Data Requirements
|
|
94
|
-
|
|
95
|
-
No new persisted data or schema changes. The private-store presence endpoint reads the in-memory `GemIndex` built from the on-disk store; the on-disk layout (`gems_path` for build artifacts, `gems_path/gems` for the served store) is unchanged.
|
|
96
|
-
|
|
97
|
-
## Integration Points
|
|
98
|
-
|
|
99
|
-
- `lib/gemkeeper/gem_syncer.rb` — reordered `sync` flow; the `cached?` replacement (presence → re-upload → build/upload); split of `build_and_upload`.
|
|
100
|
-
- `lib/gemkeeper/gem_uploader.rb` — new `has_version?(name, version)`; reuses the existing Faraday connection and error mapping.
|
|
101
|
-
- `lib/gemkeeper/compact_index_server.rb` — new `GET /gemkeeper/has/<name>/<version>` route.
|
|
102
|
-
- `lib/gemkeeper/compact_index_server/gem_index.rb` — read path used by the endpoint (presence lookup against `@index`).
|
|
103
|
-
- `lib/gemkeeper/cli/commands/sync.rb` — unchanged wiring (`GemUploader.new(config.server_url)`); the uploader now also answers presence, and `report_results` is unchanged (re-upload counts as `:synced`).
|
|
104
|
-
|
|
105
|
-
## Related Specs
|
|
106
|
-
|
|
107
|
-
| Spec | Relationship | Affected Requirements |
|
|
108
|
-
|------|-------------|---------------------|
|
|
109
|
-
| Spec 20260529-091429: Replace Geminabox with compact index server | **Modifies** — corrects the sync↔serve cache contract introduced when the in-process compact-index server replaced Geminabox, and adds a private-store presence endpoint to that server | FR-1.1, FR-2.1, AR-1.1, AR-1.3 |
|
|
110
|
-
|
|
111
|
-
## Constraints
|
|
112
|
-
|
|
113
|
-
- Quality gates must stay green: `bundle exec rubocop` clean and `bundle exec rubycritic lib --no-browser` ≥ 90.
|
|
114
|
-
- Follow the established collaborator/value-object structure of the compact-index server refactor; no Metrics-cop suppression comments.
|
|
115
|
-
- Offline-first: for pinned and `from_lockfile` versions, the happy path and recovery path must not require network or git access when a valid local artifact exists (FR-1.3); the presence endpoint must never block on an upstream probe (FR-2.1).
|
|
116
|
-
- Tests: create `sync()` orchestration tests in `test/gemkeeper/test_gem_syncer.rb` (it currently tests only `resolve_repo`); extend `test/gemkeeper/test_gem_uploader.rb` (presence method) and `test/gemkeeper/test_compact_index_server.rb` (new endpoint); add the original-bug regression to `test/integration/test_server_lifecycle_integration.rb`.
|
|
117
|
-
- Add an `[Unreleased]` CHANGELOG entry under `Fixed`.
|
|
118
|
-
|
|
119
|
-
## Out of Scope
|
|
120
|
-
|
|
121
|
-
- Flattening the `gems_path` vs `gems_path/gems` layout or eliminating the redundant flat build artifact. This is a tidiness improvement that would change the on-disk layout and require migrating existing Homebrew-service stores; deferred to a future spec.
|
|
122
|
-
- Precompiled, platform-specific gem variants (filenames like `<name>-<version>-<platform>.gem`). Internal gems are built from source with the default `ruby` platform (see Assumptions); supporting published per-platform binaries is a future spec.
|
|
123
|
-
- Removing or redesigning the HTTP `POST /upload` bridge. It is required for the shared-service mode and is retained as-is.
|
|
124
|
-
- Changing the Bundler-facing `/info`, `/names`, `/versions`, or `/gems` endpoints.
|
|
125
|
-
- Auto-refreshing the server's in-memory index from disk without an upload. The upload remains the mechanism that informs a running server of a new gem.
|
|
126
|
-
- Changing version-resolution semantics (`latest`, `from_lockfile`, tag normalization).
|
|
127
|
-
|
|
128
|
-
## Assumptions & Risks
|
|
129
|
-
|
|
130
|
-
- **Platform assumption:** internal gems are built from source via `gem build` and are therefore the default `ruby` platform, producing a single `<name>-<version>.gem` (no platform suffix). This is confirmed by evidence — every gem currently built by `sync` lacks a platform suffix. Gems with C extensions still fall under this assumption (extensions compile at install time); only deliberately cross-compiled, published per-platform binaries would not, and those are out of scope.
|
|
131
|
-
- **Presence-implies-serveable invariant:** the private-store endpoint and `/gems/<file>` both read the same `GemIndex`, so a `200` from the presence endpoint implies the binary is serveable. If a future change makes them diverge, the skip decision would become unsafe.
|
|
132
|
-
- **`sync` requires a reachable server** (already true today, since it uploads). The presence check makes this dependency explicit and surfaces it earlier (AR-1.4).
|
|
133
|
-
- **Risk — artifact identity:** a stale or wrong local `.gem` could be uploaded; mitigated by the pre-upload name+version check (FR-1.3).
|
|
134
|
-
|
|
135
|
-
## Spec Completeness Checklist
|
|
136
|
-
|
|
137
|
-
- [x] **Scope & acceptance criteria** — Goals + per-FR **Verify** lines + Out of Scope bound the change; the bug and its reproduction are in Overview/FR-1.1; `latest` is explicitly scoped (FR-1.6).
|
|
138
|
-
- [x] **Testing strategy** — Constraints list the files to create/extend (notably new `sync()` tests in `test_gem_syncer.rb`) and the original-bug regression; FR Verify lines define conditions.
|
|
139
|
-
- [x] **Existing patterns** — Reuses `GemUploader` HTTP seam, `Output` vocabulary, bare-semver normalization, the router/`RESOURCE_ROUTES` and `ResponseBuilder` patterns (AR-1.1, AR-1.2, AR-2.1, FR-1.5).
|
|
140
|
-
- [x] **Dependencies** — No new libraries; reuses the Faraday client and adds one read-only endpoint (FR-2.1, AR-1.1).
|
|
141
|
-
- [x] **Architecture & interfaces** — `has_version?` on the uploader; reordered `GemSyncer` flow with split build/upload (AR-1.1, AR-1.5); private-store endpoint reads `@index` only (AR-2.1); data model unchanged.
|
|
142
|
-
- [x] **Error handling & failure modes** — Status→outcome mapping incl. `400`/`404`/connection failure (AR-1.4); idempotent/`409` handling (FR-1.4); corrupt/mismatched artifact fallthrough (FR-1.3); offline endpoint returns `404` not `503` (FR-2.1).
|
|
143
|
-
- [x] **Security review** — Low (not N/A). The private-store endpoint removes the `/info` upstream fallback, so private gem names are no longer leaked to rubygems.org for the cache check; `<name>` is validated by `VALID_NAME` server-side (FR-2.2) and the client treats `400` as a programming error (AR-1.4). Traffic is loopback; gems are developer-controlled.
|
|
144
|
-
- [x] **Performance impact** — One extra loopback `GET /gemkeeper/has/...` per gem (no upstream probe); negligible. Re-upload avoids the far costlier clone/build for pinned/`from_lockfile` (FR-1.3).
|
|
145
|
-
- [x] **Rollout & migration** — No data migration; on-disk layout unchanged. New endpoint is additive; behavior change is limited to the skip decision; existing stores keep working.
|
|
146
|
-
- [x] **Assumptions & risks** — Platform, presence-implies-serveable, server-reachability, and artifact-identity captured in Assumptions & Risks.
|
|
147
|
-
|
|
148
|
-
---
|
|
149
|
-
|
|
150
|
-
## Change Log
|
|
151
|
-
|
|
152
|
-
### Update from critique-consolidated-v-1.md
|
|
153
|
-
|
|
154
|
-
**Applied:**
|
|
155
|
-
- Replaced the `/info`-parsing cache check with a dedicated private-store endpoint `GET /gemkeeper/has/<name>/<version>` (new Feature 2; FR-1.2, FR-2.1, FR-2.2, AR-2.1) — resolves the upstream-fallback false-positive, offline-503, and private-name-leak findings (critique A, I).
|
|
156
|
-
- Scoped the "never re-clone" guarantee to pinned and `from_lockfile` versions; added FR-1.6 making `latest`'s always-fetch behavior explicit (critique B).
|
|
157
|
-
- Added the platform assumption (default `ruby` platform, `<name>-<version>.gem`); precompiled per-platform variants moved to Out of Scope (critique C).
|
|
158
|
-
- Required a pre-upload artifact identity check (name+version match; corrupt-artifact fallthrough) in FR-1.3 (critique D).
|
|
159
|
-
- Specified the `GemSyncer` flow change — defer repo/manifest resolution until after the presence and local-artifact checks, and split `build_and_upload` (AR-1.5) (critique E).
|
|
160
|
-
- Added the presence-check status→outcome mapping (AR-1.4) and decided third-outcome counting: re-upload counts as `:synced`, differentiated by output text only, leaving the CLI summary unchanged (FR-1.5) (critique F).
|
|
161
|
-
- Clarified `has_version?` is added and `list_gems` is left as-is; noted Faraday connection reuse is acceptable (AR-1.1) (critique G, H).
|
|
162
|
-
- Updated testing constraints to create `sync()` tests in `test_gem_syncer.rb` plus the original-bug regression and listed scenarios.
|
|
163
|
-
- Downgraded the security review from N/A to low-with-mitigations.
|
|
164
|
-
|
|
165
|
-
**Rejected:**
|
|
166
|
-
- None. (Flattening the `gems_path/gems` layout was raised as adjacent messiness but was already deliberately out of scope; left out of scope.)
|
|
167
|
-
|
|
168
|
-
**Reorganized:**
|
|
169
|
-
- Split the single feature into Feature 1 (client-side sync behavior) and Feature 2 (server-side private-store endpoint) so the new server surface has its own FRs/ARs.
|