scint 0.7.1 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +138 -265
- data/VERSION +1 -1
- data/lib/scint/cache/layout.rb +5 -0
- data/lib/scint/cli/install.rb +109 -168
- data/lib/scint/index/client.rb +13 -2
- data/lib/scint/installer/extension_builder.rb +34 -9
- data/lib/scint/spec_utils.rb +4 -0
- data/lib/scint.rb +12 -4
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4b276421c3f10fed0f5211f84930f1280c73b4b087891177b2146eeb25c4ff03
|
|
4
|
+
data.tar.gz: 70c5d92d2d14c747f71499196499a4ddf58a38e578ac7ae4e27deb3d645da2dc
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 855cf61d3a05a3bc5e4fa886a8e9243a307ed471051c374196733d71190a88b09ac2119915c0444edb7d3cdcc07f9111b4d8c552208d38ddc4ee1f2cf60adda1
|
|
7
|
+
data.tar.gz: 816ae198c494d1acc655f17bca68f29663d1df1e346c404af65ac90783b02d94473a067055beb0be7ce57c7e00300eddc3962ffc7d2fb2bb5f809c612598b7c4
|
data/README.md
CHANGED
|
@@ -1,336 +1,209 @@
|
|
|
1
1
|
# Scint
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
```
|
|
4
|
+
tobi ~/t/fizzy ❯❯❯ scint install
|
|
5
|
+
💎 Scintellating Gemfile into ./.bundle (scint 0.7.1, ruby 4.0.1)
|
|
4
6
|
|
|
5
|
-
|
|
7
|
+
Fetching index https://rubygems.org
|
|
6
8
|
|
|
7
|
-
|
|
9
|
+
162 gems installed total (162 cached). (1.42s, 15 workers used)
|
|
10
|
+
```
|
|
8
11
|
|
|
9
|
-
|
|
10
|
-
2. It writes standard `Gemfile.lock`.
|
|
11
|
-
3. It interoperates with Bundler runtime layout. For example, `BUNDLE_PATH=".bundle" bundle exec ...` is expected to work.
|
|
12
|
-
4. The intent is identical behavior with better install and execution throughput.
|
|
12
|
+
Scint is an experimental Bundler replacement. Pure Ruby, no dependencies.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
It reads your `Gemfile` and `Gemfile.lock`, writes standard `Gemfile.lock`, and materializes into `.bundle/` so `bundle exec` keeps working. Same behavior, much faster installs.
|
|
15
15
|
|
|
16
|
-
|
|
17
|
-
2. Materialize project-local runtime state into `.bundle/` as efficiently as possible (hardlinks where possible).
|
|
18
|
-
3. Execute work in explicit concurrent phases coordinated by a scheduler session.
|
|
16
|
+
The core idea:
|
|
19
17
|
|
|
20
|
-
|
|
18
|
+
1. Prepare artifacts once in a global cache (`~/.cache/scint`).
|
|
19
|
+
2. Materialize into `.bundle/` via clonefile/hardlink/copy.
|
|
20
|
+
3. Warm installs are effectively instant because no fetch, extract, or compile happens when the cache is populated.
|
|
21
21
|
|
|
22
|
-
|
|
22
|
+
## Why "Scint"
|
|
23
23
|
|
|
24
|
-
|
|
24
|
+
From *scintillation*: short, high-energy flashes.
|
|
25
25
|
|
|
26
26
|
1. Event-driven scheduling.
|
|
27
27
|
2. Burst parallelism where safe.
|
|
28
28
|
3. Tight phase boundaries with clear handoffs.
|
|
29
29
|
|
|
30
|
-
##
|
|
31
|
-
|
|
32
|
-
```bash
|
|
33
|
-
scint install
|
|
34
|
-
scint exec <command>
|
|
35
|
-
scint cache list
|
|
36
|
-
scint cache clear
|
|
37
|
-
scint cache dir
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
Benchmark helpers:
|
|
41
|
-
|
|
42
|
-
```bash
|
|
43
|
-
bin/scint-vs-bundler [--force] [--test-root /tmp/scint-tests] /path/to/project
|
|
44
|
-
bin/scint-bench-matrix [--force] --root /path/to/projects
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
`bin/scint-bench-matrix` is a generic runner for a root directory where each
|
|
48
|
-
immediate subdirectory is a Ruby project under git with both `Gemfile` and
|
|
49
|
-
`Gemfile.lock`. It runs bundler cold/warm and scint cold/warm via
|
|
50
|
-
`bin/scint-vs-bundler` and writes:
|
|
51
|
-
|
|
52
|
-
1. `logs/bench-<timestamp>/summary.tsv`
|
|
53
|
-
2. `logs/bench-<timestamp>/table.md`
|
|
54
|
-
|
|
55
|
-
Optional project smoke test convention:
|
|
56
|
-
|
|
57
|
-
1. If `<root>/<project>-test.sh` exists, matrix runs it after the benchmark.
|
|
58
|
-
2. Execution is:
|
|
59
|
-
`cd <root>/<project> && scint exec ../<project>-test.sh`
|
|
60
|
-
3. The script runs against the warm scint install and is included in
|
|
61
|
-
`summary.tsv`/`table.md` status.
|
|
30
|
+
## Install Pipeline
|
|
62
31
|
|
|
63
|
-
|
|
32
|
+
Every gem flows through a deterministic cache pipeline. Resolution decides *what* to install; the pipeline defines *how*.
|
|
64
33
|
|
|
65
|
-
```
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
scint-syscall-trace /tmp/scint-sys.log -- scint install --force
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
Compatibility example:
|
|
80
|
-
|
|
81
|
-
```bash
|
|
82
|
-
BUNDLE_PATH=".bundle" bundle exec ruby -v
|
|
34
|
+
```mermaid
|
|
35
|
+
flowchart TD
|
|
36
|
+
A["Fetch + Assemble"]
|
|
37
|
+
B[Compile]
|
|
38
|
+
C[Promote]
|
|
39
|
+
D[Materialize]
|
|
40
|
+
|
|
41
|
+
A -->|"gems: download .gem, unpack\ngit: clone, checkout, export"| B
|
|
42
|
+
B -->|"native extensions built\ninside assembling tree"| C
|
|
43
|
+
C -->|"atomic move to cached/\nwrite .spec.marshal + manifest"| D
|
|
44
|
+
D -->|"clonefile / hardlink / copy\nfrom cached/ to .bundle/"| E[Done]
|
|
83
45
|
```
|
|
84
46
|
|
|
85
|
-
|
|
47
|
+
### Phase details
|
|
86
48
|
|
|
87
|
-
1.
|
|
88
|
-
|
|
49
|
+
1. **Fetch + Assemble** into `inbound/` and `assembling/`
|
|
50
|
+
- Download gem payloads to `inbound/gems/`.
|
|
51
|
+
- Clone/fetch git repos into `inbound/gits/<sha256-slug>/`.
|
|
52
|
+
- Unpack `.gem` files into `assembling/<abi>/<full_name>/`.
|
|
53
|
+
- For git sources: checkout, submodules, then export the tree into `assembling/<abi>/<full_name>/`.
|
|
54
|
+
2. **Compile** in `assembling/`
|
|
55
|
+
- Native extension builds happen inside the assembling directory so outputs are part of the final tree.
|
|
56
|
+
3. **Promote** atomically to `cached/`
|
|
57
|
+
- Move `assembling/<abi>/<full_name>/` to `cached/<abi>/<full_name>/`.
|
|
58
|
+
- Write `cached/<abi>/<full_name>.spec.marshal` and `.manifest`.
|
|
59
|
+
- Failed builds never reach `cached/`.
|
|
60
|
+
4. **Materialize** to `.bundle/`
|
|
61
|
+
- Clonefile/reflink/hardlink/copy from `cached/<abi>/`.
|
|
62
|
+
- No rebuild if the cached artifact is valid.
|
|
89
63
|
|
|
90
|
-
|
|
64
|
+
One truth source for warm installs: `cached/<abi>`.
|
|
91
65
|
|
|
92
|
-
|
|
66
|
+
## Scheduler
|
|
93
67
|
|
|
94
|
-
|
|
95
|
-
2. `assembling`
|
|
96
|
-
3. `cached`
|
|
97
|
-
4. `materialize`
|
|
98
|
-
|
|
99
|
-
Resolution/planning still decides *what* to install; this pipeline defines *how* each artifact becomes globally reusable.
|
|
100
|
-
|
|
101
|
-
### Phase Contract
|
|
102
|
-
|
|
103
|
-
1. Fetch into `inbound`
|
|
104
|
-
- Gem payloads go to `inbound/gems/`.
|
|
105
|
-
- Git repositories go to `inbound/gits/` using deterministic names (for example `https_github_com__tobi__try`).
|
|
106
|
-
2. Assemble into `assembling`
|
|
107
|
-
- For `.gem` sources: unpack into `assembling/<abi>/<full_name>/`.
|
|
108
|
-
- For git sources: fetch/checkout/submodules in `inbound/gits`, then export/copy the selected tree into `assembling/<abi>/<full_name>/`.
|
|
109
|
-
3. Compile in `assembling`
|
|
110
|
-
- Native extension build happens inside the assembling directory so successful outputs are part of the final cached tree.
|
|
111
|
-
4. Promote atomically to `cached`
|
|
112
|
-
- On success, move `assembling/<abi>/<full_name>/` to `cached/<abi>/<full_name>/`.
|
|
113
|
-
- Write `cached/<abi>/<full_name>.spec.marshal`.
|
|
114
|
-
- Write optional manifest metadata for fast materialization.
|
|
115
|
-
5. Materialize to project path (`.bundle` or `BUNDLE_PATH`)
|
|
116
|
-
- Use clonefile/reflink/hardlink/copy fallback from `cached/<abi>/...`.
|
|
117
|
-
- Do not rebuild if cached artifact is already complete.
|
|
118
|
-
|
|
119
|
-
This gives one primary truth source for warm installs: `cached/<abi>`.
|
|
68
|
+
The `Scheduler` is the install session object. It owns the job graph, worker pool, and phase coordination.
|
|
120
69
|
|
|
121
70
|
```mermaid
|
|
122
|
-
flowchart
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
71
|
+
flowchart TD
|
|
72
|
+
subgraph "Early Parallel"
|
|
73
|
+
FI[fetch_index]
|
|
74
|
+
GC[git_clone]
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
FI --> R[resolve]
|
|
78
|
+
GC --> R
|
|
79
|
+
|
|
80
|
+
R --> P[plan]
|
|
81
|
+
|
|
82
|
+
subgraph "Install DAG"
|
|
83
|
+
DL[download] --> EX[extract]
|
|
84
|
+
EX --> LN[link]
|
|
85
|
+
LN --> BE[build_ext]
|
|
86
|
+
BE --> BS[binstub]
|
|
87
|
+
LN --> BS
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
P --> DL
|
|
91
|
+
P --> LN2["link (cached)"]
|
|
92
|
+
LN2 --> BS2[binstub]
|
|
129
93
|
```
|
|
130
94
|
|
|
131
|
-
|
|
95
|
+
Job priorities (lower = dispatched first):
|
|
132
96
|
|
|
133
|
-
|
|
134
|
-
|
|
97
|
+
| Priority | Job Type | Concurrency |
|
|
98
|
+
|----------|----------|-------------|
|
|
99
|
+
| 0 | `fetch_index` | all workers |
|
|
100
|
+
| 1 | `git_clone` | all workers |
|
|
101
|
+
| 2 | `resolve` | 1 |
|
|
102
|
+
| 3 | `download` | bounded (default 8) |
|
|
103
|
+
| 4 | `extract` | IO-limited |
|
|
104
|
+
| 5 | `link` | IO-limited |
|
|
105
|
+
| 6 | `build_ext` | CPU-limited (slots x make -j) |
|
|
106
|
+
| 7 | `binstub` | 1 |
|
|
135
107
|
|
|
136
|
-
|
|
108
|
+
Workers start at 1, scale dynamically up to `cpu_count * 2` (max 50). Compile concurrency is tuned separately: a small number of slots each running `make -jN` to keep CPU saturated without thrashing.
|
|
137
109
|
|
|
138
|
-
|
|
139
|
-
2. Priority classes by job type
|
|
140
|
-
3. Worker pool scaling
|
|
141
|
-
4. Job state transitions (`pending`, `running`, `completed`, `failed`)
|
|
142
|
-
5. Follow-up chaining (for phase handoff)
|
|
143
|
-
6. Fail-fast abort state and error collection
|
|
144
|
-
7. Progress/stats snapshots used by reporting
|
|
145
|
-
|
|
146
|
-
Workers do not own global install strategy. They execute task payloads with context supplied by scheduler enqueuing and phase sequencing.
|
|
147
|
-
|
|
148
|
-
```mermaid
|
|
149
|
-
sequenceDiagram
|
|
150
|
-
participant U as User
|
|
151
|
-
participant C as scint install
|
|
152
|
-
participant S as Scheduler(Session)
|
|
153
|
-
participant W as WorkerPool
|
|
154
|
-
participant T as Task Worker
|
|
155
|
-
|
|
156
|
-
U->>C: scint install
|
|
157
|
-
C->>S: start(max_workers, fail_fast)
|
|
158
|
-
C->>S: enqueue(fetch_index/git/...)
|
|
159
|
-
S->>W: dispatch ready jobs
|
|
160
|
-
W->>T: execute payload
|
|
161
|
-
T-->>S: complete/fail + result
|
|
162
|
-
S-->>C: wait_for phase completion
|
|
163
|
-
C->>S: enqueue next phase (download/link/build_ext)
|
|
164
|
-
S-->>C: stats + errors + aborted?
|
|
165
|
-
C-->>U: summary + lockfile/runtime outputs
|
|
166
|
-
```
|
|
110
|
+
Fail-fast mode aborts scheduling after the first hard failure.
|
|
167
111
|
|
|
168
|
-
##
|
|
169
|
-
|
|
170
|
-
```mermaid
|
|
171
|
-
stateDiagram-v2
|
|
172
|
-
[*] --> pending
|
|
173
|
-
pending --> running: dependencies satisfied + worker slot available
|
|
174
|
-
running --> completed: success
|
|
175
|
-
running --> failed: exception / command failure
|
|
176
|
-
completed --> [*]
|
|
177
|
-
failed --> [*]
|
|
178
|
-
```
|
|
179
|
-
|
|
180
|
-
## Data Layout (Target)
|
|
112
|
+
## Data Layout
|
|
181
113
|
|
|
182
114
|
Global cache (`~/.cache/scint`):
|
|
183
115
|
|
|
184
|
-
```
|
|
116
|
+
```
|
|
185
117
|
~/.cache/scint/
|
|
186
118
|
inbound/
|
|
187
|
-
gems
|
|
188
|
-
|
|
189
|
-
gits/
|
|
190
|
-
<deterministic_repo_slug>/
|
|
119
|
+
gems/<full_name>.gem
|
|
120
|
+
gits/<sha256-slug>/
|
|
191
121
|
assembling/
|
|
192
|
-
<ruby-abi>/
|
|
193
|
-
<full_name>/
|
|
122
|
+
<ruby-abi>/<full_name>/
|
|
194
123
|
cached/
|
|
195
124
|
<ruby-abi>/
|
|
196
125
|
<full_name>/
|
|
197
126
|
<full_name>.spec.marshal
|
|
198
127
|
<full_name>.manifest
|
|
199
128
|
index/
|
|
129
|
+
<source-slug>/
|
|
200
130
|
```
|
|
201
131
|
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
1. `cached/ruby-3.4.5-arm64-darwin24/zlib-3.2.1/`
|
|
205
|
-
2. `cached/ruby-3.4.5-arm64-darwin24/zlib-3.2.1.spec.marshal`
|
|
132
|
+
ABI key example: `ruby-4.0.1-arm64-darwin25`
|
|
206
133
|
|
|
207
134
|
Project-local runtime (`.bundle/`):
|
|
208
135
|
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
A cached artifact is considered valid only when *all* of the following are true:
|
|
220
|
-
|
|
221
|
-
1. `cached/<abi>/<full_name>/` exists and is a fully materialized tree.
|
|
222
|
-
2. `cached/<abi>/<full_name>.spec.marshal` exists and loads successfully.
|
|
223
|
-
3. `cached/<abi>/<full_name>.manifest` exists, parses, and its `version` is supported.
|
|
224
|
-
4. Manifest fields `full_name` and `abi` match the requested spec/ABI.
|
|
225
|
-
5. If the gem has native extensions, `ext/<abi>/<full_name>/gem.build_complete` exists.
|
|
226
|
-
|
|
227
|
-
Any failure means the cache entry is *invalid* and must be rebuilt (fetch/assemble/compile).
|
|
228
|
-
|
|
229
|
-
### Manifest schema
|
|
230
|
-
|
|
231
|
-
The manifest is a single UTF-8 JSON object written to
|
|
232
|
-
`cached/<abi>/<full_name>.manifest`. It is versioned and deterministically
|
|
233
|
-
serialized for repeatable cache reuse.
|
|
234
|
-
|
|
235
|
-
**Serialization rules**:
|
|
236
|
-
|
|
237
|
-
- Top-level keys sorted lexicographically.
|
|
238
|
-
- Array ordering is deterministic (see `files` below).
|
|
239
|
-
- JSON is emitted without extra whitespace (canonical/minified).
|
|
240
|
-
- No timestamps or host-specific values are stored.
|
|
241
|
-
|
|
242
|
-
**Schema (version 1)**:
|
|
243
|
-
|
|
244
|
-
- `version` (Integer, required): schema version (starting at `1`).
|
|
245
|
-
- `full_name` (String, required): `name-version(-platform)`.
|
|
246
|
-
- `abi` (String, required): Ruby ABI key (e.g. `ruby-3.4.5-arm64-darwin24`).
|
|
247
|
-
- `source` (Object, required):
|
|
248
|
-
- `type` (String): `rubygems`, `git`, or `path`.
|
|
249
|
-
- `uri` (String): canonical source URI.
|
|
250
|
-
- `revision` (String, git only): resolved commit SHA.
|
|
251
|
-
- `path` (String, path only): absolute source path.
|
|
252
|
-
- `files` (Array, required): sorted by `path` ascending. Each entry is:
|
|
253
|
-
- `path` (String): relative to the cached root.
|
|
254
|
-
- `type` (String): `file`, `dir`, or `symlink`.
|
|
255
|
-
- `mode` (Integer): numeric permission bits (e.g. `755`).
|
|
256
|
-
- `size` (Integer): bytes (directories use `0`).
|
|
257
|
-
- `sha256` (String, optional): content hash for files/symlinks.
|
|
258
|
-
- `build` (Object, required):
|
|
259
|
-
- `extensions` (Boolean): whether the gem builds native extensions.
|
|
260
|
-
- `ext_complete` (String, optional): completion marker path when extensions exist.
|
|
261
|
-
|
|
262
|
-
### Git slug normalization + collisions
|
|
263
|
-
|
|
264
|
-
Git cache directories use a deterministic slug derived from the source URI:
|
|
136
|
+
```
|
|
137
|
+
.bundle/
|
|
138
|
+
ruby/<major.minor.0>/
|
|
139
|
+
gems/ # materialized gem trees
|
|
140
|
+
specifications/ # gemspecs
|
|
141
|
+
bin/ # gem binstubs
|
|
142
|
+
bin/ # project-level wrappers
|
|
143
|
+
scint.lock.marshal
|
|
144
|
+
```
|
|
265
145
|
|
|
266
|
-
|
|
267
|
-
parsed URI for consistent normalization).
|
|
268
|
-
2. Compute `sha256(normalized_uri)`, use the first 16 hex characters.
|
|
146
|
+
Cache root precedence: `SCINT_CACHE` > `XDG_CACHE_HOME/scint` > `~/.cache/scint`.
|
|
269
147
|
|
|
270
|
-
|
|
271
|
-
manifest `source.uri` matches the normalized URI. If it does not, treat it as a
|
|
272
|
-
collision, emit telemetry, and fall back to a longer hash (e.g. full 64 hex) or
|
|
273
|
-
an additional suffix.
|
|
148
|
+
## Cache Validity
|
|
274
149
|
|
|
275
|
-
|
|
150
|
+
A cached artifact is valid when:
|
|
276
151
|
|
|
277
|
-
|
|
278
|
-
|
|
152
|
+
1. `cached/<abi>/<full_name>/` exists.
|
|
153
|
+
2. `.spec.marshal` exists and loads.
|
|
154
|
+
3. `.manifest` exists, parses, schema version is supported, and fields match.
|
|
155
|
+
4. If the gem has native extensions, the build-complete marker exists.
|
|
279
156
|
|
|
280
|
-
|
|
281
|
-
the gemspec loads cleanly.
|
|
282
|
-
- Emit telemetry counters (per run) such as `cache.manifest.missing`,
|
|
283
|
-
`cache.manifest.unsupported`, and `cache.manifest.collision`.
|
|
284
|
-
- Log a single warning per run with counts and cache root to guide deprecation.
|
|
157
|
+
Invalid entries are rebuilt through the full pipeline. Legacy entries without manifests are read-compatible but emit telemetry for deprecation tracking.
|
|
285
158
|
|
|
286
159
|
## Warm Path Guarantees
|
|
287
160
|
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
3. Materialization should be IO-bound and close to instantaneous on warm cache.
|
|
293
|
-
4. Incomplete assemblies must never be promoted; promotion is atomic.
|
|
294
|
-
|
|
295
|
-
## Concurrency Model
|
|
161
|
+
1. If `cached/<abi>/<full_name>/` is valid, no fetch/extract/compile occurs.
|
|
162
|
+
2. Deleting `.bundle/` triggers only materialization.
|
|
163
|
+
3. Materialization is IO-bound and close to instantaneous on warm cache.
|
|
164
|
+
4. Incomplete assemblies are never promoted.
|
|
296
165
|
|
|
297
|
-
|
|
166
|
+
## CLI
|
|
298
167
|
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
168
|
+
```bash
|
|
169
|
+
scint install # install from Gemfile.lock
|
|
170
|
+
scint add <gem> # add to Gemfile and install
|
|
171
|
+
scint remove <gem> # remove from Gemfile and install
|
|
172
|
+
scint exec <cmd> # run command in bundle context
|
|
173
|
+
scint cache list # list cached gems
|
|
174
|
+
scint cache clear # clear global cache
|
|
175
|
+
scint cache dir # print cache root
|
|
176
|
+
```
|
|
304
177
|
|
|
305
|
-
|
|
178
|
+
Options: `--jobs N`, `--path P`, `--verbose`, `--force`.
|
|
306
179
|
|
|
307
|
-
|
|
180
|
+
`scint exec` sets `GEM_HOME`/`GEM_PATH`, injects load paths from `scint.lock.marshal`, and execs the command. Bundler compatibility: `BUNDLE_PATH=".bundle" bundle exec` works against the same layout.
|
|
308
181
|
|
|
309
|
-
|
|
310
|
-
2. Native build failures include full captured command output.
|
|
311
|
-
3. Final summary reports installed/failed/skipped counts.
|
|
312
|
-
4. `.gitignore` warning is emitted when `.bundle/` is not ignored.
|
|
182
|
+
## Diagnostics
|
|
313
183
|
|
|
314
|
-
|
|
184
|
+
```bash
|
|
185
|
+
# Ruby sampling profile
|
|
186
|
+
SCINT_PROFILE=/tmp/profile.json scint install --force
|
|
315
187
|
|
|
316
|
-
|
|
188
|
+
# Ruby IO trace
|
|
189
|
+
SCINT_IO_TRACE=/tmp/io.jsonl scint install --force
|
|
317
190
|
|
|
318
|
-
|
|
191
|
+
# Summarize IO trace
|
|
192
|
+
scint-io-summary /tmp/io.jsonl
|
|
319
193
|
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
4. Rebuilds runtime lock from `Gemfile.lock` + installed gems when possible if missing.
|
|
194
|
+
# Syscall trace (strace/dtruss)
|
|
195
|
+
scint-syscall-trace /tmp/sys.log -- scint install --force
|
|
196
|
+
```
|
|
324
197
|
|
|
325
|
-
##
|
|
198
|
+
## Benchmarking
|
|
326
199
|
|
|
327
|
-
|
|
200
|
+
```bash
|
|
201
|
+
bin/scint-vs-bundler [--force] /path/to/project
|
|
202
|
+
bin/scint-bench-matrix [--force] --root /path/to/projects
|
|
203
|
+
```
|
|
328
204
|
|
|
329
|
-
|
|
330
|
-
2. Isolated compile worker process with a simple line-protocol RPC (`CALL` / `RESULT`) for stronger fault isolation.
|
|
331
|
-
3. More deterministic bulk operations for extraction/linking.
|
|
332
|
-
4. Better per-phase telemetry for latency and saturation analysis.
|
|
205
|
+
`scint-bench-matrix` runs cold/warm benchmarks for every project subdirectory and writes `logs/bench-<timestamp>/summary.tsv` and `table.md`. If `<root>/<project>-test.sh` exists, it runs as a smoke test after the benchmark.
|
|
333
206
|
|
|
334
207
|
## Status
|
|
335
208
|
|
|
336
|
-
|
|
209
|
+
Experimental. Optimized for architecture iteration speed.
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.
|
|
1
|
+
0.8.0
|
data/lib/scint/cache/layout.rb
CHANGED
|
@@ -150,6 +150,11 @@ module Scint
|
|
|
150
150
|
private
|
|
151
151
|
|
|
152
152
|
def default_root
|
|
153
|
+
return Scint.cache_root if Scint.respond_to?(:cache_root)
|
|
154
|
+
|
|
155
|
+
explicit = ENV["SCINT_CACHE"]
|
|
156
|
+
return File.expand_path(explicit) unless explicit.nil? || explicit.empty?
|
|
157
|
+
|
|
153
158
|
base = ENV["XDG_CACHE_HOME"] || File.join(Dir.home, ".cache")
|
|
154
159
|
File.join(base, "scint")
|
|
155
160
|
end
|
data/lib/scint/cli/install.rb
CHANGED
|
@@ -43,7 +43,7 @@ module Scint
|
|
|
43
43
|
class Install
|
|
44
44
|
RUNTIME_LOCK = "scint.lock.marshal"
|
|
45
45
|
|
|
46
|
-
def initialize(argv = [], without: nil, with: nil)
|
|
46
|
+
def initialize(argv = [], without: nil, with: nil, output: $stderr)
|
|
47
47
|
@argv = argv
|
|
48
48
|
@jobs = nil
|
|
49
49
|
@path = nil
|
|
@@ -51,6 +51,7 @@ module Scint
|
|
|
51
51
|
@force = false
|
|
52
52
|
@without_groups = nil
|
|
53
53
|
@with_groups = nil
|
|
54
|
+
@output = output
|
|
54
55
|
@download_pool = nil
|
|
55
56
|
@download_pool_lock = Thread::Mutex.new
|
|
56
57
|
@gemspec_cache = {}
|
|
@@ -63,7 +64,7 @@ module Scint
|
|
|
63
64
|
|
|
64
65
|
def _tmark(label, t0)
|
|
65
66
|
now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
66
|
-
|
|
67
|
+
@output.puts " [timing] #{label}: #{((now - t0) * 1000).round}ms" if ENV["SCINT_TIMING"]
|
|
67
68
|
now
|
|
68
69
|
end
|
|
69
70
|
|
|
@@ -80,14 +81,15 @@ module Scint
|
|
|
80
81
|
compile_slots = compile_slots_for(worker_count)
|
|
81
82
|
git_slots = git_slots_for(worker_count)
|
|
82
83
|
per_type_limits = install_task_limits(worker_count, compile_slots, git_slots)
|
|
83
|
-
|
|
84
|
-
|
|
84
|
+
@output.puts "#{GREEN}💎#{RESET} Scintellating Gemfile into #{BOLD}#{bundle_display}#{RESET} #{DIM}(scint #{VERSION}, ruby #{RUBY_VERSION})#{RESET}"
|
|
85
|
+
@output.puts
|
|
85
86
|
|
|
86
87
|
# 0. Build credential store from config files (~/.bundle/config, XDG scint/credentials)
|
|
87
88
|
@credentials = Credentials.new
|
|
88
89
|
|
|
89
90
|
# 1. Start the scheduler with 1 worker — scale up dynamically
|
|
90
|
-
scheduler = Scheduler.new(max_workers: worker_count, fail_fast: true, per_type_limits: per_type_limits
|
|
91
|
+
scheduler = Scheduler.new(max_workers: worker_count, fail_fast: true, per_type_limits: per_type_limits,
|
|
92
|
+
progress: Progress.new(output: @output))
|
|
91
93
|
scheduler.start
|
|
92
94
|
|
|
93
95
|
begin
|
|
@@ -160,7 +162,7 @@ module Scint
|
|
|
160
162
|
elapsed_ms = elapsed_ms_since(start_time)
|
|
161
163
|
worker_count = scheduler.stats[:workers]
|
|
162
164
|
warn_missing_bundle_gitignore_entry
|
|
163
|
-
|
|
165
|
+
@output.puts "\n#{GREEN}#{total_gems}#{RESET} gems installed total#{install_breakdown(cached: cached_gems, updated: updated_gems)}. #{DIM}(#{format_run_footer(elapsed_ms, worker_count)})#{RESET}"
|
|
164
166
|
return 0
|
|
165
167
|
end
|
|
166
168
|
|
|
@@ -185,14 +187,14 @@ module Scint
|
|
|
185
187
|
errors = scheduler.errors.dup
|
|
186
188
|
stats = scheduler.stats
|
|
187
189
|
if errors.any?
|
|
188
|
-
|
|
190
|
+
@output.puts "#{RED}Some gems failed to install:#{RESET}"
|
|
189
191
|
errors.each do |err|
|
|
190
192
|
error = err[:error]
|
|
191
|
-
|
|
193
|
+
@output.puts " #{BOLD}#{err[:name]}#{RESET}: #{error.message}"
|
|
192
194
|
emit_network_error_details(error)
|
|
193
195
|
end
|
|
194
196
|
elsif stats[:failed] > 0
|
|
195
|
-
|
|
197
|
+
@output.puts "#{YELLOW}Warning: #{stats[:failed]} jobs failed but no error details captured#{RESET}"
|
|
196
198
|
end
|
|
197
199
|
|
|
198
200
|
elapsed_ms = elapsed_ms_since(start_time)
|
|
@@ -205,14 +207,14 @@ module Scint
|
|
|
205
207
|
|
|
206
208
|
if has_failures
|
|
207
209
|
warn_missing_bundle_gitignore_entry
|
|
208
|
-
|
|
210
|
+
@output.puts "\n#{RED}Bundle failed!#{RESET} #{installed_total}/#{total_gems} gems installed total#{install_breakdown(cached: cached_gems, updated: updated_gems, compiled: compiled_gems, failed: failed_count)}. #{DIM}(#{format_run_footer(elapsed_ms, worker_count)})#{RESET}"
|
|
209
211
|
1
|
|
210
212
|
else
|
|
211
213
|
# 10. Write lockfile + runtime config only for successful installs
|
|
212
214
|
write_lockfile(resolved, gemfile, lockfile)
|
|
213
215
|
write_runtime_config(resolved, bundle_path)
|
|
214
216
|
warn_missing_bundle_gitignore_entry
|
|
215
|
-
|
|
217
|
+
@output.puts "\n#{GREEN}#{total_gems}#{RESET} gems installed total#{install_breakdown(cached: cached_gems, updated: updated_gems, compiled: compiled_gems)}. #{DIM}(#{format_run_footer(elapsed_ms, worker_count)})#{RESET}"
|
|
216
218
|
0
|
|
217
219
|
end
|
|
218
220
|
ensure
|
|
@@ -468,20 +470,20 @@ module Scint
|
|
|
468
470
|
if opts[:git]
|
|
469
471
|
git_source = find_matching_git_source(Array(lockfile&.sources), opts) || find_matching_git_source(gemfile.sources, opts)
|
|
470
472
|
revision_hint = git_source&.revision || git_source&.ref || opts[:ref] || opts[:branch] || opts[:tag] || "HEAD"
|
|
471
|
-
|
|
472
|
-
if
|
|
473
|
-
clone_git_repo(opts[:git],
|
|
474
|
-
elsif
|
|
475
|
-
fetch_git_repo(
|
|
473
|
+
git_repo = cache&.git_path(opts[:git])
|
|
474
|
+
if git_repo && !Dir.exist?(git_repo)
|
|
475
|
+
clone_git_repo(opts[:git], git_repo)
|
|
476
|
+
elsif git_repo && Dir.exist?(git_repo)
|
|
477
|
+
fetch_git_repo(git_repo)
|
|
476
478
|
end
|
|
477
|
-
if
|
|
479
|
+
if git_repo && Dir.exist?(git_repo)
|
|
478
480
|
begin
|
|
479
|
-
resolved_revision = resolve_git_revision(
|
|
481
|
+
resolved_revision = resolve_git_revision(git_repo, revision_hint)
|
|
480
482
|
cache_key = "#{opts[:git]}@#{resolved_revision}"
|
|
481
483
|
git_metadata = git_source_metadata_cache[cache_key]
|
|
482
484
|
unless git_metadata
|
|
483
485
|
git_metadata = build_git_path_gems_for_revision(
|
|
484
|
-
|
|
486
|
+
git_repo,
|
|
485
487
|
resolved_revision,
|
|
486
488
|
glob: opts[:glob],
|
|
487
489
|
source_desc: opts[:git],
|
|
@@ -552,8 +554,8 @@ module Scint
|
|
|
552
554
|
nil
|
|
553
555
|
end
|
|
554
556
|
|
|
555
|
-
def find_git_gemspec(
|
|
556
|
-
gemspec_paths = gemspec_paths_in_git_revision(
|
|
557
|
+
def find_git_gemspec(git_repo, revision, gem_name, glob: nil)
|
|
558
|
+
gemspec_paths = gemspec_paths_in_git_revision(git_repo, revision)
|
|
557
559
|
return nil if gemspec_paths.empty?
|
|
558
560
|
|
|
559
561
|
path = gemspec_paths[gem_name.to_s]
|
|
@@ -564,23 +566,23 @@ module Scint
|
|
|
564
566
|
path ||= gemspec_paths.values.first
|
|
565
567
|
return nil if path.nil?
|
|
566
568
|
|
|
567
|
-
load_git_gemspec(
|
|
569
|
+
load_git_gemspec(git_repo, revision, path)
|
|
568
570
|
rescue StandardError
|
|
569
571
|
nil
|
|
570
572
|
end
|
|
571
573
|
|
|
572
|
-
def build_git_path_gems_for_revision(
|
|
573
|
-
gemspec_paths = gemspec_paths_in_git_revision(
|
|
574
|
+
def build_git_path_gems_for_revision(git_repo, revision, glob: nil, source_desc: nil)
|
|
575
|
+
gemspec_paths = gemspec_paths_in_git_revision(git_repo, revision)
|
|
574
576
|
return {} if gemspec_paths.empty?
|
|
575
577
|
|
|
576
578
|
glob_regex = glob ? git_glob_to_regex(glob) : nil
|
|
577
579
|
data = {}
|
|
578
580
|
|
|
579
|
-
|
|
581
|
+
with_git_checkout(git_repo, revision) do |checkout_dir|
|
|
580
582
|
gemspec_paths.each_value do |path|
|
|
581
583
|
next if glob_regex && !path.match?(glob_regex)
|
|
582
584
|
|
|
583
|
-
gemspec =
|
|
585
|
+
gemspec = load_gemspec_from_checkout(checkout_dir, path)
|
|
584
586
|
next unless gemspec
|
|
585
587
|
|
|
586
588
|
deps = gemspec.dependencies
|
|
@@ -725,19 +727,19 @@ module Scint
|
|
|
725
727
|
by_source = git_specs.group_by { |s| s[:source] }
|
|
726
728
|
by_source.each do |source, specs|
|
|
727
729
|
uri, revision = git_source_ref(source)
|
|
728
|
-
|
|
730
|
+
git_repo = cache.git_path(uri)
|
|
729
731
|
# Do not invalidate an otherwise-usable lockfile just because this
|
|
730
732
|
# git source has not been cached yet in the current machine/session.
|
|
731
|
-
next unless Dir.exist?(
|
|
733
|
+
next unless Dir.exist?(git_repo)
|
|
732
734
|
|
|
733
735
|
resolved_revision = begin
|
|
734
|
-
resolve_git_revision(
|
|
736
|
+
resolve_git_revision(git_repo, revision)
|
|
735
737
|
rescue InstallError
|
|
736
738
|
nil
|
|
737
739
|
end
|
|
738
740
|
return false unless resolved_revision
|
|
739
741
|
|
|
740
|
-
gemspec_paths = gemspec_paths_in_git_revision(
|
|
742
|
+
gemspec_paths = gemspec_paths_in_git_revision(git_repo, resolved_revision)
|
|
741
743
|
gemspec_names = gemspec_paths.keys.to_set
|
|
742
744
|
return false if gemspec_names.empty?
|
|
743
745
|
|
|
@@ -749,9 +751,9 @@ module Scint
|
|
|
749
751
|
true
|
|
750
752
|
end
|
|
751
753
|
|
|
752
|
-
def gemspec_paths_in_git_revision(
|
|
754
|
+
def gemspec_paths_in_git_revision(git_repo, revision)
|
|
753
755
|
out, _err, status = git_capture3(
|
|
754
|
-
"
|
|
756
|
+
"-C", git_repo,
|
|
755
757
|
"ls-tree",
|
|
756
758
|
"-r",
|
|
757
759
|
"--name-only",
|
|
@@ -771,8 +773,8 @@ module Scint
|
|
|
771
773
|
{}
|
|
772
774
|
end
|
|
773
775
|
|
|
774
|
-
def runtime_dependencies_for_git_gemspec(
|
|
775
|
-
spec = load_git_gemspec(
|
|
776
|
+
def runtime_dependencies_for_git_gemspec(git_repo, revision, gemspec_path)
|
|
777
|
+
spec = load_git_gemspec(git_repo, revision, gemspec_path)
|
|
776
778
|
return nil unless spec
|
|
777
779
|
|
|
778
780
|
spec.dependencies.select { |dep| dep.type == :runtime }
|
|
@@ -780,35 +782,28 @@ module Scint
|
|
|
780
782
|
nil
|
|
781
783
|
end
|
|
782
784
|
|
|
783
|
-
def load_git_gemspec(
|
|
785
|
+
def load_git_gemspec(git_repo, revision, gemspec_path)
|
|
784
786
|
return nil if gemspec_path.to_s.empty?
|
|
785
787
|
|
|
786
|
-
|
|
787
|
-
|
|
788
|
+
with_git_checkout(git_repo, revision) do |checkout_dir|
|
|
789
|
+
load_gemspec_from_checkout(checkout_dir, gemspec_path)
|
|
788
790
|
end
|
|
789
791
|
rescue StandardError
|
|
790
792
|
nil
|
|
791
793
|
end
|
|
792
794
|
|
|
793
|
-
def
|
|
794
|
-
worktree = Dir.mktmpdir("scint-gemspec")
|
|
795
|
+
def with_git_checkout(git_repo, revision)
|
|
795
796
|
_out, _err, status = git_capture3(
|
|
796
|
-
"
|
|
797
|
-
"
|
|
798
|
-
"checkout",
|
|
799
|
-
"--force",
|
|
800
|
-
revision,
|
|
797
|
+
"-C", git_repo,
|
|
798
|
+
"checkout", "-f", revision,
|
|
801
799
|
)
|
|
802
800
|
return nil unless status.success?
|
|
803
801
|
|
|
804
|
-
|
|
805
|
-
yield worktree if block_given?
|
|
806
|
-
ensure
|
|
807
|
-
FileUtils.rm_rf(worktree) if worktree && !worktree.empty?
|
|
802
|
+
yield git_repo if block_given?
|
|
808
803
|
end
|
|
809
804
|
|
|
810
|
-
def
|
|
811
|
-
absolute_gemspec = File.join(
|
|
805
|
+
def load_gemspec_from_checkout(checkout_dir, gemspec_path)
|
|
806
|
+
absolute_gemspec = File.join(checkout_dir, gemspec_path)
|
|
812
807
|
return nil unless File.exist?(absolute_gemspec)
|
|
813
808
|
|
|
814
809
|
SpecUtils.load_gemspec(absolute_gemspec, isolate: true)
|
|
@@ -944,7 +939,7 @@ module Scint
|
|
|
944
939
|
end
|
|
945
940
|
return if source_uri.start_with?("/") || !source_uri.start_with?("http")
|
|
946
941
|
|
|
947
|
-
return if Cache::Validity.cached_valid?(spec, cache)
|
|
942
|
+
return if Scint::Cache::Validity.cached_valid?(spec, cache)
|
|
948
943
|
|
|
949
944
|
dest_path = cache.inbound_path(spec)
|
|
950
945
|
raise InstallError, "Missing cached gem file for #{spec.name}: #{dest_path}" unless File.exist?(dest_path)
|
|
@@ -1001,94 +996,72 @@ module Scint
|
|
|
1001
996
|
def ensure_git_repo_for_spec(spec, cache, fetch:)
|
|
1002
997
|
source = spec.source
|
|
1003
998
|
uri, _revision = git_source_ref(source)
|
|
1004
|
-
|
|
999
|
+
git_repo = cache.git_path(uri)
|
|
1005
1000
|
|
|
1006
|
-
# Serialize all git operations per
|
|
1001
|
+
# Serialize all git operations per repo — git uses index.lock
|
|
1007
1002
|
# and can't handle concurrent checkouts from the same repo.
|
|
1008
|
-
git_mutex_for(
|
|
1009
|
-
if Dir.exist?(
|
|
1010
|
-
fetch_git_repo(
|
|
1003
|
+
git_mutex_for(git_repo).synchronize do
|
|
1004
|
+
if Dir.exist?(git_repo)
|
|
1005
|
+
fetch_git_repo(git_repo) if fetch
|
|
1011
1006
|
else
|
|
1012
|
-
clone_git_repo(uri,
|
|
1013
|
-
fetch_git_repo(bare_repo)
|
|
1007
|
+
clone_git_repo(uri, git_repo)
|
|
1014
1008
|
end
|
|
1015
1009
|
end
|
|
1016
1010
|
|
|
1017
|
-
|
|
1011
|
+
git_repo
|
|
1018
1012
|
end
|
|
1019
1013
|
|
|
1020
1014
|
def assemble_git_spec(entry, cache, fetch: true)
|
|
1021
1015
|
spec = entry.spec
|
|
1022
|
-
return if Cache::Validity.cached_valid?(spec, cache)
|
|
1016
|
+
return if Scint::Cache::Validity.cached_valid?(spec, cache)
|
|
1023
1017
|
|
|
1024
1018
|
source = spec.source
|
|
1025
1019
|
uri, revision = git_source_ref(source)
|
|
1026
1020
|
submodules = git_source_submodules?(source)
|
|
1027
1021
|
|
|
1028
|
-
|
|
1022
|
+
git_repo = cache.git_path(uri)
|
|
1029
1023
|
|
|
1030
|
-
# Serialize all git operations per
|
|
1024
|
+
# Serialize all git operations per repo — git uses index.lock
|
|
1031
1025
|
# and can't handle concurrent checkouts from the same repo.
|
|
1032
|
-
git_mutex_for(
|
|
1033
|
-
tmp_checkout = nil
|
|
1026
|
+
git_mutex_for(git_repo).synchronize do
|
|
1034
1027
|
tmp_assembled = nil
|
|
1035
1028
|
|
|
1036
1029
|
begin
|
|
1037
|
-
if Dir.exist?(
|
|
1038
|
-
fetch_git_repo(
|
|
1030
|
+
if Dir.exist?(git_repo)
|
|
1031
|
+
fetch_git_repo(git_repo) if fetch
|
|
1039
1032
|
else
|
|
1040
|
-
clone_git_repo(uri,
|
|
1041
|
-
fetch_git_repo(bare_repo)
|
|
1033
|
+
clone_git_repo(uri, git_repo)
|
|
1042
1034
|
end
|
|
1043
1035
|
|
|
1044
|
-
resolved_revision = resolve_git_revision(
|
|
1036
|
+
resolved_revision = resolve_git_revision(git_repo, revision)
|
|
1045
1037
|
assembling = cache.assembling_path(spec)
|
|
1046
|
-
tmp_checkout = "#{assembling}.checkout.#{Process.pid}.#{Thread.current.object_id}.tmp"
|
|
1047
1038
|
tmp_assembled = "#{assembling}.#{Process.pid}.#{Thread.current.object_id}.tmp"
|
|
1048
1039
|
promoter = cache_promoter(cache)
|
|
1049
1040
|
|
|
1050
1041
|
FileUtils.rm_rf(assembling)
|
|
1051
|
-
FileUtils.rm_rf(tmp_checkout)
|
|
1052
1042
|
FileUtils.rm_rf(tmp_assembled)
|
|
1053
1043
|
FS.mkdir_p(File.dirname(assembling))
|
|
1054
1044
|
|
|
1055
1045
|
promoter.validate_within_root!(cache.root, assembling, label: "assembling")
|
|
1056
|
-
promoter.validate_within_root!(cache.root, tmp_checkout, label: "git-checkout")
|
|
1057
1046
|
promoter.validate_within_root!(cache.root, tmp_assembled, label: "git-assembled")
|
|
1058
1047
|
|
|
1059
|
-
|
|
1060
|
-
checkout_git_tree_with_submodules(
|
|
1061
|
-
bare_repo,
|
|
1062
|
-
tmp_checkout,
|
|
1063
|
-
resolved_revision,
|
|
1064
|
-
spec,
|
|
1065
|
-
uri,
|
|
1066
|
-
)
|
|
1067
|
-
else
|
|
1068
|
-
checkout_git_tree(
|
|
1069
|
-
bare_repo,
|
|
1070
|
-
tmp_checkout,
|
|
1071
|
-
resolved_revision,
|
|
1072
|
-
spec,
|
|
1073
|
-
uri,
|
|
1074
|
-
)
|
|
1075
|
-
end
|
|
1076
|
-
|
|
1077
|
-
# Remove .git artifacts from checkout so assembled output is
|
|
1078
|
-
# deterministic and contains no git internals.
|
|
1079
|
-
Dir.glob(File.join(tmp_checkout, "**", ".git"), File::FNM_DOTMATCH).each do |path|
|
|
1080
|
-
FileUtils.rm_rf(path)
|
|
1081
|
-
end
|
|
1048
|
+
checkout_git_revision(git_repo, resolved_revision, spec, uri, submodules: submodules)
|
|
1082
1049
|
|
|
1083
|
-
gem_root = resolve_git_gem_subdir(
|
|
1084
|
-
gem_rel = git_relative_root(
|
|
1050
|
+
gem_root = resolve_git_gem_subdir(git_repo, spec)
|
|
1051
|
+
gem_rel = git_relative_root(git_repo, gem_root)
|
|
1085
1052
|
dest_root = tmp_assembled
|
|
1086
1053
|
dest_path = gem_rel.empty? ? dest_root : File.join(dest_root, gem_rel)
|
|
1087
1054
|
|
|
1088
1055
|
promoter.validate_within_root!(cache.root, dest_path, label: "git-dest")
|
|
1089
1056
|
|
|
1090
1057
|
FS.clone_tree(gem_root, dest_path)
|
|
1091
|
-
|
|
1058
|
+
|
|
1059
|
+
# Remove .git artifacts so assembled output is deterministic.
|
|
1060
|
+
Dir.glob(File.join(tmp_assembled, "**", ".git"), File::FNM_DOTMATCH).each do |path|
|
|
1061
|
+
FileUtils.rm_rf(path)
|
|
1062
|
+
end
|
|
1063
|
+
|
|
1064
|
+
copy_gemspec_root_files(git_repo, gem_root, dest_root, spec)
|
|
1092
1065
|
FS.atomic_move(tmp_assembled, assembling)
|
|
1093
1066
|
|
|
1094
1067
|
gem_subdir = begin
|
|
@@ -1103,7 +1076,6 @@ module Scint
|
|
|
1103
1076
|
promote_assembled_gem(spec, cache, assembling, gemspec, extensions: false)
|
|
1104
1077
|
end
|
|
1105
1078
|
ensure
|
|
1106
|
-
FileUtils.rm_rf(tmp_checkout) if tmp_checkout && File.exist?(tmp_checkout)
|
|
1107
1079
|
FileUtils.rm_rf(tmp_assembled) if tmp_assembled && File.exist?(tmp_assembled)
|
|
1108
1080
|
end
|
|
1109
1081
|
end
|
|
@@ -1171,61 +1143,27 @@ module Scint
|
|
|
1171
1143
|
File.basename(gem_root)
|
|
1172
1144
|
end
|
|
1173
1145
|
|
|
1174
|
-
def
|
|
1175
|
-
FileUtils.mkdir_p(destination)
|
|
1146
|
+
def checkout_git_revision(git_repo, resolved_revision, spec, uri, submodules: false)
|
|
1176
1147
|
_out, err, status = git_capture3(
|
|
1177
|
-
"
|
|
1178
|
-
"
|
|
1179
|
-
"checkout",
|
|
1180
|
-
"-f",
|
|
1181
|
-
resolved_revision,
|
|
1182
|
-
"--",
|
|
1183
|
-
".",
|
|
1148
|
+
"-C", git_repo,
|
|
1149
|
+
"checkout", "-f", resolved_revision,
|
|
1184
1150
|
)
|
|
1185
1151
|
unless status.success?
|
|
1186
1152
|
raise InstallError, "Git checkout failed for #{spec.name} (#{uri}@#{resolved_revision}): #{err.to_s.strip}"
|
|
1187
1153
|
end
|
|
1188
|
-
end
|
|
1189
1154
|
|
|
1190
|
-
|
|
1191
|
-
worktree = "#{destination}.worktree"
|
|
1192
|
-
FileUtils.rm_rf(worktree)
|
|
1155
|
+
return unless submodules
|
|
1193
1156
|
|
|
1194
|
-
|
|
1195
|
-
"
|
|
1196
|
-
"
|
|
1197
|
-
"
|
|
1198
|
-
"
|
|
1199
|
-
"--
|
|
1200
|
-
|
|
1201
|
-
resolved_revision,
|
|
1157
|
+
_sub_out, sub_err, sub_status = git_capture3(
|
|
1158
|
+
"-C", git_repo,
|
|
1159
|
+
"-c", "protocol.file.allow=always",
|
|
1160
|
+
"submodule",
|
|
1161
|
+
"update",
|
|
1162
|
+
"--init",
|
|
1163
|
+
"--recursive",
|
|
1202
1164
|
)
|
|
1203
|
-
unless
|
|
1204
|
-
raise InstallError, "Git
|
|
1205
|
-
end
|
|
1206
|
-
|
|
1207
|
-
begin
|
|
1208
|
-
_sub_out, sub_err, sub_status = git_capture3(
|
|
1209
|
-
"-C", worktree,
|
|
1210
|
-
"-c", "protocol.file.allow=always",
|
|
1211
|
-
"submodule",
|
|
1212
|
-
"update",
|
|
1213
|
-
"--init",
|
|
1214
|
-
"--recursive",
|
|
1215
|
-
)
|
|
1216
|
-
unless sub_status.success?
|
|
1217
|
-
raise InstallError, "Git submodule update failed for #{spec.name} (#{uri}@#{resolved_revision}): #{sub_err.to_s.strip}"
|
|
1218
|
-
end
|
|
1219
|
-
|
|
1220
|
-
FS.clone_tree(worktree, destination)
|
|
1221
|
-
|
|
1222
|
-
# Keep cache/extracted trees deterministic and detached from git internals.
|
|
1223
|
-
Dir.glob(File.join(destination, "**", ".git"), File::FNM_DOTMATCH).each do |path|
|
|
1224
|
-
FileUtils.rm_rf(path)
|
|
1225
|
-
end
|
|
1226
|
-
ensure
|
|
1227
|
-
git_capture3("--git-dir", bare_repo, "worktree", "remove", "--force", worktree)
|
|
1228
|
-
FileUtils.rm_rf(worktree)
|
|
1165
|
+
unless sub_status.success?
|
|
1166
|
+
raise InstallError, "Git submodule update failed for #{spec.name} (#{uri}@#{resolved_revision}): #{sub_err.to_s.strip}"
|
|
1229
1167
|
end
|
|
1230
1168
|
end
|
|
1231
1169
|
|
|
@@ -1237,32 +1175,35 @@ module Scint
|
|
|
1237
1175
|
end
|
|
1238
1176
|
end
|
|
1239
1177
|
|
|
1240
|
-
def clone_git_repo(uri,
|
|
1241
|
-
FS.mkdir_p(File.dirname(
|
|
1242
|
-
_out, err, status = git_capture3("clone",
|
|
1178
|
+
def clone_git_repo(uri, git_repo)
|
|
1179
|
+
FS.mkdir_p(File.dirname(git_repo))
|
|
1180
|
+
_out, err, status = git_capture3("clone", uri.to_s, git_repo)
|
|
1243
1181
|
unless status.success?
|
|
1244
1182
|
raise InstallError, "Git clone failed for #{uri}: #{err.to_s.strip}"
|
|
1245
1183
|
end
|
|
1246
1184
|
end
|
|
1247
1185
|
|
|
1248
|
-
def fetch_git_repo(
|
|
1186
|
+
def fetch_git_repo(git_repo)
|
|
1249
1187
|
_out, err, status = git_capture3(
|
|
1250
|
-
"
|
|
1188
|
+
"-C", git_repo,
|
|
1251
1189
|
"fetch",
|
|
1252
1190
|
"--prune",
|
|
1191
|
+
"--force",
|
|
1253
1192
|
"origin",
|
|
1254
|
-
"+refs/heads/*:refs/heads/*",
|
|
1255
|
-
"+refs/tags/*:refs/tags/*",
|
|
1256
1193
|
)
|
|
1257
1194
|
unless status.success?
|
|
1258
|
-
raise InstallError, "Git fetch failed for #{
|
|
1195
|
+
raise InstallError, "Git fetch failed for #{git_repo}: #{err.to_s.strip}"
|
|
1259
1196
|
end
|
|
1260
1197
|
end
|
|
1261
1198
|
|
|
1262
|
-
def resolve_git_revision(
|
|
1263
|
-
|
|
1199
|
+
def resolve_git_revision(git_repo, revision)
|
|
1200
|
+
# Try origin/<revision> first so we pick up fetched branch tips.
|
|
1201
|
+
out, _err, status = git_capture3("-C", git_repo, "rev-parse", "origin/#{revision}^{commit}")
|
|
1202
|
+
return out.strip if status.success?
|
|
1203
|
+
|
|
1204
|
+
out, err, status = git_capture3("-C", git_repo, "rev-parse", "#{revision}^{commit}")
|
|
1264
1205
|
unless status.success?
|
|
1265
|
-
raise InstallError, "Unable to resolve git revision #{revision.inspect} in #{
|
|
1206
|
+
raise InstallError, "Unable to resolve git revision #{revision.inspect} in #{git_repo}: #{err.to_s.strip}"
|
|
1266
1207
|
end
|
|
1267
1208
|
out.strip
|
|
1268
1209
|
end
|
|
@@ -1474,7 +1415,7 @@ module Scint
|
|
|
1474
1415
|
assembling = cache.assembling_path(entry.spec)
|
|
1475
1416
|
base = if entry.cached_path
|
|
1476
1417
|
entry.cached_path
|
|
1477
|
-
elsif Cache::Validity.cached_valid?(entry.spec, cache)
|
|
1418
|
+
elsif Scint::Cache::Validity.cached_valid?(entry.spec, cache)
|
|
1478
1419
|
cached_dir
|
|
1479
1420
|
elsif Dir.exist?(assembling)
|
|
1480
1421
|
assembling
|
|
@@ -1728,12 +1669,12 @@ module Scint
|
|
|
1728
1669
|
return if gem_names.empty?
|
|
1729
1670
|
|
|
1730
1671
|
if ENV["SCINT_TIMING"]
|
|
1731
|
-
|
|
1672
|
+
@output.puts " [timing] prelink: #{gem_names.size} gems via linker"
|
|
1732
1673
|
end
|
|
1733
1674
|
|
|
1734
1675
|
FS.bulk_link_gems(cache_abi_dir, gems_dir, gem_names)
|
|
1735
1676
|
rescue StandardError => e
|
|
1736
|
-
|
|
1677
|
+
@output.puts("bulk prelink warning: #{e.message}") if ENV["SCINT_DEBUG"]
|
|
1737
1678
|
end
|
|
1738
1679
|
|
|
1739
1680
|
def load_cached_gemspec(spec, cache, extracted_path)
|
|
@@ -1833,7 +1774,7 @@ module Scint
|
|
|
1833
1774
|
promoter.with_staging_dir(prefix: "cached") do |staging|
|
|
1834
1775
|
FS.clone_tree(assembling_path, staging)
|
|
1835
1776
|
manifest = build_cached_manifest(spec, cache, staging, extensions: extensions)
|
|
1836
|
-
Cache::Manifest.write_dotfiles(staging, manifest)
|
|
1777
|
+
Scint::Cache::Manifest.write_dotfiles(staging, manifest)
|
|
1837
1778
|
spec_payload = gemspec ? gemspec.to_ruby : nil
|
|
1838
1779
|
result = promoter.promote_tree(
|
|
1839
1780
|
staging_path: staging,
|
|
@@ -1858,11 +1799,11 @@ module Scint
|
|
|
1858
1799
|
FS.mkdir_p(File.dirname(spec_path))
|
|
1859
1800
|
|
|
1860
1801
|
FS.atomic_write(spec_path, spec_payload) if spec_payload
|
|
1861
|
-
Cache::Manifest.write(manifest_path, manifest)
|
|
1802
|
+
Scint::Cache::Manifest.write(manifest_path, manifest)
|
|
1862
1803
|
end
|
|
1863
1804
|
|
|
1864
1805
|
def build_cached_manifest(spec, cache, gem_dir, extensions:)
|
|
1865
|
-
Cache::Manifest.build(
|
|
1806
|
+
Scint::Cache::Manifest.build(
|
|
1866
1807
|
spec: spec,
|
|
1867
1808
|
gem_dir: gem_dir,
|
|
1868
1809
|
abi_key: Platform.abi_key,
|
|
@@ -2402,17 +2343,17 @@ module Scint
|
|
|
2402
2343
|
return if (headers.nil? || headers.empty?) && body.empty?
|
|
2403
2344
|
|
|
2404
2345
|
if headers && !headers.empty?
|
|
2405
|
-
|
|
2346
|
+
@output.puts " headers:"
|
|
2406
2347
|
headers.sort.each do |key, value|
|
|
2407
|
-
|
|
2348
|
+
@output.puts " #{key}: #{value}"
|
|
2408
2349
|
end
|
|
2409
2350
|
end
|
|
2410
2351
|
|
|
2411
2352
|
return if body.empty?
|
|
2412
2353
|
|
|
2413
|
-
|
|
2354
|
+
@output.puts " body:"
|
|
2414
2355
|
body.each_line do |line|
|
|
2415
|
-
|
|
2356
|
+
@output.puts " #{line.rstrip}"
|
|
2416
2357
|
end
|
|
2417
2358
|
end
|
|
2418
2359
|
|
|
@@ -2421,7 +2362,7 @@ module Scint
|
|
|
2421
2362
|
return unless File.file?(path)
|
|
2422
2363
|
return if gitignore_has_bundle_entry?(path)
|
|
2423
2364
|
|
|
2424
|
-
|
|
2365
|
+
@output.puts "#{YELLOW}Warning: .gitignore exists but does not ignore .bundle (add `.bundle/`).#{RESET}"
|
|
2425
2366
|
end
|
|
2426
2367
|
|
|
2427
2368
|
def gitignore_has_bundle_entry?(path)
|
data/lib/scint/index/client.rb
CHANGED
|
@@ -143,8 +143,19 @@ module Scint
|
|
|
143
143
|
private
|
|
144
144
|
|
|
145
145
|
def default_cache_dir
|
|
146
|
-
|
|
147
|
-
|
|
146
|
+
root =
|
|
147
|
+
if Scint.respond_to?(:cache_root)
|
|
148
|
+
Scint.cache_root
|
|
149
|
+
else
|
|
150
|
+
explicit = ENV["SCINT_CACHE"]
|
|
151
|
+
if explicit && !explicit.empty?
|
|
152
|
+
File.expand_path(explicit)
|
|
153
|
+
else
|
|
154
|
+
xdg = ENV["XDG_CACHE_HOME"] || File.join(Dir.home, ".cache")
|
|
155
|
+
File.join(xdg, "scint")
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
File.join(root, "index", Cache.slug_for(@uri))
|
|
148
159
|
end
|
|
149
160
|
|
|
150
161
|
# Fetch a top-level endpoint (names or versions).
|
|
@@ -20,6 +20,7 @@ module Scint
|
|
|
20
20
|
def build(prepared_gem, bundle_path, cache_layout, abi_key: Platform.abi_key, compile_slots: 1, output_tail: nil)
|
|
21
21
|
spec = prepared_gem.spec
|
|
22
22
|
build_ruby_dir = cache_layout.install_ruby_dir
|
|
23
|
+
source_ruby_dir = Platform.ruby_install_dir(bundle_path)
|
|
23
24
|
src_dir = prepared_gem.extracted_path
|
|
24
25
|
|
|
25
26
|
marker = build_marker_path(src_dir)
|
|
@@ -47,7 +48,17 @@ module Scint
|
|
|
47
48
|
# source-tree specific.
|
|
48
49
|
ext_build_dir = File.join(build_root, idx.to_s)
|
|
49
50
|
FS.mkdir_p(ext_build_dir)
|
|
50
|
-
compile_extension(
|
|
51
|
+
compile_extension(
|
|
52
|
+
ext_dir,
|
|
53
|
+
ext_build_dir,
|
|
54
|
+
install_dir,
|
|
55
|
+
staged_src_dir,
|
|
56
|
+
spec,
|
|
57
|
+
build_ruby_dir,
|
|
58
|
+
compile_slots,
|
|
59
|
+
output_tail,
|
|
60
|
+
source_ruby_dir,
|
|
61
|
+
)
|
|
51
62
|
end
|
|
52
63
|
|
|
53
64
|
sync_extensions_into_gem(install_dir, src_dir)
|
|
@@ -144,9 +155,9 @@ module Scint
|
|
|
144
155
|
dirs.uniq
|
|
145
156
|
end
|
|
146
157
|
|
|
147
|
-
def compile_extension(ext_dir, build_dir, install_dir, gem_dir, spec, build_ruby_dir, compile_slots, output_tail = nil)
|
|
158
|
+
def compile_extension(ext_dir, build_dir, install_dir, gem_dir, spec, build_ruby_dir, compile_slots, output_tail = nil, source_ruby_dir = nil)
|
|
148
159
|
make_jobs = adaptive_make_jobs(compile_slots)
|
|
149
|
-
env = build_env(gem_dir, build_ruby_dir, make_jobs)
|
|
160
|
+
env = build_env(gem_dir, build_ruby_dir, make_jobs, source_ruby_dir: source_ruby_dir)
|
|
150
161
|
|
|
151
162
|
if File.exist?(File.join(ext_dir, "extconf.rb"))
|
|
152
163
|
compile_extconf(ext_dir, gem_dir, build_dir, install_dir, env, make_jobs, output_tail)
|
|
@@ -245,12 +256,14 @@ module Scint
|
|
|
245
256
|
File.join(gem_dir, BUILD_MARKER)
|
|
246
257
|
end
|
|
247
258
|
|
|
248
|
-
def build_env(gem_dir, build_ruby_dir, make_jobs)
|
|
259
|
+
def build_env(gem_dir, build_ruby_dir, make_jobs, source_ruby_dir: nil)
|
|
249
260
|
ruby_bin = File.join(build_ruby_dir, "bin")
|
|
250
261
|
path = [ruby_bin, ENV["PATH"]].compact.reject(&:empty?).join(File::PATH_SEPARATOR)
|
|
262
|
+
inherited_gem_paths = ENV.fetch("GEM_PATH", "").split(File::PATH_SEPARATOR)
|
|
263
|
+
gem_path_entries = [build_ruby_dir, source_ruby_dir, *inherited_gem_paths].compact.reject(&:empty?).uniq
|
|
251
264
|
{
|
|
252
265
|
"GEM_HOME" => build_ruby_dir,
|
|
253
|
-
"GEM_PATH" =>
|
|
266
|
+
"GEM_PATH" => gem_path_entries.join(File::PATH_SEPARATOR),
|
|
254
267
|
"BUNDLE_PATH" => build_ruby_dir,
|
|
255
268
|
"BUNDLE_GEMFILE" => "",
|
|
256
269
|
"MAKEFLAGS" => "-j#{make_jobs}",
|
|
@@ -280,7 +293,7 @@ module Scint
|
|
|
280
293
|
|
|
281
294
|
# Stream output line-by-line so the UX gets live compile progress
|
|
282
295
|
# instead of waiting for the entire subprocess to finish.
|
|
283
|
-
all_output = +""
|
|
296
|
+
all_output = +"".b
|
|
284
297
|
tail_lines = []
|
|
285
298
|
cmd_label = "$ #{cmd.join(" ")}"
|
|
286
299
|
|
|
@@ -289,8 +302,8 @@ module Scint
|
|
|
289
302
|
out_err.set_encoding("ASCII-8BIT")
|
|
290
303
|
|
|
291
304
|
out_err.each_line do |line|
|
|
292
|
-
stripped = line.rstrip
|
|
293
305
|
all_output << line
|
|
306
|
+
stripped = sanitize_output(line).rstrip
|
|
294
307
|
next if stripped.empty?
|
|
295
308
|
|
|
296
309
|
tail_lines << stripped
|
|
@@ -303,7 +316,7 @@ module Scint
|
|
|
303
316
|
|
|
304
317
|
status = wait_thr.value
|
|
305
318
|
unless status.success?
|
|
306
|
-
details = all_output.strip
|
|
319
|
+
details = sanitize_output(all_output).strip
|
|
307
320
|
message = "Command failed (exit #{status.exitstatus}): #{cmd.join(" ")}"
|
|
308
321
|
message = "#{message}\n#{details}" unless details.empty?
|
|
309
322
|
raise ExtensionBuildError, message
|
|
@@ -311,6 +324,17 @@ module Scint
|
|
|
311
324
|
end
|
|
312
325
|
end
|
|
313
326
|
|
|
327
|
+
def sanitize_output(raw)
|
|
328
|
+
return "" if raw.nil? || raw.empty?
|
|
329
|
+
|
|
330
|
+
raw.to_s
|
|
331
|
+
.dup
|
|
332
|
+
.force_encoding(Encoding::BINARY)
|
|
333
|
+
.encode(Encoding::UTF_8, invalid: :replace, undef: :replace, replace: "?")
|
|
334
|
+
rescue EncodingError
|
|
335
|
+
raw.to_s.encode(Encoding::UTF_8, invalid: :replace, undef: :replace, replace: "?")
|
|
336
|
+
end
|
|
337
|
+
|
|
314
338
|
def spec_full_name(spec)
|
|
315
339
|
SpecUtils.full_name(spec)
|
|
316
340
|
end
|
|
@@ -318,7 +342,8 @@ module Scint
|
|
|
318
342
|
private_class_method :find_extension_dirs, :compile_extension,
|
|
319
343
|
:compile_extconf, :compile_cmake, :compile_rake,
|
|
320
344
|
:find_rake_executable, :link_extensions,
|
|
321
|
-
:build_env, :run_cmd, :
|
|
345
|
+
:build_env, :run_cmd, :sanitize_output,
|
|
346
|
+
:prebuilt_missing_for_ruby?
|
|
322
347
|
end
|
|
323
348
|
end
|
|
324
349
|
end
|
data/lib/scint/spec_utils.rb
CHANGED
|
@@ -108,7 +108,11 @@ module Scint
|
|
|
108
108
|
|
|
109
109
|
def load_gemspec_direct(absolute_path)
|
|
110
110
|
GEMSPEC_LOAD_MUTEX.synchronize do
|
|
111
|
+
old_stderr = $stderr
|
|
112
|
+
$stderr = StringIO.new
|
|
111
113
|
::Gem::Specification.load(absolute_path)
|
|
114
|
+
ensure
|
|
115
|
+
$stderr = old_stderr
|
|
112
116
|
end
|
|
113
117
|
end
|
|
114
118
|
private_class_method :load_gemspec_direct
|
data/lib/scint.rb
CHANGED
|
@@ -15,16 +15,24 @@ module Scint
|
|
|
15
15
|
|
|
16
16
|
# XDG-based cache root
|
|
17
17
|
def self.cache_root
|
|
18
|
-
@cache_root ||=
|
|
19
|
-
ENV.fetch("XDG_CACHE_HOME", File.join(Dir.home, ".cache")),
|
|
20
|
-
"scint"
|
|
21
|
-
)
|
|
18
|
+
@cache_root ||= default_cache_root
|
|
22
19
|
end
|
|
23
20
|
|
|
24
21
|
def self.cache_root=(path)
|
|
25
22
|
@cache_root = path
|
|
26
23
|
end
|
|
27
24
|
|
|
25
|
+
def self.default_cache_root
|
|
26
|
+
explicit = ENV["SCINT_CACHE"]
|
|
27
|
+
return File.expand_path(explicit) unless explicit.nil? || explicit.empty?
|
|
28
|
+
|
|
29
|
+
File.join(
|
|
30
|
+
ENV.fetch("XDG_CACHE_HOME", File.join(Dir.home, ".cache")),
|
|
31
|
+
"scint"
|
|
32
|
+
)
|
|
33
|
+
end
|
|
34
|
+
private_class_method :default_cache_root
|
|
35
|
+
|
|
28
36
|
# Shared data structures used across all modules
|
|
29
37
|
Dependency = Struct.new(
|
|
30
38
|
:name, :version_reqs, :source, :groups, :platforms, :require_paths,
|