scint 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 54c3f7baaea3992bcba60136b76f3b08caa580717d82906ebed3a0a69fb23314
-  data.tar.gz: 246966d0ba167f43b8c0aee1205e95377afec245dca33060fe3fcce202d0c3ab
+  metadata.gz: 4b276421c3f10fed0f5211f84930f1280c73b4b087891177b2146eeb25c4ff03
+  data.tar.gz: 70c5d92d2d14c747f71499196499a4ddf58a38e578ac7ae4e27deb3d645da2dc
 SHA512:
-  metadata.gz: 84c58296b1c391e68503090ea3889a3a469c8f63ca6c289710dc6a5f986cd33059aec6a61038ab14c2f653aefaa1675815d2e13a5ba3fb49dad6fe825cf8d3a5
-  data.tar.gz: 1cde1ae7ff37856bf44a4f909e024cb094ba1436fe062536aa4c73860ce0aa7c2494c2f5dd09ca14af5be8b226d8ada36ec212278b08a55902da708d734bea3d
+  metadata.gz: 855cf61d3a05a3bc5e4fa886a8e9243a307ed471051c374196733d71190a88b09ac2119915c0444edb7d3cdcc07f9111b4d8c552208d38ddc4ee1f2cf60adda1
+  data.tar.gz: 816ae198c494d1acc655f17bca68f29663d1df1e346c404af65ac90783b02d94473a067055beb0be7ce57c7e00300eddc3962ffc7d2fb2bb5f809c612598b7c4

data/FEATURES.md

@@ -10,4 +10,8 @@
 - the installation process involves compilation. We attempt to have compilation happen while its not blocking other operations, but also only one compilation at a time
 - we have a book keeping object that governs the worker pools and that's present during each step (fetch, extract, compile, install) and recieves the tasks for each phase from the workers. 
 - i suspect that we need to fork of a worker for compilation which we then have to communicate with via some rpc format. simple "-> CALL method, <- RESULT:\n...." type line protocol through stdin/out might work well enough there. 
+- cache validity is defined by cached tree + spec marshal + versioned manifest, scoped by ABI (with `gem.build_complete` for native extensions)
+- cached manifests are versioned JSON with deterministic ordering and a file list for fast materialization
+- git cache slugs are deterministic hashes of the normalized source URI with collision detection + telemetry
+- legacy cached entries without a manifest remain read-compatible but emit telemetry for deprecation
 

data/README.md

@@ -1,265 +1,209 @@
 # Scint
 
-Scint is an experimental Bundler/RubyGems replacement focused on high-throughput installs with a global cache and a fast local materialization step.
+```
+tobi ~/t/fizzy ❯❯❯ scint install
+💎 Scintellating Gemfile into ./.bundle (scint 0.7.1, ruby 4.0.1)
+
+Fetching index https://rubygems.org
 
-It is written in pure Ruby with no dependencies; Ruby is plenty fast for this job.
+162 gems installed total (162 cached). (1.42s, 15 workers used)
+```
 
-Scint is designed for full backwards compatibility with Bundler workflows:
+Scint is an experimental Bundler replacement. Pure Ruby, no dependencies.
 
-1. It reads `Gemfile` and `Gemfile.lock`.
-2. It writes standard `Gemfile.lock`.
-3. It interoperates with Bundler runtime layout. For example, `BUNDLE_PATH=".bundle" bundle exec ...` is expected to work.
-4. The intent is identical behavior with better install and execution throughput.
+It reads your `Gemfile` and `Gemfile.lock`, writes standard `Gemfile.lock`, and materializes into `.bundle/` so `bundle exec` keeps working. Same behavior, much faster installs.
 
-The core idea is:
+The core idea:
 
 1. Prepare artifacts once in a global cache (`~/.cache/scint`).
-2. Materialize project-local runtime state into `.bundle/` as efficiently as possible (hardlinks where possible).
-3. Execute work in explicit concurrent phases coordinated by a scheduler session.
-
-## Why Scint
+2. Materialize into `.bundle/` via clonefile/hardlink/copy.
+3. Warm installs are effectively instant because no fetch, extract, or compile happens when the cache is populated.
 
-`scint` comes from *scintillation*: short, high-energy flashes rather than continuous glow.
+## Why "Scint"
 
-That maps directly to the runtime model:
+From *scintillation*: short, high-energy flashes.
 
 1. Event-driven scheduling.
 2. Burst parallelism where safe.
 3. Tight phase boundaries with clear handoffs.
 
-## CLI
-
-```bash
-scint install
-scint exec <command>
-scint cache list
-scint cache clear
-scint cache dir
-```
-
-Benchmark helpers:
-
-```bash
-bin/scint-vs-bundler [--force] [--test-root /tmp/scint-tests] /path/to/project
-bin/scint-bench-matrix [--force] --root /path/to/projects
-```
-
-`bin/scint-bench-matrix` is a generic runner for a root directory where each
-immediate subdirectory is a Ruby project under git with both `Gemfile` and
-`Gemfile.lock`. It runs bundler cold/warm and scint cold/warm via
-`bin/scint-vs-bundler` and writes:
-
-1. `logs/bench-<timestamp>/summary.tsv`
-2. `logs/bench-<timestamp>/table.md`
+## Install Pipeline
 
-Optional project smoke test convention:
+Every gem flows through a deterministic cache pipeline. Resolution decides *what* to install; the pipeline defines *how*.
 
-1. If `<root>/<project>-test.sh` exists, matrix runs it after the benchmark.
-2. Execution is:
-   `cd <root>/<project> && scint exec ../<project>-test.sh`
-3. The script runs against the warm scint install and is included in
-   `summary.tsv`/`table.md` status.
-
-Performance and IO diagnostics:
-
-```bash
-# Ruby sampling profile (JSON)
-SCINT_PROFILE=/tmp/scint-profile.json SCINT_PROFILE_HZ=400 scint install --force
-
-# Ruby-level IO trace (JSONL)
-SCINT_IO_TRACE=/tmp/scint-io.jsonl scint install --force
-
-# Summarize high-volume IO operations for quick LLM review
-scint-io-summary /tmp/scint-io.jsonl
-
-# Syscall-level trace (Linux strace / macOS dtruss)
-scint-syscall-trace /tmp/scint-sys.log -- scint install --force
-```
-
-Compatibility example:
-
-```bash
-BUNDLE_PATH=".bundle" bundle exec ruby -v
+```mermaid
+flowchart TD
+    A["Fetch + Assemble"]
+    B[Compile]
+    C[Promote]
+    D[Materialize]
+
+    A -->|"gems: download .gem, unpack\ngit: clone, checkout, export"| B
+    B -->|"native extensions built\ninside assembling tree"| C
+    C -->|"atomic move to cached/\nwrite .spec.marshal + manifest"| D
+    D -->|"clonefile / hardlink / copy\nfrom cached/ to .bundle/"| E[Done]
 ```
 
-Defaults:
-
-1. Local install/runtime directory: `.bundle/`
-2. Global cache root: `~/.cache/scint` (or `XDG_CACHE_HOME`)
-
-## Install Architecture (Target)
-
-Scint should have one clear cache lifecycle:
+### Phase details
 
-1. `inbound`
-2. `assembling`
-3. `cached`
-4. `materialize`
+1. **Fetch + Assemble** into `inbound/` and `assembling/`
+   - Download gem payloads to `inbound/gems/`.
+   - Clone/fetch git repos into `inbound/gits/<sha256-slug>/`.
+   - Unpack `.gem` files into `assembling/<abi>/<full_name>/`.
+   - For git sources: checkout, submodules, then export the tree into `assembling/<abi>/<full_name>/`.
+2. **Compile** in `assembling/`
+   - Native extension builds happen inside the assembling directory so outputs are part of the final tree.
+3. **Promote** atomically to `cached/`
+   - Move `assembling/<abi>/<full_name>/` to `cached/<abi>/<full_name>/`.
+   - Write `cached/<abi>/<full_name>.spec.marshal` and `.manifest`.
+   - Failed builds never reach `cached/`.
+4. **Materialize** to `.bundle/`
+   - Clonefile/reflink/hardlink/copy from `cached/<abi>/`.
+   - No rebuild if the cached artifact is valid.
 
-Resolution/planning still decides *what* to install; this pipeline defines *how* each artifact becomes globally reusable.
+One truth source for warm installs: `cached/<abi>`.
 
-### Phase Contract
+## Scheduler
 
-1. Fetch into `inbound`
-   - Gem payloads go to `inbound/gems/`.
-   - Git repositories go to `inbound/gits/` using deterministic names (for example `https_github_com__tobi__try`).
-2. Assemble into `assembling`
-   - For `.gem` sources: unpack into `assembling/<abi>/<full_name>/`.
-   - For git sources: fetch/checkout/submodules in `inbound/gits`, then export/copy the selected tree into `assembling/<abi>/<full_name>/`.
-3. Compile in `assembling`
-   - Native extension build happens inside the assembling directory so successful outputs are part of the final cached tree.
-4. Promote atomically to `cached`
-   - On success, move `assembling/<abi>/<full_name>/` to `cached/<abi>/<full_name>/`.
-   - Write `cached/<abi>/<full_name>.spec.marshal`.
-   - Write optional manifest metadata for fast materialization.
-5. Materialize to project path (`.bundle` or `BUNDLE_PATH`)
-   - Use clonefile/reflink/hardlink/copy fallback from `cached/<abi>/...`.
-   - Do not rebuild if cached artifact is already complete.
-
-This gives one primary truth source for warm installs: `cached/<abi>`.
+The `Scheduler` is the install session object. It owns the job graph, worker pool, and phase coordination.
 
 ```mermaid
-flowchart LR
-    A[Resolve + Plan] --> B[Fetch to inbound]
-    B --> C[Assemble in assembling]
-    C --> D[Compile in assembling]
-    D --> E[Promote to cached]
-    E --> F[Materialize to .bundle]
-    F --> G[Write Runtime + Lockfile]
+flowchart TD
+    subgraph "Early Parallel"
+        FI[fetch_index]
+        GC[git_clone]
+    end
+
+    FI --> R[resolve]
+    GC --> R
+
+    R --> P[plan]
+
+    subgraph "Install DAG"
+        DL[download] --> EX[extract]
+        EX --> LN[link]
+        LN --> BE[build_ext]
+        BE --> BS[binstub]
+        LN --> BS
+    end
+
+    P --> DL
+    P --> LN2["link (cached)"]
+    LN2 --> BS2[binstub]
 ```
 
-## Scheduler as Session Object
-
-The `Scheduler` is more than a queue: it is the install *session object*.
-It owns global execution state and coordinates workers with phase-aware semantics.
-
-The scheduler tracks:
-
-1. Job graph and dependencies
-2. Priority classes by job type
-3. Worker pool scaling
-4. Job state transitions (`pending`, `running`, `completed`, `failed`)
-5. Follow-up chaining (for phase handoff)
-6. Fail-fast abort state and error collection
-7. Progress/stats snapshots used by reporting
+Job priorities (lower = dispatched first):
 
-Workers do not own global install strategy. They execute task payloads with context supplied by scheduler enqueuing and phase sequencing.
-
-```mermaid
-sequenceDiagram
-    participant U as User
-    participant C as scint install
-    participant S as Scheduler(Session)
-    participant W as WorkerPool
-    participant T as Task Worker
-
-    U->>C: scint install
-    C->>S: start(max_workers, fail_fast)
-    C->>S: enqueue(fetch_index/git/...)
-    S->>W: dispatch ready jobs
-    W->>T: execute payload
-    T-->>S: complete/fail + result
-    S-->>C: wait_for phase completion
-    C->>S: enqueue next phase (download/link/build_ext)
-    S-->>C: stats + errors + aborted?
-    C-->>U: summary + lockfile/runtime outputs
-```
+| Priority | Job Type | Concurrency |
+|----------|----------|-------------|
+| 0 | `fetch_index` | all workers |
+| 1 | `git_clone` | all workers |
+| 2 | `resolve` | 1 |
+| 3 | `download` | bounded (default 8) |
+| 4 | `extract` | IO-limited |
+| 5 | `link` | IO-limited |
+| 6 | `build_ext` | CPU-limited (slots x make -j) |
+| 7 | `binstub` | 1 |
 
-## Job Lifecycle
+Workers start at 1, scale dynamically up to `cpu_count * 2` (max 50). Compile concurrency is tuned separately: a small number of slots each running `make -jN` to keep CPU saturated without thrashing.
 
-```mermaid
-stateDiagram-v2
-    [*] --> pending
-    pending --> running: dependencies satisfied + worker slot available
-    running --> completed: success
-    running --> failed: exception / command failure
-    completed --> [*]
-    failed --> [*]
-```
+Fail-fast mode aborts scheduling after the first hard failure.
 
-## Data Layout (Target)
+## Data Layout
 
 Global cache (`~/.cache/scint`):
 
-```text
+```
 ~/.cache/scint/
   inbound/
-    gems/
-      <full_name>.gem
-    gits/
-      <deterministic_repo_slug>/
+    gems/<full_name>.gem
+    gits/<sha256-slug>/
   assembling/
-    <ruby-abi>/
-      <full_name>/
+    <ruby-abi>/<full_name>/
   cached/
     <ruby-abi>/
       <full_name>/
       <full_name>.spec.marshal
       <full_name>.manifest
   index/
+    <source-slug>/
 ```
 
-Example ABI key and gem directory:
-
-1. `cached/ruby-3.4.5-arm64-darwin24/zlib-3.2.1/`
-2. `cached/ruby-3.4.5-arm64-darwin24/zlib-3.2.1.spec.marshal`
+ABI key example: `ruby-4.0.1-arm64-darwin25`
 
 Project-local runtime (`.bundle/`):
 
-1. `ruby/<major.minor.0>/gems/` materialized gem trees
-2. `ruby/<major.minor.0>/specifications/` gemspecs
-3. `ruby/<major.minor.0>/bin/` gem binstubs
-4. `bin/` project-level wrappers
-5. `scint.lock.marshal` runtime lock for `scint exec`
+```
+.bundle/
+  ruby/<major.minor.0>/
+    gems/           # materialized gem trees
+    specifications/ # gemspecs
+    bin/            # gem binstubs
+  bin/              # project-level wrappers
+  scint.lock.marshal
+```
+
+Cache root precedence: `SCINT_CACHE` > `XDG_CACHE_HOME/scint` > `~/.cache/scint`.
 
-## Warm Path Guarantees
+## Cache Validity
 
-Required behavior:
+A cached artifact is valid when:
 
-1. If `cached/<abi>/<full_name>/` exists and is valid, no fetch/extract/compile occurs for that gem.
-2. Deleting only `.bundle/` should trigger only materialization work.
-3. Materialization should be IO-bound and close to instantaneous on warm cache.
-4. Incomplete assemblies must never be promoted; promotion is atomic.
+1. `cached/<abi>/<full_name>/` exists.
+2. `.spec.marshal` exists and loads.
+3. `.manifest` exists, parses, schema version is supported, and fields match.
+4. If the gem has native extensions, the build-complete marker exists.
 
-## Concurrency Model
+Invalid entries are rebuilt through the full pipeline. Legacy entries without manifests are read-compatible but emit telemetry for deprecation tracking.
 
-Scint parallelizes all non-conflicting work aggressively:
+## Warm Path Guarantees
+
+1. If `cached/<abi>/<full_name>/` is valid, no fetch/extract/compile occurs.
+2. Deleting `.bundle/` triggers only materialization.
+3. Materialization is IO-bound and close to instantaneous on warm cache.
+4. Incomplete assemblies are never promoted.
+
+## CLI
 
-1. Index fetch and git clone start early.
-2. Downloads can chain follow-up link tasks.
-3. Planner ordering prioritizes large downloads first to keep pipeline saturated.
-4. Build-ext runs after link readiness to make dependencies visible.
-5. Fail-fast mode aborts scheduling of new work after the first hard failure.
+```bash
+scint install           # install from Gemfile.lock
+scint add <gem>         # add to Gemfile and install
+scint remove <gem>      # remove from Gemfile and install
+scint exec <cmd>        # run command in bundle context
+scint cache list        # list cached gems
+scint cache clear       # clear global cache
+scint cache dir         # print cache root
+```
 
-## Error Model
+Options: `--jobs N`, `--path P`, `--verbose`, `--force`.
 
-Scint is designed to be explicit on failure:
+`scint exec` sets `GEM_HOME`/`GEM_PATH`, injects load paths from `scint.lock.marshal`, and execs the command. Bundler compatibility: `BUNDLE_PATH=".bundle" bundle exec` works against the same layout.
 
-1. Install exits non-zero on failures.
-2. Native build failures include full captured command output.
-3. Final summary reports installed/failed/skipped counts.
-4. `.gitignore` warning is emitted when `.bundle/` is not ignored.
+## Diagnostics
 
-## `scint exec` Runtime
+```bash
+# Ruby sampling profile
+SCINT_PROFILE=/tmp/profile.json scint install --force
 
-`scint exec` sets runtime env and load paths from `scint.lock.marshal`, then `exec`s the target command.
+# Ruby IO trace
+SCINT_IO_TRACE=/tmp/io.jsonl scint install --force
 
-Key behaviors:
+# Summarize IO trace
+scint-io-summary /tmp/io.jsonl
 
-1. Injects runtime load paths and bundler compatibility shim.
-2. Sets `GEM_HOME`/`GEM_PATH` to the local `.bundle` runtime.
-3. Prefers local `.bundle/bin` executables.
-4. Rebuilds runtime lock from `Gemfile.lock` + installed gems when possible if missing.
+# Syscall trace (strace/dtruss)
+scint-syscall-trace /tmp/sys.log -- scint install --force
+```
 
-## Aspirational Direction
+## Benchmarking
 
-The current architecture already separates phases and scheduling concerns. Planned direction is to make this even more explicit:
+```bash
+bin/scint-vs-bundler [--force] /path/to/project
+bin/scint-bench-matrix [--force] --root /path/to/projects
+```
 
-1. First-class session object API around scheduler state and phase transitions.
-2. Isolated compile worker process with a simple line-protocol RPC (`CALL` / `RESULT`) for stronger fault isolation.
-3. More deterministic bulk operations for extraction/linking.
-4. Better per-phase telemetry for latency and saturation analysis.
+`scint-bench-matrix` runs cold/warm benchmarks for every project subdirectory and writes `logs/bench-<timestamp>/summary.tsv` and `table.md`. If `<root>/<project>-test.sh` exists, it runs as a smoke test after the benchmark.
 
 ## Status
 
-Scint is experimental and optimized for architecture iteration speed. Behavior and internals may change quickly.
+Experimental. Optimized for architecture iteration speed.

data/VERSION

@@ -1 +1 @@
-0.7.0
+0.8.0

data/lib/scint/cache/layout.rb

@@ -23,10 +23,28 @@ module Scint
         File.join(@root, "inbound")
       end
 
+      def inbound_gems_dir
+        File.join(inbound_dir, "gems")
+      end
+
+      def inbound_gits_dir
+        File.join(inbound_dir, "gits")
+      end
+
+      def assembling_dir
+        File.join(@root, "assembling")
+      end
+
+      def cached_dir
+        File.join(@root, "cached")
+      end
+
+      # Legacy extracted cache (read-compat only).
       def extracted_dir
         File.join(@root, "extracted")
       end
 
+      # Legacy extension cache (read-compat only).
       def ext_dir
         File.join(@root, "ext")
       end
@@ -36,7 +54,7 @@ module Scint
       end
 
       def git_dir
-        File.join(inbound_dir, "git")
+        inbound_gits_dir
       end
 
       # Isolated gem home used while compiling native extensions during install.
@@ -52,17 +70,40 @@ module Scint
       # -- Per-spec paths ------------------------------------------------------
 
       def inbound_path(spec)
-        File.join(inbound_dir, "#{full_name(spec)}.gem")
+        File.join(inbound_gems_dir, "#{full_name(spec)}.gem")
+      end
+
+      def assembling_path(spec, abi_key = Platform.abi_key)
+        File.join(assembling_dir, abi_key, full_name(spec))
+      end
+
+      def cached_abi_dir(abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key)
+      end
+
+      def cached_path(spec, abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key, full_name(spec))
       end
 
+      def cached_spec_path(spec, abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key, "#{full_name(spec)}.spec.marshal")
+      end
+
+      def cached_manifest_path(spec, abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key, "#{full_name(spec)}.manifest")
+      end
+
+      # Legacy extracted cache (read-compat only).
       def extracted_path(spec)
         File.join(extracted_dir, full_name(spec))
       end
 
+      # Legacy extracted gemspec cache (read-compat only).
       def spec_cache_path(spec)
         File.join(extracted_dir, "#{full_name(spec)}.spec.marshal")
       end
 
+      # Legacy extension cache (read-compat only).
       def ext_path(spec, abi_key = Platform.abi_key)
         File.join(ext_dir, abi_key, full_name(spec))
       end
@@ -80,13 +121,13 @@ module Scint
 
       def git_path(uri)
         slug = git_slug(uri)
-        File.join(git_dir, "repos", slug)
+        File.join(git_dir, slug)
       end
 
       def git_checkout_path(uri, revision)
         slug = git_slug(uri)
         rev = revision.to_s.gsub(/[^0-9A-Za-z._-]/, "_")
-        File.join(git_dir, "checkouts", slug, rev)
+        File.join(git_dir, slug, "checkouts", rev)
       end
 
       # -- Helpers -------------------------------------------------------------
@@ -109,10 +150,19 @@ module Scint
       private
 
       def default_root
+        return Scint.cache_root if Scint.respond_to?(:cache_root)
+
+        explicit = ENV["SCINT_CACHE"]
+        return File.expand_path(explicit) unless explicit.nil? || explicit.empty?
+
         base = ENV["XDG_CACHE_HOME"] || File.join(Dir.home, ".cache")
         File.join(base, "scint")
       end
 
+      # Slug rules are defined in README.md (Cache Validity + Manifest Specification).
+      # - Index slugs prefer host/path when available, otherwise fall back to a hash.
+      # - Hash slugs are deterministic but must be paired with manifest checks for
+      #   collision detection.
       def slugify_uri(str)
         uri = URI.parse(str) rescue nil
         if uri && uri.host
@@ -125,8 +175,19 @@ module Scint
         end
       end
 
+      # Git slugs are SHA256 of the normalized URI string (uri.to_s), truncated
+      # to 16 hex chars. Callers must validate `source.uri` in the manifest to
+      # detect collisions and fall back to a longer hash if needed.
       def git_slug(uri)
-        Digest::SHA256.hexdigest(uri.to_s)[0, 16]
+        normalized = normalize_uri(uri)
+        Digest::SHA256.hexdigest(normalized)[0, 16]
+      end
+
+      def normalize_uri(uri)
+        return uri.to_s if uri.is_a?(URI)
+        URI.parse(uri.to_s).to_s
+      rescue URI::InvalidURIError
+        uri.to_s
       end
     end
   end