scint 0.6.0 → 0.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d1c0c2f09127da28473aad491848c5b3e7b976fa41405a94ce542859fa1aabf8
-  data.tar.gz: 2e7b1801b9a3004bc0d9709de9502bfff5a8b475178aa102ebb730a6cb1b6344
+  metadata.gz: d9c9934c3d5cfb4786daa6d766e031b27919cc290ffd9d3f17607e70f5f58535
+  data.tar.gz: ffc27f93068721d1fdfee4cb3fc62355beba8322c295968ae2f4cce6ccd4470e
 SHA512:
-  metadata.gz: 3d95e5006966b2025c5e21951bc918849081c330a2872e7e665cafa6d73ab522871f7e41aa2ed4c61bbed37e73363f19650dce16bbc1e472b3e4fe54c0177d97
-  data.tar.gz: 40fba5802be2359b85d201c2c732f8edac15f1e1be7ee17b2d59a39de9ac2f1ca6befd14d527fcacb4a5e763408c1296abe35301f8224a4b1138a5a4a1706096
+  metadata.gz: e722c6d0c83bf1a6bbe2f4e4b65da8282f3b4f6ac83742900f91c158a93262092780b15778979bbbc9ccc59847c7d232f2c9bc689e4b01eb6b3920dd4a875347
+  data.tar.gz: 5c7a26fe05647b14e865b60b9f08adbb4f020edb0c18f393fd577a90af701a502471485bc6eaf90d947d1f82324f2d8fa322280f5916e243705300d1ca3defea

data/FEATURES.md

@@ -10,4 +10,8 @@
 - the installation process involves compilation. We attempt to have compilation happen while its not blocking other operations, but also only one compilation at a time
 - we have a book keeping object that governs the worker pools and that's present during each step (fetch, extract, compile, install) and recieves the tasks for each phase from the workers. 
 - i suspect that we need to fork of a worker for compilation which we then have to communicate with via some rpc format. simple "-> CALL method, <- RESULT:\n...." type line protocol through stdin/out might work well enough there. 
+- cache validity is defined by cached tree + spec marshal + versioned manifest, scoped by ABI (with `gem.build_complete` for native extensions)
+- cached manifests are versioned JSON with deterministic ordering and a file list for fast materialization
+- git cache slugs are deterministic hashes of the normalized source URI with collision detection + telemetry
+- legacy cached entries without a manifest remain read-compatible but emit telemetry for deprecation
 

data/README.md

@@ -37,6 +37,29 @@ scint cache clear
 scint cache dir
 ```
 
+Benchmark helpers:
+
+```bash
+bin/scint-vs-bundler [--force] [--test-root /tmp/scint-tests] /path/to/project
+bin/scint-bench-matrix [--force] --root /path/to/projects
+```
+
+`bin/scint-bench-matrix` is a generic runner for a root directory where each
+immediate subdirectory is a Ruby project under git with both `Gemfile` and
+`Gemfile.lock`. It runs bundler cold/warm and scint cold/warm via
+`bin/scint-vs-bundler` and writes:
+
+1. `logs/bench-<timestamp>/summary.tsv`
+2. `logs/bench-<timestamp>/table.md`
+
+Optional project smoke test convention:
+
+1. If `<root>/<project>-test.sh` exists, matrix runs it after the benchmark.
+2. Execution is:
+   `cd <root>/<project> && scint exec ../<project>-test.sh`
+3. The script runs against the warm scint install and is included in
+   `summary.tsv`/`table.md` status.
+
 Performance and IO diagnostics:
 
 ```bash
@@ -64,45 +87,45 @@ Defaults:
 1. Local install/runtime directory: `.bundle/`
 2. Global cache root: `~/.cache/scint` (or `XDG_CACHE_HOME`)
 
-## Install Architecture
+## Install Architecture (Target)
+
+Scint should have one clear cache lifecycle:
 
-Scint install is phase-oriented. Each phase has explicit responsibilities and feeds the next phase.
+1. `inbound`
+2. `assembling`
+3. `cached`
+4. `materialize`
 
-1. Parse inputs (`Gemfile`, optional `Gemfile.lock`)
-2. Fetch source metadata (indexes, git clones)
-3. Resolve dependency graph
-4. Plan actions (`skip`, `link`, `download`, `build_ext`)
-5. Download/extract/cache artifacts
-6. Link into local `.bundle` runtime
-7. Build native extensions after link phase is ready
-8. Write outputs (`Gemfile.lock`, runtime lock, warnings/summary)
+Resolution/planning still decides *what* to install; this pipeline defines *how* each artifact becomes globally reusable.
+
+### Phase Contract
+
+1. Fetch into `inbound`
+   - Gem payloads go to `inbound/gems/`.
+   - Git repositories go to `inbound/gits/` using deterministic names (for example `https_github_com__tobi__try`).
+2. Assemble into `assembling`
+   - For `.gem` sources: unpack into `assembling/<abi>/<full_name>/`.
+   - For git sources: fetch/checkout/submodules in `inbound/gits`, then export/copy the selected tree into `assembling/<abi>/<full_name>/`.
+3. Compile in `assembling`
+   - Native extension build happens inside the assembling directory so successful outputs are part of the final cached tree.
+4. Promote atomically to `cached`
+   - On success, move `assembling/<abi>/<full_name>/` to `cached/<abi>/<full_name>/`.
+   - Write `cached/<abi>/<full_name>.spec.marshal`.
+   - Write optional manifest metadata for fast materialization.
+5. Materialize to project path (`.bundle` or `BUNDLE_PATH`)
+   - Use clonefile/reflink/hardlink/copy fallback from `cached/<abi>/...`.
+   - Do not rebuild if cached artifact is already complete.
+
+This gives one primary truth source for warm installs: `cached/<abi>`.
 
 ```mermaid
 flowchart LR
-    A[Gemfile + Gemfile.lock] --> B[Parse + Source Discovery]
-    B --> C[Fetch Indexes / Clone Git]
-    C --> D[Resolve Graph]
-    D --> E[Planner]
-    E -->|skip| F[Already Installed]
-    E -->|link| G[Link from Extracted Cache]
-    E -->|download| H[Download .gem]
-    H --> I[Extract + Cache Metadata]
-    I --> G
-    G --> J[All Links Complete]
-    J --> K[Native Extension Build]
-    K --> L[Runtime + Lockfile Write]
-    L --> M[Done]
-
-    subgraph GlobalCache["Global Cache (~/.cache/scint)"]
-      H
-      I
-    end
-
-    subgraph ProjectRuntime["Project Runtime (.bundle)"]
-      G
-      K
-      L
-    end
+    A[Resolve + Plan] --> B[Fetch to inbound]
+    B --> C[Assemble in assembling]
+    C --> D[Compile in assembling]
+    D --> E[Promote to cached]
+    E --> F[Materialize to .bundle]
+    F --> G[Write Runtime + Lockfile]
 ```
 
 ## Scheduler as Session Object
@@ -154,24 +177,121 @@ stateDiagram-v2
     failed --> [*]
 ```
 
-## Data Layout
+## Data Layout (Target)
 
 Global cache (`~/.cache/scint`):
 
-1. `inbound/` downloaded gem files
-2. `extracted/` unpacked gem trees
-3. `ext/` compiled extension cache keyed by ABI
-4. `index/` source metadata/index cache
-5. `git/` cached git repositories
+```text
+~/.cache/scint/
+  inbound/
+    gems/
+      <full_name>.gem
+    gits/
+      <deterministic_repo_slug>/
+  assembling/
+    <ruby-abi>/
+      <full_name>/
+  cached/
+    <ruby-abi>/
+      <full_name>/
+      <full_name>.spec.marshal
+      <full_name>.manifest
+  index/
+```
+
+Example ABI key and gem directory:
+
+1. `cached/ruby-3.4.5-arm64-darwin24/zlib-3.2.1/`
+2. `cached/ruby-3.4.5-arm64-darwin24/zlib-3.2.1.spec.marshal`
 
 Project-local runtime (`.bundle/`):
 
-1. `ruby/<major.minor.0>/gems/` linked gem trees
+1. `ruby/<major.minor.0>/gems/` materialized gem trees
 2. `ruby/<major.minor.0>/specifications/` gemspecs
 3. `ruby/<major.minor.0>/bin/` gem binstubs
 4. `bin/` project-level wrappers
 5. `scint.lock.marshal` runtime lock for `scint exec`
 
+## Cache Validity + Manifest Specification (Draft)
+
+### Cache validity criteria
+
+A cached artifact is considered valid only when *all* of the following are true:
+
+1. `cached/<abi>/<full_name>/` exists and is a fully materialized tree.
+2. `cached/<abi>/<full_name>.spec.marshal` exists and loads successfully.
+3. `cached/<abi>/<full_name>.manifest` exists, parses, and its `version` is supported.
+4. Manifest fields `full_name` and `abi` match the requested spec/ABI.
+5. If the gem has native extensions, `ext/<abi>/<full_name>/gem.build_complete` exists.
+
+Any failure means the cache entry is *invalid* and must be rebuilt (fetch/assemble/compile).
+
+### Manifest schema
+
+The manifest is a single UTF-8 JSON object written to
+`cached/<abi>/<full_name>.manifest`. It is versioned and deterministically
+serialized for repeatable cache reuse.
+
+**Serialization rules**:
+
+- Top-level keys sorted lexicographically.
+- Array ordering is deterministic (see `files` below).
+- JSON is emitted without extra whitespace (canonical/minified).
+- No timestamps or host-specific values are stored.
+
+**Schema (version 1)**:
+
+- `version` (Integer, required): schema version (starting at `1`).
+- `full_name` (String, required): `name-version(-platform)`.
+- `abi` (String, required): Ruby ABI key (e.g. `ruby-3.4.5-arm64-darwin24`).
+- `source` (Object, required):
+  - `type` (String): `rubygems`, `git`, or `path`.
+  - `uri` (String): canonical source URI.
+  - `revision` (String, git only): resolved commit SHA.
+  - `path` (String, path only): absolute source path.
+- `files` (Array, required): sorted by `path` ascending. Each entry is:
+  - `path` (String): relative to the cached root.
+  - `type` (String): `file`, `dir`, or `symlink`.
+  - `mode` (Integer): numeric permission bits (e.g. `755`).
+  - `size` (Integer): bytes (directories use `0`).
+  - `sha256` (String, optional): content hash for files/symlinks.
+- `build` (Object, required):
+  - `extensions` (Boolean): whether the gem builds native extensions.
+  - `ext_complete` (String, optional): completion marker path when extensions exist.
+
+### Git slug normalization + collisions
+
+Git cache directories use a deterministic slug derived from the source URI:
+
+1. Normalize the URI to a stable string form (`uri.to_s`; callers should pass a
+   parsed URI for consistent normalization).
+2. Compute `sha256(normalized_uri)`, use the first 16 hex characters.
+
+**Collision handling**: when a slug directory already exists, validate that the
+manifest `source.uri` matches the normalized URI. If it does not, treat it as a
+collision, emit telemetry, and fall back to a longer hash (e.g. full 64 hex) or
+an additional suffix.
+
+### Legacy read-compat + telemetry
+
+Legacy cache entries that lack a manifest (or use an unsupported schema version)
+remain *read-compatible* for now:
+
+- Treat the entry as valid only if the cached tree + `.spec.marshal` exist and
+  the gemspec loads cleanly.
+- Emit telemetry counters (per run) such as `cache.manifest.missing`,
+  `cache.manifest.unsupported`, and `cache.manifest.collision`.
+- Log a single warning per run with counts and cache root to guide deprecation.
+
+## Warm Path Guarantees
+
+Required behavior:
+
+1. If `cached/<abi>/<full_name>/` exists and is valid, no fetch/extract/compile occurs for that gem.
+2. Deleting only `.bundle/` should trigger only materialization work.
+3. Materialization should be IO-bound and close to instantaneous on warm cache.
+4. Incomplete assemblies must never be promoted; promotion is atomic.
+
 ## Concurrency Model
 
 Scint parallelizes all non-conflicting work aggressively:

data/VERSION

@@ -1 +1 @@
-0.6.0
+0.7.1

data/bin/scint

@@ -1,6 +1,15 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
+# Enable YJIT as early as possible when available.
+if defined?(RubyVM::YJIT) && RubyVM::YJIT.respond_to?(:enable)
+  begin
+    RubyVM::YJIT.enable unless RubyVM::YJIT.enabled?
+  rescue StandardError
+    # Keep startup resilient when YJIT is unavailable or disabled at runtime.
+  end
+end
+
 # Ensure lib is on the load path
 $LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
 

data/lib/bundler.rb

@@ -100,10 +100,14 @@ module Bundler
           Kernel.require(candidate)
           return true
         rescue LoadError => e
+          raise e unless retryable_load_error?(e, candidate)
+
           last_error = e
         end
       end
 
+      return true if require_matching_basename(name)
+
       raise last_error if last_error
     end
 
@@ -164,5 +168,107 @@ module Bundler
       end
       filtered.join(" ")
     end
+
+    def require_matching_basename(name)
+      target = normalize_basename(name)
+      return false if target.empty?
+
+      exact_candidates = []
+      fuzzy_candidates = []
+
+      $LOAD_PATH.each do |load_dir|
+        next unless File.directory?(load_dir)
+
+        Dir.children(load_dir).sort.each do |entry|
+          next unless entry.end_with?(".rb")
+
+          basename = entry.delete_suffix(".rb")
+          normalized = normalize_basename(basename)
+          if normalized == target
+            exact_candidates << basename
+          elsif compatible_basename?(target, normalized)
+            fuzzy_candidates << [basename, normalized]
+          end
+        end
+      rescue StandardError
+        next
+      end
+
+      exact_candidates.uniq.each do |candidate|
+        begin
+          Kernel.require(candidate)
+          return true
+        rescue LoadError => e
+          raise e unless retryable_load_error?(e, candidate)
+
+          next
+        end
+      end
+
+      fuzzy_candidates
+        .uniq
+        .sort_by { |_basename, normalized| [((target.length - normalized.length).abs), -normalized.length] }
+        .map(&:first)
+        .each do |candidate|
+          begin
+            Kernel.require(candidate)
+            return true
+          rescue LoadError => e
+            raise e unless retryable_load_error?(e, candidate)
+
+            next
+          end
+        end
+
+      false
+    end
+
+    def normalize_basename(name)
+      name.to_s.downcase.gsub(/[^a-z0-9]/, "")
+    end
+
+    def retryable_load_error?(error, candidate)
+      requested = if error.respond_to?(:path)
+                    error.path
+                  else
+                    nil
+                  end
+      return true if requested.nil? || requested.empty?
+
+      variants = [
+        candidate.to_s,
+        "#{candidate}.rb",
+        "#{candidate}.so",
+      ]
+      variants.include?(requested)
+    end
+
+    def compatible_basename?(target, normalized)
+      return false if target.length < 3 || normalized.length < 3
+
+      target_variants = basename_variants(target)
+      normalized_variants = basename_variants(normalized)
+      return true if (target_variants & normalized_variants).any?
+
+      target_variants.any? do |t|
+        normalized_variants.any? { |n| t.start_with?(n) || n.start_with?(t) }
+      end
+    end
+
+    def basename_variants(name)
+      value = name.to_s.downcase
+      variants = [value]
+
+      plural_to_s = value.sub(/ties\z/, "s")
+      plural_to_y = value.sub(/ies\z/, "y")
+      singular = value.sub(/s\z/, "")
+      strip_ruby_prefix = value.sub(/\Aruby/, "")
+
+      variants << plural_to_s unless plural_to_s == value
+      variants << plural_to_y unless plural_to_y == value
+      variants << singular unless singular == value
+      variants << strip_ruby_prefix unless strip_ruby_prefix == value
+      variants.uniq.select { |v| v.length >= 3 }
+    end
   end
 end

data/lib/scint/cache/layout.rb

@@ -2,6 +2,7 @@
 
 require_relative "../fs"
 require_relative "../platform"
+require_relative "../spec_utils"
 require "digest"
 require "uri"
 
@@ -22,10 +23,28 @@ module Scint
         File.join(@root, "inbound")
       end
 
+      def inbound_gems_dir
+        File.join(inbound_dir, "gems")
+      end
+
+      def inbound_gits_dir
+        File.join(inbound_dir, "gits")
+      end
+
+      def assembling_dir
+        File.join(@root, "assembling")
+      end
+
+      def cached_dir
+        File.join(@root, "cached")
+      end
+
+      # Legacy extracted cache (read-compat only).
       def extracted_dir
         File.join(@root, "extracted")
       end
 
+      # Legacy extension cache (read-compat only).
       def ext_dir
         File.join(@root, "ext")
       end
@@ -35,7 +54,7 @@ module Scint
       end
 
       def git_dir
-        File.join(@root, "git")
+        inbound_gits_dir
       end
 
       # Isolated gem home used while compiling native extensions during install.
@@ -45,23 +64,46 @@ module Scint
       end
 
       def install_ruby_dir
-        File.join(install_env_dir, "ruby", RUBY_VERSION.split(".")[0, 2].join(".") + ".0")
+        Platform.ruby_install_dir(install_env_dir)
       end
 
       # -- Per-spec paths ------------------------------------------------------
 
       def inbound_path(spec)
-        File.join(inbound_dir, "#{full_name(spec)}.gem")
+        File.join(inbound_gems_dir, "#{full_name(spec)}.gem")
+      end
+
+      def assembling_path(spec, abi_key = Platform.abi_key)
+        File.join(assembling_dir, abi_key, full_name(spec))
+      end
+
+      def cached_abi_dir(abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key)
+      end
+
+      def cached_path(spec, abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key, full_name(spec))
       end
 
+      def cached_spec_path(spec, abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key, "#{full_name(spec)}.spec.marshal")
+      end
+
+      def cached_manifest_path(spec, abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key, "#{full_name(spec)}.manifest")
+      end
+
+      # Legacy extracted cache (read-compat only).
       def extracted_path(spec)
         File.join(extracted_dir, full_name(spec))
       end
 
+      # Legacy extracted gemspec cache (read-compat only).
       def spec_cache_path(spec)
         File.join(extracted_dir, "#{full_name(spec)}.spec.marshal")
       end
 
+      # Legacy extension cache (read-compat only).
       def ext_path(spec, abi_key = Platform.abi_key)
         File.join(ext_dir, abi_key, full_name(spec))
       end
@@ -78,23 +120,20 @@ module Scint
       end
 
       def git_path(uri)
-        slug = Digest::SHA256.hexdigest(uri.to_s)[0, 16]
+        slug = git_slug(uri)
         File.join(git_dir, slug)
       end
 
+      def git_checkout_path(uri, revision)
+        slug = git_slug(uri)
+        rev = revision.to_s.gsub(/[^0-9A-Za-z._-]/, "_")
+        File.join(git_dir, slug, "checkouts", rev)
+      end
+
       # -- Helpers -------------------------------------------------------------
 
       def full_name(spec)
-        name = spec.respond_to?(:name) ? spec.name : spec[:name]
-        version = spec.respond_to?(:version) ? spec.version : spec[:version]
-        platform = spec.respond_to?(:platform) ? spec.platform : spec[:platform]
-
-        base = "#{name}-#{version}"
-        if platform && platform.to_s != "ruby" && platform.to_s != ""
-          "#{base}-#{platform}"
-        else
-          base
-        end
+        SpecUtils.full_name(spec)
       end
 
       # Ensure a directory exists (thread-safe, cached).
@@ -115,6 +154,10 @@ module Scint
         File.join(base, "scint")
       end
 
+      # Slug rules are defined in README.md (Cache Validity + Manifest Specification).
+      # - Index slugs prefer host/path when available, otherwise fall back to a hash.
+      # - Hash slugs are deterministic but must be paired with manifest checks for
+      #   collision detection.
       def slugify_uri(str)
         uri = URI.parse(str) rescue nil
         if uri && uri.host
@@ -126,6 +169,21 @@ module Scint
           Digest::SHA256.hexdigest(str)[0, 16]
         end
       end
+
+      # Git slugs are SHA256 of the normalized URI string (uri.to_s), truncated
+      # to 16 hex chars. Callers must validate `source.uri` in the manifest to
+      # detect collisions and fall back to a longer hash if needed.
+      def git_slug(uri)
+        normalized = normalize_uri(uri)
+        Digest::SHA256.hexdigest(normalized)[0, 16]
+      end
+
+      def normalize_uri(uri)
+        return uri.to_s if uri.is_a?(URI)
+        URI.parse(uri.to_s).to_s
+      rescue URI::InvalidURIError
+        uri.to_s
+      end
     end
   end
 end

data/lib/scint/cache/manifest.rb

@@ -0,0 +1,120 @@
+# frozen_string_literal: true
+
+require "digest"
+require "find"
+require "json"
+
+require_relative "../fs"
+require_relative "../spec_utils"
+
+module Scint
+  module Cache
+    module Manifest
+      module_function
+
+      VERSION = 1
+
+      def build(spec:, gem_dir:, abi_key:, source:, extensions:)
+        {
+          "abi" => abi_key,
+          "build" => build_block(extensions: extensions),
+          "files" => collect_files(gem_dir),
+          "full_name" => SpecUtils.full_name(spec),
+          "source" => source,
+          "version" => VERSION,
+        }
+      end
+
+      def write(path, manifest)
+        ordered = order_keys(manifest)
+        json = JSON.generate(ordered)
+        FS.atomic_write(path, json)
+      end
+
+      # Write a flat file listing (one relative path per line) into the gem dir.
+      # This enables bulk install via a single `cp -al` or `cpio -pld` call.
+      DOTFILES_NAME = ".scint-files"
+
+      def write_dotfiles(gem_dir, manifest = nil)
+        files = if manifest
+          Array(manifest["files"]).filter_map { |e| e["path"] if e["type"] != "dir" }
+        else
+          collect_file_paths(gem_dir)
+        end
+
+        dotfiles_path = File.join(gem_dir, DOTFILES_NAME)
+        File.write(dotfiles_path, files.sort.join("\n") + "\n")
+      end
+
+      def collect_file_paths(root)
+        paths = []
+        Find.find(root) do |path|
+          next if path == root
+          rel = path.delete_prefix("#{root}/")
+          next if rel == DOTFILES_NAME
+          stat = File.lstat(path)
+          paths << rel unless stat.directory?
+        end
+        paths
+      end
+
+      def collect_files(root)
+        entries = []
+        Find.find(root) do |path|
+          next if path == root
+
+          rel = path.delete_prefix("#{root}/")
+          stat = File.lstat(path)
+
+          if stat.symlink?
+            target = File.readlink(path)
+            entries << file_entry(rel, "symlink", stat, Digest::SHA256.hexdigest(target))
+          elsif stat.directory?
+            entries << dir_entry(rel, stat)
+          else
+            entries << file_entry(rel, "file", stat, Digest::SHA256.file(path).hexdigest)
+          end
+        end
+
+        entries.sort_by { |entry| entry["path"] }
+      end
+
+      def build_block(extensions:)
+        { "extensions" => !!extensions }
+      end
+
+      def order_keys(object)
+        case object
+        when Hash
+          object.keys.sort.each_with_object({}) do |key, acc|
+            acc[key] = order_keys(object[key])
+          end
+        when Array
+          object.map { |entry| order_keys(entry) }
+        else
+          object
+        end
+      end
+
+      def dir_entry(rel, stat)
+        {
+          "mode" => stat.mode & 0o777,
+          "path" => rel,
+          "size" => 0,
+          "type" => "dir",
+        }
+      end
+
+      def file_entry(rel, type, stat, sha)
+        entry = {
+          "mode" => stat.mode & 0o777,
+          "path" => rel,
+          "size" => stat.size,
+          "type" => type,
+        }
+        entry["sha256"] = sha if sha
+        entry
+      end
+    end
+  end
+end