scint 0.7.0 → 0.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 54c3f7baaea3992bcba60136b76f3b08caa580717d82906ebed3a0a69fb23314
-  data.tar.gz: 246966d0ba167f43b8c0aee1205e95377afec245dca33060fe3fcce202d0c3ab
+  metadata.gz: d9c9934c3d5cfb4786daa6d766e031b27919cc290ffd9d3f17607e70f5f58535
+  data.tar.gz: ffc27f93068721d1fdfee4cb3fc62355beba8322c295968ae2f4cce6ccd4470e
 SHA512:
-  metadata.gz: 84c58296b1c391e68503090ea3889a3a469c8f63ca6c289710dc6a5f986cd33059aec6a61038ab14c2f653aefaa1675815d2e13a5ba3fb49dad6fe825cf8d3a5
-  data.tar.gz: 1cde1ae7ff37856bf44a4f909e024cb094ba1436fe062536aa4c73860ce0aa7c2494c2f5dd09ca14af5be8b226d8ada36ec212278b08a55902da708d734bea3d
+  metadata.gz: e722c6d0c83bf1a6bbe2f4e4b65da8282f3b4f6ac83742900f91c158a93262092780b15778979bbbc9ccc59847c7d232f2c9bc689e4b01eb6b3920dd4a875347
+  data.tar.gz: 5c7a26fe05647b14e865b60b9f08adbb4f020edb0c18f393fd577a90af701a502471485bc6eaf90d947d1f82324f2d8fa322280f5916e243705300d1ca3defea

data/FEATURES.md

@@ -10,4 +10,8 @@
 - the installation process involves compilation. We attempt to have compilation happen while its not blocking other operations, but also only one compilation at a time
 - we have a book keeping object that governs the worker pools and that's present during each step (fetch, extract, compile, install) and recieves the tasks for each phase from the workers. 
 - i suspect that we need to fork of a worker for compilation which we then have to communicate with via some rpc format. simple "-> CALL method, <- RESULT:\n...." type line protocol through stdin/out might work well enough there. 
+- cache validity is defined by cached tree + spec marshal + versioned manifest, scoped by ABI (with `gem.build_complete` for native extensions)
+- cached manifests are versioned JSON with deterministic ordering and a file list for fast materialization
+- git cache slugs are deterministic hashes of the normalized source URI with collision detection + telemetry
+- legacy cached entries without a manifest remain read-compatible but emit telemetry for deprecation
 

data/README.md

@@ -212,6 +212,77 @@ Project-local runtime (`.bundle/`):
 4. `bin/` project-level wrappers
 5. `scint.lock.marshal` runtime lock for `scint exec`
 
+## Cache Validity + Manifest Specification (Draft)
+
+### Cache validity criteria
+
+A cached artifact is considered valid only when *all* of the following are true:
+
+1. `cached/<abi>/<full_name>/` exists and is a fully materialized tree.
+2. `cached/<abi>/<full_name>.spec.marshal` exists and loads successfully.
+3. `cached/<abi>/<full_name>.manifest` exists, parses, and its `version` is supported.
+4. Manifest fields `full_name` and `abi` match the requested spec/ABI.
+5. If the gem has native extensions, `ext/<abi>/<full_name>/gem.build_complete` exists.
+
+Any failure means the cache entry is *invalid* and must be rebuilt (fetch/assemble/compile).
+
+### Manifest schema
+
+The manifest is a single UTF-8 JSON object written to
+`cached/<abi>/<full_name>.manifest`. It is versioned and deterministically
+serialized for repeatable cache reuse.
+
+**Serialization rules**:
+
+- Top-level keys sorted lexicographically.
+- Array ordering is deterministic (see `files` below).
+- JSON is emitted without extra whitespace (canonical/minified).
+- No timestamps or host-specific values are stored.
+
+**Schema (version 1)**:
+
+- `version` (Integer, required): schema version (starting at `1`).
+- `full_name` (String, required): `name-version(-platform)`.
+- `abi` (String, required): Ruby ABI key (e.g. `ruby-3.4.5-arm64-darwin24`).
+- `source` (Object, required):
+  - `type` (String): `rubygems`, `git`, or `path`.
+  - `uri` (String): canonical source URI.
+  - `revision` (String, git only): resolved commit SHA.
+  - `path` (String, path only): absolute source path.
+- `files` (Array, required): sorted by `path` ascending. Each entry is:
+  - `path` (String): relative to the cached root.
+  - `type` (String): `file`, `dir`, or `symlink`.
+  - `mode` (Integer): numeric permission bits (e.g. `755`).
+  - `size` (Integer): bytes (directories use `0`).
+  - `sha256` (String, optional): content hash for files/symlinks.
+- `build` (Object, required):
+  - `extensions` (Boolean): whether the gem builds native extensions.
+  - `ext_complete` (String, optional): completion marker path when extensions exist.
+
+### Git slug normalization + collisions
+
+Git cache directories use a deterministic slug derived from the source URI:
+
+1. Normalize the URI to a stable string form (`uri.to_s`; callers should pass a
+   parsed URI for consistent normalization).
+2. Compute `sha256(normalized_uri)`, use the first 16 hex characters.
+
+**Collision handling**: when a slug directory already exists, validate that the
+manifest `source.uri` matches the normalized URI. If it does not, treat it as a
+collision, emit telemetry, and fall back to a longer hash (e.g. full 64 hex) or
+an additional suffix.
+
+### Legacy read-compat + telemetry
+
+Legacy cache entries that lack a manifest (or use an unsupported schema version)
+remain *read-compatible* for now:
+
+- Treat the entry as valid only if the cached tree + `.spec.marshal` exist and
+  the gemspec loads cleanly.
+- Emit telemetry counters (per run) such as `cache.manifest.missing`,
+  `cache.manifest.unsupported`, and `cache.manifest.collision`.
+- Log a single warning per run with counts and cache root to guide deprecation.
+
 ## Warm Path Guarantees
 
 Required behavior:

data/VERSION

@@ -1 +1 @@
-0.7.0
+0.7.1

data/lib/scint/cache/layout.rb

@@ -23,10 +23,28 @@ module Scint
         File.join(@root, "inbound")
       end
 
+      def inbound_gems_dir
+        File.join(inbound_dir, "gems")
+      end
+
+      def inbound_gits_dir
+        File.join(inbound_dir, "gits")
+      end
+
+      def assembling_dir
+        File.join(@root, "assembling")
+      end
+
+      def cached_dir
+        File.join(@root, "cached")
+      end
+
+      # Legacy extracted cache (read-compat only).
       def extracted_dir
         File.join(@root, "extracted")
       end
 
+      # Legacy extension cache (read-compat only).
       def ext_dir
         File.join(@root, "ext")
       end
@@ -36,7 +54,7 @@ module Scint
       end
 
       def git_dir
-        File.join(inbound_dir, "git")
+        inbound_gits_dir
       end
 
       # Isolated gem home used while compiling native extensions during install.
@@ -52,17 +70,40 @@ module Scint
       # -- Per-spec paths ------------------------------------------------------
 
       def inbound_path(spec)
-        File.join(inbound_dir, "#{full_name(spec)}.gem")
+        File.join(inbound_gems_dir, "#{full_name(spec)}.gem")
+      end
+
+      def assembling_path(spec, abi_key = Platform.abi_key)
+        File.join(assembling_dir, abi_key, full_name(spec))
       end
 
+      def cached_abi_dir(abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key)
+      end
+
+      def cached_path(spec, abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key, full_name(spec))
+      end
+
+      def cached_spec_path(spec, abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key, "#{full_name(spec)}.spec.marshal")
+      end
+
+      def cached_manifest_path(spec, abi_key = Platform.abi_key)
+        File.join(cached_dir, abi_key, "#{full_name(spec)}.manifest")
+      end
+
+      # Legacy extracted cache (read-compat only).
       def extracted_path(spec)
         File.join(extracted_dir, full_name(spec))
       end
 
+      # Legacy extracted gemspec cache (read-compat only).
       def spec_cache_path(spec)
         File.join(extracted_dir, "#{full_name(spec)}.spec.marshal")
       end
 
+      # Legacy extension cache (read-compat only).
       def ext_path(spec, abi_key = Platform.abi_key)
         File.join(ext_dir, abi_key, full_name(spec))
       end
@@ -80,13 +121,13 @@ module Scint
 
       def git_path(uri)
         slug = git_slug(uri)
-        File.join(git_dir, "repos", slug)
+        File.join(git_dir, slug)
       end
 
       def git_checkout_path(uri, revision)
         slug = git_slug(uri)
         rev = revision.to_s.gsub(/[^0-9A-Za-z._-]/, "_")
-        File.join(git_dir, "checkouts", slug, rev)
+        File.join(git_dir, slug, "checkouts", rev)
       end
 
       # -- Helpers -------------------------------------------------------------
@@ -113,6 +154,10 @@ module Scint
         File.join(base, "scint")
       end
 
+      # Slug rules are defined in README.md (Cache Validity + Manifest Specification).
+      # - Index slugs prefer host/path when available, otherwise fall back to a hash.
+      # - Hash slugs are deterministic but must be paired with manifest checks for
+      #   collision detection.
       def slugify_uri(str)
         uri = URI.parse(str) rescue nil
         if uri && uri.host
@@ -125,8 +170,19 @@ module Scint
         end
       end
 
+      # Git slugs are SHA256 of the normalized URI string (uri.to_s), truncated
+      # to 16 hex chars. Callers must validate `source.uri` in the manifest to
+      # detect collisions and fall back to a longer hash if needed.
       def git_slug(uri)
-        Digest::SHA256.hexdigest(uri.to_s)[0, 16]
+        normalized = normalize_uri(uri)
+        Digest::SHA256.hexdigest(normalized)[0, 16]
+      end
+
+      def normalize_uri(uri)
+        return uri.to_s if uri.is_a?(URI)
+        URI.parse(uri.to_s).to_s
+      rescue URI::InvalidURIError
+        uri.to_s
       end
     end
   end

data/lib/scint/cache/manifest.rb

@@ -0,0 +1,120 @@
+# frozen_string_literal: true
+
+require "digest"
+require "find"
+require "json"
+
+require_relative "../fs"
+require_relative "../spec_utils"
+
+module Scint
+  module Cache
+    module Manifest
+      module_function
+
+      VERSION = 1
+
+      def build(spec:, gem_dir:, abi_key:, source:, extensions:)
+        {
+          "abi" => abi_key,
+          "build" => build_block(extensions: extensions),
+          "files" => collect_files(gem_dir),
+          "full_name" => SpecUtils.full_name(spec),
+          "source" => source,
+          "version" => VERSION,
+        }
+      end
+
+      def write(path, manifest)
+        ordered = order_keys(manifest)
+        json = JSON.generate(ordered)
+        FS.atomic_write(path, json)
+      end
+
+      # Write a flat file listing (one relative path per line) into the gem dir.
+      # This enables bulk install via a single `cp -al` or `cpio -pld` call.
+      DOTFILES_NAME = ".scint-files"
+
+      def write_dotfiles(gem_dir, manifest = nil)
+        files = if manifest
+          Array(manifest["files"]).filter_map { |e| e["path"] if e["type"] != "dir" }
+        else
+          collect_file_paths(gem_dir)
+        end
+
+        dotfiles_path = File.join(gem_dir, DOTFILES_NAME)
+        File.write(dotfiles_path, files.sort.join("\n") + "\n")
+      end
+
+      def collect_file_paths(root)
+        paths = []
+        Find.find(root) do |path|
+          next if path == root
+          rel = path.delete_prefix("#{root}/")
+          next if rel == DOTFILES_NAME
+          stat = File.lstat(path)
+          paths << rel unless stat.directory?
+        end
+        paths
+      end
+
+      def collect_files(root)
+        entries = []
+        Find.find(root) do |path|
+          next if path == root
+
+          rel = path.delete_prefix("#{root}/")
+          stat = File.lstat(path)
+
+          if stat.symlink?
+            target = File.readlink(path)
+            entries << file_entry(rel, "symlink", stat, Digest::SHA256.hexdigest(target))
+          elsif stat.directory?
+            entries << dir_entry(rel, stat)
+          else
+            entries << file_entry(rel, "file", stat, Digest::SHA256.file(path).hexdigest)
+          end
+        end
+
+        entries.sort_by { |entry| entry["path"] }
+      end
+
+      def build_block(extensions:)
+        { "extensions" => !!extensions }
+      end
+
+      def order_keys(object)
+        case object
+        when Hash
+          object.keys.sort.each_with_object({}) do |key, acc|
+            acc[key] = order_keys(object[key])
+          end
+        when Array
+          object.map { |entry| order_keys(entry) }
+        else
+          object
+        end
+      end
+
+      def dir_entry(rel, stat)
+        {
+          "mode" => stat.mode & 0o777,
+          "path" => rel,
+          "size" => 0,
+          "type" => "dir",
+        }
+      end
+
+      def file_entry(rel, type, stat, sha)
+        entry = {
+          "mode" => stat.mode & 0o777,
+          "path" => rel,
+          "size" => stat.size,
+          "type" => type,
+        }
+        entry["sha256"] = sha if sha
+        entry
+      end
+    end
+  end
+end