fontist 2.1.6 → 2.2.0

data/TODO.fontist-v5.md

@@ -0,0 +1,196 @@
+# Fontist v5 Schema Implementation Plan
+
+## Overview
+
+This document outlines the implementation plan for Formula Schema v5 with
+multi-format font support. The architecture uses **v5-only classes** -
+migration from v4 to v5 is handled by a separate migration script.
+
+## Architecture
+
+### Core Principle: v5 Only
+
+All formula classes now use v5 schema with:
+- `schema_version: 5` (default)
+- Format metadata on resources (`format`, `variable_axes`)
+- Format metadata on styles (`formats`, `variable_font`, `variable_axes`)
+
+### Class Hierarchy
+
+```
+Formula (v5 only)
+├── schema_version: 5
+├── resources: ResourceCollection
+│   └── Resource
+│       ├── format: string (ttf, otf, woff, woff2, ttc, otc, dfont)
+│       └── variable_axes: array (e.g., ["wght", "wdth"])
+├── fonts: FontModel[]
+│   └── styles: FontStyle[]
+│       ├── formats: array
+│       ├── variable_font: boolean
+│       └── variable_axes: array
+└── font_collections: FontCollection[]
+```
+
+## Implementation Status
+
+### Phase 1: Core Models ✅ DONE
+
+- [x] FormatSpec model (`lib/fontist/format_spec.rb`)
+- [x] FormatMatcher service (`lib/fontist/format_matcher.rb`)
+- [x] FontFinder service (`lib/fontist/font_finder.rb`)
+- [x] v5-only Formula class (`lib/fontist/formula.rb`)
+- [x] v5-only Resource class (`lib/fontist/resource.rb`)
+- [x] v5-only FontStyle class (`lib/fontist/font_style.rb`)
+- [x] ResourceCollection class (`lib/fontist/resource_collection.rb`)
+- [x] All 65 tests passing
+
+### Phase 2: Import System Fixes ✅ DONE
+
+#### Importer Format Metadata Status
+
+| Importer | format in resources | variable_axes | Status |
+|----------|-------------------|---------------|--------|
+| Google V5 | ✅ Dynamic | ✅ Yes | DONE |
+| Google V4 | ✅ Hardcoded "ttf" | ⚠️ Skipped | N/A |
+| macOS | ✅ Via CreateFormula | ✅ Via CreateFormula | DONE |
+| SIL | ✅ Via CreateFormula | ✅ Via CreateFormula | DONE |
+
+#### Tasks
+
+- [x] Fix `CreateFormula` class to detect and add format metadata
+- [x] macOS importer uses CreateFormula (now has format detection)
+- [x] SIL importer uses CreateFormula (now has format detection)
+- [x] Verify Google V5 importer works correctly
+
+### Phase 3: Migration Script ✅ DONE
+
+Create `lib/fontist/import/v4_to_v5_migrator.rb`: ✅ Created
+
+Migration features:
+- [x] Reads v4 YAML
+- [x] Adds schema_version: 5
+- [x] Detects format from file extensions
+- [x] Detects variable fonts from filename patterns
+- [x] Writes v5 YAML
+- [x] CLI command: `fontist migrate-formulas INPUT [OUTPUT]`
+
+### Phase 4: Run Imports ✅ DONE (via migration)
+
+All existing formulas migrated to v5 schema. Fresh imports can be run if needed:
+
+```bash
+# Google Fonts (if fresh import needed)
+fontist import google --schema-version=5 --output-path=./Formulas/google --force
+
+# macOS Fonts (if fresh import needed)
+fontist import macos --schema-version=5 --output-path=./Formulas/macos --force
+
+# SIL Fonts (if fresh import needed)
+fontist import sil --schema-version=5 --output-path=./Formulas/sil --force
+```
+
+### Phase 5: Migrate Existing Formulas ✅ DONE
+
+```bash
+# Run migration on all existing v4 formulas
+fontist migrate-formulas ../formulas/Formulas ../formulas/Formulas --verbose
+```
+
+Results:
+- Migrated: 3206 formulas
+- Skipped (already v5): 474 formulas
+- Failed: 1 (malformed YAML backup file)
+- Total time: ~5 seconds
+
+## Critical Files
+
+### Core Classes (v5 only)
+
+| File | Description | Status |
+|------|-------------|--------|
+| `lib/fontist/formula.rb` | v5 formula with schema_version | ✅ Done |
+| `lib/fontist/resource.rb` | Resource with format/variable_axes | ✅ Done |
+| `lib/fontist/resource_collection.rb` | Resource collection | ✅ Done |
+| `lib/fontist/font_style.rb` | FontStyle with format metadata | ✅ Done |
+| `lib/fontist/font_model.rb` | FontModel using FontStyle | ✅ Done |
+| `lib/fontist/font_collection.rb` | FontCollection using FontModel | ✅ Done |
+
+### Services
+
+| File | Description | Status |
+|------|-------------|--------|
+| `lib/fontist/format_spec.rb` | FormatSpec model | ✅ Done |
+| `lib/fontist/format_matcher.rb` | Format matching service | ✅ Done |
+| `lib/fontist/font_finder.rb` | Font discovery by capabilities | ✅ Done |
+
+### Import System
+
+| File | Description | Status |
+|------|-------------|--------|
+| `lib/fontist/import/create_formula.rb` | Generic formula creator | ✅ Fixed |
+| `lib/fontist/import/formula_builder.rb` | Base formula builder | ✅ Works |
+| `lib/fontist/import/google/formula_builder_v5.rb` | Google V5 builder | ✅ Done |
+| `lib/fontist/import/macos_importer.rb` | macOS importer | ✅ Works |
+| `lib/fontist/import/sil_importer.rb` | SIL importer | ✅ Works |
+| `lib/fontist/import/v4_to_v5_migrator.rb` | Migration script | ✅ Created |
+
+## Next Steps
+
+1. ~~**Fix CreateFormula class**~~ - ✅ Done
+2. ~~**Fix macOS importer**~~ - ✅ Works via CreateFormula
+3. ~~**Fix SIL importer**~~ - ✅ Works via CreateFormula
+4. ~~**Create V4→V5 migrator**~~ - ✅ Created
+5. ~~**Migrate formulas**~~ - ✅ 3206 migrated, 474 skipped
+6. ~~**Commit formulas**~~ - ✅ Committed to v5 branch (3cbe032)
+7. ~~**Fix V5 builder for variable fonts**~~ - ✅ Done (8e0e568)
+8. **Run fresh Google import** - Import with v5 schema to get WOFF2/variable fonts
+9. **Verify** - Test installation with v5 formulas
+
+## v5 Formula Example (Roboto Flex)
+
+```yaml
+schema_version: 5
+resources:
+  woff2_variable:
+    format: woff2
+    variable_axes: [GRAD, XOPQ, XTRA, YOPQ, YTAS, YTDE, YTFI, YTLC, YTUC, opsz, slnt, wdth, wght]
+  ttf_variable:
+    format: ttf
+    variable_axes: [GRAD, XOPQ, XTRA, YOPQ, YTAS, YTDE, YTFI, YTLC, YTUC, opsz, slnt, wdth, wght]
+fonts:
+- styles:
+  - variable_font: true
+    variable_axes: [GRAD, XOPQ, XTRA, YOPQ, YTAS, YTDE, YTFI, YTLC, YTUC, opsz, slnt, wdth, wght]
+    formats: [ttf, woff2]
+```
+
+## Breaking Changes
+
+- v4 formulas are no longer supported directly
+- Migration or re-import required for all formulas
+- All formulas must have `schema_version: 5`
+- Resources must have `format` field in v5
+
+## Format Detection Logic
+
+### File Extension → Format
+
+| Extension | Format |
+|-----------|--------|
+| .ttf | ttf |
+| .otf | otf |
+| .woff | woff |
+| .woff2 | woff2 |
+| .ttc | ttc |
+| .otc | otc |
+| .dfont | dfont |
+
+### Filename Pattern → Variable Axes
+
+| Pattern | Axes |
+|---------|------|
+| `Font[wght].ttf` | ["wght"] |
+| `Font[wght,wdth].ttf` | ["wght", "wdth"] |
+| `Font-Variable.ttf` | Detect from font file |
+| `Font-VF.ttf` | Detect from font file |

data/fontist.gemspec

@@ -30,7 +30,7 @@ Gem::Specification.new do |spec|
   spec.executables   = ["fontist"]
 
   spec.add_dependency "down", "~> 5.0"
-  spec.add_dependency "excavate", "~> 0.3", ">= 0.3.8"
+  spec.add_dependency "excavate", "~> 1.0", ">= 1.0.3"
   spec.add_dependency "fontisan", "~> 0.2", ">= 0.2.11"
   spec.add_dependency "fuzzy_match", "~> 2.1"
   spec.add_dependency "git", "> 1.0"

data/lib/fontist/cache/store.rb

@@ -79,12 +79,17 @@ module Fontist
         return nil unless File.exist?(cache_path(key))
 
         begin
-          Marshal.load(File.read(cache_path(key)))
-        rescue ArgumentError, TypeError => e
+          # Use binary read mode for Marshal data - critical on Windows
+          Marshal.load(File.binread(cache_path(key)))
+        rescue ArgumentError, TypeError, EOFError
           # Cache file is corrupted - delete it and return nil
           # This can happen on Windows when file is read while being written,
           # or when cache files from previous runs are corrupted
-          File.delete(cache_path(key)) rescue nil
+          begin
+            File.delete(cache_path(key))
+          rescue StandardError
+            nil
+          end
           nil
         end
       end
@@ -92,15 +97,20 @@ module Fontist
       def write_entry(key, entry)
         # Use temp file + atomic rename to prevent race conditions
         # This ensures readers never see partial writes, even on Windows
-        temp_path = cache_path(key) + ".tmp"
+        temp_path = "#{cache_path(key)}.tmp"
 
-        File.write(temp_path, Marshal.dump(entry))
+        # Use binary write mode for Marshal data - critical on Windows
+        File.binwrite(temp_path, Marshal.dump(entry))
         # Atomic rename (overwrites target atomically)
         # File.rename is atomic on all platforms for same filesystem
         File.rename(temp_path, cache_path(key))
-      rescue => e
+      rescue StandardError => e
         # Clean up temp file if rename fails
-        File.delete(temp_path) rescue nil
+        begin
+          File.delete(temp_path)
+        rescue StandardError
+          nil
+        end
         raise e
       end
 

data/lib/fontist/cli.rb

@@ -116,6 +116,31 @@ module Fontist
            type: :string, aliases: :l,
            enum: ["fontist", "user", "system"],
            desc: "Install location: fontist (default), user, system"
+    # Format selection options
+    option :format,
+           type: :string,
+           desc: "Font format to install (ttf, otf, woff, woff2, ttc, otc). " \
+                 "If format not available, will transcode from available formats."
+    option :variable_axes,
+           type: :string,
+           desc: "Variable axes to match (comma-separated, e.g., 'wght,wdth')"
+    option :prefer_variable,
+           type: :boolean,
+           desc: "Prefer variable fonts over static fonts"
+    option :prefer_format,
+           type: :string,
+           desc: "Preferred format when multiple available"
+    option :transcode_path,
+           type: :string,
+           desc: "Directory to save transcoded fonts (default: same as install location)"
+    option :keep_original,
+           type: :boolean,
+           default: true,
+           desc: "Keep original font after transcoding"
+    # Collection options
+    option :collection_index,
+           type: :numeric,
+           desc: "Extract specific font from TTC/OTC collection (0-indexed)"
     def install(*fonts)
       handle_class_options(options)
 
@@ -216,6 +241,52 @@ module Fontist
       handle_error(e)
     end
 
+    desc "find", "Find fonts by capabilities"
+    option :axes, type: :string,
+                  desc: "Variable axes to match (comma-separated, e.g., 'wght,wdth')"
+    option :variable, type: :boolean,
+                      desc: "Find all variable fonts"
+    option :category, type: :string,
+                      desc: "Filter by category (sans-serif, serif, monospace, display)"
+    option :format, type: :string,
+                    desc: "Filter by format (ttf, otf, woff2)"
+    option :json, type: :boolean,
+                  desc: "Output as JSON"
+    def find
+      handle_class_options(options)
+
+      require_relative "font_finder"
+      require_relative "format_spec"
+
+      finder = FontFinder.new(
+        format_spec: FormatSpec.new(format: options[:format]),
+        category: options[:category],
+      )
+
+      results = if options[:variable]
+                  finder.variable_fonts
+                elsif options[:axes]
+                  axes = options[:axes].split(",").map(&:strip)
+                  finder.by_axes(axes)
+                elsif options[:category]
+                  finder.by_category(options[:category])
+                else
+                  error("Please specify --axes, --variable, or --category",
+                        STATUS_UNKNOWN_ERROR)
+                  return
+                end
+
+      if options[:json]
+        Fontist.ui.say(JSON.pretty_generate(results.map(&:to_h)))
+      else
+        print_find_results(results)
+      end
+
+      success
+    rescue Fontist::Errors::GeneralError => e
+      handle_error(e)
+    end
+
     desc "update", "Update formulas"
     def update
       handle_class_options(options)
@@ -227,6 +298,34 @@ module Fontist
       STATUS_REPO_COULD_NOT_BE_UPDATED
     end
 
+    desc "migrate-formulas INPUT [OUTPUT]",
+         "Migrate v4 formulas to v5 schema"
+    option :verbose, type: :boolean, desc: "Show detailed progress"
+    option :dry_run, type: :boolean,
+                     desc: "Show what would be done without making changes"
+    def migrate_formulas(input, output = nil)
+      handle_class_options(options)
+
+      require_relative "import/v4_to_v5_migrator"
+
+      migrator = Fontist::Import::V4ToV5Migrator.new(input, output,
+                                                     verbose: options[:verbose],
+                                                     dry_run: options[:dry_run])
+
+      results = migrator.migrate_all
+
+      if results[:failed].positive?
+        Fontist.ui.error("Migration completed with #{results[:failed]} error(s)")
+        STATUS_UNKNOWN_ERROR
+      else
+        Fontist.ui.success("Migrated #{results[:migrated]} formula(s), " \
+                           "skipped #{results[:skipped]} already v5 formula(s)")
+        success
+      end
+    rescue Fontist::Errors::GeneralError => e
+      handle_error(e)
+    end
+
     desc "manifest SUBCOMMAND ...ARGS", "Manage font manifests"
     subcommand "manifest", Fontist::ManifestCLI
 
@@ -239,6 +338,7 @@ module Fontist
                                 "Uses `fnmatch` patterns."
     option :name_prefix, desc: "Prefix to add to all font family names, " \
                                "e.g. 'Wine ' for compatibility fonts"
+    option :schema_version, type: :numeric, default: 5, desc: "Formula schema version (default: 5)"
     def create_formula(url)
       handle_class_options(options)
       name = Fontist::Import::CreateFormula.new(url, options).call
@@ -371,5 +471,21 @@ module Fontist
       end
     end
     # rubocop:enable Metrics/AbcSize, Metrics/MethodLength
+
+    def print_find_results(results)
+      if results.empty?
+        Fontist.ui.say("No fonts found matching criteria")
+        return
+      end
+
+      Fontist.ui.say("Found #{results.count} fonts:\n")
+
+      results.each do |match|
+        Fontist.ui.say("  #{match.name}")
+        Fontist.ui.say("    Axes: #{match.axes.join(', ')}") if match.axes&.any?
+        Fontist.ui.say("    Format: #{match.format}") if match.format
+        Fontist.ui.say("    Category: #{match.category}") if match.category
+      end
+    end
   end
 end

data/lib/fontist/errors.rb

@@ -53,6 +53,13 @@ module Fontist
       end
     end
 
+    class UnsupportedSchemaVersionError < GeneralError
+      def initialize(version)
+        super("Unsupported formula schema version: #{version}. " \
+              "Supported versions: 4, 5")
+      end
+    end
+
     class MainRepoNotFoundError < FormulaIndexNotFoundError; end
 
     class InvalidConfigAttributeError < GeneralError; end
@@ -197,5 +204,59 @@ module Fontist
         end.join("\n")
       end
     end
+
+    # Format not available for font
+    class FormatNotAvailableError < GeneralError
+      def initialize(font_name, requested_format, available_formats)
+        super("Format '#{requested_format}' not available for font " \
+              "'#{font_name}'. Available formats: #{available_formats.join(', ')}")
+      end
+    end
+
+    # Variable axes not supported
+    class VariableAxesNotSupportedError < GeneralError
+      def initialize(font_name, requested_axes)
+        super("Variable axes #{requested_axes.join(', ')} not supported by " \
+              "font '#{font_name}'")
+      end
+    end
+
+    # Transcoding errors
+    class UnsupportedTranscodeError < GeneralError
+      def initialize(source_format, target_format)
+        super("Cannot transcode from '#{source_format}' to '#{target_format}'. " \
+              "Supported: TTF/OTF to WOFF/WOFF2")
+      end
+    end
+
+    class TranscodeToolNotFoundError < GeneralError
+      def initialize(tool_name)
+        super("Transcode tool '#{tool_name}' not found. " \
+              "Please install the required package.")
+      end
+    end
+
+    class SourceNotFoundError < GeneralError
+      def initialize(path)
+        super("Source font file not found: #{path}")
+      end
+    end
+
+    class CollectionIndexError < GeneralError
+      def initialize(index, max_index)
+        super("Collection index #{index} out of range. " \
+              "Valid range: 0-#{max_index}")
+      end
+    end
+
+    # License-related transcoding error
+    class TranscodeLicenseNotAcceptedError < GeneralError
+      def initialize(font_name)
+        super("Font '#{font_name}' requires license agreement. " \
+              "Transcoding is a form of modification that may not be permitted " \
+              "by all licenses. Please accept the license with " \
+              "--accept-all-licenses to proceed with transcoding.")
+      end
+    end
   end
 end

data/lib/fontist/extract.rb

@@ -21,5 +21,18 @@ module Fontist
       map "file", to: :file
       map "options", to: :options
     end
+
+    def empty?
+      format.nil? && file.nil? && options_empty?
+    end
+
+    private
+
+    def options_empty?
+      return true if options.nil?
+      return options.empty? if options.respond_to?(:empty?)
+      return options.file.nil? && options.fonts_sub_dir.nil? if options.is_a?(ExtractOptions)
+      false
+    end
   end
 end

data/lib/fontist/font.rb

@@ -1,3 +1,5 @@
+require_relative "format_spec"
+
 module Fontist
   class Font
     def initialize(options = {})
@@ -14,6 +16,9 @@ module Fontist
       @update_fontconfig = options[:update_fontconfig]
       @install_location = options[:location] || options[:install_location]
 
+      # Accept FormatSpec or create from options
+      @format_spec = options[:format_spec] || FormatSpec.from_options(options)
+
       validate_location_parameter!
       check_or_create_fontist_path!
     end
@@ -179,6 +184,8 @@ module Fontist
       options = {
         no_progress: @no_progress,
         location: @install_location,
+        format_spec: @format_spec,
+        confirmation: @confirmation,
       }
 
       if @by_formula
@@ -194,7 +201,8 @@ module Fontist
                           size_limit: @size_limit,
                           version: @version,
                           smallest: @smallest,
-                          newest: @newest)
+                          newest: @newest,
+                          format_spec: @format_spec)
           .call(downloadable_formulas)
     end
 

data/lib/fontist/font_collection.rb

@@ -1,6 +1,7 @@
 require "lutaml/model"
 
 module Fontist
+  # FontCollection - uses FontModel with v5 format metadata
   class FontCollection < Lutaml::Model::Serializable
     attribute :filename, :string
     attribute :source_filename, :string

data/lib/fontist/font_finder.rb

@@ -0,0 +1,154 @@
+require_relative "format_spec"
+require_relative "format_matcher"
+
+module Fontist
+  # Find fonts by their capabilities (axes, formats, etc.)
+  #
+  # Supports:
+  # - Find fonts with specific variable axes
+  # - Find fonts with any variable support
+  # - Find fonts by category (sans-serif, monospace, etc.)
+  #
+  # Examples:
+  #   FontFinder.by_axes(["wght", "wdth"])  # Fonts with both axes
+  #   FontFinder.variable_fonts             # All variable fonts
+  #   FontFinder.by_category("monospace")   # Monospace fonts
+  #
+  class FontFinder
+    def initialize(format_spec: nil, category: nil)
+      @format_spec = format_spec
+      @category = category
+    end
+
+    # Find fonts that support ALL specified axes
+    def by_axes(axes)
+      raise ArgumentError, "axes must be an array" unless axes.is_a?(Array)
+
+      matching_formulas.flat_map do |formula|
+        next [] unless formula.v5?
+
+        resources = each_resource(formula)
+        resources = apply_format_filter(resources)
+        resources.select do |resource|
+          resource.variable_font? && axes_supported?(resource, axes)
+        end.map do |resource|
+          build_font_match(formula, resource.name, resource)
+        end
+      end.flatten
+    end
+
+    # Find all variable fonts
+    def variable_fonts
+      matching_formulas.flat_map do |formula|
+        next [] unless formula.v5?
+
+        resources = each_resource(formula)
+        resources = apply_format_filter(resources)
+        resources.select(&:variable_font?).map do |resource|
+          build_font_match(formula, resource.name, resource)
+        end
+      end.flatten
+    end
+
+    # Find fonts by category
+    def by_category(category)
+      matching_formulas.select do |formula|
+        extract_category(formula) == category
+      end.map do |formula|
+        resource_names = extract_resource_names(formula)
+        FontMatch.new(
+          name: formula.name,
+          resources: resource_names,
+          category: category,
+        )
+      end
+    end
+
+    private
+
+    def each_resource(formula)
+      return [] unless formula.resources
+
+      Array(formula.resources)
+    end
+
+    def extract_resource_names(formula)
+      return [] unless formula.resources
+
+      Array(formula.resources).map(&:name).compact
+    end
+
+    def build_font_match(formula, name, resource)
+      FontMatch.new(
+        name: formula.name,
+        resource: name,
+        axes: resource.axes_tags,
+        format: resource.format,
+        category: extract_category(formula),
+      )
+    end
+
+    def matching_formulas
+      @matching_formulas ||= Formula.all.select do |formula|
+        next false if @category && extract_category(formula) != @category
+
+        true
+      end
+    end
+
+    def axes_supported?(resource, required_axes)
+      available = resource.axes_tags
+      required_axes.all? { |axis| available.include?(axis.to_s) }
+    end
+
+    def extract_category(formula)
+      # Extract from formula metadata if available
+      # Note: Formula does not have a category attribute currently
+      # Category is detected from name heuristics
+      detect_category_from_name(formula.name)
+    end
+
+    def detect_category_from_name(name)
+      # Heuristics for common patterns
+      return "monospace" if name.match?(/mono/i)
+      return "sans-serif" if name.match?(/sans[-\s]?serif/i)
+      return "serif" if name.match?(/serif/i)
+
+      "sans-serif"
+    end
+
+    def apply_format_filter(resources)
+      return resources unless @format_spec&.has_constraints?
+
+      matcher = FormatMatcher.new(@format_spec)
+      matcher.filter_resources(resources)
+    end
+
+  end
+
+  # Result object for font matches
+  class FontMatch
+    attr_reader :name, :resource, :axes, :format, :category, :resources
+
+    def initialize(name:, resource: nil, axes: [], format: nil,
+                   category: nil, resources: nil)
+      @name = name
+      @resource = resource
+      @axes = axes
+      @format = format
+      @category = category
+      @resources = resources
+    end
+
+    def to_h
+      {
+        name: name,
+        resource: resource,
+        resources: resources,
+        axes: axes,
+        format: format,
+        category: category,
+      }.compact
+    end
+  end
+end