im 0.1.6 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3ef66ecbb4745e878d911ec789567b204abe1bb174f57cee6b04ca6caddb76d9
-  data.tar.gz: a8cb56445189f2eb3d4fc4b7ff3b49c028b5b43cc7d8e20961481b125171d777
+  metadata.gz: a146ab43403d85ad1ba24849e235c4c3f9c6f1f344c350f93f7815163ad0ec17
+  data.tar.gz: 88f76dc7b193e2566b6603f4828ba07cade314dc08853b4941bce01df442954f
 SHA512:
-  metadata.gz: f4fed0705aca8dfd1eb036ea78c5e5a71f64837134669937a31c04daf97224744d6c881363a688adaf61fa01322c0b462c1d6fa57ac5bf53b2eee72deb633634
-  data.tar.gz: a3e0d1b7e940c0afd22d4415d855964b73a043390fd6538f075affe1eb29964c5b7c3b4cf675a738a6b999ab73cba72ce2ec86af790797866ff97ee685979fb5
+  metadata.gz: 53b9300337db2c3c716b9527f16ad78b29e65629da09306e22f48312907ef18d9e66f34e5b7a627dd7cf5daee00fd79953b222984937e43c49dd9f20934d6af2
+  data.tar.gz: 706be184a19a39e665c8324c69b98bcd847f908c206382338b933470e803a7503c3b376227c6be6913ebe8630c83f4c865d9f943a76bcaaa224931139b8e09f9

data/MIT-LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2023 Chris Salzberg
+Copyright (c) 2019 Xavier Noria
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -1,64 +1,277 @@
 # Im
 
-Im is a (currently) experimental implementation of import modules in Ruby,
-similar to systems like CommonJS, ESmodules, and RequireJS in Javascript. It
-requires a patched version of Ruby which you will need to build in order to use
-the gem.
+[![Build Status](https://github.com/shioyama/im/actions/workflows/ci.yml/badge.svg)][actions]
 
-## Installation
+[actions]: https://github.com/shioyama/im/actions
 
-Im requires a patched version of Ruby to use; if not installed, the gem will raise and exit.
+<!-- TOC -->
 
-You can find the patched version of Ruby here:
+- [Introduction](#introduction)
+- [Synopsis](#synopsis)
+- [File structure](#file-structure)
+  - [File paths match constant paths under loader](#file-paths-match-constant-paths-under-loader)
+  - [Root directories](#root-directories)
+  - [Relative and absolute cpaths](#relative-and-absolute-cpaths)
+- [Usage](#usage)
+- [Motivation](#motivation)
+- [License](#license)
 
-https://github.com/shioyama/ruby/tree/import_modules
+<!-- /TOC -->
 
-Once installed, install the gem and the error should go away.
+<a id="markdown-introduction" name="introduction"></a>
+## Introduction
 
-## Usage
+Im is a thread-safe code loader for anonymous-rooted namespaces in Ruby. It
+allows you to share any nested, autoloaded set of code without polluting or in
+any way touching the global namespace.
+
+To do this, Im leverages code autoloading, Zeitwerk conventions around file
+structure and naming, and two features added in Ruby 3.2: `Kernel#load`
+with a module argument[^1] and `Module#const_added`[^2]. Since these Ruby
+features are essential to its design, it cannot be used with earlier versions
+of Ruby.
+
+Im started its life as a fork of Zeitwerk and has a very similar interface. The
+gem strives to follow the Zeitwerk pattern as much as possible. Im and Zeitwerk
+can be used alongside each other provided there is no overlap between file
+paths managed by each gem.
+
+Im is in active development and should be considered experimental until the
+eventual release of version 1.0. Versions 0.1.6 and earlier of the gem were
+part of a different experiment and are unrelated to the current gem.
+
+<a id="markdown-synopsis" name="synopsis"></a>
+## Synopsis
+
+Im follows an interface that is in most respects identical to Zeitwerk.
+The central difference is that whereas Zeitwerk loads constants into the global
+namespace (rooted in `Object`), Im loads them into anonymous namespaces rooted
+on the loader itself. Loaders in Im are a subclass of the `Module` class, and
+thus each one can define its own namespace. Since there can be arbitrarily many
+loaders, there can also be arbitrarily many autoloaded namespaces.
 
-Extend `Im` and use `import` to import files or gems under an anonymous module namespace:
+Im's gem interface looks like this:
 
 ```ruby
+# lib/my_gem.rb (main file)
+
 require "im"
+loader = Im::Loader.for_gem
+loader.setup # ready!
+
+module loader::MyGem
+  # ...
+end
+
+loader.eager_load # optionally
+```
+
+The generic interface is likewise identical to Zeitwerk's:
+
+```ruby
+loader = Zeitwerk::Loader.new
+loader.push_dir(...)
+loader.setup # ready!
+```
+
+Other than gem names, the only difference here is in the definition of `MyGem`
+under the loader namespace in the gem code. Unlike Zeitwerk, with Im the gem
+namespace is not defined at toplevel:
+
+```ruby
+Object.const_defined?(:MyGem)
+#=> false
+```
+
+In order to prevent leakage, the gem's entrypoint, in this case
+`lib/my_gem.rb`), must not define anything at toplevel, hence the use of
+`module loader::MyGem`.
+
+Once the entrypoint has been required, all constants defined within the gem's
+file structure are autoloadable from the loader itself:
+
+```ruby
+# lib/my_gem/foo.rb
+
+module MyGem
+  class Foo
+    def hello_world
+      "Hello World!"
+    end
+  end
+end
+```
+
+```ruby
+foo = loader::MyGem::Foo
+# loads `Foo` from lib/my_gem/foo.rb
+
+foo.new.hello_world
+# "Hello World!"
+```
+
+Constants under the loader can be given permanent names that are different from
+the one defined in the gem itself:
+
+```ruby
+Bar = loader::MyGem::Foo
+Bar.new.hello_world
+# "Hello World!"
+```
+
+Since Im uses loaders as its namespace roots, it is important that consumers of
+gems have a way to fetch the loader for a given file path.
+
+The loader variable can go out of scope. Like Zeitwerk, Im keeps a registry
+with all of them, and so the object won't be garbage collected. For
+convenience, Im also provides a method, `Im#import`, to fetch a loader for
+a given file path:
+
+```ruby
+require "im"
+require "my_gem"
 
 extend Im
+my_gem = import "my_gem"
+#=> loader::MyGem is autoloadable
+```
 
-mod = import "active_model"
-#=> <#Im::Import root: active_model>
+Reloading works like Zeitwerk:
+
+```ruby
+loader = Im::Loader.new
+loader.push_dir(...)
+loader.enable_reloading # you need to opt-in before setup
+loader.setup
+...
+loader.reload
 ```
 
-Constants in the imported files are under the returned module, not the root namespace:
+You can assign a permanent name to an autoloaded constant, and it will be
+reloaded when the loader is reloaded:
 
 ```ruby
-ActiveModel
-#=> uninitialized constant ActiveModel (NameError)
+Foo = loader::Foo
+loader.reload # Object::Foo is replaced by an autoload
+Foo #=> autoload is triggered, reloading loader::Foo
+```
 
-mod::ActiveModel
-#=> ActiveModel
+Like Zeitwerk, you can eager-load all the code at once:
+
+```ruby
+loader.eager_load
 ```
 
-You can now assign your custom parent namespace and use the gem constants as always:
+Alternatively, you can broadcast `eager_load` to all loader instances:
 
 ```ruby
-MyRails = mod
+Im::Loader.eager_load_all
+```
 
-class EmailContact
-  include MyRails::ActiveModel::API
+<a id="markdown-file-structure" name="file-structure"></a>
+## File structure
 
-  attr_accessor :name
-  validates :name, presence: true
-end
+<a id="markdown-the-idea-file-paths-match-constant-paths-under-loader" name="the-idea-file-paths-match-constant-paths-under-loader"></a>
+### File paths match constant paths under loader
 
-contact = EmailContact.new
-contact.valid?
-#=> false
+File structure is identical to Zeitwerk, again with the difference that
+constants are loaded from the loader's namespace rather than the root one:
+
+```
+lib/my_gem.rb         -> loader::MyGem
+lib/my_gem/foo.rb     -> loader::MyGem::Foo
+lib/my_gem/bar_baz.rb -> loader::MyGem::BarBaz
+lib/my_gem/woo/zoo.rb -> loader::MyGem::Woo::Zoo
+```
+
+Im inherits support for collapsing directories and custom inflection, see
+Zeitwerk's documentation for details on usage of these features.
+
+<a id="markdown-root-directories" name="root-directories"></a>
+### Root directories
+
+Internally, each loader in Im can have one or more _root directories_ from which
+it loads code onto itself. Root directories are added to the loader using
+`Im::Loader#push_dir`:
+
+```ruby
+loader.push_dir("#{__dir__}/models")
+loader.push_dir("#{__dir__}/serializers"))
+```
+
+Note that concept of a _root namespace_, which Zeitwerk uses to load code
+under a given node of the global namespace, is absent in Im. Custom root
+namespaces are likewise not supported. These features were removed as they add
+complexity for little gain given Im's flexibility to anchor a namespace
+anywhere in the global namespace.
+
+<a id="markdown-relative-and-absolute-cpaths" name="relative-and-absolute-cpaths"></a>
+### Relative and absolute cpaths
+
+Im uses two types of constant paths: relative and absolute, wherever possible
+defaulting to relative ones. A relative cpath is a constant name relative to
+the loader in which it was originally defined, regardless of any other names it
+was assigned. Whereas Zeitwerk uses absolute cpaths, Im uses relative cpaths for
+all external loader APIs (see usage for examples).
 
-contact.name = "foo"
-contact.valid?
-#=> true
+To understand these concepts, it is important first to distinguish between two
+types of names in Ruby: _temporary names_ and _permanent names_.
+
+A temporary name is a constant name on an anonymous-rooted namespace, for
+example a loader:
+
+```ruby
+my_gem = import "my_gem"
+my_gem::Foo
+my_gem::Foo.name
+#=> "#<Im::Loader ...>::Foo"
+```
+
+Here, the string `"#<Im::Loader ...>::Foo"` is called a temporary name. We can
+give this module a permanent name by assigning it to a toplevel constant:
+
+```ruby
+Bar = my_gem::Foo
+my_gem::Foo.name
+#=> "Bar"
+```
+
+Now its name is `"Bar"`, and it is near impossible to get back its original
+temporary name.
+
+This property of module naming in Ruby is problematic since cpaths are used as
+keys in Im's internal registries to index constants and their autoloads, which
+is critical for successful autoloading.
+
+To get around this issue, Im tracks all module names and uses relative naming
+inside loader code. You can get the name of a module relative to the loader
+that loaded it with `Im::Loader#relative_cpath`:
+
+```ruby
+my_gem.relative_cpath(my_gem::Foo)
+#=> "Foo"
 ```
 
+Using relative cpaths frees Im from depending on `Module#name` for
+registry keys like Zeitwerk does, which does not work with anonymous
+namespaces. All public methods in Im that take a `cpath` take the _relative_
+cpath, i.e. the cpath relative to the loader as toplevel, regardless of any
+toplevel-rooted constant a module may have been assigned to.
+
+<a id="markdown-usage" name="usage"></a>
+## Usage
+
+(TODO)
+
+<a id="markdown-motivation" name="motivation"></a>
+## Motivation
+
+(TODO)
+
+<a id="markdown-license" name="license"></a>
 ## License
 
-The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+Released under the MIT License, Copyright (c) 2023 Chris Salzberg and 2019–<i>ω</i> Xavier Noria.
+
+[^1]: https://bugs.ruby-lang.org/issues/6210
+[^2]: https://bugs.ruby-lang.org/issues/17881

data/lib/im/const_path.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module Im
+  module ConstPath
+    UNBOUND_METHOD_MODULE_NAME = Module.instance_method(:name)
+    UNBOUND_METHOD_MODULE_TO_S = Module.instance_method(:to_s)
+    private_constant :UNBOUND_METHOD_MODULE_NAME, :UNBOUND_METHOD_MODULE_TO_S
+
+    # @sig (Module) -> String
+    def cpath(mod)
+      real_mod_name(mod) || real_mod_to_s(mod)
+    end
+
+    # @sig (Module) -> String?
+    def permanent_cpath(mod)
+      name = real_mod_name(mod)
+      return name unless temporary_name?(name)
+    end
+
+    # @sig (Module) -> Boolean
+    def permanent_cpath?(mod)
+      !temporary_cpath?(mod)
+    end
+
+    # @sig (Module) -> Boolean
+    def temporary_cpath?(mod)
+      temporary_name?(real_mod_name(mod))
+    end
+
+    private
+
+    # @sig (Module) -> String
+    def real_mod_to_s(mod)
+      UNBOUND_METHOD_MODULE_TO_S.bind_call(mod)
+    end
+
+    # @sig (Module) -> String?
+    def real_mod_name(mod)
+      UNBOUND_METHOD_MODULE_NAME.bind_call(mod)
+    end
+
+    # @sig (String) -> Boolean
+    def temporary_name?(name)
+      # There should be a nicer way to get this in Ruby.
+      name.nil? || name.start_with?("#")
+    end
+  end
+end

data/lib/im/error.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Im
+  class Error < StandardError
+  end
+
+  class ReloadingDisabledError < Error
+    def initialize
+      super("can't reload, please call loader.enable_reloading before setup")
+    end
+  end
+
+  class NameError < ::NameError
+  end
+
+  class SetupRequired < Error
+    def initialize
+      super("please, finish your configuration and call Im::Loader#setup once all is ready")
+    end
+  end
+
+  class InvalidModuleName < Error
+  end
+end

data/lib/im/explicit_namespace.rb

@@ -0,0 +1,96 @@
+# frozen_string_literal: true
+
+module Im
+  # Centralizes the logic for the trace point used to detect the creation of
+  # explicit namespaces, needed to descend into matching subdirectories right
+  # after the constant has been defined.
+  #
+  # The implementation assumes an explicit namespace is managed by one loader.
+  # Loaders that reopen namespaces owned by other projects are responsible for
+  # loading their constant before setup. This is documented.
+  module ExplicitNamespace # :nodoc: all
+    class << self
+      extend Internal
+
+      # Maps constant paths that correspond to explicit namespaces according to
+      # the file system, to the loader responsible for them.
+      #
+      # @sig Hash[String, [String, Im::Loader]]
+      attr_reader :cpaths
+      private :cpaths
+
+      # @sig Mutex
+      attr_reader :mutex
+      private :mutex
+
+      # @sig TracePoint
+      attr_reader :tracer
+      private :tracer
+
+      # Asserts `cpath` corresponds to an explicit namespace for which `loader`
+      # is responsible.
+      #
+      # @sig (String, Im::Loader) -> void
+      internal def register(cpath, module_name, loader)
+        mutex.synchronize do
+          cpaths[cpath] = [module_name, loader]
+          # We check enabled? because, looking at the C source code, enabling an
+          # enabled tracer does not seem to be a simple no-op.
+          tracer.enable unless tracer.enabled?
+        end
+      end
+
+      # @sig (Im::Loader) -> void
+      internal def unregister_loader(loader)
+        cpaths.delete_if { |_cpath, (_, l)| l == loader }
+        disable_tracer_if_unneeded
+      end
+
+      # @sig (String, String) -> void
+      internal def update_cpaths(prefix, replacement)
+        pattern = /^#{prefix}/
+        mutex.synchronize do
+          cpaths.transform_keys! do |key|
+            key.start_with?(prefix) ? key.gsub(pattern, replacement) : key
+          end
+        end
+      end
+
+      # @sig () -> void
+      private def disable_tracer_if_unneeded
+        mutex.synchronize do
+          tracer.disable if cpaths.empty?
+        end
+      end
+
+      # @sig (TracePoint) -> void
+      private def tracepoint_class_callback(event)
+        # If the class is a singleton class, we won't do anything with it so we
+        # can bail out immediately. This is several orders of magnitude faster
+        # than accessing its name.
+        return if event.self.singleton_class?
+
+        # It might be tempting to return if name.nil?, to avoid the computation
+        # of a hash code and delete call. But Ruby does not trigger the :class
+        # event on Class.new or Module.new, so that would incur in an extra call
+        # for nothing.
+        #
+        # On the other hand, if we were called, cpaths is not empty. Otherwise
+        # the tracer is disabled. So we do need to go ahead with the hash code
+        # computation and delete call.
+        relative_cpath, loader = cpaths.delete(Im.cpath(event.self))
+        if loader
+          loader.on_namespace_loaded(relative_cpath)
+          disable_tracer_if_unneeded
+        end
+      end
+    end
+
+    @cpaths = {}
+    @mutex  = Mutex.new
+
+    # We go through a method instead of defining a block mainly to have a better
+    # label when profiling.
+    @tracer = TracePoint.new(:class, &method(:tracepoint_class_callback))
+  end
+end

data/lib/im/gem_inflector.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module Im
+  class GemInflector < Inflector
+    # @sig (String) -> void
+    def initialize(root_file)
+      namespace     = File.basename(root_file, ".rb")
+      lib_dir       = File.dirname(root_file)
+      @version_file = File.join(lib_dir, namespace, "version.rb")
+    end
+
+    # @sig (String, String) -> String
+    def camelize(basename, abspath)
+      abspath == @version_file ? "VERSION" : super
+    end
+  end
+end

data/lib/im/gem_loader.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+module Im
+  # @private
+  class GemLoader < Loader
+    # Users should not create instances directly, the public interface is
+    # `Im::Loader.for_gem`.
+    private_class_method :new
+
+    # @private
+    # @sig (String, bool) -> Im::GemLoader
+    def self._new(root_file, warn_on_extra_files:)
+      new(root_file, warn_on_extra_files: warn_on_extra_files)
+    end
+
+    # @sig (String, bool) -> void
+    def initialize(root_file, warn_on_extra_files:)
+      super()
+
+      @tag                 = File.basename(root_file, ".rb")
+      @inflector           = GemInflector.new(root_file)
+      @root_file           = File.expand_path(root_file)
+      @lib                 = File.dirname(root_file)
+      @warn_on_extra_files = warn_on_extra_files
+
+      push_dir(@lib)
+    end
+
+    # @sig () -> void
+    def setup
+      warn_on_extra_files if @warn_on_extra_files
+      super
+    end
+
+    private
+
+    # @sig () -> void
+    def warn_on_extra_files
+      expected_namespace_dir = @root_file.delete_suffix(".rb")
+
+      ls(@lib) do |basename, abspath|
+        next if abspath == @root_file
+        next if abspath == expected_namespace_dir
+
+        basename_without_ext = basename.delete_suffix(".rb")
+        cname = inflector.camelize(basename_without_ext, abspath).to_sym
+        ftype = dir?(abspath) ? "directory" : "file"
+
+        warn(<<~EOS)
+          WARNING: Im defines the constant #{cname} after the #{ftype}
+
+              #{abspath}
+
+          To prevent that, please configure the loader to ignore it:
+
+              loader.ignore("\#{__dir__}/#{basename}")
+
+          Otherwise, there is a flag to silence this warning:
+
+              Im::Loader.for_gem(warn_on_extra_files: false)
+        EOS
+      end
+    end
+  end
+end

data/lib/im/inflector.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module Im
+  class Inflector
+    # Very basic snake case -> camel case conversion.
+    #
+    #   inflector = Im::Inflector.new
+    #   inflector.camelize("post", ...)             # => "Post"
+    #   inflector.camelize("users_controller", ...) # => "UsersController"
+    #   inflector.camelize("api", ...)              # => "Api"
+    #
+    # Takes into account hard-coded mappings configured with `inflect`.
+    #
+    # @sig (String, String) -> String
+    def camelize(basename, _abspath)
+      overrides[basename] || basename.split('_').each(&:capitalize!).join
+    end
+
+    # Configures hard-coded inflections:
+    #
+    #   inflector = Im::Inflector.new
+    #   inflector.inflect(
+    #     "html_parser"   => "HTMLParser",
+    #     "mysql_adapter" => "MySQLAdapter"
+    #   )
+    #
+    #   inflector.camelize("html_parser", abspath)      # => "HTMLParser"
+    #   inflector.camelize("mysql_adapter", abspath)    # => "MySQLAdapter"
+    #   inflector.camelize("users_controller", abspath) # => "UsersController"
+    #
+    # @sig (Hash[String, String]) -> void
+    def inflect(inflections)
+      overrides.merge!(inflections)
+    end
+
+    private
+
+    # Hard-coded basename to constant name user maps that override the default
+    # inflection logic.
+    #
+    # @sig () -> Hash[String, String]
+    def overrides
+      @overrides ||= {}
+    end
+  end
+end

data/lib/im/internal.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+# This is a private module.
+module Im::Internal
+  def internal(method_name)
+    private method_name
+
+    mangled = "__#{method_name}"
+    alias_method mangled, method_name
+    public mangled
+  end
+end