reference_extractor 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: bac9513331865230939596e19a86fb7e446498d4c4f9194dcb68a97b5377a668
+  data.tar.gz: e5db87762acef10e792fb4f56b6cd333ea8f08a43b2f40f9d7c9f5206eb7daa0
+SHA512:
+  metadata.gz: 5439eaacdaf0df2b261122de2d5853542b4a51841945197aca3ee04b33dc259348946401598125faea3c7f941a826f06d1ef5dd265ee3a56dece72e944d0a6c8
+  data.tar.gz: 5970480b43f52c8abfe2c4d0a54dfffca9b024113d19a2642b6f93b10e4b5cfbf540dec4b0887ec0380d500846032358f889f96c1cfd70c0399172c57a562ce3

data/.standard.yml

@@ -0,0 +1,5 @@
+# For available configuration options, see:
+#   https://github.com/standardrb/standard
+ruby_version: 3.4
+ignore:
+  - 'test/fixtures/formats/**/*'

data/.tool-versions

@@ -0,0 +1 @@
+ruby 3.4.7

data/CHANGELOG.md

@@ -0,0 +1,7 @@
+# Changelog
+
+## [Unreleased]
+
+## [0.1.0] - 2025-10-30
+
+- Initial release

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,132 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Philip Theus (prev. Mueller)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Philip Theus
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,72 @@
+# 🚧 [WIP] ReferenceExtractor [WIP] 🚧
+
+## Introduction
+
+ReferenceExtractor parses Ruby files and returns the constants they reference, using your Zeitwerk autoloaders to resolve what lives where. It gives you a graph you can use for architecture rules like layering or dependency checks.
+
+ReferenceExtractor is in _prototype_ stage. It works in general but is not battle tested.
+
+It is based on [packwerk](https://github.com/Shopify/packwerk).
+
+## Usage
+
+```ruby
+extractor = ReferenceExtractor::Extractor.new(
+  autoloaders: Rails.autoloaders,
+  root_path: Rails.root
+)
+
+# From a string snippet
+extractor.references_from_string("Order.find(1)")
+# => [#<ReferenceExtractor::Reference constant=#<ReferenceExtractor::ConstantContext name=\"::Order\" ...>>]
+
+# From a file relative to root_path
+extractor.references_from_file("app/models/user.rb")
+# => [#<ReferenceExtractor::Reference ...>, ...]
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Publishing to RubyGems
+
+1. Update the version in `lib/reference_extractor/version.rb`. Push / merge to main.
+2. Build the gem:
+
+   ```bash
+   gem build reference_extractor.gemspec
+   ```
+
+   This should produce `reference_extractor-<version>.gem`.
+3. Sign in to RubyGems (only needed once):
+
+   ```bash
+   gem signin
+   ```
+
+4. Push the built gem:
+
+   ```bash
+   gem push reference_extractor-<version>.gem
+   ```
+
+5. Tag the release:
+
+   ```bash
+   git tag v<version> && git push origin v<version>
+   ```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/exterm/reference_extractor. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/exterm/reference_extractor/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the ReferenceExtractor project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/exterm/reference_extractor/blob/main/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "minitest/test_task"
+
+Minitest::TestTask.create
+
+require "standard/rake"
+
+task default: %i[test standard]

data/exe/reference_extractor

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+
+require "reference_extractor"

data/lib/reference_extractor/constant_context.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module ReferenceExtractor
+  ConstantContext = Struct.new(:name, :location)
+end

data/lib/reference_extractor/extractor.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+module ReferenceExtractor
+  # Public API for extracting constant references from Ruby code (that is autoloaded via Zeitwerk).
+  #
+  # @example
+  #   extractor = ReferenceExtractor::Extractor.new(autoloaders: Rails.autoloaders, root_path: Rails.root)
+  #   references = extractor.references_from_string("Order.find(1)")
+  #   references = extractor.references_from_file("app/models/user.rb")
+  class Extractor
+    attr_reader :root_path
+
+    # @param autoloaders [Enumerable] Collection of Zeitwerk loaders, e.g. from `Rails.autoloaders`
+    # @param root_path [String, Pathname] The root path of the application, e.g. from `Rails.root`
+    def initialize(autoloaders:, root_path:)
+      @autoloaders = autoloaders
+      @root_path = Pathname.new(root_path)
+      @context_provider = Internal::ConstantDiscovery.new(root_path:, loaders: @autoloaders)
+    end
+
+    # Extract constant references from a Ruby code string.
+    #
+    # @param snippet [String] The Ruby code to analyze
+    # @return [Array<Reference>] Array of references to autoloaded constants in project files
+    def references_from_string(snippet)
+      ast = parse_ruby_string(snippet)
+      return [] unless ast
+
+      extract_references(ast, relative_path: "<snippet>")
+    end
+
+    # Extract constant references from a Ruby file.
+    #
+    # @param file_path [String, Pathname] Path to the Ruby file (relative to root_path or absolute)
+    # @return [Array<Reference>] Array of references to autoloaded constants in project files
+    def references_from_file(file_path)
+      absolute_path = Pathname.new(file_path).expand_path(root_path)
+      return [] unless File.exist?(absolute_path)
+
+      ast = parse_file(absolute_path)
+      return [] unless ast
+
+      relative_path = Pathname.new(absolute_path).relative_path_from(root_path).to_s
+      extract_references(ast, relative_path:)
+    end
+
+    private
+
+    def parse_ruby_string(snippet)
+      parser = Internal::Parsers::Ruby.new
+      parser.call(io: StringIO.new(snippet))
+    end
+
+    def parse_file(file_path)
+      parser = Internal::Parsers::Factory.instance.for_path(file_path.to_s)
+      raise ArgumentError, "Unsupported file type: #{file_path}" unless parser
+
+      File.open(file_path, "r") do |io|
+        parser.call(io:, file_path: file_path.to_s)
+      end
+    end
+
+    def extract_references(root_node, relative_path:)
+      ast_reference_extractor.extract_references(root_node, relative_path:)
+    end
+
+    def ast_reference_extractor
+      @ast_reference_extractor ||= Internal::AstReferenceExtractor.new(
+        # TO DO: Add association inspector
+        constant_name_inspectors: [Internal::ConstNodeInspector.new],
+        context_provider: @context_provider,
+        root_path:
+      )
+    end
+  end
+end

data/lib/reference_extractor/internal/ast_reference_extractor.rb

@@ -0,0 +1,120 @@
+# frozen_string_literal: true
+
+module ReferenceExtractor
+  module Internal
+    # Extracts a possible constant reference from a given AST node.
+    class AstReferenceExtractor
+      class << self
+        def get_fully_qualified_references_from(unresolved_references, context_provider)
+          fully_qualified_references = []
+
+          unresolved_references.each do |unresolved_reference|
+            constant =
+              context_provider.context_for(
+                unresolved_reference.constant_name,
+                current_namespace_path: unresolved_reference.namespace_path
+              )
+
+            next if constant.nil?
+
+            # Ignore references that resolve to the same file they originate from.
+            next if constant.location.to_s == unresolved_reference.relative_path.to_s
+
+            fully_qualified_references << Reference.new(
+              relative_path: unresolved_reference.relative_path,
+              constant: constant,
+              source_location: unresolved_reference.source_location
+            )
+          end
+
+          fully_qualified_references
+        end
+      end
+
+      def initialize(
+        constant_name_inspectors:,
+        context_provider:,
+        root_path:
+      )
+        @constant_name_inspectors = constant_name_inspectors
+        @context_provider = context_provider
+        @root_path = root_path
+      end
+
+      # Extract and resolve all references from the AST in one step
+      def extract_references(root_node, relative_path:)
+        local_constant_definitions = ParsedConstantDefinitions.new(root_node: root_node)
+        unresolved_references = collect_references(
+          root_node,
+          ancestors: [],
+          relative_path: relative_path,
+          local_constant_definitions: local_constant_definitions
+        )
+        self.class.get_fully_qualified_references_from(unresolved_references, @context_provider)
+      end
+
+      def reference_from_node(node, ancestors:, relative_path:, local_constant_definitions:)
+        constant_name = nil
+
+        @constant_name_inspectors.each do |inspector|
+          constant_name = inspector.constant_name_from_node(node, ancestors:)
+
+          break if constant_name
+        end
+
+        if constant_name
+          reference_from_constant(
+            constant_name,
+            node:,
+            ancestors:,
+            relative_path:,
+            local_constant_definitions:
+          )
+        end
+      end
+
+      private
+
+      def reference_from_constant(constant_name, node:, ancestors:, relative_path:, local_constant_definitions:)
+        namespace_path = NodeHelpers.enclosing_namespace_path(node, ancestors: ancestors)
+
+        return if local_reference?(constant_name, NodeHelpers.name_location(node), namespace_path, local_constant_definitions)
+
+        UnresolvedReference.new(
+          constant_name:,
+          namespace_path:,
+          relative_path: relative_path,
+          source_location: NodeHelpers.location(node)
+        )
+      end
+
+      def local_reference?(constant_name, name_location, namespace_path, local_constant_definitions)
+        local_constant_definitions.local_reference?(
+          constant_name,
+          location: name_location,
+          namespace_path: namespace_path
+        )
+      end
+
+      def collect_references(node, ancestors:, relative_path:, local_constant_definitions:)
+        reference = reference_from_node(
+          node,
+          ancestors:,
+          relative_path:,
+          local_constant_definitions:
+        )
+
+        child_references = NodeHelpers.each_child(node).flat_map do |child|
+          collect_references(
+            child,
+            ancestors: [node] + ancestors,
+            relative_path:,
+            local_constant_definitions:
+          )
+        end
+
+        ([reference] + child_references).compact
+      end
+    end
+  end
+end

data/lib/reference_extractor/internal/const_node_inspector.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module ReferenceExtractor
+  module Internal
+    # Extracts a constant name from an AST node of type :const
+    class ConstNodeInspector
+      def constant_name_from_node(node, ancestors:)
+        return nil unless NodeHelpers.constant?(node)
+
+        parent = ancestors.first
+
+        # Only process the root `const` node for namespaced constant references. For example, in the
+        # reference `Spam::Eggs::Thing`, we only process the const node associated with `Spam`.
+        return nil unless root_constant?(parent)
+
+        if parent && constant_in_module_or_class_definition?(node, parent: parent)
+          fully_qualify_constant(ancestors)
+        else
+          begin
+            NodeHelpers.constant_name(node)
+          rescue NodeHelpers::TypeError
+            nil
+          end
+        end
+      end
+
+      private
+
+      def root_constant?(parent)
+        !(parent && NodeHelpers.constant?(parent))
+      end
+
+      def constant_in_module_or_class_definition?(node, parent:)
+        parent_name = NodeHelpers.module_name_from_definition(parent)
+        parent_name && parent_name == NodeHelpers.constant_name(node)
+      end
+
+      def fully_qualify_constant(ancestors)
+        # We're defining a class with this name, in which case the constant is implicitly fully qualified by its
+        # enclosing namespace
+        "::" + NodeHelpers.parent_module_name(ancestors: ancestors)
+      end
+    end
+  end
+end

data/lib/reference_extractor/internal/constant_discovery.rb

@@ -0,0 +1,104 @@
+# frozen_string_literal: true
+
+module ReferenceExtractor
+  module Internal
+    # Get information about unresolved constants without loading the application code.
+    # Information gathered: Fully qualified name and path to file containing the definition.
+    #
+    # The implementation makes a few assumptions about the code base:
+    # - `Something::SomeOtherThing` is defined in a path of either `something/some_other_thing.rb` or `something.rb`,
+    #   relative to the load path. Rails' `zeitwerk` autoloader makes the same assumption.
+    # - It is OK to not always infer the exact file defining the constant. For example, when a constant is inherited, we
+    #   have no way of inferring the file it is defined in. You could argue though that inheritance means that another
+    #   constant with the same name exists in the inheriting class, and this view is sufficient for all our use cases.
+    class ConstantDiscovery
+      class Error < StandardError; end
+
+      def initialize(root_path:, loaders:)
+        @root_path = root_path
+        @loaders = loaders
+      end
+
+      # Analyze a constant via its name.
+      # If the constant is unresolved, we need the current namespace path to correctly infer its full name
+      #
+      # @param const_name [String] The unresolved constant's name.
+      # @param current_namespace_path [Array<String>] (optional) The namespace of the context in which the constant is
+      #   used, e.g. ["Apps", "Models"] for `Apps::Models`. Defaults to [] which means top level.
+      # @return [ConstantContext]
+      def context_for(const_name, current_namespace_path: [])
+        current_namespace_path = [] if const_name.start_with?("::")
+        const_name, location = resolve_constant(const_name.delete_prefix("::"), current_namespace_path)
+
+        return unless location
+
+        relative_location = Pathname.new(location).relative_path_from(@root_path)
+        ConstantContext.new(const_name, relative_location)
+      end
+
+      # Analyze the constants and raise errors if any potential issues are encountered that would prevent
+      # resolving the context for constants, such as ambiguous constant locations.
+      #
+      # @return [ConstantDiscovery]
+      def validate_constants
+        const_locations
+        true
+      end
+
+      private
+
+      def const_locations
+        return @const_locations unless @const_locations.nil?
+
+        all_cpaths = @loaders.inject({}) do |cpaths, loader|
+          paths = loader.all_expected_cpaths.filter do |cpath, _const|
+            cpath.ends_with?(".rb")
+          end
+          cpaths.merge(paths)
+        end
+        paths_by_const = all_cpaths.invert
+        validate_constant_paths(paths_by_const, all_cpaths: all_cpaths)
+        @const_locations = paths_by_const
+      end
+
+      def resolve_constant(const_name, current_namespace_path, original_name: const_name)
+        namespace, location = resolve_traversing_namespace_path(const_name, current_namespace_path)
+        if location
+          ["::" + namespace.push(original_name).join("::"), location]
+        elsif !const_name.include?("::")
+          # constant could not be resolved to a file in the given load paths
+          [nil, nil]
+        else
+          parent_constant = const_name.split("::")[0..-2].join("::")
+          resolve_constant(parent_constant, current_namespace_path, original_name:)
+        end
+      end
+
+      def resolve_traversing_namespace_path(const_name, current_namespace_path)
+        fully_qualified_name_guess = (current_namespace_path + [const_name]).join("::")
+
+        location = const_locations[fully_qualified_name_guess]
+        if location || fully_qualified_name_guess == const_name
+          [current_namespace_path, location]
+        else
+          resolve_traversing_namespace_path(const_name, current_namespace_path[0..-2])
+        end
+      end
+
+      def validate_constant_paths(paths_by_constant, all_cpaths:)
+        raise(Error, "Could not find any ruby files.") if all_cpaths.empty?
+
+        is_ambiguous = all_cpaths.size != paths_by_constant.size
+        raise(Error, ambiguous_constants_hint(paths_by_constant, all_cpaths: all_cpaths)) if is_ambiguous
+      end
+
+      def ambiguous_constants_hint(paths_by_constant, all_cpaths:)
+        ambiguous_constants = all_cpaths.except(*paths_by_constant.invert.keys).values
+        <<~MSG
+          Ambiguous constant definition:
+          #{ambiguous_constants.map { |const| " - #{const}" }.join("\n")}
+        MSG
+      end
+    end
+  end
+end

data/lib/reference_extractor/internal/node.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module ReferenceExtractor
+  module Internal
+    module Node
+      Location = Struct.new(:line, :column)
+    end
+  end
+end