to_llm 0.1.2 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ada141bafdd63b00f12af14c5d5150a8de09deb030f6dd9605780af6acf2c01f
-  data.tar.gz: 05ac140d21cd33acb30f88914673a695ea9859479d0d115d8154826c04ee090b
+  metadata.gz: 79e005ef579c63795093a10bc0318048894fdaef71f959eaac97ec46d9d12340
+  data.tar.gz: 297867c42b4a9caf4d803798edd51deae0fcf63815f47a3e742ddd60cbf918ab
 SHA512:
-  metadata.gz: f63a54f041713178db91321fb2efd3fdce44103007da3de130f451df8892a93728cb72e816cf9f953627d44626fe2561e5bdfc464f7ca71628c47b6703c54960
-  data.tar.gz: 046b4b51252bbe3d38203b9e5f61eebe7ebd8decc69057eb178690cb8f36d3cfec06748a7497ed1cc26147c15efd09cf7552db898450fa6decfcc2f6242048fd
+  metadata.gz: 3aaa9070c20e815f41aebb34bb933e4ac289e45b0640edb0abe80af02b818ed3e42d7c902194df3571363229c0b5fd839cd3fc3c8d90e04f6bed6f61f15523a1
+  data.tar.gz: 9bde15f9c6be326a6659fd7d6d497234de245928c59179c0d5bc2487c254d446228684be518cbca768ec5a479e1449de2410db75ac02ff545cfd1426b7f17e65

data/README.md

@@ -1,166 +1,75 @@
 # to_llm
 
 [![Gem Version](https://badge.fury.io/rb/to_llm.svg)](https://badge.fury.io/rb/to_llm)
-[![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE.txt)
 
-**to_llm** is a lightweight Ruby gem that provides a simple set of tasks (or commands) to extract code from a Rails application into text files. This is useful if you want to feed your Rails codebase into a Large Language Model (LLM) or any text-based analysis tool.
-
----
-
-## Table of Contents
-
-- [Features](#features)
-- [Installation](#installation)
-- [Usage](#usage)
-- [Commands](#commands)
-- [Configuration](#configuration)
-- [Examples](#examples)
-- [Contributing](#contributing)
-- [License](#license)
-
----
+**to_llm** is a lightweight Ruby gem that allows you to extract code from your Rails application into text files. It’s especially handy if you want to feed your Rails codebase into a Large Language Model (LLM) or any text-based analysis tool.
 
 ## Features
 
-- **Simple extraction**: Automatically scans your Rails app for files in `app/models`, `app/views`, `app/controllers`, etc.  
-- **Configurable**: By default, extracts `.rb`, `.erb`, `.js`, and `.yml` file types, but you can easily adjust.  
-- **Rake tasks**: Provides a Rake task (`to_llm:extract`) to keep usage straightforward.  
-- **Keeps files separate**: Creates separate `.txt` output files (e.g., `models.txt`, `views.txt`) for easier organization (or combine them, if you prefer).
-
----
+- **Simple extraction**: Automatically scans core Rails directories (`app/models`, `app/controllers`, `app/views`, etc.).
+- **Configurable**: Extracts `.rb`, `.erb`, `.js`, `.yml`, `.ts`, and `.tsx` by default; easy to adjust.
+- **Rake tasks**: Single entry point for extracting everything or selective directories.
+- **Flexible formats**: Outputs to `.txt` or `.md`.
 
 ## Installation
 
-Add this line to your Rails project’s `Gemfile`:
+Add to your `Gemfile`:
 
 ```ruby
-gem 'to_llm', git: 'https://github.com/jcmaciel/to_llm.git'
+gem 'to_llm'
 ```
-
-> Or using RubyGems
-> ```ruby
-> gem 'to_llm'
-> ```
-
-Then execute:
+And then:
 
 ```bash
 bundle install
 ```
 
----
+*(Alternatively, you can point `gem 'to_llm', git: 'https://github.com/jcmaciel/to_llm.git'` to the GitHub repo.)*
 
 ## Usage
 
-Once installed, the gem integrates with your Rails app via a Railtie that exposes one or more rake tasks. By default, you can run:
+With **v0.1.3**, you can run the following commands inside your Rails app:
 
 ```bash
-rails to_llm:extract -[ALL|MODELS|CONTROLLERS|VIEWS|CONFIG|SCHEMA]
+rails "to_llm:extract[TYPE,FORMAT]"
 ```
 
-Each of these commands will scan the relevant folders in your Rails app and produce text files containing all the code it finds.
-
----
-
-## Commands
-
-### 1. `rails to_llm:extract -ALL`
-
-- **Description**: Extracts from all supported Rails directories:
-  - `app/models` -> `models.txt`  
-  - `app/controllers` -> `controllers.txt`  
-  - `app/views` -> `views.txt`  
-  - `app/helpers` -> `helpers.txt`  
-  - `config` -> `config.txt` (including `config/initializers`)  
-  - `db/schema.rb` -> `schema.txt`  
-- **Result**: Creates a folder named `to_llm/` with separate `.txt` files.
-
-### 2. `rails to_llm:extract -MODELS`
-
-- **Description**: Extracts only files from `app/models`.
-- **Output**: Creates (or overwrites) `to_llm/models.txt`.
-
-### 3. `rails to_llm:extract -CONTROLLERS`
-
-- **Description**: Extracts only from `app/controllers`.
-- **Output**: `to_llm/controllers.txt`.
-
-### 4. `rails to_llm:extract -VIEWS`
-
-- **Description**: Extracts from `app/views`.
-- **Output**: `to_llm/views.txt`.
-
-### 5. `rails to_llm:extract -CONFIG`
-
-- **Description**: Extracts from `config`, including `config/initializers`.
-- **Output**: `to_llm/config.txt`.
-
-### 6. `rails to_llm:extract -SCHEMA`
-
-- **Description**: Extracts only `db/schema.rb`.
-- **Output**: `to_llm/schema.txt`.
+Where:
+- **TYPE** can be `ALL`, `MODELS`, `CONTROLLERS`, `VIEWS`, `CONFIG`, `SCHEMA`, `JAVASCRIPT`, `HELPERS`.
+- **FORMAT** can be `txt` or `md`.
 
----
+**Examples**:
 
-## Configuration
-
-If you want to customize what extensions are included or which directories map to which output files, open the `lib/tasks/to_llm.rake` file (inside this gem) and edit:
-
-```ruby
-file_extensions = %w[.rb .erb .js .yml]
-directories_to_files = {
-  "app/models"      => "models.txt",
-  "app/controllers" => "controllers.txt",
-  # ...
-}
-```
-
-This gem is intentionally minimalist. Feel free to fork or override these settings in your local app.
-
----
-
-## Examples
-
-1. **Extract everything**:
+1. Extract *everything* to Markdown:
    ```bash
-   rails to_llm:extract -ALL
-   ```
-   Generates:
+   rails "to_llm:extract[ALL,md]"
    ```
-   to_llm/
-     models.txt
-     controllers.txt
-     views.txt
-     helpers.txt
-     config.txt
-     schema.txt
-   ```
-
-2. **Extract only views**:
+2. Extract only models to plain text:
    ```bash
-   rails to_llm:extract -VIEWS
-   ```
-   Generates:
+   rails "to_llm:extract[MODELS,txt]"
    ```
-   to_llm/
-     views.txt
-   ```
-   (Note that any previously existing files in `to_llm` will be untouched or overwritten if they have the same filename.)
+3. If you omit the second parameter, it defaults to `.txt` (old usage triggers a warning but still works).
+
+## Examples
+
+- **Extract everything**:
+  ```bash
+  rails "to_llm:extract[ALL,md]"
+  ```
+  Produces a `to_llm/` folder with separate `.md` files (e.g., `models.md`, `views.md`, etc.).
 
----
+- **Extract only controllers**:
+  ```bash
+  rails "to_llm:extract[CONTROLLERS,txt]"
+  ```
+  Creates/overwrites `to_llm/controllers.txt`.
 
 ## Contributing
 
-1. Fork the repo on GitHub.
-2. Create a new branch for your feature (`git checkout -b my-new-feature`).
+1. Fork the repository.
+2. Create a new branch (`git checkout -b my-new-feature`).
 3. Commit your changes (`git commit -am 'Add new feature'`).
 4. Push to your branch (`git push origin my-new-feature`).
-5. Create a Pull Request on GitHub.
-
-We welcome issues, PRs, and general feedback!
-
----
-
-## License
+5. Open a Pull Request on GitHub.
 
-This project is available as open source under the terms of the [MIT License](./LICENSE.txt). Feel free to use it in your own projects.
+Feedback, issues, and PRs are always welcome!

data/lib/tasks/to_llm.rake

@@ -1,39 +1,29 @@
+# lib/tasks/to_llm.rake
 # frozen_string_literal: true
 
-#
-# rails to_llm:extract -ALL
-# rails to_llm:extract -MODELS
-# rails to_llm:extract -CONTROLLERS
-# rails to_llm:extract -VIEWS
-# rails to_llm:extract -CONFIG
-# rails to_llm:extract -SCHEMA
-#
-
-
 namespace :to_llm do
-  desc "Extract code for LLM usage. Usage: rails to_llm:extract [ALL|MODELS|CONTROLLERS|VIEWS|CONFIG|SCHEMA]"
-  task :extract, [:type] => :environment do |_t, args|
-    # Default to "ALL" if no argument
-    extract_type = args[:type]&.upcase || "ALL"
-    
-    puts "----- to_llm:extract started with type=#{extract_type} -----"
+  def run_extraction(extract_type, output_format)
+    extract_type  = extract_type&.upcase || "ALL"
+    output_format = output_format&.downcase || "txt"
+
+    puts ">> to_llm:extract started with type=#{extract_type}, format=#{output_format}"
 
     # Directory where extracted files will be stored
     output_dir = "to_llm"
     FileUtils.mkdir_p(output_dir)
 
-    # Define which directories to process
+    
     directories_to_files = {
-      "app/models"      => "models.txt",
-      "app/controllers" => "controllers.txt",
-      "app/views"       => "views.txt",
-      "app/helpers"     => "helpers.txt",
-      "app/javascript"  => "javascript.txt",
-      "config"          => "config.txt",
-      "db/schema.rb"    => "schema.txt"
+      "app/models"      => "models",
+      "app/controllers" => "controllers",
+      "app/views"       => "views",
+      "app/helpers"     => "helpers",
+      "app/javascript"  => "javascript",
+      "config"          => "config",
+      "db/schema.rb"    => "schema"
     }
 
-    # We can filter which directories to process based on the user input
+    
     filter_map = {
       "ALL"         => %w[app/models app/controllers app/views app/helpers app/javascript config db/schema.rb],
       "MODELS"      => %w[app/models],
@@ -41,50 +31,120 @@ namespace :to_llm do
       "VIEWS"       => %w[app/views],
       "CONFIG"      => %w[config],
       "SCHEMA"      => %w[db/schema.rb],
-      "JAVASCRIPT" => %w[app/javascript],
-      "HELPERS" => %w[app/helpers]
+      "JAVASCRIPT"  => %w[app/javascript],
+      "HELPERS"     => %w[app/helpers]
     }
 
-    # Define which extensions you want to extract
+    # File extensions to extract
     file_extensions = %w[.rb .erb .js .yml .ts .tsx]
 
-    directories_to_files.values.each do |filename|
-      file_path = File.join(output_dir, filename)
-      File.delete(file_path) if File.exist?(file_path)
+    # Clean old outputs
+    directories_to_files.values.each do |base_filename|
+      path = File.join(output_dir, "#{base_filename}.#{output_format}")
+      File.delete(path) if File.exist?(path)
     end
 
-    def extract_files(dir_path, file_extensions, output_file, output_dir)
-      Dir.glob(File.join(dir_path, "**", "*")) do |path|
-        next if File.directory?(path)
-        next unless file_extensions.include?(File.extname(path))
-
-        File.open(File.join(output_dir, output_file), "a") do |f|
-          f.puts "#{path}:"
-          f.puts "----------------------------------------------------"
-          f.puts File.read(path)
-          f.puts "\n"
-        end
+    # Detects a language for syntax highlighting if output_format = md
+    def detect_language(ext)
+      case ext
+      when ".rb"  then "ruby"
+      when ".erb" then "erb"
+      when ".js"  then "javascript"
+      when ".ts", ".tsx" then "typescript"
+      when ".yml" then "yaml"
+      else "plaintext"
       end
     end
 
-    (filter_map[extract_type] || []).each do |dir_or_file|
-      next unless directories_to_files.key?(dir_or_file)
-      output_name = directories_to_files[dir_or_file]
+    # Writes file content in either txt or md format
+    def write_content(path, content, output_format, output_file)
+      if output_format == "md"
+        language = detect_language(File.extname(path))
+        <<~MARKDOWN
+          ## #{path}
+          ```#{language}
+          #{content}
+          ```
+        MARKDOWN
+      else
+        # txt format
+        <<~TXT
+          #{path}:
+          ----------------------------------------------------
+          #{content}
+
+        TXT
+      end
+    end
 
+    # Method to handle directories or single-file extraction
+    def extract_content(dir_or_file, file_extensions, output_format, base_output_name, output_dir)
       if File.file?(dir_or_file)
-        if File.exist?(dir_or_file)
-          File.open(File.join(output_dir, output_name), "a") do |f|
-            f.puts "#{dir_or_file}:"
-            f.puts "----------------------------------------------------"
-            f.puts File.read(dir_or_file)
-            f.puts "\n"
-          end
+        
+        return unless File.exist?(dir_or_file)
+
+        content = File.read(dir_or_file)
+        formatted = write_content(dir_or_file, content, output_format, base_output_name)
+        File.open(File.join(output_dir, "#{base_output_name}.#{output_format}"), "a") do |f|
+          f.puts formatted
         end
       else
-        extract_files(dir_or_file, file_extensions, output_name, output_dir)
+        
+        Dir.glob(File.join(dir_or_file, "**", "*")) do |path|
+          next if File.directory?(path)
+          next unless file_extensions.include?(File.extname(path))
+
+          content = File.read(path)
+          formatted = write_content(path, content, output_format, base_output_name)
+          File.open(File.join(output_dir, "#{base_output_name}.#{output_format}"), "a") do |f|
+            f.puts formatted
+          end
+        end
       end
     end
 
-    puts "----- to_llm:extract finished! Output is located in '#{output_dir}' -----"
+    
+    (filter_map[extract_type] || []).each do |dir_or_file|
+      next unless directories_to_files.key?(dir_or_file)
+
+      base_output_name = directories_to_files[dir_or_file]
+      extract_content(dir_or_file, file_extensions, output_format, base_output_name, output_dir)
+    end
+
+    puts ">> to_llm:extract finished! See '#{output_dir}' for .#{output_format} files."
+  end
+
+  
+  desc "Extract code into txt or md. Usage: rails \"to_llm:extract[TYPE,FORMAT]\"\n" \
+       "Examples:\n" \
+       "  rails \"to_llm:extract[ALL,md]\"\n" \
+       "  rails \"to_llm:extract[MODELS,txt]\""
+  task :extract, [:type_and_format] => :environment do |_t, args|
+    if args[:type_and_format].blank?
+      # No parameters passed -> show help message
+      puts "--------------------------------------------------------------------"
+      puts "Usage: rails \"to_llm:extract[TYPE,FORMAT]\""
+      puts "Types: ALL, MODELS, CONTROLLERS, VIEWS, CONFIG, SCHEMA, JAVASCRIPT, HELPERS"
+      puts "Format: txt or md (Markdown)"
+      puts "Examples:"
+      puts "  rails \"to_llm:extract[ALL,md]\""
+      puts "  rails \"to_llm:extract[MODELS,txt]\""
+      puts "--------------------------------------------------------------------"
+      next
+    end
+
+    # Attempt to parse "TYPE,FORMAT"
+    type, format = args[:type_and_format].split(",")
+
+    if format.nil?
+      # Old usage
+      puts "You are using the old usage: rails to_llm:extract #{type}"
+      puts "Please use: rails \"to_llm:extract[#{type},md]\" or rails \"to_llm:extract[#{type},txt]\"."
+      
+      run_extraction(type, "txt")
+    else
+      # New usage
+      run_extraction(type, format)
+    end
   end
 end

data/lib/to_llm/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ToLLM
-  VERSION = "0.1.0"
+  VERSION = "0.1.3"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: to_llm
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.3
 platform: ruby
 authors:
 - José Carlos Maciel
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-12-31 00:00:00.000000000 Z
+date: 2025-01-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -25,7 +25,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '5.0'
 description: A simple gem that provides a Rails command or Rake tasks to export .rb,
-  .erb, .js, and .yml files into .txt for LLM ingestion.
+  .erb, .js, and .yml files into .txt or .md for LLM ingestion.
 email:
 - jcmacielp@gmail.com
 executables: []