grainery 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ab9b7d46f3a3716cf0f043060f4f267a9be926ffae12cdb477dd85df016529e3
+  data.tar.gz: 44edc342af67c5fce8f77559830eada34772b18f5ac226c76886d1fe65a55fa0
+SHA512:
+  metadata.gz: 5eea1be971b0ee1b02619377c1b2ec44fc68e8ea302653164a754d04493d31bbe914906391061ca2eaa773794890f6b4a49239c15aefbdc0c8386daa20bef518
+  data.tar.gz: 40258587446259c0e5770b61b1d7282fb7750d64b14aed6983cee287338e4d65338a7b363455c292f1fe56cfcc7f9dd93e71af12f1618a506c7dec71ef5fa35e

data/CHANGELOG.md

@@ -0,0 +1,33 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2025-10-01
+
+### Added
+- Initial release
+- Automatic database detection and configuration generation
+- Multi-database support (SQL Server, MySQL, PostgreSQL)
+- Dependency-aware seed loading with topological sort
+- One seed file per table organization
+- Configurable per-project via `config/grainery.yml`
+- Lookup tables support (harvest all records)
+- Test database management tasks
+- Preserves custom `db/seeds.rb` (loaded last)
+- Clean separation of harvested vs custom seeds
+- Rake tasks:
+  - `grainery:init_config` - Initialize configuration
+  - `grainery:generate` - Harvest with limit (100 records per table)
+  - `grainery:generate_all` - Harvest ALL records
+  - `grainery:load` - Load seeds in dependency order
+  - `grainery:clean` - Clean grainery directory
+  - `test:db:setup_for_grainery` - Setup clean test database
+  - `test:db:seed_with_grainery` - Seed test database
+  - `test:db:reset_with_grainery` - Reset and seed test database
+  - `test:db:clean` - Truncate all test tables
+  - `test:db:stats` - Show test database statistics
+
+[0.1.0]: https://github.com/mpantel/grainery/releases/tag/v0.1.0

data/MIT-LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Michail Pantelelis
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,313 @@
+# Grainery
+
+Database seed storage system for Rails applications. Extract database records and generate seed files organized by database with automatic dependency resolution. Like a grainery stores grain, this gem stores and organizes your database seeds.
+
+## Features
+
+- ✅ Automatic database detection
+- ✅ Dependency-aware loading (topological sort)
+- ✅ Multi-database support
+- ✅ Configurable per project
+- ✅ Preserves custom seeds
+- ✅ One seed file per table
+- ✅ Clean separation of concerns
+- ✅ Supports SQL Server, MySQL, PostgreSQL
+- ✅ Test database management tasks
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'grainery', path: 'grainery'
+```
+
+And then execute:
+
+```bash
+bundle install
+```
+
+## Usage
+
+### 1. Initialize Configuration
+
+```bash
+rake grainery:init_config
+```
+
+This auto-detects all databases and creates `config/grainery.yml`.
+
+### 2. Harvest Data
+
+```bash
+# Harvest with limit (100 records per table)
+rake grainery:generate
+
+# Harvest ALL records (use with caution)
+rake grainery:generate_all
+```
+
+### 3. Load Seeds
+
+```bash
+rake grainery:load
+```
+
+This loads:
+1. Harvested seeds (in dependency order)
+2. Custom seeds from `db/seeds.rb` (last)
+
+## Directory Structure
+
+```
+db/
+├── grainery/                   # Harvested seeds (auto-generated)
+│   ├── load_order.txt         # Load order respecting dependencies
+│   ├── primary/               # Primary database
+│   │   ├── users.rb
+│   │   ├── posts.rb
+│   │   └── comments.rb
+│   ├── other/                 # Other database
+│   │   └── projects.rb
+│   └── banking/               # Banking database
+│       └── employees.rb
+└── seeds.rb                   # Custom seeds (loaded last)
+```
+
+## Configuration
+
+`config/grainery.yml`:
+
+```yaml
+# Path for harvested seed files
+grainery_path: db/grainery
+
+# Database connection mappings
+database_connections:
+  primary:
+    connection: test
+    adapter: sqlserver
+    model_base_class: ApplicationRecord
+  other:
+    connection: other
+    adapter: sqlserver
+    model_base_class: OtherDB
+  # ... other databases
+
+# Lookup tables (harvest all records)
+lookup_tables: []
+```
+
+## Available Rake Tasks
+
+### Grainery Tasks
+
+```bash
+# Initialize configuration
+rake grainery:init_config
+
+# Harvest data (with limit)
+rake grainery:generate
+
+# Harvest ALL records
+rake grainery:generate_all
+
+# Load harvested + custom seeds
+rake grainery:load
+
+# Clean grainery directory
+rake grainery:clean
+```
+
+### Test Database Tasks
+
+```bash
+# Setup clean test database (schema only)
+rake test:db:setup_for_grainery
+# or: rake db:test:setup_for_grainery
+
+# Seed test database with grainery data
+rake test:db:seed_with_grainery
+
+# Reset and seed (one command)
+rake test:db:reset_with_grainery
+# or: rake db:test:reset_with_grainery
+
+# Clean test database (truncate all tables)
+rake test:db:clean
+# or: rake db:test:clean
+
+# Show test database statistics
+rake test:db:stats
+# or: rake db:test:stats
+```
+
+## Dependency Resolution
+
+Grainer automatically:
+1. Analyzes `belongs_to` associations
+2. Builds dependency graph
+3. Performs topological sort
+4. Generates `load_order.txt`
+
+### Example Load Order
+
+```
+# PRIMARY Database
+primary/users.rb
+primary/categories.rb
+primary/posts.rb
+primary/comments.rb
+
+# OTHER Database
+other/departments.rb
+other/projects.rb
+```
+
+## Lookup Tables
+
+For small reference tables (statuses, types, categories), grainer can load **all records** instead of samples.
+
+Add to `config/grainery.yml`:
+```yaml
+lookup_tables:
+  - invoice_statuses
+  - user_roles
+  - categories
+```
+
+## Seed File Format
+
+Each table gets its own seed file:
+
+```ruby
+# Harvested from primary database: users
+# Records: 100
+# Generated: 2025-10-01 10:30:00
+
+User.create!(
+  {
+    email: "user1@example.com",
+    name: "John Doe",
+    active: true
+  },
+  {
+    email: "user2@example.com",
+    name: "Jane Smith",
+    active: true
+  }
+)
+```
+
+## Custom Seeds
+
+Your custom seed logic in `db/seeds.rb` is **preserved and loaded last**.
+
+Example `db/seeds.rb`:
+```ruby
+# Custom seed logic
+puts "Creating admin user..."
+User.find_or_create_by!(email: 'admin@example.com') do |user|
+  user.name = 'Admin'
+  user.role = 'admin'
+end
+
+puts "Setting up application defaults..."
+Setting.create!(key: 'app_name', value: 'My App')
+```
+
+## Use Cases
+
+### Development
+```bash
+# Harvest production-like data for development
+rake grainery:generate
+rake grainery:load
+```
+
+### Testing
+```bash
+# Create test fixtures
+rake grainery:generate
+# In test setup, load specific seeds as needed
+```
+
+### Staging
+```bash
+# Harvest production data (anonymized)
+rake grainery:generate_all
+# Deploy to staging
+# Load on staging server
+rake grainery:load
+```
+
+## Safety Features
+
+1. **Separate Directories**: Harvested seeds never touch `db/seeds.rb`
+2. **Dependency Order**: Foreign keys respected automatically
+3. **Custom Preservation**: Your `db/seeds.rb` always loads last
+4. **Clean Command**: `rake grainery:clean` removes only harvested files
+
+## Best Practices
+
+1. **Use Limits**: Start with `rake grainery:generate` (100 records)
+2. **Review Load Order**: Check `db/grainery/load_order.txt`
+3. **Test Loading**: Run `rake grainery:load` on clean database first
+4. **Commit Selectively**: Consider `.gitignore` for large grainery files
+5. **Custom Seeds Last**: Keep application-specific logic in `db/seeds.rb`
+
+## Troubleshooting
+
+### Circular Dependencies
+If you see "Circular dependency detected", check for:
+- Self-referential associations
+- Circular foreign keys
+
+Solution: Temporarily remove `optional: true` or `foreign_key: false`
+
+### Missing Records
+If records fail to load:
+1. Check `load_order.txt` for correct ordering
+2. Verify foreign key constraints
+3. Review error messages in console output
+
+### Large Files
+If seed files are too large:
+```bash
+# Use limit parameter
+rake grainery:generate  # 100 records per table (default)
+```
+
+## Example Workflow
+
+```bash
+# 1. Initialize on first use
+rake grainery:init_config
+
+# 2. Harvest from production (with VPN/SSH tunnel)
+RAILS_ENV=production rake grainery:generate
+
+# 3. Review generated files
+ls -la db/grainery/
+
+# 4. Commit grainery files (optional)
+git add db/grainery/
+git commit -m "Add production seed data"
+
+# 5. On another machine, pull and load
+git pull
+rake db:reset
+rake grainery:load
+
+# 6. Your custom seeds run automatically last
+# db/seeds.rb is executed after all harvested seeds
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/mpantel/grainery.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/grainery/grainer.rb

@@ -0,0 +1,519 @@
+require 'fileutils'
+require 'yaml'
+
+module Grainery
+  class Grainer
+    attr_reader :database_configs, :lookup_tables
+
+    # Security: Whitelist of allowed base class patterns
+    ALLOWED_BASE_CLASS_PATTERN = /\A(ApplicationRecord|ActiveRecord::Base|[A-Z][a-zA-Z0-9]*Record|[A-Z][a-zA-Z0-9]*(DB|Database|Connection))\z/
+
+    def initialize
+      @config = load_config
+      @grainery_path = load_grainery_path
+      @database_configs = load_database_connections
+      @lookup_tables = load_lookup_tables
+    end
+
+    def load_config
+      config_path = Rails.root.join('config/grainery.yml')
+
+      unless File.exist?(config_path)
+        puts "  Warning: config/grainery.yml not found. Creating default configuration..."
+        create_default_config
+      end
+
+      YAML.safe_load_file(config_path, permitted_classes: [Symbol, Date, Time], aliases: true) || {}
+    rescue => e
+      puts "  Warning: Could not load grainery.yml: #{e.message}"
+      {}
+    end
+
+    def create_default_config
+      config_path = Rails.root.join('config/grainery.yml')
+
+      # Detect databases and model base classes dynamically
+      detected_databases = detect_databases_and_models
+
+      # Build configuration hash
+      config = {
+        'database_connections' => detected_databases,
+        'grainery_path' => 'db/grainery',
+        'lookup_tables' => [],
+        'last_updated' => Time.now.to_s
+      }
+
+      # Write with custom formatting for better readability
+      write_config_file(config_path, config)
+
+      puts "  ✓ Created config/grainery.yml with #{detected_databases.size} detected databases"
+    end
+
+    def detect_databases_and_models
+      puts "  → Detecting databases and model base classes..."
+
+      Rails.application.eager_load!
+
+      # Find all model base classes
+      base_classes = ObjectSpace.each_object(Class).select do |klass|
+        klass < ActiveRecord::Base &&
+        !klass.abstract_class? &&
+        klass != ActiveRecord::Base &&
+        klass.descendants.any?
+      rescue
+        false
+      end
+
+      connections_map = {}
+
+      base_classes.each do |base_class|
+        begin
+          connection_config = base_class.connection_db_config
+          connection_name = connection_config.name.to_s
+          adapter = connection_config.adapter.to_s
+
+          logical_name = infer_logical_name(connection_name, base_class.name)
+
+          connections_map[logical_name] = {
+            'connection' => connection_name,
+            'adapter' => adapter,
+            'model_base_class' => base_class.name
+          }
+
+          puts "    ✓ Detected: #{logical_name} → #{base_class.name} (#{adapter})"
+        rescue => e
+          puts "    ⚠ Warning: Could not detect connection for #{base_class.name}: #{e.message}"
+        end
+      end
+
+      # Ensure primary database is included
+      unless connections_map.key?('primary')
+        begin
+          primary_config = ApplicationRecord.connection_db_config
+          connections_map['primary'] = {
+            'connection' => primary_config.name.to_s,
+            'adapter' => primary_config.adapter.to_s,
+            'model_base_class' => 'ApplicationRecord'
+          }
+          puts "    ✓ Detected: primary → ApplicationRecord (#{primary_config.adapter})"
+        rescue => e
+          puts "    ⚠ Warning: Could not detect primary database: #{e.message}"
+        end
+      end
+
+      connections_map
+    end
+
+    def infer_logical_name(connection_name, class_name)
+      logical = class_name.gsub(/DB$|Database$|Connection$/, '')
+      logical = logical.gsub(/([A-Z]+)([A-Z][a-z])/, '\\1_\\2')
+                       .gsub(/([a-z\d])([A-Z])/, '\\1_\\2')
+                       .downcase
+
+      if connection_name != 'test' && connection_name.length > logical.length
+        logical = connection_name.gsub(/_db$|_database$/, '')
+      end
+
+      logical
+    end
+
+    def write_config_file(path, config)
+      content = []
+
+      content << "# Data Harvest Configuration"
+      content << "# This file contains configuration for the data harvesting system"
+      content << "#"
+      content << "# Database Connections:"
+      content << "# Map of logical database names to connection name, adapter, and model base class"
+      content << "#"
+      content << "# Harvest Path:"
+      content << "# Where harvested seed files are stored (default: db/grainery)"
+      content << "#"
+      content << "# Lookup Tables:"
+      content << "# Tables to harvest ALL records (not just samples)"
+      content << ""
+      content << "# Path for harvested seed files"
+      content << "grainery_path: #{config['grainery_path'] || 'db/grainery'}"
+      content << ""
+      content << "# Database connection mappings"
+      content << "database_connections:"
+
+      config['database_connections'].each do |db_name, db_config|
+        content << "  #{db_name}:"
+        content << "    connection: #{db_config['connection']}"
+        content << "    adapter: #{db_config['adapter']}"
+        content << "    model_base_class: #{db_config['model_base_class']}"
+      end
+
+      content << ""
+      content << "# Lookup tables (harvest all records)"
+      content << "lookup_tables: #{config['lookup_tables'].inspect}"
+      content << ""
+      content << "# Metadata"
+      content << "last_updated: #{config['last_updated']}"
+      content << ""
+
+      File.write(path, content.join("\n"))
+    end
+
+    def load_database_connections
+      connections = @config['database_connections'] || {}
+      result = {}
+
+      connections.each do |db_name, config|
+        db_key = db_name.to_sym
+        if config.is_a?(Hash)
+          result[db_key] = {
+            connection: config['connection'].to_sym,
+            adapter: config['adapter']&.to_sym || :sqlserver,
+            model_base_class: config['model_base_class'] || 'ApplicationRecord'
+          }
+        else
+          result[db_key] = {
+            connection: config.to_sym,
+            adapter: :sqlserver,
+            model_base_class: 'ApplicationRecord'
+          }
+        end
+      end
+      result
+    end
+
+    def load_grainery_path
+      path = @config['grainery_path'] || 'db/grainery'
+      path = path.sub(/\/$/, '')
+
+      # Security: Validate path is within Rails.root to prevent path traversal
+      full_path = Rails.root.join(path).expand_path
+      unless full_path.to_s.start_with?(Rails.root.to_s)
+        raise SecurityError, "Invalid grainery_path '#{path}': must be within Rails application directory"
+      end
+
+      path
+    rescue SecurityError => e
+      puts "  Error: #{e.message}"
+      raise
+    rescue => e
+      puts "  Warning: Could not load grainery_path: #{e.message}"
+      'db/grainery'
+    end
+
+    def load_lookup_tables
+      (@config['lookup_tables'] || []).to_set
+    rescue => e
+      puts "  Warning: Could not load lookup tables: #{e.message}"
+      Set.new
+    end
+
+    # Security: Safe constant resolution with whitelist
+    def safe_const_get(class_name)
+      unless class_name.match?(ALLOWED_BASE_CLASS_PATTERN)
+        raise SecurityError, "Unauthorized base class '#{class_name}'. Only ActiveRecord model base classes are allowed."
+      end
+
+      Object.const_get(class_name)
+    rescue NameError => e
+      raise NameError, "Could not find constant '#{class_name}': #{e.message}"
+    end
+
+    def get_all_models
+      Rails.application.eager_load!
+      models = []
+
+      @database_configs.each do |db_name, db_config|
+        base_class_name = db_config[:model_base_class]
+        next unless base_class_name
+
+        begin
+          base_class = safe_const_get(base_class_name)
+          models += base_class.descendants
+        rescue => e
+          puts "  Warning: Could not load models from '#{base_class_name}': #{e.message}"
+        end
+      end
+
+      models.uniq.compact
+    end
+
+    def detect_database(model)
+      @database_configs.each do |db_name, db_config|
+        base_class_name = db_config[:model_base_class]
+        next unless base_class_name
+
+        begin
+          base_class = safe_const_get(base_class_name)
+          return db_name if model < base_class
+        rescue
+          next
+        end
+      end
+
+      :primary
+    end
+
+    def harvest_all(limit: nil)
+      all_models = get_all_models
+
+      models_to_harvest = all_models.reject do |model|
+        model.abstract_class? ||
+        model.name.start_with?('HABTM_', 'ActiveRecord::') ||
+        model.table_name.nil?
+      end
+
+      harvest_models(models_to_harvest, limit: limit)
+    end
+
+    def harvest_models(models, limit: nil)
+      models = Array(models)
+      return if models.empty?
+
+      puts "\n" + "="*80
+      puts "Grainer - Extracting Database Seeds"
+      puts "="*80
+      puts "Total models: #{models.size}"
+      puts "Limit per table: #{limit || 'ALL RECORDS'}"
+      puts "="*80 + "\n"
+
+      # Group by database
+      grouped_models = models.group_by { |model| detect_database(model) }
+
+      # Calculate dependencies for load order
+      dependency_graph = build_dependency_graph(models)
+      load_order = topological_sort(dependency_graph)
+
+      # Create harvest directories
+      grouped_models.each do |db_name, _|
+        db_dir = Rails.root.join(@grainery_path, db_name.to_s)
+        FileUtils.mkdir_p(db_dir)
+      end
+
+      # Harvest in dependency order
+      load_order.each do |model|
+        next unless models.include?(model)
+
+        begin
+          db_name = detect_database(model)
+          harvest_table(model, db_name, limit: limit)
+        rescue => e
+          puts "  ✗ Error harvesting #{model.name}: #{e.message}"
+        end
+      end
+
+      # Create load order file
+      create_load_order_file(load_order, models)
+
+      puts "\n" + "="*80
+      puts "Data harvest complete!"
+      puts "Seed files created in #{@grainery_path}/"
+      puts "Load with: rake grainery:load"
+      puts "="*80
+    end
+
+    def build_dependency_graph(models)
+      graph = {}
+
+      models.each do |model|
+        graph[model] = []
+
+        # Find belongs_to associations (dependencies)
+        model.reflect_on_all_associations(:belongs_to).each do |assoc|
+          begin
+            if assoc.klass && !assoc.polymorphic? && models.include?(assoc.klass)
+              graph[model] << assoc.klass
+            end
+          rescue
+            next
+          end
+        end
+      end
+
+      graph
+    end
+
+    def topological_sort(graph)
+      sorted = []
+      visited = Set.new
+      visiting = Set.new
+
+      visit = lambda do |node|
+        return if visited.include?(node)
+        raise "Circular dependency detected" if visiting.include?(node)
+
+        visiting.add(node)
+
+        (graph[node] || []).each do |dependency|
+          visit.call(dependency) if graph.key?(dependency)
+        end
+
+        visiting.delete(node)
+        visited.add(node)
+        sorted << node
+      end
+
+      graph.keys.each { |node| visit.call(node) }
+
+      sorted
+    end
+
+    def harvest_table(model, db_name, limit: nil)
+      table_name = model.table_name
+      is_lookup = @lookup_tables.include?(table_name)
+
+      # Determine how many records to harvest
+      records = if is_lookup
+        model.all.to_a
+      elsif limit
+        model.limit(limit).to_a
+      else
+        model.all.to_a
+      end
+
+      if records.empty?
+        puts "  ⚠ #{model.name.ljust(50)} → skipped (no data)"
+        return
+      end
+
+      # Generate seed file
+      seed_content = generate_seed_content(model, records, db_name)
+      seed_path = get_seed_path(model, db_name)
+
+      File.write(seed_path, seed_content)
+
+      record_info = is_lookup ? " (lookup: #{records.size} records)" : " (#{records.size} records)"
+      puts "  ✓ #{model.name.ljust(50)} → #{table_name}.rb#{record_info}"
+    end
+
+    def get_seed_path(model, db_name)
+      db_dir = File.join(@grainery_path, db_name.to_s)
+      FileUtils.mkdir_p(Rails.root.join(db_dir))
+      Rails.root.join(db_dir, "#{model.table_name}.rb")
+    end
+
+    def generate_seed_content(model, records, db_name)
+      table_name = model.table_name
+
+      # Get columns to export (exclude id, timestamps)
+      columns = model.columns.reject do |col|
+        %w[id created_at updated_at].include?(col.name)
+      end
+
+      content = []
+      content << "# Harvested from #{db_name} database: #{table_name}"
+      content << "# Records: #{records.size}"
+      content << "# Generated: #{Time.now}"
+      content << ""
+      content << "#{model.name}.create!("
+
+      records.each_with_index do |record, idx|
+        content << "  {" if idx == 0
+        content << "  }," if idx > 0
+        content << "  {" if idx > 0
+
+        columns.each_with_index do |col, col_idx|
+          value = record.send(col.name)
+          formatted_value = format_seed_value(value, col)
+          comma = col_idx < columns.size - 1 ? ',' : ''
+          content << "    #{col.name}: #{formatted_value}#{comma}"
+        end
+      end
+
+      content << "  }"
+      content << ")"
+      content << ""
+
+      content.join("\n")
+    end
+
+    def format_seed_value(value, column)
+      return 'nil' if value.nil?
+
+      case column.type
+      when :string, :text
+        value.to_s.inspect
+      when :integer, :bigint
+        value.to_i
+      when :decimal, :float
+        value.to_f
+      when :boolean
+        value ? 'true' : 'false'
+      when :date
+        "Date.parse(#{value.to_s.inspect})"
+      when :datetime, :timestamp
+        "Time.parse(#{value.to_s.inspect})"
+      when :json, :jsonb
+        value.to_json
+      else
+        value.inspect
+      end
+    end
+
+    def create_load_order_file(load_order, models)
+      order_path = Rails.root.join(@grainery_path, 'load_order.txt')
+
+      grouped_by_db = load_order.select { |m| models.include?(m) }.group_by { |m| detect_database(m) }
+
+      content = []
+      content << "# Load order for harvested seeds"
+      content << "# Load files in this order to respect foreign key dependencies"
+      content << ""
+
+      grouped_by_db.each do |db_name, db_models|
+        content << "# #{db_name.to_s.upcase} Database"
+        db_models.each do |model|
+          content << "#{db_name}/#{model.table_name}.rb"
+        end
+        content << ""
+      end
+
+      File.write(order_path, content.join("\n"))
+      puts "\n  ✓ Load order written to #{@grainery_path}/load_order.txt"
+    end
+
+    def load_seeds
+      order_file = Rails.root.join(@grainery_path, 'load_order.txt')
+
+      unless File.exist?(order_file)
+        puts "  ✗ Load order file not found. Run 'rake grainery:generate' first."
+        return
+      end
+
+      puts "\n" + "="*80
+      puts "Loading Harvested Seeds"
+      puts "="*80
+
+      # Load harvested seeds in dependency order
+      File.readlines(order_file).each do |line|
+        line = line.strip
+        next if line.empty? || line.start_with?('#')
+
+        seed_file = Rails.root.join(@grainery_path, line)
+        if File.exist?(seed_file)
+          puts "  → Loading #{line}..."
+          begin
+            load seed_file
+          rescue => e
+            puts "  ✗ Error loading #{line}: #{e.message}"
+          end
+        end
+      end
+
+      # Load custom seeds last (if they exist)
+      custom_seeds = Rails.root.join('db/seeds.rb')
+      if File.exist?(custom_seeds) && File.read(custom_seeds).strip.present?
+        puts "\n" + "-"*80
+        puts "Loading Custom Seeds"
+        puts "-"*80
+        puts "  → Loading db/seeds.rb..."
+        begin
+          load custom_seeds
+        rescue => e
+          puts "  ✗ Error loading custom seeds: #{e.message}"
+        end
+      end
+
+      puts "\n" + "="*80
+      puts "Seed loading complete!"
+      puts "="*80
+    end
+  end
+end

data/lib/grainery/railtie.rb

@@ -0,0 +1,12 @@
+require 'grainery/grainer'
+
+module Grainery
+  class Railtie < Rails::Railtie
+    railtie_name :grainery
+
+    rake_tasks do
+      load 'tasks/grainery_tasks.rake'
+      load 'tasks/test_db_tasks.rake'
+    end
+  end
+end

data/lib/grainery/version.rb

@@ -0,0 +1,3 @@
+module Grainery
+  VERSION = "0.1.0"
+end

data/lib/grainery.rb

@@ -0,0 +1,8 @@
+require "grainery/version"
+require "grainery/railtie" if defined?(Rails)
+
+module Grainery
+  class Error < StandardError; end
+
+  # Your code goes here...
+end

data/lib/tasks/grainery_tasks.rake

@@ -0,0 +1,59 @@
+namespace :grainery do
+  desc "Initialize grainery.yml configuration file"
+  task init_config: :environment do
+    config_path = Rails.root.join('config/grainery.yml')
+
+    if File.exist?(config_path)
+      puts "Configuration file already exists at config/grainery.yml"
+      exit
+    end
+
+    puts "Initializing grainery configuration..."
+    grainery = Grainery::Grainer.new
+
+    puts "\n✓ Configuration file created successfully!"
+    puts "\nNext steps:"
+    puts "1. Review config/grainery.yml"
+    puts "2. Run 'rake grainery:generate' to harvest data"
+  end
+
+  desc "Harvest data from all tables (respecting dependencies)"
+  task generate: :environment do
+    puts "Harvesting data from all tables..."
+
+    grainery = Grainery::Grainer.new
+    grainery.harvest_all(limit: 100) # Default limit of 100 records per table
+
+    puts "\n✓ Data harvested to db/grainery/"
+  end
+
+  desc "Harvest all records from all tables (no limit)"
+  task generate_all: :environment do
+    puts "⚠  WARNING: This will harvest ALL records from ALL tables"
+    puts "   This may take a while and create large files\n\n"
+
+    grainery = Grainery::Grainer.new
+    grainery.harvest_all(limit: nil)
+
+    puts "\n✓ All data harvested to db/grainery/"
+  end
+
+  desc "Load harvested seeds into database (in dependency order)"
+  task load: :environment do
+    grainery = Grainery::Grainer.new
+    grainery.load_seeds
+  end
+
+  desc "Clean grainery directory"
+  task clean: :environment do
+    grainery = Grainery::Grainer.new
+    grainery_path = Rails.root.join(grainery.instance_variable_get(:@grainery_path))
+
+    if Dir.exist?(grainery_path)
+      FileUtils.rm_rf(grainery_path)
+      puts "✓ Cleaned #{grainery_path}"
+    else
+      puts "  Nothing to clean"
+    end
+  end
+end

data/lib/tasks/test_db_tasks.rake

@@ -0,0 +1,174 @@
+namespace :test do
+  namespace :db do
+    desc "Setup clean test database for grainery-based testing"
+    task setup_for_grainery: :environment do
+      puts "Setting up clean test database for grainery-based testing..."
+
+      # Switch to test environment
+      Rails.env = 'test'
+      ActiveRecord::Base.establish_connection(:test)
+
+      begin
+        # Drop existing test database (optional - remove if you want to preserve structure)
+        puts "  → Dropping test database..."
+        ActiveRecord::Tasks::DatabaseTasks.drop_current('test')
+      rescue => e
+        puts "    ⚠ Could not drop database: #{e.message}"
+      end
+
+      # Create test database
+      puts "  → Creating test database..."
+      ActiveRecord::Tasks::DatabaseTasks.create_current('test')
+
+      # Load schema
+      puts "  → Loading schema..."
+      ActiveRecord::Base.establish_connection(:test)
+      ActiveRecord::Tasks::DatabaseTasks.load_schema_for 'test', :ruby
+
+      # Reconnect
+      ActiveRecord::Base.establish_connection(:test)
+
+      puts "\n✓ Test database is ready for grainery-based testing!"
+      puts "\nNext steps:"
+      puts "  1. Load seeds: RAILS_ENV=test rake grainery:load"
+      puts "  2. Run tests:  bundle exec rspec"
+      puts "\nOr use the combined task:"
+      puts "  rake test:db:reset_with_grainery"
+    end
+
+    desc "Seed test database with grainery data"
+    task seed_with_grainery: :environment do
+      Rails.env = 'test'
+      ActiveRecord::Base.establish_connection(:test)
+
+      puts "Seeding test database with grainery data..."
+
+      begin
+        # Check if grainery seeds exist
+        grainery_path = Rails.root.join('db/grainery')
+        load_order_file = grainery_path.join('load_order.txt')
+
+        unless File.exist?(load_order_file)
+          puts "\n⚠ No grainery seeds found!"
+          puts "\nTo generate seeds:"
+          puts "  1. Harvest from another environment:"
+          puts "     RAILS_ENV=development rake grainery:generate"
+          puts "\n  2. Then load into test:"
+          puts "     RAILS_ENV=test rake grainery:load"
+          puts "\nOr run this task which does both:"
+          puts "  rake test:db:reset_with_grainery"
+          exit 1
+        end
+
+        # Load grainery seeds
+        puts "  → Loading grainery seeds..."
+        Rake::Task['grainery:load'].invoke
+
+        puts "\n✓ Test database seeded successfully with grainery data!"
+      rescue => e
+        puts "Error seeding database: #{e.message}"
+        puts e.backtrace.first(5).join("\n")
+      end
+    end
+
+    desc "Reset test database and seed with grainery"
+    task reset_with_grainery: [:setup_for_grainery, :seed_with_grainery]
+
+    desc "Clean test database (truncate all tables)"
+    task clean: :environment do
+      Rails.env = 'test'
+      ActiveRecord::Base.establish_connection(:test)
+
+      puts "Cleaning test database..."
+
+      # Get all table names
+      tables = ActiveRecord::Base.connection.tables.reject do |table|
+        table == 'schema_migrations' || table == 'ar_internal_metadata'
+      end
+
+      puts "  → Truncating #{tables.size} tables..."
+
+      # Disable foreign key checks temporarily (SQL Server specific)
+      ActiveRecord::Base.connection.execute("EXEC sp_MSforeachtable 'ALTER TABLE ? NOCHECK CONSTRAINT ALL'")
+
+      tables.each do |table|
+        begin
+          quoted_table = ActiveRecord::Base.connection.quote_table_name(table)
+          ActiveRecord::Base.connection.execute("TRUNCATE TABLE #{quoted_table}")
+        rescue => e
+          puts "    ⚠ Could not truncate #{table}: #{e.message}"
+        end
+      end
+
+      # Re-enable foreign key checks
+      ActiveRecord::Base.connection.execute("EXEC sp_MSforeachtable 'ALTER TABLE ? CHECK CONSTRAINT ALL'")
+
+      puts "\n✓ Test database cleaned!"
+    end
+
+    desc "Show test database statistics"
+    task stats: :environment do
+      Rails.env = 'test'
+      ActiveRecord::Base.establish_connection(:test)
+
+      puts "Test Database Statistics"
+      puts "=" * 80
+
+      # Get all tables and their row counts
+      tables = ActiveRecord::Base.connection.tables.reject do |table|
+        table == 'schema_migrations' || table == 'ar_internal_metadata'
+      end
+
+      total_rows = 0
+      table_data = []
+
+      tables.each do |table|
+        begin
+          quoted_table = ActiveRecord::Base.connection.quote_table_name(table)
+          count = ActiveRecord::Base.connection.select_value("SELECT COUNT(*) FROM #{quoted_table}")
+          total_rows += count
+          table_data << [table, count] if count > 0
+        rescue => e
+          # Skip tables we can't query
+        end
+      end
+
+      # Sort by row count descending
+      table_data.sort_by! { |_, count| -count }
+
+      if table_data.empty?
+        puts "\n  Database is empty ✓"
+        puts "\n  Run 'rake test:db:setup_for_grainery' to initialize"
+      else
+        puts "\n  Total tables with data: #{table_data.size}"
+        puts "  Total rows: #{total_rows.to_s.reverse.gsub(/(\d{3})(?=\d)/, '\\1,').reverse}"
+        puts "\n  Top 20 tables by row count:"
+        puts "  " + "-" * 76
+
+        table_data.first(20).each do |table, count|
+          formatted_count = count.to_s.reverse.gsub(/(\d{3})(?=\d)/, '\\1,').reverse
+          puts "  #{table.ljust(50)} #{formatted_count.rjust(10)} rows"
+        end
+      end
+
+      puts "=" * 80
+    end
+  end
+end
+
+# Aliases for convenience
+namespace :db do
+  namespace :test do
+    desc "Alias for test:db:setup_for_grainery"
+    task setup_for_grainery: 'test:db:setup_for_grainery'
+
+    desc "Alias for test:db:clean"
+    task clean: 'test:db:clean'
+
+    desc "Alias for test:db:stats"
+    task stats: 'test:db:stats'
+
+    desc "Alias for test:db:reset_with_grainery"
+    task reset_with_grainery: 'test:db:reset_with_grainery'
+  end
+end

metadata

@@ -0,0 +1,95 @@
+--- !ruby/object:Gem::Specification
+name: grainery
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Michail Pantelelis
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '6.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '6.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.4'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.4'
+description: Extract database records and generate seed files organized by database
+  with automatic dependency resolution. Like a grainery stores grain, this gem stores
+  and organizes your database seeds.
+email:
+- mpantel@aegean.gr
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- MIT-LICENSE
+- README.md
+- lib/grainery.rb
+- lib/grainery/grainer.rb
+- lib/grainery/railtie.rb
+- lib/grainery/version.rb
+- lib/tasks/grainery_tasks.rake
+- lib/tasks/test_db_tasks.rake
+homepage: https://github.com/mpantel/grainery
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/mpantel/grainery
+  changelog_uri: https://github.com/mpantel/grainery/blob/main/CHANGELOG.md
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.7.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.7.1
+specification_version: 4
+summary: Database seed storage system for Rails applications
+test_files: []