bulk_csv_parser 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: dea086882061e780057b9254ba51d007762ee28a59b5dd2f48f7e4c29e425c18
+  data.tar.gz: 120ece99293ee3037523996b65f9a74b2199e147cd8ebfb0cd9f8c9fc08ec113
+SHA512:
+  metadata.gz: b684e7677a8a438d30e93342c51d71ec8c33598317823eb6099de5d1d5b6a6ab478485db0f53c358923afedf0bc94d0cdad2badc1dc4f509929d6889a9654d2d
+  data.tar.gz: c75d50cbf48eec783b677a0e549e3ff667fa2b622c6c72b956a974d41a4c7bf1dddb6766151a8285558ab8e157ed5bd363a5e7464da29f01aadb187b3f49c7b1

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jagrit Bharara
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,213 @@
+# BulkCsvParser
+
+Generic bulk CSV import/update for any ActiveRecord model. You declare:
+
+- **which model** to import into
+- **which attributes** identify an existing record (`find_by`)
+- **which attributes** should be created/updated (`attributes`)
+
+...and the gem handles the rest: streaming large files without loading them
+fully into memory, batching, and (optionally) running the whole import in a
+background job.
+
+## Installation
+
+Add to your Rails app's Gemfile:
+
+```ruby
+gem "bulk_csv_parser", path: "../csv_gem" # or git: "..." once pushed, or from rubygems once published
+```
+
+```bash
+bundle install
+bin/rails generate bulk_csv_parser:install
+```
+
+This creates `config/initializers/bulk_csv_parser.rb` where you can set
+defaults (batch size, strategy, queue name, upload dir).
+
+## Basic usage (synchronous)
+
+```ruby
+result = BulkCsvParser.import(
+  file_path: params[:file].path,
+  model: User,
+  find_by: [:email],
+  attributes: [:first_name, :last_name, :status]
+)
+
+result.success_count # => 980
+result.failed_count  # => 20
+result.errors        # => [{ row: {...}, error: "Email can't be blank" }, ...]
+```
+
+CSV headers are normalized automatically (`"First Name"` -> `:first_name`).
+If your CSV headers don't match your model's attribute names, map them:
+
+```ruby
+BulkCsvParser.import(
+  file_path: "/tmp/users.csv",
+  model: User,
+  find_by: [:email],
+  attributes: [:first_name, :status],
+  header_mapping: { "Email Address" => :email, "Full Name" => :first_name }
+)
+```
+
+Need custom logic per row (parsing dates, computing derived fields, skipping
+rows)? Pass `transform`, which receives the normalized row hash and returns
+the hash to use (or `nil` to skip the row):
+
+```ruby
+BulkCsvParser.import(
+  file_path: "/tmp/users.csv",
+  model: User,
+  find_by: [:email],
+  attributes: [:status, :activated_at],
+  transform: ->(row) {
+    row[:status] = row[:status].to_s.downcase
+    row[:activated_at] = Date.parse(row[:activated_at]) rescue nil
+    row
+  }
+)
+```
+
+## Background processing (large files, and workers on a different machine)
+
+Uploaded tempfiles disappear once the request ends, and in production your
+job queue is very often processed on a **different machine** than the one
+that received the upload (a separate Sidekiq/Resque/GoodJob fleet) with no
+access to that web server's local disk. Passing a bare `file_path` across
+that boundary silently breaks.
+
+`BulkCsvParser.enqueue` solves this via `BulkCsvParser::Storage`: it uploads
+the file to whatever service Active Storage is configured with
+(`config/storage.yml` — S3, GCS, Azure) and passes a small signed reference
+to the job instead of a path. Any worker with the same storage credentials
+can fetch it back, regardless of which machine it runs on:
+
+```ruby
+class ImportsController < ApplicationController
+  def create
+    BulkCsvParser.enqueue(
+      file: params[:file], # anything responding to #path
+      model: User,
+      find_by: [:email],
+      attributes: [:first_name, :last_name, :status],
+      notify_email: current_user.email
+    )
+
+    redirect_to imports_path, notice: "Import queued"
+  end
+end
+```
+
+This requires Active Storage to be set up (`bin/rails active_storage:install`
+and a non-`:local` service configured for any multi-machine deployment — the
+local disk service has the exact same cross-machine problem the file upload
+itself does). `config.storage` defaults to `:active_storage` automatically
+whenever Active Storage is loaded.
+
+If web and worker genuinely share a disk (single-machine deploys, or a
+shared NFS/EFS mount), you can opt into the simpler local-copy adapter:
+
+```ruby
+BulkCsvParser.configure { |c| c.storage = :local } # config/initializers/bulk_csv_parser.rb
+```
+
+You can also enqueue the job directly if you already have a storage
+reference:
+
+```ruby
+BulkCsvParser::ImportJob.perform_later(
+  file_reference: some_active_storage_blob.signed_id,
+  model: "User",
+  find_by: [:email],
+  attributes: [:first_name, :last_name],
+  notify_email: "ops@example.com"
+)
+```
+
+The job runs on whatever ActiveJob adapter your app uses (Sidekiq, Resque,
+GoodJob, etc.) on the `bulk_csv_parser` queue (configurable), and always
+deletes/purges the stored file when it finishes, even on failure.
+
+## Email notification of results
+
+Pass `notify_email:` to `BulkCsvParser.enqueue` or `ImportJob.perform_later`
+and, once the import finishes, `BulkCsvParser::ImportMailer` sends an email
+with the success count, failure count, and the reason each failed row
+failed (capped at `config.notify_error_limit`, default 25, so a file with
+thousands of bad rows doesn't produce a giant email):
+
+```ruby
+BulkCsvParser.enqueue(
+  file: params[:file],
+  model: User,
+  find_by: [:email],
+  attributes: [:first_name, :status],
+  notify_email: "ops@example.com"
+)
+```
+
+Set the sender address in the initializer:
+
+```ruby
+BulkCsvParser.configure { |c| c.mailer_sender = "imports@yourapp.com" }
+```
+
+## Strategies
+
+- `:save` (default) — `find_or_initialize_by` + `assign_attributes` + `save`
+  per row. Runs validations and callbacks. Safer, slower.
+- `:upsert` — batches rows into a single `upsert_all` call per batch. Much
+  faster for very large files, but **skips validations/callbacks** and
+  requires a DB-level unique index covering the `find_by` columns.
+  `created_at` is only set on insert and left alone on conflict, so
+  re-importing an existing record won't clobber when it was originally
+  created — only `updated_at` and the declared `attributes` change on update.
+
+```ruby
+BulkCsvParser.import(
+  file_path: "/tmp/huge_import.csv",
+  model: Product,
+  find_by: [:sku],
+  attributes: [:name, :price, :stock],
+  strategy: :upsert,
+  batch_size: 5000
+)
+```
+
+## Large files
+
+The processor uses `CSV.foreach` to stream row by row and only holds
+`batch_size` rows in memory at a time, so multi-GB files are safe to import.
+Tune `batch_size` (default 1000) to trade off memory usage vs. number of DB
+round-trips.
+
+## Configuration
+
+```ruby
+BulkCsvParser.configure do |config|
+  config.batch_size = 1000
+  config.strategy    = :save
+  config.queue_name  = :bulk_csv_parser
+  config.upload_dir  = Rails.root.join("tmp", "bulk_csv_parser")
+end
+```
+
+## Handling row-level errors
+
+```ruby
+BulkCsvParser.import(
+  file_path: "/tmp/users.csv",
+  model: User,
+  find_by: [:email],
+  attributes: [:status],
+  on_row_error: ->(row, error) { Rails.logger.error("Row failed: #{row} - #{error.message}") }
+)
+```
+
+## License
+
+MIT

data/app/mailers/bulk_csv_parser/import_mailer.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module BulkCsvParser
+  class ImportMailer < ActionMailer::Base
+    default from: -> { BulkCsvParser.configuration.mailer_sender }
+
+    # @param to [String, Array<String>]
+    # @param model [String, Class] model name/class the import ran against
+    # @param result [BulkCsvParser::Result]
+    def result_email(to:, model:, result:)
+      @model = model
+      @result = result
+
+      limit = BulkCsvParser.configuration.notify_error_limit
+      @errors = result.errors.first(limit)
+      @more_errors_count = [result.errors.size - @errors.size, 0].max
+
+      status = result.success? ? "succeeded" : "completed with errors"
+
+      mail(
+        to: to,
+        subject: "CSV import for #{@model} #{status}: " \
+                 "#{result.success_count} succeeded, #{result.failed_count} failed"
+      )
+    end
+  end
+end

data/app/views/bulk_csv_parser/import_mailer/result_email.html.erb

@@ -0,0 +1,33 @@
+<h2>CSV import for <%= @model %></h2>
+
+<ul>
+  <li>Succeeded: <strong><%= @result.success_count %></strong></li>
+  <li>Failed: <strong><%= @result.failed_count %></strong></li>
+  <li>Total: <%= @result.total %></li>
+</ul>
+
+<% if @errors.any? %>
+  <h3>Failed rows</h3>
+  <table border="1" cellpadding="6" cellspacing="0">
+    <thead>
+      <tr>
+        <th>Row</th>
+        <th>Reason</th>
+      </tr>
+    </thead>
+    <tbody>
+      <% @errors.each do |error| %>
+        <tr>
+          <td><%= error[:row].inspect %></td>
+          <td><%= error[:error] %></td>
+        </tr>
+      <% end %>
+    </tbody>
+  </table>
+
+  <% if @more_errors_count.positive? %>
+    <p>...and <%= @more_errors_count %> more failed row(s) not shown.</p>
+  <% end %>
+<% else %>
+  <p>No failed rows.</p>
+<% end %>

data/app/views/bulk_csv_parser/import_mailer/result_email.text.erb

@@ -0,0 +1,18 @@
+CSV import for <%= @model %>
+
+Succeeded: <%= @result.success_count %>
+Failed:    <%= @result.failed_count %>
+Total:     <%= @result.total %>
+
+<% if @errors.any? -%>
+Failed rows:
+<% @errors.each do |error| -%>
+- <%= error[:row].inspect %>
+  reason: <%= error[:error] %>
+<% end -%>
+<% if @more_errors_count.positive? -%>
+... and <%= @more_errors_count %> more failed row(s) not shown.
+<% end -%>
+<% else -%>
+No failed rows.
+<% end -%>

data/lib/bulk_csv_parser/configuration.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module BulkCsvParser
+  class Configuration
+    # Rows read into memory per batch before a DB round-trip happens.
+    attr_accessor :batch_size
+
+    # :save    -> find_or_initialize_by + assign_attributes + save (runs validations/callbacks)
+    # :upsert  -> ActiveRecord#upsert_all (fast, bulk, skips validations/callbacks)
+    attr_accessor :strategy
+
+    # ActiveJob queue used by BulkCsvParser::ImportJob
+    attr_accessor :queue_name
+
+    # :active_storage -> uploads the CSV to whatever service is configured in
+    #                    config/storage.yml (S3/GCS/Azure). Required whenever
+    #                    the job's queue is processed by a different machine
+    #                    than the one that received the upload, since a local
+    #                    file path from the web server won't resolve on the
+    #                    worker box. Falls back to :local automatically when
+    #                    ActiveStorage isn't loaded.
+    # :local          -> copies the file to `upload_dir` on the local disk.
+    #                    Only safe when web and worker share that disk (e.g.
+    #                    single-machine deploys, or a shared NFS mount).
+    attr_accessor :storage
+
+    # Directory used by the :local storage adapter.
+    attr_accessor :upload_dir
+
+    # Default "from" address for BulkCsvParser::ImportMailer.
+    attr_accessor :mailer_sender
+
+    # Max number of row errors listed in the result notification email
+    # (the rest are summarized as a count, so the email doesn't balloon
+    # on a file with thousands of bad rows).
+    attr_accessor :notify_error_limit
+
+    def initialize
+      @batch_size = 1000
+      @strategy = :save
+      @queue_name = :bulk_csv_parser
+      @storage = defined?(::ActiveStorage) ? :active_storage : :local
+      @upload_dir = defined?(Rails) ? Rails.root.join("tmp", "bulk_csv_parser") : "tmp/bulk_csv_parser"
+      @mailer_sender = "no-reply@example.com"
+      @notify_error_limit = 25
+    end
+  end
+end

data/lib/bulk_csv_parser/engine.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module BulkCsvParser
+  if defined?(::Rails::Engine)
+    # Registers app/mailers and app/views so the host Rails app can autoload
+    # BulkCsvParser::ImportMailer and find its view templates.
+    class Engine < ::Rails::Engine
+    end
+  end
+end

data/lib/bulk_csv_parser/import_job.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module BulkCsvParser
+  # Runs BulkCsvParser::Processor in the background via ActiveJob, so large
+  # CSV uploads don't block a web request. Enqueue it via BulkCsvParser.enqueue
+  # rather than calling perform_later directly -- that helper is what puts the
+  # file somewhere the worker (often a different machine than the one that
+  # received the upload) can actually read it back from.
+  class ImportJob < ActiveJob::Base
+    queue_as { BulkCsvParser.configuration.queue_name }
+
+    discard_on BulkCsvParser::Processor::ConfigurationError
+
+    def perform(
+      file_reference:,
+      model:,
+      find_by:,
+      attributes:,
+      header_mapping: {},
+      batch_size: nil,
+      strategy: nil,
+      notify_email: nil
+    )
+      local_file = BulkCsvParser::Storage.fetch(file_reference)
+      path = local_file.respond_to?(:path) ? local_file.path : local_file
+
+      result = BulkCsvParser::Processor.new(
+        file_path: path,
+        model: model,
+        find_by: find_by,
+        attributes: attributes,
+        header_mapping: header_mapping,
+        batch_size: batch_size,
+        strategy: strategy
+      ).call
+
+      log(model, result)
+      notify(notify_email, model, result)
+      result
+    ensure
+      local_file.close! if local_file.respond_to?(:close!)
+      BulkCsvParser::Storage.cleanup(file_reference)
+    end
+
+    private
+
+    def log(model, result)
+      return unless defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger
+
+      Rails.logger.info(
+        "[BulkCsvParser] model=#{model} success=#{result.success_count} " \
+        "failed=#{result.failed_count} total=#{result.total}"
+      )
+    end
+
+    def notify(notify_email, model, result)
+      return unless notify_email
+      return unless defined?(BulkCsvParser::ImportMailer)
+
+      BulkCsvParser::ImportMailer.result_email(to: notify_email, model: model, result: result).deliver_now
+    end
+  end
+end

data/lib/bulk_csv_parser/processor.rb

@@ -0,0 +1,160 @@
+# frozen_string_literal: true
+
+require "csv"
+
+module BulkCsvParser
+  # Generic, model-agnostic CSV importer/updater.
+  #
+  # Streams the CSV row by row (CSV.foreach) so arbitrarily large files never
+  # get loaded into memory at once, batches rows, and either upserts them in
+  # bulk or finds/updates each record with full validation support.
+  #
+  # Example:
+  #
+  #   BulkCsvParser::Processor.new(
+  #     file_path: "/tmp/users.csv",
+  #     model: User,
+  #     find_by: [:email],
+  #     attributes: [:first_name, :last_name, :status],
+  #     header_mapping: { "Email Address" => :email }
+  #   ).call
+  #
+  class Processor
+    class ConfigurationError < StandardError; end
+
+    def initialize(
+      model:,
+      find_by:,
+      attributes:,
+      file_path: nil,
+      file: nil,
+      header_mapping: {},
+      batch_size: nil,
+      strategy: nil,
+      transform: nil,
+      on_row_error: nil
+    )
+      @model = model.is_a?(String) ? model.constantize : model
+      @file_path = file_path
+      @file = file
+      @find_by = Array(find_by).map(&:to_sym)
+      @attributes = Array(attributes).map(&:to_sym)
+      @header_mapping = header_mapping.each_with_object({}) { |(k, v), h| h[k.to_s] = v.to_sym }
+      @batch_size = batch_size || BulkCsvParser.configuration.batch_size
+      @strategy = (strategy || BulkCsvParser.configuration.strategy).to_sym
+      @transform = transform
+      @on_row_error = on_row_error
+      @result = Result.new
+
+      validate_arguments!
+    end
+
+    def call
+      each_batch do |batch|
+        next if batch.empty?
+
+        strategy == :upsert ? process_upsert(batch) : process_save(batch)
+      end
+      result
+    end
+
+    private
+
+    attr_reader :model, :find_by, :attributes, :header_mapping, :batch_size,
+                :strategy, :transform, :on_row_error, :result
+
+    def validate_arguments!
+      raise ConfigurationError, "model must respond to .find_or_initialize_by / .upsert_all" unless model.respond_to?(:find_or_initialize_by)
+      raise ConfigurationError, "find_by can't be empty" if find_by.empty?
+      raise ConfigurationError, "attributes can't be empty" if attributes.empty?
+      raise ConfigurationError, "provide either file_path or file" if @file_path.nil? && @file.nil?
+    end
+
+    # Yields arrays of Hash(attribute_symbol => value) with at most batch_size elements.
+    def each_batch
+      batch = []
+
+      CSV.foreach(csv_source, headers: true, encoding: "bom|utf-8:utf-8") do |row|
+        attrs = extract_attributes(row)
+        next if attrs.nil? # allow transform to skip/filter a row by returning nil
+
+        batch << attrs
+
+        if batch.size >= batch_size
+          yield batch
+          batch = []
+        end
+      end
+
+      yield batch unless batch.empty?
+    end
+
+    def csv_source
+      @file_path || @file
+    end
+
+    def extract_attributes(row)
+      hash = {}
+
+      row.headers.each do |header|
+        next if header.nil?
+
+        key = header_mapping[header] || normalize_header(header)
+        hash[key] = row[header]
+      end
+
+      transform ? transform.call(hash) : hash
+    end
+
+    def normalize_header(header)
+      header.to_s.strip.downcase.gsub(/\s+/, "_").to_sym
+    end
+
+    # Safe, validation-aware strategy: one find_or_initialize_by + save per row.
+    def process_save(batch)
+      batch.each do |attrs|
+        find_attrs = attrs.slice(*find_by)
+        update_attrs = attrs.slice(*attributes)
+
+        record = model.find_or_initialize_by(find_attrs)
+        record.assign_attributes(update_attrs)
+
+        if record.save
+          result.increment_success
+        else
+          result.add_error(attrs, record.errors.full_messages.join(", "))
+        end
+      rescue StandardError => e
+        on_row_error&.call(attrs, e)
+        result.add_error(attrs, e.message)
+      end
+    end
+
+    # Fast, bulk strategy: a single upsert_all per batch. Skips validations and
+    # callbacks -- requires a DB unique index covering the find_by columns.
+    def process_upsert(batch)
+      now = defined?(Time.current) ? Time.current : Time.now
+      has_created_at = model.column_names.include?("created_at")
+      has_updated_at = model.column_names.include?("updated_at")
+
+      rows = batch.map do |attrs|
+        row = attrs.slice(*attributes).merge(attrs.slice(*find_by))
+        row[:created_at] = now if has_created_at
+        row[:updated_at] = now if has_updated_at
+        row
+      end
+
+      # update_only excludes created_at from the ON CONFLICT ... SET clause,
+      # so it's only written on insert -- otherwise upsert_all would
+      # overwrite an existing record's created_at on every re-import.
+      update_only = attributes.dup
+      update_only << :updated_at if has_updated_at
+
+      model.upsert_all(rows, unique_by: find_by, update_only: update_only)
+      result.increment_success(rows.size)
+    rescue StandardError => e
+      on_row_error&.call(batch, e)
+      result.add_error(batch, e.message)
+    end
+  end
+end

data/lib/bulk_csv_parser/result.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module BulkCsvParser
+  # Collects the outcome of a Processor run: how many rows succeeded,
+  # how many failed, and the reason each failure happened.
+  class Result
+    attr_reader :success_count, :failed_count, :errors
+
+    def initialize
+      @success_count = 0
+      @failed_count = 0
+      @errors = []
+    end
+
+    def increment_success(by = 1)
+      @success_count += by
+    end
+
+    def add_error(row, message)
+      @failed_count += 1
+      @errors << { row: row, error: message }
+    end
+
+    def total
+      success_count + failed_count
+    end
+
+    def success?
+      failed_count.zero?
+    end
+
+    def to_h
+      { success_count: success_count, failed_count: failed_count, total: total, errors: errors }
+    end
+  end
+end

data/lib/bulk_csv_parser/storage.rb

@@ -0,0 +1,103 @@
+# frozen_string_literal: true
+
+require "tempfile"
+
+module BulkCsvParser
+  # Moves the uploaded CSV from the machine that received the HTTP request to
+  # wherever ImportJob actually executes -- which, in production, is very
+  # often a separate worker box (Sidekiq/Resque/GoodJob fleet) with no access
+  # to the web server's local disk or its request-scoped tempfile.
+  #
+  # `store` is called on the web process and returns a small, serializable
+  # "reference" (a String) that's safe to pass as an ActiveJob argument.
+  # `fetch` is called on the worker process and turns that reference back
+  # into a local file it can stream with CSV.foreach. `cleanup` removes the
+  # persisted copy once the job is done with it.
+  class Storage
+    class UnknownAdapterError < StandardError; end
+
+    class << self
+      def store(file)
+        adapter.store(file)
+      end
+
+      # Returns an object responding to #path; call #cleanup_local(result) on
+      # it when done reading, if it responds to that (see Local::Handle).
+      def fetch(reference)
+        adapter.fetch(reference)
+      end
+
+      def cleanup(reference)
+        adapter.cleanup(reference)
+      end
+
+      private
+
+      def adapter
+        case BulkCsvParser.configuration.storage
+        when :active_storage
+          ActiveStorageAdapter
+        when :local
+          LocalAdapter
+        else
+          raise UnknownAdapterError, "unknown storage adapter #{BulkCsvParser.configuration.storage.inspect}"
+        end
+      end
+    end
+
+    # Copies to local disk. Only correct when the enqueuing process and the
+    # job's execution process share that disk.
+    module LocalAdapter
+      module_function
+
+      def store(file)
+        dir = BulkCsvParser.configuration.upload_dir
+        FileUtils.mkdir_p(dir)
+
+        destination = File.join(dir.to_s, "#{SecureRandom.uuid}.csv")
+        FileUtils.cp(file.path, destination)
+        destination
+      end
+
+      def fetch(reference)
+        reference # already a local path
+      end
+
+      def cleanup(reference)
+        File.delete(reference) if reference && File.exist?(reference)
+      end
+    end
+
+    # Uploads to whatever service config/storage.yml points at (S3, GCS,
+    # Azure, ...). The reference is an ActiveStorage signed_id: safe to hand
+    # to a job that runs on any machine with the same storage credentials.
+    module ActiveStorageAdapter
+      module_function
+
+      def store(file)
+        filename = file.respond_to?(:original_filename) ? file.original_filename : File.basename(file.path)
+
+        blob = ::ActiveStorage::Blob.create_and_upload!(
+          io: File.open(file.path),
+          filename: filename,
+          content_type: "text/csv"
+        )
+        blob.signed_id
+      end
+
+      def fetch(reference)
+        blob = ::ActiveStorage::Blob.find_signed!(reference)
+
+        tempfile = Tempfile.new(["bulk_csv_parser", ".csv"])
+        tempfile.binmode
+        blob.download { |chunk| tempfile.write(chunk) }
+        tempfile.rewind
+        tempfile
+      end
+
+      def cleanup(reference)
+        ::ActiveStorage::Blob.find_signed(reference)&.purge
+      end
+    end
+  end
+end

data/lib/bulk_csv_parser/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module BulkCsvParser
+  VERSION = "0.1.0"
+end

data/lib/bulk_csv_parser.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+require "fileutils"
+require "securerandom"
+require "bulk_csv_parser/version"
+require "bulk_csv_parser/configuration"
+require "bulk_csv_parser/result"
+require "bulk_csv_parser/processor"
+require "bulk_csv_parser/storage"
+require "bulk_csv_parser/import_job" if defined?(ActiveJob)
+require "bulk_csv_parser/engine" if defined?(Rails::Engine)
+
+module BulkCsvParser
+  class << self
+    attr_writer :configuration
+
+    def configuration
+      @configuration ||= Configuration.new
+    end
+
+    def configure
+      yield(configuration)
+    end
+
+    # Synchronous import, runs in the current process/request.
+    def import(**options)
+      Processor.new(**options).call
+    end
+
+    # Stores the uploaded file (via BulkCsvParser::Storage -- Active Storage
+    # by default, so it lands on S3/GCS/Azure and any worker machine can read
+    # it back) then enqueues BulkCsvParser::ImportJob. `file` can be an
+    # ActionDispatch::Http::UploadedFile, Rack::Test::UploadedFile, or
+    # anything responding to #path.
+    #
+    # Pass notify_email: "someone@example.com" to have ImportJob email the
+    # success/failure counts and per-row failure reasons once it's done.
+    def enqueue(file:, model:, find_by:, attributes:, **options)
+      reference = Storage.store(file)
+
+      ImportJob.perform_later(
+        file_reference: reference,
+        model: model.to_s,
+        find_by: find_by,
+        attributes: attributes,
+        **options
+      )
+    end
+  end
+end

data/lib/generators/bulk_csv_parser/install/install_generator.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+
+module BulkCsvParser
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      source_root File.expand_path("templates", __dir__)
+      desc "Creates a BulkCsvParser initializer"
+
+      def copy_initializer
+        template "bulk_csv_parser.rb", "config/initializers/bulk_csv_parser.rb"
+      end
+    end
+  end
+end

data/lib/generators/bulk_csv_parser/install/templates/bulk_csv_parser.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+BulkCsvParser.configure do |config|
+  # Rows read into memory per batch before a DB round-trip happens.
+  config.batch_size = 1000
+
+  # :save   -> validations/callbacks run, safe default
+  # :upsert -> ActiveRecord#upsert_all, fastest for huge files, needs a DB
+  #            unique index on the find_by columns, skips validations/callbacks
+  config.strategy = :save
+
+  # ActiveJob queue used by BulkCsvParser::ImportJob
+  config.queue_name = :bulk_csv_parser
+
+  # :active_storage (default when Active Storage is installed) uploads the
+  # CSV to whatever service config/storage.yml points at (S3/GCS/Azure) so
+  # any worker machine can read it back -- REQUIRED if your job queue is
+  # processed on a different machine than your web servers.
+  #
+  # :local copies the file to `upload_dir` on local disk. Only safe when web
+  # and worker processes share that disk.
+  config.storage = :active_storage
+
+  # Used only by the :local storage adapter.
+  config.upload_dir = Rails.root.join("tmp", "bulk_csv_parser")
+
+  # "From" address for the result notification email
+  # (BulkCsvParser::ImportMailer), sent when you pass notify_email: to
+  # BulkCsvParser.enqueue.
+  config.mailer_sender = "no-reply@#{Rails.application.class.module_parent_name.underscore.dasherize}.com"
+
+  # Max number of failed rows listed in the notification email.
+  config.notify_error_limit = 25
+end

metadata

@@ -0,0 +1,181 @@
+--- !ruby/object:Gem::Specification
+name: bulk_csv_parser
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Jagrit Bharara
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: csv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: activejob
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+- !ruby/object:Gem::Dependency
+  name: activerecord
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: actionmailer
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+description: Drop-in gem to bulk-import or bulk-update large CSV files into any ActiveRecord
+  model by declaring which model, which columns identify existing records, and which
+  columns to update. Streams the file so it never loads fully into memory, and ships
+  an ActiveJob worker so imports run in the background.
+email:
+- jagritbharara12@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE.txt
+- README.md
+- app/mailers/bulk_csv_parser/import_mailer.rb
+- app/views/bulk_csv_parser/import_mailer/result_email.html.erb
+- app/views/bulk_csv_parser/import_mailer/result_email.text.erb
+- lib/bulk_csv_parser.rb
+- lib/bulk_csv_parser/configuration.rb
+- lib/bulk_csv_parser/engine.rb
+- lib/bulk_csv_parser/import_job.rb
+- lib/bulk_csv_parser/processor.rb
+- lib/bulk_csv_parser/result.rb
+- lib/bulk_csv_parser/storage.rb
+- lib/bulk_csv_parser/version.rb
+- lib/generators/bulk_csv_parser/install/install_generator.rb
+- lib/generators/bulk_csv_parser/install/templates/bulk_csv_parser.rb
+homepage: https://example.com/bulk_csv_parser
+licenses:
+- MIT
+metadata: {}
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.7.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.9
+specification_version: 4
+summary: Generic bulk CSV import/update for any ActiveRecord model, with a background
+  worker.
+test_files: []