data_drain 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: a9745b689c4134b2dd8ac402e1d8371b5e92d112a516f310d323b2eb3957a665
+  data.tar.gz: 3cb740933d8d7031446ef2f3c9c4564c0a27b1a93b7e210ca2e449f9213fd453
+SHA512:
+  metadata.gz: c31dbd33cf14556384ca9abaa1fb056bfa52efa324cd34e829027babff82183d694418ea12f5d4d99fa43d459733c5466d75c510d4d61d900b49ca3561da84d4
+  data.tar.gz: 3a1968bdd650604d7699a3225ca79e9c5c174749927c77684284cf736f29d7b4b97a80e5570ba77ebb97f769a0c1e5282d64bb67b4eb586af600390d378fd0b6

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.rubocop.yml

@@ -0,0 +1,8 @@
+AllCops:
+  TargetRubyVersion: 3.2
+
+Style/StringLiterals:
+  EnforcedStyle: double_quotes
+
+Style/StringLiteralsInInterpolation:
+  EnforcedStyle: double_quotes

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## [Unreleased]
+
+## [0.1.0] - 2026-03-11
+
+- Initial release

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,132 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Gabriel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,172 @@
+# DataDrain 🚰
+
+DataDrain es un micro-framework de nivel empresarial diseñado para extraer, archivar y purgar datos históricos desde bases de datos PostgreSQL transaccionales, así como para **ingerir archivos crudos (CSV, JSON, Parquet)**, hacia un Data Lake analítico.
+
+Utiliza **DuckDB** en memoria para lograr velocidades de procesamiento y compresión extremas. Garantiza la retención segura de datos mediante chequeos de integridad estrictos antes de purgar las bases de datos de origen, y automatiza la conversión y subida de archivos pesados a la nube.
+
+## Características Principales
+
+* **ETL de Alto Rendimiento:** Transfiere millones de registros desde Postgres a Parquet utilizando DuckDB sin cargar los objetos en la memoria RAM de Ruby.
+* **File Ingestion:** Convierte archivos crudos masivos (ej. logs de Netflow en CSV) a Parquet (ZSTD) y los sube directamente a S3 en milisegundos.
+* **Hive Partitioning:** Organiza automáticamente los archivos en carpetas optimizadas para consultas (`year=X/month=Y/tenant_id=Z`).
+* **Storage Adapters:** Soporte nativo y transparente para almacenamiento en Disco Local y AWS S3.
+* **Integridad Garantizada:** Verifica matemáticamente que los datos exportados coincidan exactamente con el origen antes de ejecutar sentencias `DELETE`.
+* **ORM Analítico Integrado:** Incluye una clase base (`DataDrain::Record`) compatible con `ActiveModel` para consultar y destruir particiones históricas de forma idiomática.
+
+## Instalación
+
+Agrega esta línea al `Gemfile` de tu aplicación o microservicio:
+
+```ruby
+gem 'data_drain', git: '[https://github.com/tu-organizacion/data_drain.git](https://github.com/tu-organizacion/data_drain.git)', branch: 'main'
+```
+
+Y ejecuta:
+```bash
+$ bundle install
+```
+
+## Configuración
+
+Crea un inicializador en tu aplicación (ej. `config/initializers/data_drain.rb`) para configurar las credenciales y el comportamiento del motor:
+
+```ruby
+DataDrain.configure do |config|
+  # Almacenamiento (:local o :s3)
+  config.storage_mode = ENV.fetch('STORAGE_MODE', 'local').to_sym
+
+  # AWS S3 (Requerido solo si storage_mode es :s3)
+  # config.aws_region = ENV['AWS_REGION']
+  # config.aws_access_key_id = ENV['AWS_ACCESS_KEY_ID']
+  # config.aws_secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
+
+  # Base de Datos PostgreSQL de Origen (Requerido solo para DataDrain::Engine)
+  config.db_host = ENV.fetch('DB_HOST', '127.0.0.1')
+  config.db_port = ENV.fetch('DB_PORT', '5432')
+  config.db_user = ENV.fetch('DB_USER', 'postgres')
+  config.db_pass = ENV.fetch('DB_PASS', '')
+  config.db_name = ENV.fetch('DB_NAME', 'core_production')
+
+  # Rendimiento y Tuning
+  config.batch_size     = 5000 # Registros a borrar por transacción
+  config.throttle_delay = 0.5  # Segundos de pausa entre borrados
+  config.logger         = Rails.logger
+end
+```
+
+## Uso
+
+El framework provee tres herramientas principales: **Ingestor de Archivos**, **Drenaje de Base de Datos**, y el **ORM Analítico**.
+
+### 1. Ingestión de Archivos Crudos (FileIngestor)
+
+Ideal para servicios que generan grandes volúmenes de datos (ej. métricas de Netflow). Toma un archivo local, lo transforma, lo comprime a Parquet y lo sube particionado a S3.
+
+```ruby
+# Un archivo generado temporalmente por tu servicio
+archivo_temporal = "/tmp/netflow_metrics_1600.csv"
+
+ingestor = DataDrain::FileIngestor.new(
+  bucket: 'my-bucket-store',
+  source_path: archivo_temporal,
+  folder_name: 'netflow',
+  # Particionamos dinámicamente según columnas extraídas al vuelo
+  partition_keys: %w[year month isp_id],
+  # Transformación SQL ejecutada por DuckDB durante la lectura
+  select_sql: "*, EXTRACT(YEAR FROM timestamp) AS year, EXTRACT(MONTH FROM timestamp) AS month",
+  delete_after_upload: true # Limpia el archivo temporal al terminar
+)
+
+ingestor.call
+```
+
+### 2. Extracción y Purga de BD (Engine)
+
+Ideal para crear Ventanas Rodantes de retención (ej. mantener solo 6 meses de datos vivos en Postgres y archivar el resto).
+
+```ruby
+# lib/tasks/archive.rake
+task versions: :environment do
+  target_date = 6.months.ago.beginning_of_month
+
+  select_sql = <<~SQL
+    id, item_type, item_id, event, whodunnit,
+    object::VARCHAR AS object,
+    object_changes::VARCHAR AS object_changes,
+    created_at,
+    EXTRACT(YEAR FROM created_at)::INT AS year,
+    EXTRACT(MONTH FROM created_at)::INT AS month,
+    isp_id
+  SQL
+
+  engine = DataDrain::Engine.new(
+    bucket:         'my-bucket-store',
+    start_date:     target_date.beginning_of_month,
+    end_date:       target_date.end_of_month,
+    table_name:     'versions',
+    select_sql:     select_sql,
+    partition_keys: %w[year month isp_id],
+    where_clause:   "event = 'update'"
+  )
+
+  # Cuenta, exporta a Parquet, verifica integridad y purga Postgres.
+  engine.call
+end
+```
+
+### 3. Consultar el Data Lake (Record)
+
+Para consultar los datos archivados sin salir de Ruby, crea un modelo que herede de `DataDrain::Record`.
+
+```ruby
+# app/models/archived_version.rb
+class ArchivedVersion < DataDrain::Record
+  self.bucket = 'my-bucket-storage'
+  self.folder_name = 'versions'
+  self.partition_keys = [:year, :month, :isp_id]
+
+  attribute :id, :string
+  attribute :item_type, :string
+  attribute :item_id, :string
+  attribute :event, :string
+  attribute :whodunnit, :string
+  attribute :created_at, :datetime
+
+  # Utiliza el tipo :json provisto por la gema para hidratar Hashes
+  attribute :object, :json
+  attribute :object_changes, :json
+end
+```
+
+Consultas altamente optimizadas mediante Hive Partitioning:
+
+```ruby
+# Búsqueda puntual hiper-rápida aislando las particiones
+version = ArchivedVersion.find("un-uuid", year: 2026, month: 3, isp_id: 42)
+puts version.object_changes # => {"status" => ["active", "suspended"]}
+
+# Colecciones
+history = ArchivedVersion.where(limit: 10, year: 2026, month: 3, isp_id: 42)
+```
+
+### 4. Destrucción de Datos (Retención y Cumplimiento)
+
+El framework permite eliminar físicamente carpetas completas en S3 o Local utilizando comodines.
+
+```ruby
+# Elimina todo el historial de un cliente en específico a través de todos los años
+ArchivedVersion.destroy_all(isp_id: 42)
+
+# Elimina todos los datos de marzo de 2024 globalmente
+ArchivedVersion.destroy_all(year: 2024, month: 3)
+```
+
+## Arquitectura
+
+DataDrain implementa el patrón **Storage Adapter**, lo que permite aislar completamente la lógica del sistema de archivos de los motores de procesamiento. 
+* DuckDB mantiene una conexión persistente (`Thread-Safe`) para maximizar el rendimiento de las consultas web.
+* El ORM Analítico incluye sanitización de parámetros para prevenir Inyección SQL al consultar archivos Parquet.
+
+## Licencia
+
+La gema está disponible como código abierto bajo los términos de la Licencia MIT.

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/data_drain.gemspec

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+require_relative "lib/data_drain/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "data_drain"
+  spec.version = DataDrain::VERSION
+  spec.authors = ["Gabriel"]
+  spec.email = ["gab.edera@gmail.com"]
+
+  spec.summary = "Micro-framework para drenar datos de PostgreSQL a Parquet vía DuckDB."
+  spec.description = "Extrae datos transaccionales, los archiva en un Data Lake (S3/Local) " \
+                     "en formato Parquet usando Hive Partitioning, y purga el origen de forma segura."
+  spec.homepage = "https://github.com/gedera/data_drain"
+  spec.required_ruby_version = ">= 3.0.0"
+
+  spec.files = Dir.chdir(__dir__) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      (File.expand_path(f) == __FILE__) ||
+        f.start_with?(*%w[bin/ test/ spec/ features/ .git .github appveyor Gemfile])
+    end
+  end
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  # 💡 Dependencias Core de la Gema
+  spec.add_dependency "activemodel", ">= 6.0"
+  spec.add_dependency "aws-sdk-s3", "~> 1.114"
+  spec.add_dependency "duckdb", "~> 1.4"
+  spec.add_dependency "pg", ">= 1.2"
+end

data/lib/data_drain/configuration.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+require "logger"
+
+module DataDrain
+  # Contenedor para todas las opciones de configuración del motor DataDrain.
+  class Configuration
+    attr_accessor :storage_mode, :aws_region,
+      :aws_access_key_id, :aws_secret_access_key,
+      :db_host, :db_port, :db_user, :db_pass, :db_name,
+      :batch_size, :throttle_delay, :logger
+
+    def initialize
+      @storage_mode   = :local
+      @db_host        = "127.0.0.1"
+      @db_port        = 5432
+      @batch_size     = 5000
+      @throttle_delay = 0.5
+      @logger         = Logger.new($stdout)
+    end
+
+    # @return [String] Cadena de conexión optimizada para DuckDB.
+    def duckdb_connection_string
+      "host=#{@db_host} port=#{@db_port} dbname=#{@db_name} user=#{@db_user} password=#{@db_pass}"
+    end
+  end
+end

data/lib/data_drain/engine.rb

@@ -0,0 +1,177 @@
+# frozen_string_literal: true
+
+require "duckdb"
+require "pg"
+
+module DataDrain
+  # Motor principal de extracción y purga de datos (DataDrain).
+  #
+  # Orquesta el flujo ETL desde PostgreSQL hacia un Data Lake analítico
+  # delegando la interacción del almacenamiento al adaptador configurado.
+  class Engine
+    # Inicializa una nueva instancia del motor de extracción.
+    #
+    # @param options [Hash] Diccionario de configuración para la extracción.
+    # @option options [Time, DateTime, Date] :start_date Fecha y hora de inicio.
+    # @option options [Time, DateTime, Date] :end_date Fecha y hora de fin.
+    # @option options [String] :table_name Nombre de la tabla en PostgreSQL.
+    # @option options [String] :folder_name (Opcional) Nombre de la carpeta destino.
+    # @option options [String] :select_sql (Opcional) Sentencia SELECT personalizada.
+    # @option options [Array<String, Symbol>] :partition_keys Columnas para particionar.
+    # @option options [String] :primary_key (Opcional) Clave primaria para borrado. Por defecto 'id'.
+    # @option options [String] :where_clause (Opcional) Condición SQL extra.
+    def initialize(options)
+      @start_date     = options.fetch(:start_date).beginning_of_day
+      @end_date       = options.fetch(:end_date).end_of_day
+      @table_name     = options.fetch(:table_name)
+      @folder_name    = options.fetch(:folder_name, @table_name)
+      @select_sql     = options.fetch(:select_sql, "*")
+      @partition_keys = options.fetch(:partition_keys)
+      @primary_key    = options.fetch(:primary_key, "id")
+      @where_clause   = options[:where_clause]
+      @bucket         = options[:bucket]
+
+      @config  = DataDrain.configuration
+      @logger  = @config.logger
+      @adapter = DataDrain::Storage.adapter
+
+      database = DuckDB::Database.open(":memory:")
+      @duckdb  = database.connect
+    end
+
+    # Ejecuta el flujo completo del motor: Setup, Conteo, Exportación, Verificación y Purga.
+    #
+    # @return [Boolean] `true` si el proceso finalizó con éxito, `false` si falló la integridad.
+    def call
+      @logger.info "[DataDrain Engine] 🚀 Preparando '#{@table_name}' (#{@start_date.to_date} a #{@end_date.to_date})..."
+
+      setup_duckdb
+
+      @pg_count = get_postgres_count
+
+      if @pg_count.zero?
+        @logger.info "[DataDrain Engine] ⏭️ No hay registros que cumplan las condiciones."
+        return true
+      end
+
+      @logger.info "[DataDrain Engine] 📦 Exportando #{@pg_count} registros a Parquet..."
+      export_to_parquet
+
+      if verify_integrity
+        purge_from_postgres
+        @logger.info "[DataDrain Engine] ✅ Proceso completado exitosamente para '#{@table_name}'."
+        true
+      else
+        @logger.error "[DataDrain Engine] ❌ ERROR de integridad en '#{@table_name}'. Abortando purga."
+        false
+      end
+    end
+
+    private
+
+    # @api private
+    # @return [String]
+    def base_where_sql
+      sql = "created_at >= '#{@start_date.to_fs(:db)}' AND created_at <= '#{@end_date.to_fs(:db)}'"
+      sql += " AND #{@where_clause}" if @where_clause && !@where_clause.empty?
+      sql
+    end
+
+    # @api private
+    def setup_duckdb
+      @duckdb.query("INSTALL postgres; LOAD postgres;")
+      @duckdb.query("SET max_memory='2GB';")
+
+      # 💡 Magia del Adapter: Él sabe si cargar httpfs y setear credenciales o no hacer nada
+      @adapter.setup_duckdb(@duckdb)
+    end
+
+    # @api private
+    # @return [Integer]
+    def get_postgres_count
+      query = <<~SQL
+        SELECT COUNT(*)
+        FROM postgres_scan('#{@config.duckdb_connection_string}', 'public', '#{@table_name}')
+        WHERE #{base_where_sql}
+      SQL
+      @duckdb.query(query).first.first
+    end
+
+    # @api private
+    def export_to_parquet
+      # 💡 Magia del Adapter: Si es local crea las carpetas, si es S3 no hace nada.
+      @adapter.prepare_export_path(@bucket, @folder_name)
+
+      # Determinamos el path base de destino según el adaptador
+      dest_path = @config.storage_mode.to_sym == :s3 ? "s3://#{@bucket}/#{@folder_name}/" : File.join(@bucket, @folder_name, "")
+
+      query = <<~SQL
+        COPY (
+          SELECT #{@select_sql}
+          FROM postgres_scan('#{@config.duckdb_connection_string}', 'public', '#{@table_name}')
+          WHERE #{base_where_sql}
+        ) TO '#{dest_path}'
+        (
+          FORMAT PARQUET,
+          PARTITION_BY (#{@partition_keys.join(', ')}),
+          COMPRESSION 'ZSTD',
+          OVERWRITE_OR_IGNORE 1
+        );
+      SQL
+      @duckdb.query(query)
+    end
+
+    # @api private
+    # @return [Boolean]
+    def verify_integrity
+      # 💡 Magia del Adapter: Construye la ruta de búsqueda global ('**/*.parquet')
+      archive_path = @adapter.build_path(@bucket, @folder_name, nil)
+
+      begin
+        query = <<~SQL
+          SELECT COUNT(*)
+          FROM read_parquet('#{archive_path}')
+          WHERE #{base_where_sql}
+        SQL
+        parquet_result = @duckdb.query(query).first.first
+      rescue DuckDB::Error => e
+        @logger.error "[DataDrain Engine] ❌ Error leyendo Parquet: #{e.message}"
+        return false
+      end
+
+      @logger.info "[DataDrain Engine] 📊 Verificación -> Postgres: #{@pg_count} | Parquet: #{parquet_result}"
+      @pg_count == parquet_result
+    end
+
+    # @api private
+    def purge_from_postgres
+      @logger.info "[DataDrain Engine] 🗑️ Purgando en base de datos (Lotes de #{@config.batch_size})..."
+
+      conn = PG.connect(
+        host:     @config.db_host,
+        port:     @config.db_port,
+        user:     @config.db_user,
+        password: @config.db_pass,
+        dbname:   @config.db_name
+      )
+
+      loop do
+        sql = <<~SQL
+          DELETE FROM #{@table_name}
+          WHERE #{@primary_key} IN (
+            SELECT #{@primary_key} FROM #{@table_name}
+            WHERE #{base_where_sql}
+            LIMIT #{@config.batch_size}
+          )
+        SQL
+
+        result = conn.exec(sql)
+        break if result.cmd_tuples.zero?
+
+        sleep(@config.throttle_delay) if @config.throttle_delay.positive?
+      end
+    ensure
+      conn&.close
+    end
+  end
+end

data/lib/data_drain/errors.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module DataDrain
+  # Clase base para todos los errores originados en el framework DataDrain.
+  class Error < StandardError; end
+
+  # Levantado cuando falta configuración obligatoria.
+  class ConfigurationError < Error; end
+
+  # Levantado cuando la verificación matemática entre Postgres y Parquet no coincide.
+  class IntegrityError < Error; end
+
+  # Levantado cuando hay problemas interactuando con DuckDB, el disco local o AWS S3.
+  class StorageError < Error; end
+end

data/lib/data_drain/file_ingestor.rb

@@ -0,0 +1,111 @@
+# frozen_string_literal: true
+
+module DataDrain
+  # Clase encargada de ingerir archivos locales (CSV, JSON, Parquet)
+  # generados por otros servicios (ej. Netflow) y subirlos al Data Lake
+  # aplicando compresión ZSTD y particionamiento Hive.
+  class FileIngestor
+    # @param options [Hash] Opciones de ingestión.
+    # @option options [String] :source_path Ruta absoluta al archivo local.
+    # @option options [String] :folder_name Nombre de la carpeta destino en el Data Lake.
+    # @option options [Array<String, Symbol>] :partition_keys (Opcional) Columnas para particionar.
+    # @option options [String] :select_sql (Opcional) Sentencia SELECT para transformar datos al vuelo.
+    # @option options [Boolean] :delete_after_upload (Opcional) Borra el archivo local al terminar. Por defecto true.
+    def initialize(options)
+      @source_path         = options.fetch(:source_path)
+      @folder_name         = options.fetch(:folder_name)
+      @partition_keys      = options.fetch(:partition_keys, [])
+      @select_sql          = options.fetch(:select_sql, "*")
+      @delete_after_upload = options.fetch(:delete_after_upload, true)
+      @bucket              = options[:bucket]
+
+      @config  = DataDrain.configuration
+      @logger  = @config.logger
+      @adapter = DataDrain::Storage.adapter
+
+      database = DuckDB::Database.open(":memory:")
+      @duckdb  = database.connect
+    end
+
+    # Ejecuta el flujo de ingestión.
+    # @return [Boolean] true si el proceso fue exitoso.
+    def call
+      @logger.info "[DataDrain FileIngestor] 🚀 Iniciando ingestión de '#{@source_path}'..."
+
+      unless File.exist?(@source_path)
+        @logger.error "[DataDrain FileIngestor] ❌ El archivo origen no existe: #{@source_path}"
+        return false
+      end
+
+      @adapter.setup_duckdb(@duckdb)
+
+      # Determinamos la función lectora de DuckDB según la extensión del archivo
+      reader_function = determine_reader
+
+      # 1. Conteo de seguridad
+      source_count = @duckdb.query("SELECT COUNT(*) FROM #{reader_function}").first.first
+      @logger.info "[DataDrain FileIngestor] 📊 Encontrados #{source_count} registros para procesar."
+
+      if source_count.zero?
+        cleanup_local_file
+        return true
+      end
+
+      # 2. Exportación / Subida
+      @adapter.prepare_export_path(@bucket, @folder_name)
+      dest_path = @config.storage_mode.to_sym == :s3 ? "s3://#{@bucket}/#{@folder_name}/" : File.join(@bucket, @folder_name, "")
+
+      partition_clause = @partition_keys.any? ? "PARTITION_BY (#{@partition_keys.join(', ')})," : ""
+
+      query = <<~SQL
+        COPY (
+          SELECT #{@select_sql}
+          FROM #{reader_function}
+        ) TO '#{dest_path}'
+        (
+          FORMAT PARQUET,
+          #{partition_clause}
+          COMPRESSION 'ZSTD',
+          OVERWRITE_OR_IGNORE 1
+        );
+      SQL
+
+      @logger.info "[DataDrain FileIngestor] ☁️ Escribiendo en el Data Lake..."
+      @duckdb.query(query)
+
+      @logger.info "[DataDrain FileIngestor] ✅ Archivo ingerido y comprimido exitosamente."
+
+      cleanup_local_file
+      true
+    rescue DuckDB::Error => e
+      @logger.error "[DataDrain FileIngestor] ❌ Error de DuckDB durante la ingestión: #{e.message}"
+      false
+    ensure
+      @duckdb&.close
+    end
+
+    private
+
+    # @api private
+    def determine_reader
+      case File.extname(@source_path).downcase
+      when '.csv'
+        "read_csv_auto('#{@source_path}')"
+      when '.json'
+        "read_json_auto('#{@source_path}')"
+      when '.parquet'
+        "read_parquet('#{@source_path}')"
+      else
+        raise DataDrain::Error, "Formato de archivo no soportado para ingestión: #{@source_path}"
+      end
+    end
+
+    # @api private
+    def cleanup_local_file
+      if @delete_after_upload && File.exist?(@source_path)
+        File.delete(@source_path)
+        @logger.info "[DataDrain FileIngestor] 🗑️ Archivo temporal local eliminado."
+      end
+    end
+  end
+end

data/lib/data_drain/record.rb

@@ -0,0 +1,127 @@
+# frozen_string_literal: true
+
+require "active_model"
+require "duckdb"
+
+module DataDrain
+  # Clase base que actúa como un ORM (Object-Relational Mapper) de solo lectura y purga
+  # para interactuar con el Data Lake en formato Parquet utilizando DuckDB.
+  #
+  # @abstract Subclasifica este modelo para cada tabla archivada.
+  # @example
+  #   class ArchivedVersion < DataDrain::Record
+  #     self.folder_name = 'versions'
+  #     self.partition_keys = [:year, :month, :isp_id]
+  #     attribute :event, :string
+  #   end
+  class Record
+    include ActiveModel::Model
+    include ActiveModel::Attributes
+
+    class_attribute :bucket
+    class_attribute :folder_name
+    class_attribute :partition_keys
+
+    # Retorna la conexión persistente a DuckDB en memoria para el hilo (Thread) actual.
+    # Esto previene tener que recargar extensiones (como httpfs) en cada consulta.
+    #
+    # @return [DuckDB::Connection] Conexión activa a DuckDB.
+    def self.connection
+      Thread.current[:data_drain_duckdb_conn] ||= begin
+        db = DuckDB::Database.open(":memory:")
+        conn = db.connect
+        DataDrain::Storage.adapter.setup_duckdb(conn)
+        conn
+      end
+    end
+
+    # Consulta registros en el Data Lake filtrando por claves de partición.
+    #
+    # @param limit [Integer] Cantidad máxima de registros a retornar.
+    # @param partitions [Hash] Pares clave-valor correspondientes a las particiones.
+    # @return [Array<DataDrain::Record>] Colección de registros instanciados.
+    def self.where(limit: 50, **partitions)
+      path = build_query_path(partitions)
+
+      sql = <<~SQL
+        SELECT #{attribute_names.join(', ')}
+        FROM read_parquet('#{path}')
+        ORDER BY created_at DESC
+        LIMIT #{limit}
+      SQL
+
+      execute_and_instantiate(sql, attribute_names)
+    end
+
+    # Busca un registro específico por su ID.
+    # Implementa sanitización básica para prevenir Inyección SQL.
+    #
+    # @param id [String, Integer] Identificador único del registro.
+    # @param partitions [Hash] Pares clave-valor de las particiones donde buscar.
+    # @return [DataDrain::Record, nil] El registro encontrado o nil.
+    def self.find(id, **partitions)
+      path = build_query_path(partitions)
+      # Sanitización básica: duplicar comillas simples para anular escapes SQL
+      safe_id = id.to_s.gsub("'", "''")
+
+      sql = <<~SQL
+        SELECT #{attribute_names.join(', ')}
+        FROM read_parquet('#{path}')
+        WHERE id = '#{safe_id}'
+        LIMIT 1
+      SQL
+
+      execute_and_instantiate(sql, attribute_names).first
+    end
+
+    # Elimina físicamente los directorios o prefijos de S3.
+    #
+    # @param partitions [Hash] Particiones a eliminar.
+    # @return [Integer] Cantidad de particiones físicas eliminadas.
+    def self.destroy_all(**partitions)
+      adapter = DataDrain::Storage.adapter
+      DataDrain.configuration.logger.info "[DataDrain] 🗑️ Ejecutando destroy_all en #{folder_name} con: #{partitions.inspect}"
+
+      adapter.destroy_partitions(bucket, folder_name, partition_keys, partitions)
+    end
+
+    # @return [String] Representación legible en consola.
+    def inspect
+      inspection = attributes.map do |name, value|
+        "#{name}: #{value.nil? ? 'nil' : value.inspect}"
+      end.compact.join(", ")
+
+      "#<#{self.class} #{inspection}>"
+    end
+
+    class << self
+      private
+
+      # @api private
+      # @param partitions [Hash]
+      # @return [String]
+      def build_query_path(partitions)
+        partition_path = partitions.map { |k, v| "#{k}=#{v}" }.join("/")
+        DataDrain::Storage.adapter.build_path(bucket, folder_name, partition_path)
+      end
+
+      # @api private
+      # @param sql [String]
+      # @param columns [Array<String>]
+      # @return [Array<DataDrain::Record>]
+      def execute_and_instantiate(sql, columns)
+        begin
+          result = connection.query(sql)
+        rescue DuckDB::Error => e
+          DataDrain.configuration.logger.warn "[DataDrain] ⚠️ Ruta o archivo no encontrado: #{e.message}"
+          return []
+        end
+
+        result.map do |row|
+          attributes_hash = columns.zip(row).to_h
+          new(attributes_hash)
+        end
+      end
+    end
+  end
+end

data/lib/data_drain/storage/base.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+module DataDrain
+  module Storage
+    # Interfaz abstracta para los adaptadores de almacenamiento de DataDrain.
+    # Define los métodos obligatorios que cada proveedor (Local, S3, etc.)
+    # debe implementar para interactuar con DuckDB y el sistema de archivos.
+    #
+    # @abstract
+    class Base
+      # @return [DataDrain::Configuration] Configuración actual del framework.
+      attr_reader :config
+
+      # Inicializa el adaptador con la configuración proveída.
+      #
+      # @param config [DataDrain::Configuration]
+      def initialize(config)
+        @config = config
+      end
+
+      # Configura las extensiones y credenciales necesarias en la conexión de DuckDB.
+      #
+      # @param connection [DuckDB::Connection] Conexión activa a DuckDB.
+      # @raise [NotImplementedError] Si la subclase no lo implementa.
+      def setup_duckdb(connection)
+        raise NotImplementedError, "#{self.class} debe implementar #setup_duckdb"
+      end
+
+      # Prepara el directorio destino antes de una exportación (ej. crear carpetas).
+      #
+      # @param bucket [String] nombre del bucket tanto local o de S3.
+      # @param folder_name [String] Nombre de la carpeta principal de la tabla.
+      def prepare_export_path(bucket, folder_name)
+        # Operación nula por defecto. Las subclases pueden sobreescribirlo.
+      end
+
+      # Construye la ruta de lectura compatible con la función `read_parquet` de DuckDB.
+      #
+      # @param bucket [String] nombre del bucket tanto local o de S3.
+      # @param folder_name [String] Carpeta de la tabla (ej. 'versions').
+      # @param partition_path [String, nil] Ruta parcial de particiones (ej. 'year=2026/month=3').
+      # @return [String] Ruta completa con comodines (ej. '.../**/*.parquet').
+      def build_path(bucket, folder_name, partition_path)
+        raise NotImplementedError, "#{self.class} debe implementar #build_path"
+      end
+
+      # Elimina físicamente las particiones que coincidan con los criterios.
+      #
+      # @param bucket [String] nombre del bucket tanto local o de S3.
+      # @param folder_name [String] Carpeta de la tabla.
+      # @param partition_keys [Array<Symbol>] Claves de partición esperadas.
+      # @param partitions [Hash] Valores de las particiones a eliminar (puede contener nulos).
+      # @return [Integer] Cantidad de particiones o archivos eliminados.
+      def destroy_partitions(bucket, folder_name, partition_keys, partitions)
+        raise NotImplementedError, "#{self.class} debe implementar #destroy_partitions"
+      end
+    end
+  end
+end

data/lib/data_drain/storage/local.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+require "fileutils"
+
+module DataDrain
+  module Storage
+    # Implementación del adaptador de almacenamiento para el disco local.
+    class Local < Base
+      # (DuckDB ya soporta archivos locales de forma nativa, no requiere extensiones extras)
+      # @param connection [DuckDB::Connection]
+      def setup_duckdb(connection)
+        # No-op
+      end
+
+      # Crea la jerarquía de carpetas en el disco si no existe.
+      # @param bucket [String]
+      # @param folder_name [String]
+      def prepare_export_path(bucket, folder_name)
+        FileUtils.mkdir_p(File.join(bucket, folder_name))
+      end
+
+      # @param bucket [String]
+      # @param folder_name [String]
+      # @param partition_path [String, nil]
+      # @return [String]
+      def build_path(bucket, folder_name, partition_path)
+        base = File.join(bucket, folder_name)
+        base = File.join(base, partition_path) if partition_path && !partition_path.empty?
+        "#{base}/**/*.parquet"
+      end
+
+      # @param bucket [String]
+      # @param folder_name [String]
+      # @param partition_keys [Array<Symbol>]
+      # @param partitions [Hash]
+      # @return [Integer]
+      def destroy_partitions(bucket, folder_name, partition_keys, partitions)
+        path_parts = partition_keys.map do |key|
+          val = partitions[key]
+          val.nil? || val.to_s.empty? ? "#{key}=*" : "#{key}=#{val}"
+        end
+
+        pattern = File.join(bucket, folder_name, path_parts.join("/"))
+        folders_to_delete = Dir.glob(pattern)
+
+        return 0 if folders_to_delete.empty?
+
+        folders_to_delete.each { |folder| FileUtils.rm_rf(folder) }
+        folders_to_delete.size
+      end
+    end
+  end
+end

data/lib/data_drain/storage/s3.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+module DataDrain
+  module Storage
+    # Implementación del adaptador de almacenamiento para Amazon S3.
+    class S3 < Base
+      # Carga la extensión httpfs en DuckDB e inyecta las credenciales de AWS.
+      # @param connection [DuckDB::Connection]
+      def setup_duckdb(connection)
+        connection.query("INSTALL httpfs; LOAD httpfs;")
+        connection.query("SET s3_region='#{@config.aws_region}';")
+        connection.query("SET s3_access_key_id='#{@config.aws_access_key_id}';")
+        connection.query("SET s3_secret_access_key='#{@config.aws_secret_access_key}';")
+      end
+
+      # @param bucket [String]
+      # @param folder_name [String]
+      # @param partition_path [String, nil]
+      # @return [String]
+      def build_path(bucket, folder_name, partition_path)
+        # En S3, el base_path actúa como el nombre del bucket
+        base = File.join(bucket, folder_name)
+        base = File.join(base, partition_path) if partition_path && !partition_path.empty?
+        "s3://#{base}/**/*.parquet"
+      end
+
+      # @param bucket [String]
+      # @param folder_name [String]
+      # @param partition_keys [Array<Symbol>]
+      # @param partitions [Hash]
+      # @return [Integer]
+      def destroy_partitions(bucket, folder_name, partition_keys, partitions)
+        client = Aws::S3::Client.new(
+          region: @config.aws_region,
+          access_key_id: @config.aws_access_key_id,
+          secret_access_key: @config.aws_secret_access_key
+        )
+
+        regex_parts = partition_keys.map do |key|
+          val = partitions[key]
+          val.nil? || val.to_s.empty? ? "#{key}=[^/]+" : "#{key}=#{val}"
+        end
+        pattern_regex = Regexp.new("^#{folder_name}/#{regex_parts.join('/')}")
+
+        objects_to_delete = []
+        prefix = "#{folder_name}/"
+        first_key = partition_keys.first
+        prefix += "#{first_key}=#{partitions[first_key]}/" if partitions[first_key]
+
+        client.list_objects_v2(bucket: bucket, prefix: prefix).each do |response|
+          response.contents.each do |obj|
+            objects_to_delete << { key: obj.key } if obj.key.match?(pattern_regex)
+          end
+        end
+
+        delete_in_batches(client, bucket, objects_to_delete)
+      end
+
+      private
+
+      # @api private
+      def delete_in_batches(client, bucket, objects_to_delete)
+        return 0 if objects_to_delete.empty?
+
+        deleted_count = 0
+        objects_to_delete.each_slice(1000) do |batch|
+          client.delete_objects(bucket: bucket, delete: { objects: batch, quiet: true })
+          deleted_count += batch.size
+        end
+        deleted_count
+      end
+    end
+  end
+end

data/lib/data_drain/storage.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+require_relative "storage/base"
+require_relative "storage/local"
+require_relative "storage/s3"
+
+module DataDrain
+  # Espacio de nombres para las estrategias de almacenamiento físico.
+  module Storage
+    # Excepción lanzada cuando se intenta usar un modo de almacenamiento no registrado.
+    class InvalidAdapterError < DataDrain::Error; end
+
+    # Resuelve e instancia el adaptador de almacenamiento correspondiente
+    # basándose en la configuración actual del framework.
+    #
+    # @return [DataDrain::Storage::Base] Una instancia de Local o S3.
+    # @raise [InvalidAdapterError] Si el storage_mode no es válido.
+    def self.adapter
+      mode = DataDrain.configuration.storage_mode
+      case mode.to_sym
+      when :local
+        Local.new(DataDrain.configuration)
+      when :s3
+        S3.new(DataDrain.configuration)
+      else
+        raise InvalidAdapterError, "Storage mode '#{mode}' no está soportado."
+      end
+    end
+  end
+end

data/lib/data_drain/types/json_type.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+require "json"
+
+module DataDrain
+  module Types
+    # Tipo personalizado para ActiveModel que maneja la conversión de
+    # cadenas JSON de DuckDB hacia Hashes de Ruby.
+    class JsonType < ActiveModel::Type::Value
+      # @param value [String, Hash, Array, nil]
+      # @return [Hash, Array, String, nil]
+      def cast(value)
+        return value if value.is_a?(Hash) || value.is_a?(Array) || value.nil?
+
+        begin
+          JSON.parse(value.to_s)
+        rescue JSON::ParserError
+          value
+        end
+      end
+    end
+  end
+end

data/lib/data_drain/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module DataDrain
+  VERSION = "0.1.0"
+end

data/lib/data_drain.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require "active_model"
+require_relative "data_drain/version"
+require_relative "data_drain/errors"
+require_relative "data_drain/configuration"
+require_relative "data_drain/storage"
+require_relative "data_drain/engine"
+require_relative "data_drain/record"
+require_relative "data_drain/file_ingestor"
+
+# Registramos el tipo JSON personalizado de ActiveModel
+require_relative "data_drain/types/json_type"
+ActiveModel::Type.register(:json, DataDrain::Types::JsonType)
+
+module DataDrain
+  class << self
+    # @return [DataDrain::Configuration]
+    def configuration
+      @configuration ||= Configuration.new
+    end
+
+    # @yieldparam config [DataDrain::Configuration]
+    def configure
+      yield(configuration)
+    end
+
+    # @api private
+    def reset_configuration!
+      @configuration = Configuration.new
+    end
+  end
+end

data/sig/data_drain.rbs

@@ -0,0 +1,4 @@
+module DataDrain
+  VERSION: String
+  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+end

metadata

@@ -0,0 +1,120 @@
+--- !ruby/object:Gem::Specification
+name: data_drain
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Gabriel
+autorequire:
+bindir: exe
+cert_chain: []
+date: 2026-03-12 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activemodel
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+- !ruby/object:Gem::Dependency
+  name: aws-sdk-s3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.114'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.114'
+- !ruby/object:Gem::Dependency
+  name: duckdb
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.4'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.4'
+- !ruby/object:Gem::Dependency
+  name: pg
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.2'
+description: Extrae datos transaccionales, los archiva en un Data Lake (S3/Local)
+  en formato Parquet usando Hive Partitioning, y purga el origen de forma segura.
+email:
+- gab.edera@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".rspec"
+- ".rubocop.yml"
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- LICENSE.txt
+- README.md
+- Rakefile
+- data_drain.gemspec
+- lib/data_drain.rb
+- lib/data_drain/configuration.rb
+- lib/data_drain/engine.rb
+- lib/data_drain/errors.rb
+- lib/data_drain/file_ingestor.rb
+- lib/data_drain/record.rb
+- lib/data_drain/storage.rb
+- lib/data_drain/storage/base.rb
+- lib/data_drain/storage/local.rb
+- lib/data_drain/storage/s3.rb
+- lib/data_drain/types/json_type.rb
+- lib/data_drain/version.rb
+- sig/data_drain.rbs
+homepage: https://github.com/gedera/data_drain
+licenses: []
+metadata: {}
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.4.19
+signing_key:
+specification_version: 4
+summary: Micro-framework para drenar datos de PostgreSQL a Parquet vía DuckDB.
+test_files: []