data_drain 0.3.1 → 0.4.0

data/docs/glue-jobs-lifecycle.md

@@ -0,0 +1,159 @@
+# Glue Jobs Lifecycle
+
+Gestión completa de AWS Glue Jobs desde la gema.
+
+## Métodos
+
+### `job_exists?(job_name)` → Boolean
+
+Verifica si un job existe en Glue.
+
+```ruby
+DataDrain::GlueRunner.job_exists?("my-job")
+# => true
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza `Aws::Glue::Errors::EntityNotFoundException` si el job no existe.
+- Lanza otros errores de AWS sin atrapar.
+
+### `get_job(job_name)` → Aws::Glue::Types::Job
+
+Obtiene la configuración completa de un job.
+
+```ruby
+job = DataDrain::GlueRunner.get_job("my-job")
+job.name               # => "my-job"
+job.role               # => "arn:aws:iam::123:role/GlueRole"
+job.command            # => { name: "glueetl", python_version: "3", script_location: "s3://..." }
+job.default_arguments  # => { "--extra-files" => "s3://..." }
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza `Aws::Glue::Errors::EntityNotFoundException` si el job no existe.
+
+### `create_job(job_name, role_arn:, script_location:, ...)` → Aws::Glue::Types::Job
+
+Crea un nuevo job en Glue y retorna el job creado.
+
+**Parámetros requeridos:**
+- `job_name` (String): nombre del job
+- `role_arn` (String): ARN del IAM role de Glue
+- `script_location` (String): path S3 del script Python
+
+**Parámetros opcionales:**
+- `command_name` (String): nombre del comando (`"glueetl"`, `"pythonshell"`). Default: `"glueetl"`.
+- `default_arguments` (Hash): argumentos default del job
+- `description` (String): descripción del job
+- `timeout` (Integer): timeout en minutos. Default: `2880` (48h)
+- `max_retries` (Integer): reintentos. Default: `0`
+- `allocated_capacity` (Integer): DPU legacy. Preferir `worker_type` + `number_of_workers`
+- `worker_type` (String): `"Standard"`, `"G.1X"`, `"G.2X"`, `"G.4X"`, `"G.8X"`
+- `number_of_workers` (Integer): número de workers (requiere `worker_type`)
+- `glue_version` (String): versión de Glue (ej. `"4.0"`)
+
+```ruby
+job = DataDrain::GlueRunner.create_job(
+  "my-job",
+  role_arn: "arn:aws:iam::123:role/GlueServiceRole",
+  script_location: "s3://my-bucket/scripts/export.py",
+  default_arguments: { "--extra-files" => "s3://my-bucket/scripts/udf.py" },
+  timeout: 1440,
+  max_retries: 2,
+  worker_type: "G.1X",
+  number_of_workers: 10
+)
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza errores de AWS sin atrapar (nombre duplicado, rol inválido, etc.)
+
+### `update_job(job_name, ...)` → Aws::Glue::Types::Job
+
+Actualiza un job existente y retorna el job actualizado.
+
+Mismos parámetros que `create_job`, todos opcionales. Solo los parámetros provistos se actualizan.
+
+```ruby
+job = DataDrain::GlueRunner.update_job(
+  "my-job",
+  script_location: "s3://my-bucket/scripts/export-v2.py",
+  timeout: 720
+)
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza `Aws::Glue::Errors::EntityNotFoundException` si el job no existe.
+
+### `delete_job(job_name)` → nil
+
+Elimina un job de Glue.
+
+```ruby
+DataDrain::GlueRunner.delete_job("my-job")
+# => nil
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza `Aws::Glue::Errors::EntityNotFoundException` si el job no existe.
+
+### `ensure_job(job_name, role_arn:, script_location:, ...)` → Aws::Glue::Types::Job
+
+Crea o actualiza un job de forma idempotente.
+
+- Si el job existe → `update_job`
+- Si el job no existe → `create_job`
+
+```ruby
+job = DataDrain::GlueRunner.ensure_job(
+  "my-job",
+  role_arn: "arn:aws:iam::123:role/GlueServiceRole",
+  script_location: "s3://my-bucket/scripts/export.py",
+  timeout: 1440
+)
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza errores de AWS sin atrapar.
+
+### `run_and_wait(job_name, arguments = {}, ...)` → Boolean
+
+Ejecuta un job existente y espera a que complete.
+
+```ruby
+DataDrain::GlueRunner.run_and_wait(
+  "my-job",
+  { "--start_date" => "2025-01-01", "--end_date" => "2025-02-01" },
+  polling_interval: 60,
+  max_wait_seconds: 7200
+)
+# => true (SUCCEEDED)
+```
+
+- Lanza `RuntimeError` si el job falla (`FAILED`, `STOPPED`, `TIMEOUT`).
+- Lanza `DataDrain::Error` si `max_wait_seconds` excede.
+
+## Convenciones de nombres
+
+AWS Glue permite: letras (`a-zA-Z`), números (`0-9`), guiones (`-`). No permite guiones bajos ni espacios.
+
+```ruby
+# Válido
+DataDrain::GlueRunner.job_exists?("my-export-job-v2")
+
+# Inválido — lanza ConfigurationError
+DataDrain::GlueRunner.job_exists?("my_export_job")
+# DataDrain::ConfigurationError: job_name 'my_export_job' no es un nombre válido para Glue Job
+```
+
+## Eventos de telemetría
+
+| Evento | Nivel | Descripción |
+|--------|-------|-------------|
+| `glue_runner.start` | INFO | Antes de `start_job_run` |
+| `glue_runner.job_exists` | INFO | Job encontrado en `ensure_job` |
+| `glue_runner.job_created` | INFO | Job creado en `ensure_job` |
+| `glue_runner.polling` | INFO | Chequeo de estado durante `run_and_wait` |
+| `glue_runner.complete` | INFO | Job terminó `SUCCEEDED` |
+| `glue_runner.failed` | ERROR | Job falló con `FAILED\|STOPPED\|TIMEOUT` |
+| `glue_runner.timeout` | ERROR | `max_wait_seconds` excedido |

data/lib/data_drain/engine.rb

@@ -159,7 +159,7 @@ module DataDrain
     # @api private
     # @return [Integer]
     def get_postgres_count
-      pg_sql = "SELECT count() AS row_count FROM public.#{@table_name} WHERE #{base_where_sql}"
+      pg_sql = "SELECT COUNT(*) AS row_count FROM public.#{@table_name} WHERE #{base_where_sql}"
       pg_sql = pg_sql.gsub("'", "''")
       query = "SELECT row_count FROM postgres_query('pg_source', '#{pg_sql}')"
       @duckdb.query(query).first.first
@@ -204,7 +204,7 @@ module DataDrain
 
       begin
         query = <<~SQL
-          SELECT count()
+          SELECT COUNT(*)
           FROM read_parquet('#{archive_path}')
           WHERE #{base_where_sql}
         SQL

data/lib/data_drain/file_ingestor.rb

@@ -82,7 +82,7 @@ module DataDrain
 
     # @api private
     def step_count_source
-      source_count = timed(:source_query) { @duckdb.query("SELECT count() FROM #{@reader_function}").first.first }
+      source_count = timed(:source_query) { @duckdb.query("SELECT COUNT(*) FROM #{@reader_function}").first.first }
       safe_log(:info, "file_ingestor.count", {
                  source_path: @source_path,
                  count: source_count,

data/lib/data_drain/glue_runner.rb

@@ -19,10 +19,109 @@ module DataDrain
     # @return [Boolean] true si el Job terminó exitosamente (SUCCEEDED).
     # @raise [DataDrain::Error] si max_wait_seconds excede antes de SUCCEEDED.
     # @raise [RuntimeError] si el Job falla o se detiene.
+    def self.client
+      @client ||= Aws::Glue::Client.new(region: DataDrain.configuration.aws_region)
+    end
+
+    class << self
+      attr_writer :client
+    end
+
+    def self.job_exists?(job_name)
+      DataDrain::Validations.validate_glue_name!(:job_name, job_name)
+      get_job(job_name)
+      true
+    rescue Aws::Glue::Errors::EntityNotFoundException
+      false
+    end
+
+    def self.get_job(job_name)
+      DataDrain::Validations.validate_glue_name!(:job_name, job_name)
+      client.get_job(job_name: job_name).job
+    end
+
+    def self.create_job(job_name, role_arn:, script_location:, command_name: "glueetl",
+                        default_arguments: {}, description: nil, worker_type: nil, number_of_workers: nil,
+                        timeout: 2880, max_retries: 0, allocated_capacity: nil, glue_version: nil)
+      DataDrain::Validations.validate_glue_name!(:job_name, job_name)
+      opts = {
+        name: job_name,
+        role: role_arn,
+        command: {
+          name: command_name,
+          python_version: "3",
+          script_location: script_location
+        }
+      }
+      opts[:default_arguments] = default_arguments unless default_arguments.empty?
+      opts[:description] = description if description
+      opts[:timeout] = timeout if timeout
+      opts[:max_retries] = max_retries if max_retries
+      opts[:allocated_capacity] = allocated_capacity if allocated_capacity
+      opts[:worker_type] = worker_type if worker_type
+      opts[:number_of_workers] = number_of_workers if number_of_workers
+      opts[:glue_version] = glue_version if glue_version
+
+      client.create_job(**opts)
+      get_job(job_name)
+    end
+
+    def self.update_job(job_name, role_arn: nil, command_name: nil, script_location: nil,
+                        default_arguments: nil, description: nil, worker_type: nil,
+                        number_of_workers: nil, timeout: nil, max_retries: nil, allocated_capacity: nil,
+                        glue_version: nil)
+      DataDrain::Validations.validate_glue_name!(:job_name, job_name)
+      job_update = {}
+      job_update[:role] = role_arn if role_arn
+      if command_name && script_location
+        job_update[:command] =
+          { name: command_name, python_version: "3", script_location: script_location }
+      end
+      job_update[:default_arguments] = default_arguments if default_arguments
+      job_update[:description] = description if description
+      job_update[:timeout] = timeout if timeout
+      job_update[:max_retries] = max_retries if max_retries
+      job_update[:allocated_capacity] = allocated_capacity if allocated_capacity
+      job_update[:worker_type] = worker_type if worker_type
+      job_update[:number_of_workers] = number_of_workers if number_of_workers
+      job_update[:glue_version] = glue_version if glue_version
+
+      client.update_job(job_name: job_name, job_update: job_update)
+      get_job(job_name)
+    end
+
+    def self.delete_job(job_name)
+      DataDrain::Validations.validate_glue_name!(:job_name, job_name)
+      client.delete_job(job_name: job_name)
+      nil
+    end
+
+    def self.ensure_job(job_name, role_arn:, script_location:, command_name: "glueetl",
+                        default_arguments: {}, description: nil, worker_type: nil,
+                        number_of_workers: nil, timeout: 2880, max_retries: 0,
+                        allocated_capacity: nil, glue_version: nil)
+      if job_exists?(job_name)
+        safe_log(:info, "glue_runner.job_exists", { job: job_name })
+        update_job(job_name, role_arn: role_arn, command_name: command_name,
+                             script_location: script_location, default_arguments: default_arguments,
+                             description: description, worker_type: worker_type,
+                             number_of_workers: number_of_workers, timeout: timeout,
+                             max_retries: max_retries, allocated_capacity: allocated_capacity,
+                             glue_version: glue_version)
+      else
+        safe_log(:info, "glue_runner.job_created", { job: job_name })
+        create_job(job_name, role_arn: role_arn, script_location: script_location,
+                             command_name: command_name, default_arguments: default_arguments,
+                             description: description, worker_type: worker_type,
+                             number_of_workers: number_of_workers, timeout: timeout,
+                             max_retries: max_retries, allocated_capacity: allocated_capacity,
+                             glue_version: glue_version)
+      end
+    end
+
     def self.run_and_wait(job_name, arguments = {}, polling_interval: 30, max_wait_seconds: nil)
       config = DataDrain.configuration
       config.validate!
-      client = Aws::Glue::Client.new(region: config.aws_region)
       start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
 
       @logger = config.logger

data/lib/data_drain/validations.rb

@@ -6,9 +6,17 @@ module DataDrain
     # Regex que valida identificadores SQL (tablas, columnas, etc.).
     # Permite letras, guiones bajos y números (no al inicio).
     IDENTIFIER_REGEX = /\A[a-zA-Z_][a-zA-Z0-9_]*\z/
+    GLUE_NAME_REGEX = /\A[a-zA-Z0-9][a-zA-Z0-9-]*\z/
 
     module_function
 
+    def validate_glue_name!(name, value)
+      return if GLUE_NAME_REGEX.match?(value.to_s)
+
+      raise DataDrain::ConfigurationError,
+            "#{name} '#{value}' no es un nombre válido para Glue Job (usa solo letras, números y guiones)"
+    end
+
     def validate_identifier!(name, value)
       return if IDENTIFIER_REGEX.match?(value.to_s)
 

data/lib/data_drain/version.rb

@@ -2,5 +2,5 @@
 
 module DataDrain
   # @return [String] versión semver de la gema
-  VERSION = "0.3.1"
+  VERSION = "0.4.0"
 end

data/skill/references/eventos-telemetria.md

@@ -115,6 +115,14 @@ Catálogo completo de eventos KV emitidos por DataDrain. Formato Wispro-Observab
 **Nivel:** INFO. Emite antes de `start_job_run`.
 **Campos:** `job`.
 
+### `glue_runner.job_exists`
+**Nivel:** INFO. Emite en `ensure_job` cuando el job ya existe y se actualiza.
+**Campos:** `job`.
+
+### `glue_runner.job_created`
+**Nivel:** INFO. Emite en `ensure_job` cuando el job se crea.
+**Campos:** `job`.
+
 ### `glue_runner.polling`
 **Nivel:** INFO. Emite cada chequeo de estado mientras Job no terminó.
 **Campos:** `job`, `run_id`, `status`, `next_check_in_s`.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_drain
 version: !ruby/object:Gem::Version
-  version: 0.3.1
+  version: 0.4.0
 platform: ruby
 authors:
 - Gabriel
@@ -102,9 +102,12 @@ files:
 - docs/execution/archive/v0.2.0.md
 - docs/execution/archive/v0.3.0-OBSERVACIONES.md
 - docs/execution/archive/v0.3.0.md
+- docs/execution/archive/v0.3.1-OBSERVACIONES.md
+- docs/execution/archive/v0.3.1.md
 - docs/execution/v0.2.2.md
-- docs/execution/v0.3.1-OBSERVACIONES.md
-- docs/execution/v0.3.1.md
+- docs/execution/v0.4.0-OBSERVACIONES.md
+- docs/execution/v0.4.0.md
+- docs/glue-jobs-lifecycle.md
 - docs/glue_pyspark_example.py
 - lib/data_drain.rb
 - lib/data_drain/configuration.rb