data_drain 0.3.2 → 0.5.0

data/docs/glue-jobs-lifecycle.md

@@ -0,0 +1,330 @@
+# Glue Jobs Lifecycle
+
+Gestión completa de AWS Glue Jobs desde la gema.
+
+## Métodos
+
+### `job_exists?(job_name)` → Boolean
+
+Verifica si un job existe en Glue.
+
+```ruby
+DataDrain::GlueRunner.job_exists?("my-job")
+# => true
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza `Aws::Glue::Errors::EntityNotFoundException` si el job no existe.
+- Lanza otros errores de AWS sin atrapar.
+
+### `get_job(job_name)` → Aws::Glue::Types::Job
+
+Obtiene la configuración completa de un job.
+
+```ruby
+job = DataDrain::GlueRunner.get_job("my-job")
+job.name               # => "my-job"
+job.role               # => "arn:aws:iam::123:role/GlueRole"
+job.command            # => { name: "glueetl", python_version: "3", script_location: "s3://..." }
+job.default_arguments  # => { "--extra-files" => "s3://..." }
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza `Aws::Glue::Errors::EntityNotFoundException` si el job no existe.
+
+### Subir scripts locales (v0.5.0+)
+
+Desde v0.5.0 la gema puede subir scripts PySpark a S3 automáticamente.
+
+```ruby
+# Opción moderna: script local subido por la gema
+DataDrain::GlueRunner.create_job(
+  "my-job",
+  script_path: "scripts/glue/export.py",  # local
+  script_bucket: "my-bucket",
+  script_folder: "scripts",
+  role_arn: "arn:aws:iam::123:role/GlueRole"
+)
+# → Sube scripts/glue/export.py a s3://my-bucket/scripts/export.py
+# → Crea el job
+```
+
+**Parámetros para upload:**
+- `script_path` (String): ruta local al script Python.
+- `script_bucket` (String): bucket S3 destino. **Requerido si se usa `script_path`.**
+- `script_folder` (String): folder dentro del bucket. Default: `"scripts"`.
+- `script_filename` (String, nil): override del nombre en S3. Default: basename del archivo.
+
+**`script_location` vs `script_path`:**
+- `script_location:` → comportamiento anterior, no hay upload.
+- `script_path:` + `script_bucket:` → la gema sube a S3 primero, luego crea el Job.
+- Si se pasan ambos → `DataDrain::ConfigurationError`.
+- Si no se pasa ninguno → `ArgumentError`.
+
+**Importante:** el upload **sobrescribe** cualquier archivo existente en el mismo path.
+No es idempotente en sentido estricto. Usar `script_filename:` con hash o timestamp
+si necesitás versionado.
+
+### Concurrencia (limitación conocida)
+
+No hay lock distribuido. Si dos procesos llaman `upload_script` con el mismo destino
+simultáneamente, el último `put_object` en llegar a S3 gana. Para scripts PySpark
+esto es típicamente bajo riesgo (scripts son pequeños, rara vez hay writes
+concurrentes al mismo path).
+
+### Permisos IAM mínimos
+
+El IAM role/user que ejecuta `upload_script` necesita:
+
+```json
+{
+  "Effect": "Allow",
+  "Action": ["s3:PutObject"],
+  "Resource": "arn:aws:s3:::my-bucket/scripts/*"
+}
+```
+
+Para usar con `create_job`/`ensure_job` también se necesitan los permisos de Glue
+(ver sección "Permisos Glue" al inicio de este documento) + permiso para que el
+IAM role del Glue Job pueda leer el script:
+
+```json
+{
+  "Effect": "Allow",
+  "Action": ["s3:GetObject"],
+  "Resource": "arn:aws:s3:::my-bucket/scripts/*"
+}
+```
+
+(Este último en el role del Glue Job, no en el role de la aplicación Ruby.)
+
+### API standalone: `upload_script`
+
+Para casos donde solo querés subir (sin crear Job):
+
+```ruby
+s3_path = DataDrain::GlueRunner.upload_script(
+  local_path: "scripts/glue/export.py",
+  bucket: "my-bucket",
+  folder: "scripts"
+)
+# => "s3://my-bucket/scripts/export.py"
+```
+
+Requiere `storage_mode = :s3`.
+
+### `create_job(job_name, role_arn:, ...)` → Aws::Glue::Types::Job
+
+Crea un nuevo job en Glue y retorna el job creado.
+
+**Parámetros requeridos:**
+- `job_name` (String): nombre del job
+- `role_arn` (String): ARN del IAM role de Glue
+
+**Parámetros de script (mutuamente excluyentes):**
+- `script_location` (String): path S3 del script Python (comportamiento anterior)
+- `script_path` + `script_bucket` (String): upload local a S3 primero (v0.5.0+)
+
+**Parámetros opcionales:**
+- `script_folder` (String): folder S3. Default: `"scripts"`.
+- `script_filename` (String, nil): override del nombre en S3.
+- `command_name` (String): nombre del comando (`"glueetl"`, `"pythonshell"`). Default: `"glueetl"`.
+- `default_arguments` (Hash): argumentos default del job
+- `description` (String): descripción del job
+- `timeout` (Integer): timeout en minutos. Default: `2880` (48h)
+- `max_retries` (Integer): reintentos. Default: `0`
+- `allocated_capacity` (Integer): DPU legacy. Preferir `worker_type` + `number_of_workers`
+- `worker_type` (String): `"Standard"`, `"G.1X"`, `"G.2X"`, `"G.4X"`, `"G.8X"`
+- `number_of_workers` (Integer): número de workers (requiere `worker_type`)
+- `glue_version` (String): versión de Glue (ej. `"4.0"`)
+
+```ruby
+job = DataDrain::GlueRunner.create_job(
+  "my-job",
+  role_arn: "arn:aws:iam::123:role/GlueServiceRole",
+  script_location: "s3://my-bucket/scripts/export.py",
+  default_arguments: { "--extra-files" => "s3://my-bucket/scripts/udf.py" },
+  timeout: 1440,
+  max_retries: 2,
+  worker_type: "G.1X",
+  number_of_workers: 10
+)
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza errores de AWS sin atrapar (nombre duplicado, rol inválido, etc.)
+
+### `update_job(job_name, ...)` → Aws::Glue::Types::Job
+
+Actualiza un job existente y retorna el job actualizado.
+
+Mismos parámetros que `create_job`, todos opcionales. Solo los parámetros provistos se actualizan.
+
+```ruby
+job = DataDrain::GlueRunner.update_job(
+  "my-job",
+  script_location: "s3://my-bucket/scripts/export-v2.py",
+  timeout: 720
+)
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza `Aws::Glue::Errors::EntityNotFoundException` si el job no existe.
+
+### `delete_job(job_name)` → Boolean
+
+Elimina un job de Glue. Es idempotente.
+
+```ruby
+DataDrain::GlueRunner.delete_job("my-job")
+# => true (job existía y fue eliminado)
+
+DataDrain::GlueRunner.delete_job("nonexistent")
+# => false (job no existía)
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza otros errores de AWS sin atrapar.
+
+### `ensure_job(job_name, role_arn:, ...)` → Aws::Glue::Types::Job
+
+Crea o actualiza un job de forma idempotente con diffing de configuración.
+
+- Si el job no existe → `create_job`
+- Si el job existe con config diferente → `update_job`
+- Si el job existe con config idéntica → no-op, retorna el job actual (`:unchanged`)
+
+```ruby
+job = DataDrain::GlueRunner.ensure_job(
+  "my-job",
+  role_arn: "arn:aws:iam::123:role/GlueServiceRole",
+  script_location: "s3://my-bucket/scripts/export.py",
+  timeout: 1440
+)
+```
+
+También soporta upload de script local (v0.5.0+):
+
+```ruby
+job = DataDrain::GlueRunner.ensure_job(
+  "my-job",
+  script_path: "scripts/glue/export.py",
+  script_bucket: "my-bucket",
+  script_folder: "scripts",
+  role_arn: "arn:aws:iam::123:role/GlueServiceRole",
+  timeout: 1440
+)
+```
+
+- Lanza `DataDrain::ConfigurationError` si `job_name` es inválido.
+- Lanza errores de AWS sin atrapar.
+
+### `run_and_wait(job_name, arguments = {}, ...)` → Boolean
+
+Ejecuta un job existente y espera a que complete.
+
+```ruby
+DataDrain::GlueRunner.run_and_wait(
+  "my-job",
+  { "--start_date" => "2025-01-01", "--end_date" => "2025-02-01" },
+  polling_interval: 60,
+  max_wait_seconds: 7200
+)
+# => true (SUCCEEDED)
+```
+
+- Lanza `RuntimeError` si el job falla (`FAILED`, `STOPPED`, `TIMEOUT`).
+- Lanza `DataDrain::Error` si `max_wait_seconds` excede.
+
+## Patrón completo: ensure_job + run_and_wait + PySpark
+
+Workflow end-to-end para archivar y purgar tablas PostgreSQL usando AWS Glue:
+
+```ruby
+# 1. Asegurar que el Glue Job existe con la config deseada (idempotente)
+DataDrain::GlueRunner.ensure_job(
+  "my-export-job",
+  role_arn: "arn:aws:iam::123:role/GlueServiceRole",
+  script_location: "s3://my-bucket/scripts/glue_pyspark_export.py",
+  glue_version: "4.0",
+  worker_type: "G.1X",
+  number_of_workers: 10,
+  timeout: 1440
+)
+
+# 2. Ejecutar el export (delegado a Glue Spark distribuido)
+DataDrain::GlueRunner.run_and_wait(
+  "my-export-job",
+  {
+    "--start_date"   => start_date.to_fs(:db),
+    "--end_date"     => end_date.to_fs(:db),
+    "--s3_bucket"    => bucket,
+    "--s3_folder"    => table,
+    "--db_url"       => "jdbc:postgresql://#{host}:#{port}/#{db}",
+    "--db_user"      => db_user,
+    "--db_password"  => db_password,
+    "--db_table"     => table,
+    "--partition_by" => partition_keys.join(",")
+  },
+  polling_interval: 60,
+  max_wait_seconds: 7200
+)
+
+# 3. Verificar integridad y purgar Postgres (DataDrain solo lee Parquet)
+DataDrain::Engine.new(
+  bucket: bucket,
+  folder_name: table,
+  start_date: start_date,
+  end_date: end_date,
+  table_name: table,
+  partition_keys: partition_keys,
+  skip_export: true  # export ya lo hizo Glue
+).call
+```
+
+### Prerequisites
+
+1. **Subir el script a S3:**
+   ```bash
+   aws s3 cp glue_pyspark_export.py s3://my-bucket/scripts/
+   ```
+
+2. **IAM Role** con permisos para: Glue, S3 (lectura del script + escritura del bucket destino), RDS/Postgres (vía JDBC)
+
+3. **Script PySpark** en `s3://my-bucket/scripts/glue_pyspark_export.py` (ver [ejemplo](../glue_pyspark_example.py))
+
+## Convenciones de nombres
+
+AWS Glue permite: letras (`a-zA-Z`), números (`0-9`), guiones (`-`), guiones bajos (`_`). No permite espacios ni caracteres especiales.
+
+```ruby
+# Válido
+DataDrain::GlueRunner.job_exists?("my-export-job-v2")
+DataDrain::GlueRunner.job_exists?("my_export_job")
+
+# Inválido — lanza ConfigurationError
+DataDrain::GlueRunner.job_exists?("-starts-with-dash")
+# DataDrain::ConfigurationError: job_name '-starts-with-dash' no es un nombre válido para Glue Job
+```
+
+## Eventos de telemetría
+
+| Evento | Nivel | Descripción |
+|--------|-------|-------------|
+| `glue_runner.start` | INFO | Antes de `start_job_run` |
+| `glue_runner.job_create` | INFO | Job creado exitosamente |
+| `glue_runner.job_update` | INFO | Job actualizado (incluye `changed_fields`) |
+| `glue_runner.job_delete` | INFO | Job eliminado exitosamente |
+| `glue_runner.job_delete_skipped` | INFO | `delete_job` sobre job inexistente |
+| `glue_runner.job_exists` | INFO | Job encontrado en `ensure_job` (y difiere) |
+| `glue_runner.job_created` | INFO | Job creado en `ensure_job` |
+| `glue_runner.job_unchanged` | INFO | Job existe con config idéntica en `ensure_job` |
+| `glue_runner.job_create_error` | ERROR | Error en `create_job` |
+| `glue_runner.job_update_error` | ERROR | Error en `update_job` |
+| `glue_runner.job_delete_error` | ERROR | Error en `delete_job` |
+| `glue_runner.script_uploaded` | INFO | Script subido a S3 (v0.5.0+) |
+| `glue_runner.script_upload_error` | ERROR | Error al subir script a S3 (v0.5.0+) |
+| `glue_runner.polling` | INFO | Chequeo de estado durante `run_and_wait` |
+| `glue_runner.complete` | INFO | Job terminó `SUCCEEDED` |
+| `glue_runner.failed` | ERROR | Job falló con `FAILED\|STOPPED\|TIMEOUT` |
+| `glue_runner.timeout` | ERROR | `max_wait_seconds` excedido |

data/docs/glue_pyspark_example.py

@@ -1,11 +1,29 @@
 """
 Script de AWS Glue (PySpark) compatible con DataDrain::GlueRunner.
 
-Crear el Job en la consola de AWS Glue (Spark 4.0+) y usar este script como base.
-Argumentos requeridos: JOB_NAME, start_date, end_date, s3_bucket, s3_folder,
+Para crear el Glue Job programmatically (en vez de la consola):
+
+    # Opcion moderna: script local subido por la gema (v0.5.0+)
+    DataDrain::GlueRunner.ensure_job(
+      "my-export-job",
+      script_path: "docs/glue_pyspark_example.py",
+      script_bucket: "my-bucket",
+      script_folder: "scripts",
+      role_arn: "arn:aws:iam::123:role/GlueServiceRole",
+      worker_type: "G.1X",
+      number_of_workers: 10,
+      timeout: 1440
+    )
+    # -> Sube este archivo a s3://my-bucket/scripts/glue_pyspark_example.py
+    # -> Crea el Job apuntando a ese path
+
+    # Ejecutar
+    DataDrain::GlueRunner.run_and_wait("my-export-job", { "--start_date" => "2025-01-01", ... })
+
+Argumentos requeridos del job: JOB_NAME, start_date, end_date, s3_bucket, s3_folder,
 db_url, db_user, db_password, db_table, partition_by.
 
-Personalizar la sección de columnas derivadas según las partition_keys de cada tabla.
+Personalizar la seccion de columnas derivadas segun las partition_keys de cada tabla.
 """
 
 import sys
@@ -15,27 +33,38 @@ from awsglue.context import GlueContext
 from awsglue.job import Job
 from pyspark.sql.functions import col, year, month
 
-args = getResolvedOptions(sys.argv, [
-    'JOB_NAME', 'start_date', 'end_date', 's3_bucket', 's3_folder',
-    'db_url', 'db_user', 'db_password', 'db_table', 'partition_by'
-])
+args = getResolvedOptions(
+    sys.argv,
+    [
+        "JOB_NAME",
+        "start_date",
+        "end_date",
+        "s3_bucket",
+        "s3_folder",
+        "db_url",
+        "db_user",
+        "db_password",
+        "db_table",
+        "partition_by",
+    ],
+)
 
 sc = SparkContext()
 glueContext = GlueContext(sc)
 spark = glueContext.spark_session
 job = Job(glueContext)
-job.init(args['JOB_NAME'], args)
+job.init(args["JOB_NAME"], args)
 
 options = {
-    "url": args['db_url'],
-    "dbtable": args['db_table'],
-    "user": args['db_user'],
-    "password": args['db_password'],
+    "url": args["db_url"],
+    "dbtable": args["db_table"],
+    "user": args["db_user"],
+    "password": args["db_password"],
     "sampleQuery": (
         f"SELECT * FROM {args['db_table']} "
         f"WHERE created_at >= '{args['start_date']}' "
         f"AND created_at < '{args['end_date']}'"
-    )
+    ),
 }
 
 df = spark.read.format("jdbc").options(**options).load()
@@ -43,18 +72,19 @@ df = spark.read.format("jdbc").options(**options).load()
 # Agregar columnas derivadas necesarias para las particiones.
 # isp_id ya existe en la tabla fuente — solo agregar las que se calculan.
 # Personalizar esta seccion segun las partition_keys de cada tabla.
-df_final = (
-    df.withColumn("year", year(col("created_at")))
-      .withColumn("month", month(col("created_at")))
+df_final = df.withColumn("year", year(col("created_at"))).withColumn(
+    "month", month(col("created_at"))
 )
 
 output_path = f"s3://{args['s3_bucket']}/{args['s3_folder']}/"
-partitions = args['partition_by'].split(",")
+partitions = args["partition_by"].split(",")
 
-(df_final.write.mode("overwrite")
+(
+    df_final.write.mode("overwrite")
     .partitionBy(*partitions)
     .format("parquet")
     .option("compression", "zstd")
-    .save(output_path))
+    .save(output_path)
+)
 
 job.commit()

data/lib/data_drain/glue_runner.rb

@@ -19,10 +19,245 @@ module DataDrain
     # @return [Boolean] true si el Job terminó exitosamente (SUCCEEDED).
     # @raise [DataDrain::Error] si max_wait_seconds excede antes de SUCCEEDED.
     # @raise [RuntimeError] si el Job falla o se detiene.
+    def self.client
+      @client ||= Aws::Glue::Client.new(region: DataDrain.configuration.aws_region)
+    end
+
+    class << self
+      attr_writer :client
+    end
+
+    def self.job_exists?(job_name)
+      DataDrain::Validations.validate_glue_name!(:job_name, job_name)
+      get_job(job_name)
+      true
+    rescue Aws::Glue::Errors::EntityNotFoundException
+      false
+    end
+
+    def self.get_job(job_name)
+      DataDrain::Validations.validate_glue_name!(:job_name, job_name)
+      client.get_job(job_name: job_name).job
+    end
+
+    def self.create_job(job_name, role_arn:, script_location: nil, script_path: nil,
+                        script_bucket: nil, script_folder: "scripts", script_filename: nil,
+                        command_name: "glueetl", default_arguments: {}, description: nil,
+                        worker_type: nil, number_of_workers: nil, timeout: 2880,
+                        max_retries: 0, allocated_capacity: nil, glue_version: nil)
+      @logger = DataDrain.configuration.logger
+      DataDrain::Validations.validate_glue_name!(:job_name, job_name)
+
+      final_script_location = resolve_script_location(
+        script_location: script_location,
+        script_path: script_path,
+        script_bucket: script_bucket,
+        script_folder: script_folder,
+        script_filename: script_filename
+      )
+
+      opts = {
+        name: job_name,
+        role: role_arn,
+        command: {
+          name: command_name,
+          python_version: "3",
+          script_location: final_script_location
+        }
+      }
+      opts[:default_arguments] = default_arguments unless default_arguments.empty?
+      opts[:description] = description if description
+      opts[:timeout] = timeout if timeout
+      opts[:max_retries] = max_retries if max_retries
+      opts[:allocated_capacity] = allocated_capacity if allocated_capacity
+      opts[:worker_type] = worker_type if worker_type
+      opts[:number_of_workers] = number_of_workers if number_of_workers
+      opts[:glue_version] = glue_version if glue_version
+
+      client.create_job(**opts)
+      safe_log(:info, "glue_runner.job_create", {
+                 job: job_name,
+                 glue_version: glue_version,
+                 worker_type: worker_type,
+                 number_of_workers: number_of_workers
+               })
+      get_job(job_name)
+    rescue Aws::Glue::Errors::ServiceError => e
+      safe_log(:error, "glue_runner.job_create_error",
+               { job: job_name }.merge(exception_metadata(e)))
+      raise
+    end
+
+    def self.update_job(job_name, role_arn: nil, command_name: nil, script_location: nil,
+                        default_arguments: nil, description: nil, worker_type: nil,
+                        number_of_workers: nil, timeout: nil, max_retries: nil, allocated_capacity: nil,
+                        glue_version: nil)
+      @logger = DataDrain.configuration.logger
+      DataDrain::Validations.validate_glue_name!(:job_name, job_name)
+      job_update = {}
+      job_update[:role] = role_arn if role_arn
+      if command_name && script_location
+        job_update[:command] =
+          { name: command_name, python_version: "3", script_location: script_location }
+      end
+      job_update[:default_arguments] = default_arguments if default_arguments
+      job_update[:description] = description if description
+      job_update[:timeout] = timeout if timeout
+      job_update[:max_retries] = max_retries if max_retries
+      job_update[:allocated_capacity] = allocated_capacity if allocated_capacity
+      job_update[:worker_type] = worker_type if worker_type
+      job_update[:number_of_workers] = number_of_workers if number_of_workers
+      job_update[:glue_version] = glue_version if glue_version
+
+      client.update_job(job_name: job_name, job_update: job_update)
+      safe_log(:info, "glue_runner.job_update", {
+                 job: job_name,
+                 changed_fields: job_update.keys.map(&:to_s)
+               })
+      get_job(job_name)
+    rescue Aws::Glue::Errors::ServiceError => e
+      safe_log(:error, "glue_runner.job_update_error",
+               { job: job_name }.merge(exception_metadata(e)))
+      raise
+    end
+
+    def self.delete_job(job_name)
+      @logger = DataDrain.configuration.logger
+      DataDrain::Validations.validate_glue_name!(:job_name, job_name)
+      client.delete_job(job_name: job_name)
+      safe_log(:info, "glue_runner.job_delete", { job: job_name })
+      true
+    rescue Aws::Glue::Errors::EntityNotFoundException
+      safe_log(:info, "glue_runner.job_delete_skipped", { job: job_name, reason: "not_found" })
+      false
+    rescue Aws::Glue::Errors::ServiceError => e
+      safe_log(:error, "glue_runner.job_delete_error",
+               { job: job_name }.merge(exception_metadata(e)))
+      raise
+    end
+
+    def self.ensure_job(job_name, role_arn:, script_location: nil, script_path: nil,
+                        script_bucket: nil, script_folder: "scripts", script_filename: nil,
+                        command_name: "glueetl", default_arguments: {}, description: nil,
+                        worker_type: nil, number_of_workers: nil, timeout: 2880,
+                        max_retries: 0, allocated_capacity: nil, glue_version: nil)
+      @logger = DataDrain.configuration.logger
+
+      final_script_location = resolve_script_location(
+        script_location: script_location,
+        script_path: script_path,
+        script_bucket: script_bucket,
+        script_folder: script_folder,
+        script_filename: script_filename
+      )
+
+      if job_exists?(job_name)
+        current = get_job(job_name)
+        desired = {
+          role: role_arn,
+          command_name: command_name,
+          script_location: final_script_location,
+          default_arguments: default_arguments,
+          description: description,
+          worker_type: worker_type,
+          number_of_workers: number_of_workers,
+          timeout: timeout,
+          max_retries: max_retries,
+          glue_version: glue_version
+        }
+        changed = changed_fields(desired, current)
+        if changed.empty?
+          safe_log(:info, "glue_runner.job_unchanged", { job: job_name })
+          current
+        else
+          safe_log(:info, "glue_runner.job_exists", { job: job_name })
+          update_job(job_name, role_arn: role_arn, command_name: command_name,
+                               script_location: final_script_location, default_arguments: default_arguments,
+                               description: description, worker_type: worker_type,
+                               number_of_workers: number_of_workers, timeout: timeout,
+                               max_retries: max_retries, allocated_capacity: allocated_capacity,
+                               glue_version: glue_version)
+        end
+      else
+        safe_log(:info, "glue_runner.job_created", { job: job_name })
+        create_job(job_name, role_arn: role_arn, script_location: final_script_location,
+                             command_name: command_name, default_arguments: default_arguments,
+                             description: description, worker_type: worker_type,
+                             number_of_workers: number_of_workers, timeout: timeout,
+                             max_retries: max_retries, allocated_capacity: allocated_capacity,
+                             glue_version: glue_version)
+      end
+    end
+
+    def self.changed_fields(desired, current)
+      changed = []
+      changed << :role if current.role != desired[:role]
+      changed << :command if current.command.name != desired[:command_name] ||
+                             current.command.script_location != desired[:script_location]
+      changed << :default_arguments if current.default_arguments != desired[:default_arguments]
+      changed << :description if current.description != desired[:description]
+      changed << :worker_type if current.worker_type != desired[:worker_type]
+      changed << :number_of_workers if current.number_of_workers != desired[:number_of_workers]
+      changed << :timeout if current.timeout != desired[:timeout]
+      changed << :max_retries if current.max_retries != desired[:max_retries]
+      changed << :glue_version if current.glue_version != desired[:glue_version]
+      changed
+    end
+    private_class_method :changed_fields
+
+    def self.resolve_script_location(script_location:, script_path:, script_bucket:, script_folder:, script_filename:)
+      both_set = script_location && script_path
+      raise DataDrain::ConfigurationError, "provee script_location o script_path, no ambos" if both_set
+
+      return script_location if script_location
+      raise ArgumentError, "script_location o script_path es requerido" unless script_path
+      raise DataDrain::ConfigurationError, "script_path requiere script_bucket" unless script_bucket
+
+      upload_script(
+        local_path: script_path,
+        bucket: script_bucket,
+        folder: script_folder,
+        filename: script_filename
+      )
+    end
+    private_class_method :resolve_script_location
+
+    def self.upload_script(local_path:, bucket:, folder: "scripts", filename: nil)
+      @logger = DataDrain.configuration.logger
+
+      unless File.exist?(local_path)
+        raise DataDrain::ConfigurationError,
+              "Script local '#{local_path}' no existe"
+      end
+
+      actual_filename = filename || File.basename(local_path)
+      s3_key = "#{folder.chomp("/")}/#{actual_filename}"
+      bytes = File.size(local_path)
+
+      adapter = DataDrain::Storage.adapter
+      unless adapter.is_a?(DataDrain::Storage::S3)
+        raise DataDrain::ConfigurationError,
+              "upload_script requiere storage_mode = :s3, actual: #{DataDrain.configuration.storage_mode}"
+      end
+
+      s3_path = adapter.upload_file(local_path, bucket, s3_key, content_type: "text/x-python")
+
+      safe_log(:info, "glue_runner.script_uploaded", {
+                 local_path: local_path,
+                 s3_path: s3_path,
+                 bytes: bytes
+               })
+
+      s3_path
+    rescue Aws::S3::Errors::ServiceError => e
+      safe_log(:error, "glue_runner.script_upload_error",
+               { local_path: local_path, bucket: bucket }.merge(exception_metadata(e)))
+      raise
+    end
+
     def self.run_and_wait(job_name, arguments = {}, polling_interval: 30, max_wait_seconds: nil)
       config = DataDrain.configuration
       config.validate!
-      client = Aws::Glue::Client.new(region: config.aws_region)
       start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
 
       @logger = config.logger