pg_objects 1.4.7 → 1.5.0

data/README.md

@@ -32,6 +32,18 @@ Run the installation procedure to initialize directories structure and configura
 bundle exec rails generate pg_objects:install
 ```
 
+## Supported object types
+
+The following `CREATE` statements are recognized as manageable objects:
+
+`AGGREGATE`, `CONVERSION`, `DOMAIN`, `EVENT TRIGGER`, `EXTENSION`, `FUNCTION`,
+`INDEX`, `MATERIALIZED VIEW`, `OPERATOR`, `OPERATOR CLASS`, `POLICY`, `RULE`,
+`SEQUENCE`, `TABLE`, `TEXT SEARCH PARSER`, `TEXT SEARCH TEMPLATE`, `TRIGGER`,
+`TYPE` (composite, enum, range, base, and shell forms), `VIEW`.
+
+Files containing any other statement raise
+`PgObjects::UnknownObjectTypeError` during loading.
+
 ## Usage
 
 Store DB objects as CREATE (or CREATE OR UPDATE) queries in files within a directory structure (default: *db/objects*).
@@ -46,6 +58,21 @@ CREATE FUNCTION my_func()
 
 The string after the directive should be the name of the file that the dependency refers to, without the file extension.
 
+### Directive syntax
+
+- The directive must start the line — no leading whitespace.
+- Both `--!` (SQL comment) and `#!` prefixes are supported.
+- List several dependencies on one line, separated by commas and/or whitespace,
+  or use a separate directive line for each:
+
+```sql
+--!depends_on func_a, func_b, func_c
+--!depends_on func_d
+#!depends_on func_e
+CREATE FUNCTION my_func()
+...
+```
+
 ## Configuration
 
 You have the option to configure the gem using either a YAML file or a Ruby initializer. The priority order for configuration is as follows:
@@ -55,7 +82,13 @@ You have the option to configure the gem using either a YAML file or a Ruby init
 
 ### YAML
 
-Create `pg_objects.yml` in the application *config* directory:
+Create `pg_objects.yml` in the application *config* directory.
+
+The file is loaded on first configuration access and resolved against
+`Rails.root` in Rails applications, so it is found even when the gem is
+required before the process changes into the app root (e.g. under the Spring
+preloader). Outside Rails the path is relative to the current working
+directory.
 
 ```yaml
 # pg_objects.yml
@@ -70,8 +103,18 @@ extensions:
   - sql
   - txt
 
-# Specify whether to suppress output to console
+# Suppress non-error output to console (error messages are always printed)
 silent: false
+
+# Whether to wrap each object-creation run in a database transaction so a
+# failure rolls back everything created in that run (default: true).
+# Note: some PostgreSQL statements cannot run inside a transaction
+# (e.g. CREATE INDEX CONCURRENTLY, VACUUM) — set to false if your
+# object files contain them.
+transactional: true
+
+# Whether to install the Rake hooks that auto-create objects (default: true)
+auto_hook_migrations: true
 ```
 
 ### Initializer
@@ -83,7 +126,8 @@ PgObjects.configure do |config|
   config.before_path = 'path/to/objects/before' # default: 'db/objects/before'
   config.after_path = 'path/to/objects/after' # default: 'db/objects/after'
   config.extensions = ['sql', 'txt'] # default: 'sql'
-  config.silent = true # whether to suppress output to console, default: false
+  config.silent = true # suppress non-error output to console (errors are always printed), default: false
+  config.transactional = false # opt out of the wrapping transaction, default: true
 end
 ```
 
@@ -91,6 +135,118 @@ Otherwise, the default values will be used.
 
 Please make sure to verify that the specified directories actually exist.
 
+> [!NOTE]
+> Object creation runs inside a single database transaction by default, so a
+> failure mid-run rolls back every object created in that run. PostgreSQL
+> rejects some statements inside a transaction block — for example
+> `CREATE INDEX CONCURRENTLY`, `VACUUM`, or `ALTER TYPE ... ADD VALUE` on
+> PostgreSQL versions before 12. If your object files contain such statements,
+> set `config.transactional = false`.
+
+### Rake tasks that trigger object creation
+
+Object creation is hooked into Rake tasks. By default:
+
+| Task | `before` objects | `after` objects |
+| --- | :---: | :---: |
+| `db:migrate` | ✓ | ✓ |
+| `db:schema:load` | | ✓ |
+| `db:migrate:redo` | ✓ | ✓ |
+
+`db:rollback` is **not** hooked — rolling a migration back should not recreate
+objects.
+
+You can also invoke the underlying tasks directly: `db:create_objects:before`
+and `db:create_objects:after`.
+
+### Multiple databases
+
+By default objects are created over the global connection
+(`ActiveRecord::Base.connection`). In a Rails multi-database setup, target a
+specific database by passing its connection to the manager:
+
+```ruby
+PgObjects::Manager.new(connection: AnimalsRecord.connection).load_files(:before).create_objects
+```
+
+For the rake tasks, set `PG_OBJECTS_CONNECTION_CLASS` to the name of the
+Active Record class whose connection should be used:
+
+```sh
+PG_OBJECTS_CONNECTION_CLASS=AnimalsRecord bin/rails db:create_objects:before
+```
+
+To disable all hooks entirely, set `auto_hook_migrations` to `false` in an
+initializer; then create objects only by invoking those tasks manually:
+
+```ruby
+PgObjects.configure do |config|
+  config.auto_hook_migrations = false # default: true
+end
+```
+
+### Re-running and idempotency
+
+The hooked tasks re-execute **every** object file on each run — the gem keeps
+no state about what was already created. Files must therefore contain
+re-runnable (idempotent) SQL, or the second `db:migrate` fails with errors
+like `relation "..." already exists`.
+
+What PostgreSQL offers per object type:
+
+| Object type | Idempotent form |
+| --- | --- |
+| `FUNCTION`, `VIEW` | `CREATE OR REPLACE` |
+| `RULE` | `CREATE OR REPLACE` |
+| `AGGREGATE` | `CREATE OR REPLACE` (PostgreSQL 12+) |
+| `TRIGGER` | `CREATE OR REPLACE` (PostgreSQL 14+), otherwise drop-and-recreate |
+| `TABLE`, `INDEX`, `SEQUENCE`, `MATERIALIZED VIEW`, `EXTENSION` | `IF NOT EXISTS` |
+| `TYPE`, `DOMAIN`, `POLICY`, `CONVERSION`, `OPERATOR`, `OPERATOR CLASS`, `EVENT TRIGGER`, `TEXT SEARCH PARSER/TEMPLATE` | none — use a guard (below) |
+
+For types without an idempotent form, either drop first:
+
+```sql
+DROP POLICY IF EXISTS user_policy ON users;
+CREATE POLICY user_policy ON users USING (user_id = current_user_id());
+```
+
+or swallow the duplicate error in a `DO` block:
+
+```sql
+DO $$ BEGIN
+  CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy');
+EXCEPTION WHEN duplicate_object THEN NULL;
+END $$;
+```
+
+> [!NOTE]
+> Object files are classified by parsing their **first** statement. When that
+> statement is a `DROP` or a `DO` block (as in the guards above), the SQL
+> object name cannot be extracted. The file still executes normally, but it
+> can only be referenced by its **file identifiers** — the extensionless file
+> name or the file path — while `--!depends_on` directives referring to the
+> SQL object name (or its schema-qualified form) will not match it.
+> The plain `IF NOT EXISTS` / `OR REPLACE` forms keep full name resolution.
+
+Also note that object creation runs inside a single transaction by default
+(see above), so one failing statement rolls back the entire run — a
+half-idempotent set of files either all applies or not at all.
+
+Override `hook_tasks` to customize which tasks/stages are hooked (e.g. an empty
+hash also disables all hooks). Configure it in a Rails initializer: the hooks are installed when the
+gem's rake tasks load, which happens after initializers run, so an initializer
+value is always picked up. Changing `hook_tasks` later (after task loading) has
+no effect on which hooks are installed.
+
+```ruby
+PgObjects.configure do |config|
+  config.hook_tasks = {
+    'db:migrate' => %i[before after],
+    'db:schema:load' => %i[after]
+  }
+end
+```
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -145,3 +301,18 @@ Full Workflow Performance:
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/marinazzio/pg_objects.
+
+Every pull request that changes gem behavior must add an entry to the
+`[Unreleased]` section of [CHANGELOG.md](CHANGELOG.md) (following the
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format). Pure
+refactorings, spec-only, and dependency-bump PRs may skip this.
+
+## Releasing
+
+1. Move the `[Unreleased]` entries in `CHANGELOG.md` under a new version
+   heading with the release date, and update the comparison links at the
+   bottom of the file.
+2. Bump `PgObjects::VERSION` in `lib/pg_objects/version.rb` accordingly
+   (semantic versioning).
+3. Tag the commit `vX.Y.Z` and publish a GitHub release — the publish
+   workflow builds and pushes the gem to RubyGems.

data/Rakefile

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require 'bundler/gem_tasks'
 require 'rspec/core/rake_task'
 

data/bin/benchmark

@@ -1,6 +1,8 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require 'bundler/setup'
+require 'active_support/core_ext/string/inflections' # String#classify / #constantize used by the parser
 require 'pg_objects'
 require 'benchmark'
 require 'fileutils'
@@ -242,6 +244,7 @@ class PgObjectsBenchmark
     parser = PgObjects::Parser.new
     successful_parses = 0
     parse_errors = 0
+    sample_error = nil
 
     result = Benchmark.measure do
       files.each do |file|
@@ -249,8 +252,9 @@ class PgObjectsBenchmark
         begin
           parser.load(content).fetch_object_name
           successful_parses += 1
-        rescue StandardError
+        rescue StandardError => e
           parse_errors += 1
+          sample_error ||= "#{e.class}: #{e.message}"
         end
       end
     end
@@ -260,12 +264,14 @@ class PgObjectsBenchmark
       files_count: files.size,
       successful_parses: successful_parses,
       parse_errors: parse_errors,
+      sample_error: sample_error,
       throughput: files.size / result.real
     }
 
     puts "Parsed #{files.size} files in #{result.real.round(4)}s"
     puts "Successful parses: #{successful_parses}"
     puts "Parse errors: #{parse_errors}"
+    puts "  Sample error: #{sample_error}" if sample_error
     puts "Throughput: #{(files.size / result.real).round(2)} files/second"
     puts "Average time per file: #{(result.real / files.size * 1000).round(2)}ms"
     puts
@@ -308,6 +314,8 @@ class PgObjectsBenchmark
     # Test the parsing workflow without database operations
     files = Dir[File.join(@temp_dir, '**', '*.sql')]
     db_objects = []
+    workflow_errors = 0
+    sample_error = nil
 
     result = Benchmark.measure do
       files.each do |file_path|
@@ -327,18 +335,23 @@ class PgObjectsBenchmark
           dependencies: dependencies,
           content: content
         }
-      rescue StandardError
-        # Count parse errors but continue
+      rescue StandardError => e
+        workflow_errors += 1
+        sample_error ||= "#{e.class}: #{e.message}"
       end
     end
 
     @results[:full_workflow] = {
       time: result.real,
       objects_count: db_objects.size,
+      errors: workflow_errors,
+      sample_error: sample_error,
       throughput: db_objects.size / result.real
     }
 
     puts "Processed #{db_objects.size} objects in #{result.real.round(4)}s"
+    puts "Errors: #{workflow_errors}"
+    puts "  Sample error: #{sample_error}" if sample_error
     puts "Throughput: #{(db_objects.size / result.real).round(2)} objects/second"
     puts "Average time per object: #{(result.real / db_objects.size * 1000).round(2)}ms"
     puts
@@ -354,6 +367,8 @@ class PgObjectsBenchmark
     files = Dir[File.join(@temp_dir, '**', '*.sql')]
     parser = PgObjects::Parser.new
     parsed_objects = []
+    memory_errors = 0
+    sample_error = nil
 
     files.each do |file|
       content = File.read(file)
@@ -361,11 +376,15 @@ class PgObjectsBenchmark
         object_name = parser.load(content).fetch_object_name
         dependencies = parser.fetch_directives[:depends_on]
         parsed_objects << { file: file, name: object_name, deps: dependencies }
-      rescue StandardError
-        # Ignore parse errors for memory analysis
+      rescue StandardError => e
+        memory_errors += 1
+        sample_error ||= "#{e.class}: #{e.message}"
       end
     end
 
+    puts "Parse errors during memory analysis: #{memory_errors}" if memory_errors.positive?
+    puts "  Sample error: #{sample_error}" if sample_error
+
     end_memory = memory_usage
     memory_diff = end_memory - start_memory
 
@@ -386,9 +405,25 @@ class PgObjectsBenchmark
   end
 
   def memory_usage
-    `ps -o rss= -p #{Process.pid}`.to_i * 1024 # Convert KB to bytes
+    rss_from_proc_status || rss_from_ps || 0
+  end
+
+  # Linux: parse VmRSS from /proc/self/status (portable, no shell, locale-safe)
+  def rss_from_proc_status
+    return nil unless File.readable?('/proc/self/status')
+
+    kb = File.read('/proc/self/status')[/^VmRSS:\s+(\d+)\s+kB/, 1]
+    kb && (kb.to_i * 1024) # Convert KB to bytes
+  rescue StandardError
+    nil
+  end
+
+  # Portable fallback (e.g. macOS): RSS in KB via ps
+  def rss_from_ps
+    kb = `ps -o rss= -p #{Process.pid}`.strip
+    kb.empty? ? nil : kb.to_i * 1024
   rescue StandardError
-    0
+    nil
   end
 
   def format_memory(bytes)

data/bin/console

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require 'bundler/setup'
 require 'pg_objects'

data/lib/generators/pg_objects/install/install_generator.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 ##
 # Creates default directories structure
 class PgObjects::InstallGenerator < Rails::Generators::Base

data/lib/pg_objects/config.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require_relative 'yaml_configurable'
 
 module PgObjects
@@ -10,7 +12,7 @@ module PgObjects
   #     # or full
   #     config.after_path = '/var/tmp/alternate/after'
   #     config.extensions = ['sql', 'txt']
-  #     # suppress output to console
+  #     # suppress non-error output to console
   #     config.silent = true
   #   end
   class << self
@@ -32,6 +34,55 @@ module PgObjects
     setting :extensions, default: ['sql']
     setting :silent, default: false
 
-    load_from_yaml 'config/pg_objects.yml'
+    # Wraps each create_objects run in a database transaction so a failure
+    # mid-run rolls back every object created in that run. Set to false to
+    # execute statements without a wrapping transaction — required when object
+    # files contain statements PostgreSQL rejects inside a transaction block
+    # (e.g. CREATE INDEX CONCURRENTLY, VACUUM).
+    setting :transactional, default: true
+
+    # Master switch for the Rake hooks. When false, no hooks are installed and
+    # objects are created only by invoking db:create_objects:before / :after
+    # manually. Default true for backward compatibility.
+    setting :auto_hook_migrations, default: true
+
+    # Rake tasks that trigger object creation, mapped to the stages they run.
+    # +:before+ creates objects from the "before" folder ahead of the task,
+    # +:after+ creates objects from the "after" folder once the task finishes.
+    # Override (or empty) this to opt out. db:rollback is intentionally absent
+    # (a rollback should not recreate objects).
+    setting :hook_tasks, default: {
+      'db:migrate' => %i[before after],
+      'db:schema:load' => %i[after],
+      'db:migrate:redo' => %i[before after]
+    }
+
+    DEFAULT_YAML_PATH = 'config/pg_objects.yml'
+
+    class << self
+      # YAML loading is deferred to the first config access so the path is
+      # resolved against Rails.root (when available) instead of the
+      # require-time working directory — preloaders like Spring may require
+      # the gem before the process chdirs into the app root.
+      def config
+        ensure_yaml_loaded
+        super
+      end
+
+      private
+
+      def ensure_yaml_loaded
+        return if @yaml_loaded
+
+        @yaml_loaded = true
+        load_from_yaml(yaml_config_path)
+      end
+
+      def yaml_config_path
+        return DEFAULT_YAML_PATH unless defined?(Rails) && Rails.respond_to?(:root) && Rails.root
+
+        Rails.root.join(DEFAULT_YAML_PATH).to_s
+      end
+    end
   end
 end

data/lib/pg_objects/db_object.rb

@@ -1,16 +1,20 @@
+# frozen_string_literal: true
+
 ##
 # Represents DB object as it is described in file
 #
 # [name]  name of file without extension
 # [full_name] full pathname of file
 # [object_name] name of function, trigger etc. if it was successfully parsed, otherwise - nil
+# [qualified_object_name] schema-qualified object name (+schema.name+) when a schema is
+#   present, otherwise same as object_name; nil if parsing failed
 class PgObjects::DbObject
   include Memery
 
   include Import['parser']
 
   attr_accessor :status
-  attr_reader :full_name, :object_name
+  attr_reader :full_name, :object_name, :qualified_object_name
 
   def initialize(path, status = :new, parser:)
     @full_name = path
@@ -21,6 +25,7 @@ class PgObjects::DbObject
   def create
     parser.load(sql_query)
     @object_name = parser.fetch_object_name
+    @qualified_object_name = parser.fetch_qualified_object_name
     @status = :pending
 
     self

data/lib/pg_objects/db_object_factory.rb

@@ -3,11 +3,19 @@
 ##
 # Factory for DbObject
 #
+# Each DbObject gets its own Parser instance so the parser's mutable `@source`
+# is never shared between objects. The parser class is
+# injectable so callers/tests can substitute an alternate parser.
 class PgObjects::DbObjectFactory
-  include Import['parser']
+  def initialize(parser_class: PgObjects::Parser)
+    @parser_class = parser_class
+  end
 
   def create_instance(path, status: :new)
-    db_object = PgObjects::DbObject.new(path, status, parser:)
-    db_object.create
+    PgObjects::DbObject.new(path, status, parser: parser_class.new).create
   end
+
+  private
+
+  attr_reader :parser_class
 end

data/lib/pg_objects/logger.rb

@@ -1,10 +1,41 @@
+# frozen_string_literal: true
+
 ##
-# Console output
-#
+# Console output with severity levels. Writes to the injected +stream+
+# (default +$stdout+). With +config.silent+ enabled, +info+/+warn+ messages
+# are suppressed; +error+ messages are always written.
 class PgObjects::Logger
   include Import['config']
 
+  def initialize(stream: $stdout, **deps)
+    super(**deps)
+    @stream = stream
+  end
+
+  # Backward-compatible entry point; behaves like +info+.
   def write(str)
-    puts "== #{str} ".ljust(80, '=') unless config.silent
+    log(str)
+  end
+
+  def info(str)
+    log(str)
+  end
+
+  def warn(str)
+    log("[WARN] #{str}")
+  end
+
+  def error(str)
+    log("[ERROR] #{str}", force: true)
+  end
+
+  private
+
+  attr_reader :stream
+
+  def log(str, force: false)
+    return if config.silent && !force
+
+    stream.puts "== #{str} ".ljust(80, '=')
   end
 end

data/lib/pg_objects/manager.rb

@@ -1,23 +1,39 @@
+# frozen_string_literal: true
+
 ##
 # Manages process to create objects
 #
-# Usage:
+# Usage (dependencies are auto-injected, keyword overrides optional):
 #
-#   Manager.new(config, logger).load_files(:before).create_objects
+#   Manager.new.load_files(:before).create_objects
 #
 # or
 #
-#   Manager.new(config, logger).load_files(:after).create_objects
+#   Manager.new(config: custom_config, logger: custom_logger).load_files(:after).create_objects
+#
+# Pass +connection:+ to run against a specific database connection instead of
+# the global one (Rails 6+ multi-DB):
+#
+#   Manager.new(connection: AnimalsRecord.connection).load_files(:before).create_objects
 class PgObjects::Manager
   include Import['db_object_factory', 'config', 'logger']
 
+  def initialize(connection: nil, **deps)
+    super(**deps)
+    @connection = connection
+  end
+
   ##
   # event: +:before+ or +:after+
   #
   # used to reference configuration settings +before_path+ and +after_path+
+  #
+  # Resets the object list before loading, so each call reflects only the
+  # files for the given event.
   def load_files(event)
     validate_workability
 
+    objects.clear
     dir = config.send "#{event}_path"
     Dir[File.join(dir, '**', "*.{#{config.extensions.join(',')}}")].each do |path|
       objects << db_object_factory.create_instance(path)
@@ -27,7 +43,8 @@ class PgObjects::Manager
   end
 
   def create_objects
-    objects.each { create_object(_1) }
+    build_objects_index
+    within_transaction { objects.each { |obj| create_object(obj) } }
   end
 
   def objects
@@ -36,34 +53,68 @@ class PgObjects::Manager
 
   private
 
+  def connection
+    @connection || ActiveRecord::Base.connection
+  end
+
   def validate_workability
-    raise PgObjects::UnsupportedAdapterError if ActiveRecord::Base.connection.adapter_name != 'PostgreSQL'
+    raise PgObjects::UnsupportedAdapterError if connection.adapter_name != 'PostgreSQL'
+  end
+
+  def within_transaction(&)
+    return yield unless config.transactional
+
+    connection.transaction(&)
   end
 
-  def create_object(obj)
+  def create_object(obj, stack = [])
     return if obj.status == :done
-    raise PgObjects::CyclicDependencyError, obj.name if obj.status == :processing
+    raise PgObjects::CyclicDependencyError, cycle_path(obj, stack) if obj.status == :processing
 
     obj.status = :processing
 
-    create_dependencies(obj.dependencies)
+    create_dependencies(obj, stack + [obj.name])
 
     logger.write("creating #{obj.name}")
-    ActiveRecord::Base.connection.execute(obj.sql_query)
+    connection.execute(obj.sql_query)
 
     obj.status = :done
   end
 
-  def create_dependencies(dependencies)
-    dependencies.each { |dep_name| create_object(find_object(dep_name)) }
+  def create_dependencies(obj, stack)
+    obj.dependencies.each { |dep_name| create_object(find_object(dep_name, obj), stack) }
   end
 
-  def find_object(dep_name)
-    result = objects.select { |obj| [obj.name, obj.full_name, obj.object_name].compact.include? dep_name }
+  # Resolution chain that closed the cycle: from the first occurrence of the
+  # revisited object down to the point of revisit, closed with the object
+  # itself (e.g. ["a", "b", "a"]).
+  def cycle_path(obj, stack)
+    first_seen = stack.index(obj.name)
+    (first_seen ? stack[first_seen..] : stack) + [obj.name]
+  end
 
-    raise PgObjects::AmbiguousDependencyError, dep_name if result.size > 1
-    raise PgObjects::DependencyNotExistError, dep_name if result.empty?
+  def build_objects_index
+    @objects_index = objects.each_with_object({}) do |obj, index|
+      [obj.name, obj.full_name, obj.object_name, obj.qualified_object_name].compact.uniq.each do |key|
+        (index[key] ||= []) << obj
+      end
+    end
+  end
+
+  def find_object(dep_name, referrer = nil)
+    result = @objects_index[dep_name] || []
+
+    raise ambiguous_error(dep_name, result, referrer) if result.size > 1
+    raise PgObjects::DependencyNotExistError.new(dep_name, referrer: referrer&.full_name) if result.empty?
 
     result[0]
   end
+
+  def ambiguous_error(dep_name, result, referrer)
+    PgObjects::AmbiguousDependencyError.new(
+      dep_name,
+      candidates: result.map(&:full_name),
+      referrer: referrer&.full_name
+    )
+  end
 end

data/lib/pg_objects/parsed_object/aggregate.rb

@@ -1,8 +1,16 @@
+# frozen_string_literal: true
+
 #
 # AGGREGATE object representation
 #
 class PgObjects::ParsedObject::Aggregate < PgObjects::ParsedObject::Base
   def name
-    stmt.define_stmt.defnames[0].string.sval
+    extract_name { stmt.define_stmt.defnames.last.string.sval }
+  end
+
+  private
+
+  def schema
+    extract_name { qualifier(stmt.define_stmt.defnames) }
   end
 end