pilipinas 1.1.1 → 1.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 989723b7dcd5133a2d2240aa124e0c5400a981f521d0904d8b2858fb71f15e9e
-  data.tar.gz: c01c0377bf9572381844a38d1c77dbe45aed0e554076159a9964371f5f7aa851
+  metadata.gz: '069c53de29b6607d4f19d961e085870e474ee3f53e5a9950d5564211d05b7286'
+  data.tar.gz: fbaa56417882627d836b445625b65fab4cd2077fc4f901019dc2c841b38f6c4f
 SHA512:
-  metadata.gz: b2589f13688bd95ad3ad176d9dccc2412a0df9a3484a173129e3dc66e7f78dad67ea1f7920f7bee9136f412b5c6221a7776b2f7a682ef1e0fdb55d2ffd1cd307
-  data.tar.gz: 73bc97b32dcaa44f0a9181e63d450770535e12f73254fcea500551a46bb788a0dfe2bf9ed7de897a84ab1c22f0a19545d050e2b7a1c8cd76c7d3515a55b60d0d
+  metadata.gz: 9bf493ab7a8d154f11c17b992d93abb8695e3c92b9286a8be3c14d0aad1d95568f2abb5467cbb86416a082cfda21eda542103cffd0e039aae32bbe34d9b3575f
+  data.tar.gz: a11da75e206b70b6c1fb8cee5f0e3f132ddaa74b10beb3a51e946271f415367511cddce3f6a9c307ba8eb46ba7fcd64f8094c5ca2fbf5604de3c5c86ee4cc395

data/CHANGELOG.md

@@ -7,6 +7,53 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [1.1.3] - 2026-06-13
+
+### Fixed
+
+- `rake pilipinas:load` now upserts ActiveRecord seed rows by `location_id` instead of `code`, matching the stable identifier from `lib/data/pilipinas_data.yml`.
+- `rails generate pilipinas:code_indexes` now generates a migration that adds unique `location_id` indexes to all four `pilipinas_*` tables, matching the loader's upsert conflict target.
+
+### Changed
+
+- Updated loader upgrade guidance and README wording to reference the required `location_id` unique indexes.
+
+### Tests
+
+- Added regression coverage that asserts `Loader.bulk_insert` uses `location_id` as the `upsert_all` unique key and reports the matching missing-index guidance.
+
+---
+
+## [1.1.2] - 2026-06-12
+
+### Fixed
+
+- `rake pilipinas:load` now seeds the ActiveRecord tables from `lib/data/pilipinas_data.yml`, the complete bundled data source. Previously the loader used the compact file-backed YAML files, so database rows were populated with only `code` and `name` while columns such as `location_id`, `parent_id`, `lft`, `rgt`, coordinates, and classification fields remained `NULL`.
+
+- Re-running `rake pilipinas:load` now removes stale compact rows created by earlier loader versions. This fixes cases where old rows such as provinces with `location_id: nil` remained in the database because their compact internal codes did not match the full PSA-style codes used by `pilipinas_data.yml`.
+
+### Changed
+
+- Improved loader memory behavior by transforming rows directly from the parsed full-data table and keeping only the current insert batch in memory.
+
+### Tests
+
+- Added regression coverage for complete-column seeding, stale compact-row cleanup, legacy seed behavior, and exact batch-boundary inserts.
+
+---
+
+## [1.1.1] - 2026-06-11
+
+### Added
+
+- `rails generate pilipinas:code_indexes` — new migration generator that adds `UNIQUE` indexes on the `code` column to all four `pilipinas_*` tables. Run this if your database was created with a pre-1.0 migration and `rake pilipinas:load` raises `ArgumentError: No unique index found for code`.
+
+### Fixed
+
+- `Loader.bulk_insert` now rescues the bare `ArgumentError` raised by `upsert_all` when the unique index on `code` is absent, and re-raises it as a descriptive `Pilipinas::Error` that tells the user exactly which commands to run (`rails generate pilipinas:code_indexes && rails db:migrate`). Previously the error surfaced as an unguided `ArgumentError` from deep inside ActiveRecord.
+
+---
+
 ## [1.1.0] - 2026-05-13
 
 ### Added
@@ -18,19 +65,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
     self.enforce_readonly = false
   end
   ```
-- `lib/pilipinas/testing/rspec.rb` — a ready-made RSpec helper that disables
-  the read-only guard on all four DB models for the entire test suite.
-  Require it once in `rails_helper.rb`:
+- `lib/pilipinas/testing/rspec.rb` — a ready-made RSpec helper that disables the read-only guard on all four DB models for the entire test suite. Require it once in `rails_helper.rb`:
   ```ruby
   require 'pilipinas/testing/rspec'
   ```
-- Spec coverage for `StaticRecord` — 8 examples covering default behaviour,
-  `enforce_readonly = false`, subclass inheritance, and class-level isolation.
+- Spec coverage for `StaticRecord` — 8 examples covering default behaviour, `enforce_readonly = false`, subclass inheritance, and class-level isolation.
 
 ### Changed
 
-- `readonly?` now gates on `self.class.enforce_readonly && !new_record?` instead
-  of unconditionally returning `!new_record?`.
+- `readonly?` now gates on `self.class.enforce_readonly && !new_record?` instead of unconditionally returning `!new_record?`.
 
 ---
 
@@ -40,26 +83,20 @@ Complete rewrite of the gem. Zero runtime dependencies.
 
 ### Added
 
-- In-memory layer with thread-safe `Pilipinas::Cache` (Mutex + double-checked
-  locking) and O(1) look-ups via separate code/name hash indices.
+- In-memory layer with thread-safe `Pilipinas::Cache` (Mutex + double-checked locking) and O(1) look-ups via separate code/name hash indices.
 - Immutable value objects — every entity instance is frozen.
-- `Pilipinas::Region`, `Province`, `City`, `Barangay` with `.all`, `.count`,
-  `.first`, `.last`, `.find_by`, `.find_by_code`, `.find_by_name`.
+- `Pilipinas::Region`, `Province`, `City`, `Barangay` with `.all`, `.count`, `.first`, `.last`, `.find_by`, `.find_by_code`, `.find_by_name`.
 - Hierarchy traversal: `region.provinces`, `province.cities`, `city.barangays`.
-- Optional ActiveRecord layer (`Pilipinas::Db::*`) with memory-efficient scopes
-  (`.lite`, `.by_code`, `.by_name`, `.find_lite_by_code`, `.find_lite_by_name`).
-- `StaticRecord` concern: disables STI, adds lean SELECT scopes, enforces
-  read-only on persisted records.
-- Migration generator (`rails generate pilipinas:migration`) and
-  `rake pilipinas:load` seeding task.
+- Optional ActiveRecord layer (`Pilipinas::Db::*`) with memory-efficient scopes (`.lite`, `.by_code`, `.by_name`, `.find_lite_by_code`, `.find_lite_by_name`).
+- `StaticRecord` concern: disables STI, adds lean SELECT scopes, enforces read-only on persisted records.
+- Migration generator (`rails generate pilipinas:migration`) and `rake pilipinas:load` seeding task.
 - Full RSpec suite (57 examples).
 - GitHub Actions CI pipeline.
 
 ### Changed
 
 - Requires Ruby ≥ 3.4 (developed against Ruby 4.0).
-- Removed all runtime gem dependencies (previously depended on `yaml_db` and
-  others).
+- Removed all runtime gem dependencies (previously depended on `yaml_db` and others).
 
 ---
 
@@ -74,6 +111,8 @@ Complete rewrite of the gem. Zero runtime dependencies.
 - Rails generator for migrations.
 - Railtie for automatic Rake task loading in Rails apps.
 
+[1.1.2]: https://github.com/denmarkmeralpis/pilipinas/compare/v1.1.1...v1.1.2
+[1.1.1]: https://github.com/denmarkmeralpis/pilipinas/compare/v1.1.0...v1.1.1
 [1.1.0]: https://github.com/denmarkmeralpis/pilipinas/compare/v1.0.0...v1.1.0
 [1.0.0]: https://github.com/denmarkmeralpis/pilipinas/compare/v0.0.1...v1.0.0
 [0.0.1]: https://github.com/denmarkmeralpis/pilipinas/releases/tag/v0.0.1

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    pilipinas (1.1.1)
+    pilipinas (1.1.3)
 
 GEM
   remote: https://rubygems.org/
@@ -76,7 +76,7 @@ GEM
       prism (>= 1.3.0)
       rdoc (>= 4.0.0)
       reline (>= 0.4.2)
-    json (2.19.8)
+    json (2.19.9)
     language_server-protocol (3.17.0.5)
     lint_roller (1.1.0)
     logger (1.7.0)
@@ -280,7 +280,7 @@ CHECKSUMS
   i18n (1.14.8) sha256=285778639134865c5e0f6269e0b818256017e8cde89993fdfcbfb64d088824a5
   io-console (0.8.2) sha256=d6e3ae7a7cc7574f4b8893b4fca2162e57a825b223a177b7afa236c5ef9814cc
   irb (1.18.0) sha256=de9454a0703a54704b9811a5ef31a60c86949fbf4013fcf244fabc7c775248e3
-  json (2.19.8) sha256=6354310fd76ef69b87d5bd1f38b40d730613baf90b6803d2d0a48f618d32dfaa
+  json (2.19.9) sha256=9b9025b7cdddafa38d316eca0b2358488e42d417045c1b90d216a9fefe46b79a
   language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc
   lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87
   logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203
@@ -296,7 +296,7 @@ CHECKSUMS
   nokogiri (1.19.3-x86_64-linux-musl) sha256=248c906d2166eca5efb56d52fdee5f9a1f51d69a72e2b64fdac647b4ce39ea3f
   parallel (2.1.0) sha256=b35258865c2e31134c5ecb708beaaf6772adf9d5efae28e93e99260877b09356
   parser (3.3.11.1) sha256=d17ace7aabe3e72c3cc94043714be27cc6f852f104d81aa284c2281aecc65d54
-  pilipinas (1.1.1)
+  pilipinas (1.1.3)
   pp (0.6.3) sha256=2951d514450b93ccfeb1df7d021cae0da16e0a7f95ee1e2273719669d0ab9df6
   prettyprint (0.2.0) sha256=2bc9e15581a94742064a3cc8b0fb9d45aae3d03a1baa6ef80922627a0766f193
   prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85

data/README.md

@@ -177,7 +177,7 @@ rake pilipinas:load
 > **Upgrading from an older version?**
 >
 > If `rake pilipinas:load` raises
-> `Pilipinas::Error: pilipinas:load requires a unique index on the code column`,
+> `Pilipinas::Error: pilipinas:load requires a unique index on the location_id column`,
 > your database was created with an earlier migration that did not include those
 > indexes. Add them with the dedicated generator:
 >

data/lib/generators/pilipinas/code_indexes_generator.rb

@@ -4,13 +4,13 @@ require 'rails/generators/base'
 require 'rails/generators/active_record'
 
 module Pilipinas
-  # Rails generator that adds unique indexes on the +code+ column to all four
-  # pilipinas_* tables.
+  # Rails generator that adds unique indexes on the +location_id+ column to all
+  # four pilipinas_* tables.
   #
   # Run this if your database was created with an older version of the gem
   # that did not include these indexes, and +rake pilipinas:load+ raises:
   #
-  #   ArgumentError: No unique index found for code
+  #   ArgumentError: No unique index found for location_id
   #
   # @example
   #   rails generate pilipinas:code_indexes
@@ -23,7 +23,7 @@ module Pilipinas
 
     def generate_migration
       migration_template 'templates/add_pilipinas_code_indexes.rb',
-                         'db/migrate/add_pilipinas_code_indexes.rb'
+                         'db/migrate/add_pilipinas_location_id_indexes.rb'
     end
 
     # @param _dir [String] unused (required by the interface)

data/lib/generators/templates/add_pilipinas_code_indexes.rb

@@ -1,8 +1,8 @@
-class AddPilipinasCodeIndexes < ActiveRecord::Migration<%= migration_version %>
+class AddPilipinasLocationIdIndexes < ActiveRecord::Migration<%= migration_version %>
   def change
-    add_index :pilipinas_regions,   :code, unique: true, if_not_exists: true
-    add_index :pilipinas_provinces, :code, unique: true, if_not_exists: true
-    add_index :pilipinas_cities,    :code, unique: true, if_not_exists: true
-    add_index :pilipinas_barangays, :code, unique: true, if_not_exists: true
+    add_index :pilipinas_regions,   :location_id, unique: true, if_not_exists: true
+    add_index :pilipinas_provinces, :location_id, unique: true, if_not_exists: true
+    add_index :pilipinas_cities,    :location_id, unique: true, if_not_exists: true
+    add_index :pilipinas_barangays, :location_id, unique: true, if_not_exists: true
   end
 end

data/lib/pilipinas/loader.rb

@@ -9,8 +9,9 @@ module Pilipinas
   # * **Idempotent** — uses +upsert_all+ (Rails 6.1+) so re-running the Rake
   #   task is safe.  Falls back to +insert_all+ (Rails 6.0) or individual
   #   +create!+ calls on older versions.
-  # * **Memory-efficient** — rows are inserted in batches of {BATCH_SIZE}
-  #   (default 500) so the process never holds a 42 k-row Array in memory.
+  # * **Memory-aware** — rows are transformed and inserted in batches of
+  #   {BATCH_SIZE} (default 500) so the process never holds a full
+  #   ActiveRecord insert payload in memory.
   # * **Atomic** — all four tables are seeded inside a single transaction; a
   #   failure rolls back everything, leaving no partial data.
   #
@@ -21,22 +22,109 @@ module Pilipinas
     # Number of rows inserted per SQL statement.
     # 500 balances SQL statement size against the number of round-trips.
     BATCH_SIZE = 500
+    FULL_DATA_FILE = 'pilipinas_data.yml'
+    LOCATION_TABLE = 'pilipinas_locations'
+
+    FULL_DATA_SEEDS = [
+      ['Region', 'Locations::Region', %w[location_id lft rgt code name longitude latitude]],
+      ['Province', 'Locations::Province', %w[location_id parent_id lft rgt code name longitude latitude]],
+      [
+        'City',
+        'Locations::Town',
+        %w[location_id parent_id lft rgt code name city income_class urban_rural district longitude latitude]
+      ],
+      ['Barangay', 'Locations::Barangay', %w[location_id parent_id lft rgt code name urban_rural]]
+    ].freeze
+    private_constant :FULL_DATA_FILE, :LOCATION_TABLE, :FULL_DATA_SEEDS
 
     class << self
       # Seed all four geographic tables inside a single transaction.
       #
       # @return [void]
       def run
-        ActiveRecord::Base.transaction do
-          seed(Db::Region,   'regions.yml')
-          seed(Db::Province, 'provinces.yml')
-          seed(Db::City,     'cities.yml')
-          seed(Db::Barangay, 'barangays.yml')
+        records = nil
+        column_indexes, records = full_location_table
+
+        ActiveRecord::Base.uncached do
+          ActiveRecord::Base.transaction do
+            FULL_DATA_SEEDS.each do |model_name, type, attributes|
+              model = Db.const_get(model_name)
+              delete_stale_compact_rows(model)
+              seed_full_data(model, records, column_indexes, type, attributes)
+            end
+          end
         end
+      ensure
+        records&.clear
+        clear_query_cache
       end
 
       private
 
+      # Load the full Rails fixture-style location dump bundled with the gem.
+      #
+      # The older per-table YAML files only contain +code+ and +name+ for the
+      # file-backed API.  Database seeding needs the complete dump so lft/rgt,
+      # parent links, coordinates, and classification fields are populated.
+      #
+      # @return [Array(Hash, Array<Array>)]
+      def full_location_table
+        data = Psych.load_file(File.join(DATA_DIR, FULL_DATA_FILE)) || {}
+        table = data.fetch(LOCATION_TABLE)
+        columns = table.fetch('columns').map(&:to_s)
+
+        [columns.each_with_index.to_h, table.fetch('records')]
+      end
+
+      # Insert or update rows for one split table from the full location dump.
+      #
+      # @param model          [Class] ActiveRecord model class
+      # @param records        [Array<Array>] full location dump rows
+      # @param column_indexes [Hash] source column names mapped to row indexes
+      # @param type           [String] source STI type to select
+      # @param attributes     [Array<String>] destination table attributes
+      # @return [void]
+      def seed_full_data(model, records, column_indexes, type, attributes)
+        type_index = column_indexes.fetch('type')
+        attribute_indexes = attributes.to_h { |attribute| [attribute, column_indexes.fetch(attribute)] }
+        now = timestamp
+        batch = []
+
+        records.each do |record|
+          next unless record[type_index] == type
+
+          batch << full_data_attributes(record, attribute_indexes, now)
+
+          next if batch.size < BATCH_SIZE
+
+          bulk_insert(model, batch)
+          batch.clear
+        end
+
+        bulk_insert(model, batch) unless batch.empty?
+        batch.clear
+      end
+
+      # @param record            [Array] full location dump row
+      # @param attribute_indexes [Hash] destination attributes mapped to row indexes
+      # @param now               [Time] timestamp shared by the current table seed
+      # @return [Hash]
+      def full_data_attributes(record, attribute_indexes, now)
+        attribute_indexes.to_h { |attribute, index| [attribute, record[index]] }
+                         .merge('created_at' => now, 'updated_at' => now)
+      end
+
+      # Older loader versions seeded compact file-backed rows with only +code+
+      # and +name+. Those rows cannot be upserted into the full-data records
+      # because their +code+ values differ from the PSA codes in
+      # +pilipinas_data.yml+, so remove them before inserting the canonical set.
+      #
+      # @param model [Class] ActiveRecord model class
+      # @return [void]
+      def delete_stale_compact_rows(model)
+        model.where(location_id: nil).delete_all
+      end
+
       # Insert or update rows for one table from a YAML file.
       #
       # @param model    [Class]  ActiveRecord model class
@@ -54,16 +142,17 @@ module Pilipinas
         records.each_slice(BATCH_SIZE) do |slice|
           batch = slice.map { |r| r.transform_keys(&:to_s).merge('created_at' => now, 'updated_at' => now) }
           bulk_insert(model, batch)
+          batch.clear
         end
       end
 
       # Perform the most capable bulk-insert available for this AR version.
       #
       # +upsert_all+ (Rails 6.1+) is idempotent — it updates existing rows
-      # matched by the unique index on +code+, which is present in every YAML
-      # record.  Using +code+ (not +location_id+) as the conflict column is
-      # essential because +location_id+ is nullable and NULL != NULL in SQL —
-      # a unique index on a nullable column cannot reliably detect conflicts.
+      # matched by the unique index on +location_id+, the stable identifier in
+      # the full location dump. Compact rows from older loader versions have a
+      # NULL +location_id+ and are deleted before the canonical full-data rows
+      # are inserted.
       #
       # +insert_all+ (Rails 6.0) silently skips conflicts.
       # Legacy path: individual +create!+ calls (raises on conflict).
@@ -74,10 +163,10 @@ module Pilipinas
       def bulk_insert(model, batch)
         if model.respond_to?(:upsert_all)
           begin
-            model.upsert_all(batch, unique_by: :code)
+            model.upsert_all(batch, unique_by: :location_id)
           rescue ArgumentError
             raise Pilipinas::Error,
-                  'pilipinas:load requires a unique index on the `code` column, ' \
+                  'pilipinas:load requires a unique index on the `location_id` column, ' \
                   "which is missing from #{model.table_name}. " \
                   'Your database was likely created with an older version of the gem. ' \
                   "Run the following to add the missing indexes and retry:\n\n  " \
@@ -89,6 +178,17 @@ module Pilipinas
         else
           batch.each { |attrs| model.unscoped.create!(attrs) }
         end
+      ensure
+        clear_query_cache
+      end
+
+      # Clear any per-connection query-cache state so a long-running Rails
+      # process does not retain loader queries or bind values between batches.
+      #
+      # @return [void]
+      def clear_query_cache
+        connection = ActiveRecord::Base.connection
+        connection.clear_query_cache if connection.respond_to?(:clear_query_cache)
       end
 
       # @return [Time] current UTC time, compatible with or without ActiveSupport

data/lib/pilipinas/version.rb

@@ -4,5 +4,5 @@ module Pilipinas
   # Semantic version of the gem.
   #
   # Follows {https://semver.org Semantic Versioning}: MAJOR.MINOR.PATCH.
-  VERSION = '1.1.1'
+  VERSION = '1.1.3'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pilipinas
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 1.1.3
 platform: ruby
 authors:
 - Nujian Den Mark Meralpis