schema-model 0.6.11 → 0.7.0

data/README.md

@@ -2,19 +2,8 @@
 
 A powerful Ruby gem for data transformation, validation, and type safety. Schema provides a flexible and intuitive way to define data models with support for complex nested structures, dynamic associations, and robust validation.
 
-[![Build Status](https://travis-ci.org/dougyouch/schema.svg?branch=master)](https://travis-ci.org/dougyouch/schema)
-[![Maintainability](https://api.codeclimate.com/v1/badges/c142d46a7a37d4a8c2e5/maintainability)](https://codeclimate.com/github/dougyouch/schema/maintainability)
-[![Test Coverage](https://api.codeclimate.com/v1/badges/c142d46a7a37d4a8c2e5/test_coverage)](https://codeclimate.com/github/dougyouch/schema/test_coverage)
-
-## Features
-
-- **Type Safety**: Strong typing with automatic parsing and validation
-- **Flexible Attributes**: Support for aliases and custom data types
-- **Nested Models**: Complex data structures with nested associations
-- **Dynamic Associations**: Runtime type-based model creation
-- **ActiveModel Integration**: Seamless integration with ActiveModel validations
-- **Error Handling**: Comprehensive error collection and reporting
-- **CSV Support**: Built-in CSV parsing capabilities
+[![CI](https://github.com/dougyouch/schema/actions/workflows/ci.yml/badge.svg)](https://github.com/dougyouch/schema/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/dougyouch/schema/graph/badge.svg)](https://codecov.io/gh/dougyouch/schema)
 
 ## Installation
 
@@ -32,8 +21,6 @@ $ bundle install
 
 ## Quick Start
 
-Here's a simple example to get you started:
-
 ```ruby
 class UserSchema
   include Schema::All
@@ -41,140 +28,429 @@ class UserSchema
   attribute :name, :string
   attribute :age, :integer
   attribute :email, :string
+  attribute :active, :boolean, default: false
   attribute :tags, :array, separator: ',', data_type: :string
 
   validates :name, presence: true
   validates :email, presence: true, format: { with: URI::MailTo::EMAIL_REGEXP }
 end
 
-# Usage
-user_data = {
+user = UserSchema.from_hash(
   name: 'John Doe',
   age: '30',
   email: 'john@example.com',
+  active: 'yes',
   tags: 'ruby,rails,developer'
-}
+)
 
-user = UserSchema.from_hash(user_data)
-if user.valid?
-  puts "User created: #{user.name}"
-else
-  puts "Validation errors: #{user.errors.full_messages}"
-end
+user.valid?        # => true
+user.name          # => "John Doe"
+user.age           # => 30 (parsed to integer)
+user.active        # => true (parsed from "yes")
+user.tags          # => ["ruby", "rails", "developer"]
+```
+
+## Data Types
+
+### Basic Types
+
+```ruby
+attribute :name, :string              # String values
+attribute :count, :integer            # Integer values (parses "123" to 123)
+attribute :price, :float              # Float values (parses "9.99" to 9.99)
+attribute :active, :boolean           # Boolean (accepts: 1, t, true, on, y, yes)
+attribute :notes, :string_or_nil      # String, but returns nil if empty
+```
+
+### Date and Time Types
+
+```ruby
+attribute :created_at, :time          # ISO 8601 format (Time.xmlschema)
+attribute :birth_date, :date          # Date.parse format
+attribute :us_date, :american_date    # MM/DD/YYYY format
+attribute :us_time, :american_time    # MM/DD/YYYY HH:MM:SS format
 ```
 
-## Core Concepts
+### Complex Types
 
-### Attributes
+```ruby
+attribute :tags, :array                           # Array of values
+attribute :tags, :array, separator: ','           # Parse "a,b,c" into ["a","b","c"]
+attribute :tags, :array, separator: ',', data_type: :integer  # Parse and convert elements
+attribute :metadata, :hash                        # Hash/dictionary values
+attribute :config, :json                          # Parse JSON strings
+```
+
+## Attribute Options
 
-Attributes define the structure of your data model. Each attribute has:
-- A name
-- A type
-- Optional aliases
-- Custom parsing rules
+### Aliases
 
 ```ruby
+# Single alias
 attribute :name, :string, alias: 'FullName'
-attribute :age, :integer
-attribute :tags, :array, separator: ',', data_type: :string
+
+# Multiple aliases
+attribute :name, :string, aliases: [:full_name, :display_name]
 ```
 
-### Associations
+### Default Values
 
-Schema supports various types of associations:
+```ruby
+attribute :status, :string, default: 'pending'
+attribute :count, :integer, default: 0
+attribute :tags, :array, default: []
+```
+
+### Checking If Attribute Was Set
 
-1. **Has One**: Single nested model
-2. **Has Many**: Multiple nested models
-3. **Dynamic Associations**: Type-based model creation
+Every attribute generates a `_was_set?` predicate method:
 
 ```ruby
-has_one(:profile) do
-  attribute :bio, :string
-  attribute :website, :string
-end
+user = UserSchema.from_hash(name: 'John')
+user.name_was_set?   # => true
+user.email_was_set?  # => false (not provided)
 
-has_many(:posts) do
-  attribute :title, :string
-  attribute :content, :string
-end
+# Useful for distinguishing "not provided" vs "provided as nil"
+user = UserSchema.from_hash(name: nil)
+user.name_was_set?   # => true (explicitly set to nil)
 ```
 
-### Dynamic Types
+## Associations
 
-Create different model structures based on a type field:
+### Has One
 
 ```ruby
-has_many(:vehicles, type_field: :type) do
-  attribute :type, :string
-  attribute :color, :string
+class OrderSchema
+  include Schema::All
+
+  attribute :id, :integer
 
-  add_type('car') do
-    attribute :doors, :integer
+  has_one :customer do
+    attribute :name, :string
+    attribute :email, :string
   end
+end
+
+order = OrderSchema.from_hash(
+  id: 1,
+  customer: { name: 'Alice', email: 'alice@example.com' }
+)
+order.customer.name  # => "Alice"
+```
 
-  add_type('truck') do
-    attribute :bed_length, :float
+### Has Many
+
+```ruby
+class OrderSchema
+  include Schema::All
+
+  attribute :id, :integer
+
+  has_many :items do
+    attribute :sku, :string
+    attribute :quantity, :integer
   end
 end
+
+order = OrderSchema.from_hash(
+  id: 1,
+  items: [
+    { sku: 'ABC', quantity: 2 },
+    { sku: 'XYZ', quantity: 1 }
+  ]
+)
+order.items.length       # => 2
+order.items.first.sku    # => "ABC"
+```
+
+### Association Options
+
+```ruby
+# Default values - creates empty model/array if not provided
+has_one :profile, default: true
+has_many :tags, default: true
+
+# Reuse existing schema class
+has_one :shipping_address, base_class: AddressSchema
+has_one :billing_address, base_class: AddressSchema
+
+# Has many from hash (keyed by field)
+has_many :items, from: :hash, hash_key_field: :id do
+  attribute :id, :string
+  attribute :name, :string
+end
+
+# Input: { items: { 'abc' => { name: 'Item 1' }, 'xyz' => { name: 'Item 2' } } }
+# Result: items[0].id => 'abc', items[1].id => 'xyz'
 ```
 
-## Advanced Features
+### Appending to Has Many
 
-### Custom Parsers
+```ruby
+order = OrderSchema.from_hash(id: 1, items: [])
+order.append_to_items(sku: 'NEW', quantity: 5)
+order.items.length  # => 1
+```
 
-Define custom parsing logic for your attributes:
+## Dynamic Types
+
+Create different model structures based on a type field:
 
 ```ruby
-attribute :custom_field, :custom_type do
-  def parse_custom_type(field_name, errors, value)
-    # Custom parsing logic
+class CompanySchema
+  include Schema::All
+
+  has_many :employees, type_field: :type do
+    attribute :type, :string
+    attribute :name, :string
+
+    add_type('engineer') do
+      attribute :programming_languages, :array, separator: ','
+    end
+
+    add_type('manager') do
+      attribute :department, :string
+      attribute :team_size, :integer
+    end
+
+    default_type do
+      # Fallback for unknown types
+    end
   end
 end
+
+company = CompanySchema.from_hash(
+  employees: [
+    { type: 'engineer', name: 'Alice', programming_languages: 'ruby,python' },
+    { type: 'manager', name: 'Bob', department: 'Engineering', team_size: 5 }
+  ]
+)
+
+company.employees[0].programming_languages  # => ["ruby", "python"]
+company.employees[1].team_size              # => 5
 ```
 
-### CSV Integration
+### Dynamic Type Options
 
-Parse CSV data directly into your models:
+```ruby
+# Type field within nested data (default)
+has_many :items, type_field: :kind do
+  # looks for :kind in each item's data
+end
+
+# Type determined by parent field
+has_one :details, external_type_field: :category do
+  # uses parent's :category field to determine type
+end
+
+# Case-insensitive type matching
+has_many :items, type_field: :type, type_ignorecase: true do
+  add_type('widget') { }  # matches "Widget", "WIDGET", etc.
+end
+```
+
+## Validation and Error Handling
+
+### ActiveModel Validations
 
 ```ruby
-class UserCSVSchema
-  include Schema::CSVParser
-  
+class UserSchema
+  include Schema::All
+
   attribute :name, :string
   attribute :email, :string
+  attribute :age, :integer
+
+  validates :name, presence: true
+  validates :email, format: { with: URI::MailTo::EMAIL_REGEXP }
+  validates :age, numericality: { greater_than: 0 }, allow_nil: true
+end
+```
+
+### Parsing Errors vs Validation Errors
+
+```ruby
+user = UserSchema.from_hash(name: 'John', age: 'not-a-number')
+
+# Parsing errors (type conversion failures)
+user.parsing_errors.empty?  # => false
+user.parsed?                # => false
+
+# Validation errors (business rules)
+user.valid?                 # => false (also runs validations)
+user.errors.full_messages   # => ["Age is invalid", ...]
+```
+
+### Raising Exceptions
+
+```ruby
+user = UserSchema.from_hash(age: 'invalid')
+
+user.parsed!      # raises Schema::ParsingException if parsing errors
+user.valid_model! # raises Schema::ValidationException if validation errors
+user.valid!       # raises either (checks both)
+
+# Exception includes the model and errors
+begin
+  user.valid!
+rescue Schema::ParsingException => e
+  e.schema  # => the model instance
+  e.errors  # => the errors object
 end
+```
+
+### Unknown Attributes
+
+By default, unknown attributes are captured as parsing errors:
+
+```ruby
+user = UserSchema.from_hash(name: 'John', unknown_field: 'value')
+user.parsing_errors[:unknown_field]  # => ["unknown_attribute"]
 
-users = UserCSVSchema.parse_csv(csv_data)
+# Disable this behavior
+UserSchema.capture_unknown_attributes = false
 ```
 
-## Using `skip_fields` to Protect Internal Fields
+## Serialization
+
+### to_hash / as_json
+
+```ruby
+user = UserSchema.from_hash(name: 'John', email: nil)
+
+user.to_hash                          # => { name: "John", email: nil }
+user.as_json                          # => { name: "John" } (excludes nils)
+user.as_json(include_nils: true)      # => { name: "John", email: nil }
 
-When instantiating a schema with `from_hash`, you can use the `skip_fields` argument to prevent certain fields (such as `id`, `created_at`, `updated_at`) from being set by user input. This is especially useful for fields managed by the database or internal logic, ensuring end users cannot override these values.
+# Filter fields
+user.as_json(select_filter: ->(name, value, opts) { name == :name })
+user.as_json(reject_filter: ->(name, value, opts) { value.nil? })
+```
+
+## Protecting Fields with skip_fields
 
-**Example:**
+Prevent certain fields from being set by user input:
 
 ```ruby
 user_data = {
-  id: 123, # Should be managed by DB
+  id: 123,
   name: 'John Doe',
-  email: 'john@example.com',
-  created_at: '2024-06-01T12:00:00Z', # Should be managed by DB
-  updated_at: '2024-06-01T12:00:00Z'  # Should be managed by DB
+  created_at: '2024-01-01'
 }
 
-# Skip DB-managed fields
-user = UserSchema.from_hash(user_data, [:id, :created_at, :updated_at])
+# Skip database-managed fields
+user = UserSchema.from_hash(user_data, [:id, :created_at])
+
+user.id          # => nil (not set)
+user.name        # => "John Doe"
+user.created_at  # => nil (not set)
+
+# Nested skip_fields for associations
+order = OrderSchema.from_hash(data, [:id, { items: [:id] }])
+```
+
+## Array and CSV Support
+
+### Schema::Arrays Module
+
+Convert models to/from flat arrays (useful for CSV/spreadsheet data):
+
+```ruby
+class UserSchema
+  include Schema::All
+  schema_include Schema::Arrays
+
+  attribute :name, :string
+  attribute :email, :string
+end
+
+# Get headers
+UserSchema.to_headers  # => ["name", "email"]
+
+# Convert to array
+user = UserSchema.from_hash(name: 'John', email: 'john@example.com')
+user.to_a  # => ["John", "john@example.com"]
+
+# Create from array
+headers = ['name', 'email']
+mapped = UserSchema.map_headers_to_attributes(headers)
+user = UserSchema.from_array(['Jane', 'jane@example.com'], mapped)
+```
+
+### Schema::CSVParser Module
+
+Parse CSV data directly into models:
+
+```ruby
+class UserCSVSchema
+  include Schema::Model
+  include Schema::CSVParser
+
+  attribute :name, :string
+  attribute :email, :string
+end
+
+csv_data = CSV.parse("name,email\nJohn,john@example.com", headers: true)
+parser = Schema::CSVParser.new(csv_data, UserCSVSchema)
 
-puts user.id          # => nil (not set from user input)
-puts user.created_at  # => nil (not set from user input)
-puts user.updated_at  # => nil (not set from user input)
-puts user.name        # => 'John Doe'
+parser.each do |user|
+  puts user.name
+end
 ```
 
-**Benefit:**
-- Prevents end users from setting or changing internal DB values.
-- Ensures only safe, intended fields are settable from external input.
-- Helps maintain data integrity and security in your application.
+### Schema::ArrayHeaders Module
+
+Map CSV/array headers to schema attributes:
+
+```ruby
+class UserSchema
+  include Schema::All
+  schema_include Schema::ArrayHeaders
+
+  attribute :name, :string, alias: 'FullName'
+  attribute :email, :string
+end
+
+headers = ['FullName', 'email', 'unknown_column']
+mapped = UserSchema.map_headers_to_attributes(headers)
+# => { name: { index: 0 }, email: { index: 1 } }
+
+UserSchema.get_mapped_field_names(mapped)    # => ["name", "email"]
+UserSchema.get_unmapped_field_names(mapped)  # => []
+```
+
+## Extending Schemas
+
+### schema_include
+
+Add modules to a schema and all its nested associations:
+
+```ruby
+class OrderSchema
+  include Schema::All
+
+  has_many :items do
+    attribute :name, :string
+  end
+end
+
+# Add Arrays support to OrderSchema and OrderSchema::SchemaHasManyItems
+OrderSchema.schema_include Schema::Arrays
+```
+
+## CLI Tools
+
+### schema-json2csv
+
+Convert JSON data to CSV using a schema:
+
+```bash
+# Basic usage
+schema-json2csv --require ./my_schema.rb --schema MySchema --json data.json --csv output.csv
+
+# From stdin
+cat data.json | schema-json2csv --require ./my_schema.rb --schema MySchema - --csv output.csv
+```
 
 ## Contributing
 
@@ -187,4 +463,3 @@ puts user.name        # => 'John Doe'
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
-

data/lib/schema/active_model_validations.rb

@@ -11,13 +11,13 @@ module Schema
     end
 
     def valid_model!
-      unless valid?
-        raise ValidationException.new(
-                "invalid values for attributes #{errors.map(&:attribute).join(', ')}",
-                self,
-                errors
-              )
-      end
+      return if valid?
+
+      raise ValidationException.new(
+        "invalid values for attributes #{errors.map(&:attribute).join(', ')}",
+        self,
+        errors
+      )
     end
 
     def valid!
@@ -36,13 +36,13 @@ module Schema
       end
 
       def parsed!
-        unless parsed?
-          raise ParsingException.new(
-                  "schema parsing failed for attributes #{parsing_errors.errors.map(&:attribute).join(', ')}",
-                  self,
-                  parsing_errors
-                )
-        end
+        return if parsed?
+
+        raise ParsingException.new(
+          "schema parsing failed for attributes #{parsing_errors.errors.map(&:attribute).join(', ')}",
+          self,
+          parsing_errors
+        )
       end
     end
   end

data/lib/schema/array_headers.rb

@@ -41,13 +41,15 @@ module Schema
         fields
       end
 
+      MAX_ARRAY_INDEX = 10_000
+
       private
 
       def get_model_field_names(field_name, field_options, mapped_headers, header_prefix, mapped)
         mapped_model = mapped_headers[field_name] || {}
         const_get(field_options[:class_name]).get_field_names(
           mapped_model,
-          header_prefix || (field_options[:aliases] && field_options[:aliases].first),
+          header_prefix || field_options[:aliases]&.first,
           mapped
         )
       end
@@ -62,7 +64,7 @@ module Schema
 
       def generate_field_name(field_name, field_options, header_prefix)
         field_name = field_options[:aliases].first if field_options[:aliases]
-        field_name = header_prefix + 'X' + field_name.to_s if header_prefix
+        field_name = "#{header_prefix}X#{field_name}" if header_prefix
         field_name.to_s
       end
 
@@ -114,9 +116,9 @@ module Schema
         cnt = field_options[:starting_index] || 1
         indexes = []
         # finding all headers that look like Company1Name through CompanyXName
-        while (index = headers.index(header_prefix + cnt.to_s + field_options[:key]))
-          cnt += 1
+        while cnt <= MAX_ARRAY_INDEX && (index = headers.index(header_prefix + cnt.to_s + field_options[:key]))
           indexes << index
+          cnt += 1
         end
         indexes
       end

data/lib/schema/arrays.rb

@@ -15,7 +15,7 @@ module Schema
 
       def to_empty_array
         data = []
-        schema.each do |_, field_options|
+        schema.each_value do |field_options|
           next if field_options[:alias_of]
 
           data <<
@@ -24,8 +24,6 @@ module Schema
               const_get(field_options[:class_name]).to_empty_array
             when :has_many
               field_options[:size].times.map { const_get(field_options[:class_name]).to_empty_array }
-            else
-              nil
             end
         end
         data
@@ -33,16 +31,16 @@ module Schema
 
       def to_headers(prefix = nil)
         headers = []
-        schema.each do |_, field_options|
+        schema.each_value do |field_options|
           next if field_options[:alias_of]
 
           headers <<
             case field_options[:type]
             when :has_one
-              const_get(field_options[:class_name]).to_headers(prefix.to_s + field_options[:key] + '.')
+              const_get(field_options[:class_name]).to_headers("#{prefix}#{field_options[:key]}.")
             when :has_many
               field_options[:size].times.map do |i|
-                const_get(field_options[:class_name]).to_headers(prefix.to_s + field_options[:key] + "[#{i +1}].")
+                const_get(field_options[:class_name]).to_headers(prefix.to_s + field_options[:key] + "[#{i + 1}].")
               end
             else
               prefix.to_s + field_options[:key]
@@ -54,7 +52,7 @@ module Schema
 
     def to_a
       data = []
-      self.class.schema.each do |_, field_options|
+      self.class.schema.each_value do |field_options|
         next if field_options[:alias_of]
 
         value = public_send(field_options[:getter])
@@ -76,7 +74,7 @@ module Schema
     end
 
     def update_attributes_with_array(array, mapped_headers, offset = nil)
-      self.class.schema.each do |_, field_options|
+      self.class.schema.each_value do |field_options|
         next unless (mapped_field = mapped_headers[field_options[:name]])
 
         if offset
@@ -103,7 +101,7 @@ module Schema
     end
 
     def update_nested_has_one_associations_from_array(array, mapped_headers, current_offset = nil)
-      self.class.schema.each do |_, field_options|
+      self.class.schema.each_value do |field_options|
         next unless field_options[:type] == :has_one
         next unless (mapped_model = mapped_headers[field_options[:name]])
 
@@ -115,7 +113,7 @@ module Schema
     end
 
     def update_nested_has_many_associations_from_array(array, mapped_headers)
-      self.class.schema.each do |_, field_options|
+      self.class.schema.each_value do |field_options|
         next unless field_options[:type] == :has_many
         next unless (mapped_model = mapped_headers[field_options[:name]])
 
@@ -136,10 +134,10 @@ module Schema
 
     def largest_number_of_indexes_from_map(mapped_model)
       size = 0
-      mapped_model.each do |_, info|
+      mapped_model.each_value do |info|
         if info[:indexes]
-          size = info[:indexes].size if info[:indexes] && info[:indexes].size > size
-        else
+          size = info[:indexes].size if info[:indexes].size > size
+        elsif info.is_a?(Hash)
           new_size = largest_number_of_indexes_from_map(info)
           size = new_size if new_size > size
         end