grainery 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ab9b7d46f3a3716cf0f043060f4f267a9be926ffae12cdb477dd85df016529e3
-  data.tar.gz: 44edc342af67c5fce8f77559830eada34772b18f5ac226c76886d1fe65a55fa0
+  metadata.gz: 24d14592917d3f6680818f8096acd7f68f63a7e47bbb26aadbc19084ee106845
+  data.tar.gz: 9eb6671fcd81703bc1a6c450a84ef58213b396c0e3eb731a116dba7d5c22eec6
 SHA512:
-  metadata.gz: 5eea1be971b0ee1b02619377c1b2ec44fc68e8ea302653164a754d04493d31bbe914906391061ca2eaa773794890f6b4a49239c15aefbdc0c8386daa20bef518
-  data.tar.gz: 40258587446259c0e5770b61b1d7282fb7750d64b14aed6983cee287338e4d65338a7b363455c292f1fe56cfcc7f9dd93e71af12f1618a506c7dec71ef5fa35e
+  metadata.gz: d934e0497f34309c019774b752ba83e83e6f083ab61961ebf7f16235290492c1674a7209fda111c56d9044464f598be7f99ac35b6ba2ef5074dfe95b5b62be1a
+  data.tar.gz: 321aaf5825073cf57e747ceda98ae4d4ba1034e180b60a5583474422db80d13fe0507fda4df8b3d71d67b71dc58e25952d22c3abb9e6dba8e7cff8519aa34f21

data/CHANGELOG.md

@@ -5,6 +5,69 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.2.0] - 2025-10-01
+
+### Added
+- Database schema dumping functionality for all related databases
+- Schema files now generated in each database directory (e.g., `db/grainery/primary/schema.rb`)
+- New rake task: `grainery:load_with_schema` - Load schemas and seeds together
+- New rake task: `grainery:generate_data_only` - Harvest data without schema dump
+- New rake task: `grainery:generate_raw` - Harvest data without anonymization
+- Schema loading option in `load_seeds` method with `load_schema` parameter
+- **Production environment protection**: Destructive tasks now blocked in production by default
+  - Affects: `grainery:load`, `grainery:load_with_schema`, and all `test:db:*` tasks
+  - Override with `GRAINERY_ALLOW_PRODUCTION=true` environment variable
+  - Includes 5-second safety countdown when override is used
+- **Data Anonymization**: Automatic anonymization of sensitive fields using Faker gem
+  - **Automatic field detection**: `grainery:init_config` scans database schema and auto-detects anonymizable fields
+  - **Scoped anonymization**: Support for table-specific and database-specific field configuration using `table.field` or `database.table.field` notation
+  - Automatic scoping when duplicate field names are detected across multiple tables
+  - Default anonymization for common fields (email, name, phone, address, SSN, credit cards, passwords, tokens, API keys)
+  - Greek-specific document anonymization:
+    - `greek_vat` - Greek VAT number (AFM - 9 digits)
+    - `greek_amka` - Greek Social Security Number (11 digits: DDMMYY + 5 digits)
+    - `greek_personal_number` - Greek Personal Number (12 characters: 2 digits + letter + 9-digit AFM)
+    - `greek_ada` - Greek ADA/Diavgeia Decision Number (15 characters: 4 Greek letters + 2 digits + 4 Greek letters + dash + 1 digit + 2 Greek letters, e.g., "ΨΜΦΡ69ΟΤΝΡ-9ΤΟ")
+    - `greek_adam` - Greek ADAM/Public Procurement Publicity identifier (14-15 characters: 2 digits + PROC or REQ + 9 digits, e.g., "24REQ187755230" or "23PROC456789012")
+    - `iban` - Greek IBAN (27 characters)
+  - `date_of_birth` - Anonymizes birth dates while preserving approximate age (±2 years, minimum age 18 to preserve adulthood)
+  - Selective anonymization: Use `skip` value to preserve real data for specific non-sensitive fields
+  - All fake values automatically respect database column size limits
+  - Configurable anonymization via `anonymize_fields` in `config/grainery.yml`
+
+### Changed
+- `grainery:generate` and `grainery:generate_all` now dump database schemas by default
+- All generation tasks (`generate`, `generate_all`, `generate_data_only`) now anonymize data by default
+- `grainery:load` continues to load only seed data (schemas optional)
+- Schema dump includes table definitions, columns with attributes, and indexes
+- Updated test framework from RSpec to Minitest
+- Rails dependency updated to support versions 6.1 through 8.x
+- Ruby requirement updated to >= 3.2.0
+- Anonymization happens during harvest, not during load
+- Generated seed files contain anonymized data safe for version control
+
+### Security
+- Added production environment safeguards to prevent accidental data loss
+- Destructive operations require explicit opt-in via environment variable in production
+- Sensitive data is now anonymized by default during harvest
+- Safe to commit anonymized seed files to version control
+- Lookup tables are not anonymized (reference data)
+
+### Technical Details
+- Schema dumps use `ActiveRecord::Schema.define` format
+- Each database connection gets its own schema file
+- Schema loading occurs before seed data when enabled
+- Automatic detection and skip of internal Rails tables (schema_migrations, ar_internal_metadata)
+- Anonymization uses Faker gem for realistic fake data
+- Automatic field detection uses pattern matching on column names (email, phone, ssn, afm, amka, ada, adam, etc.)
+- Detected anonymizable fields are automatically added to `config/grainery.yml` during initialization
+- Scoped field resolution with priority: `database.table.field` > `table.field` > `field`
+- When duplicate field names detected, automatically uses scoped configuration
+- Type-aware anonymization respects column data types and size limits
+- String fields automatically truncated to match column maximum length
+- Numeric fields maintain their data type
+- Date of birth anonymization preserves age categories while protecting actual birth dates
+
 ## [0.1.0] - 2025-10-01
 
 ### Added
@@ -30,4 +93,5 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - `test:db:clean` - Truncate all test tables
   - `test:db:stats` - Show test database statistics
 
+[0.2.0]: https://github.com/mpantel/grainery/releases/tag/v0.2.0
 [0.1.0]: https://github.com/mpantel/grainery/releases/tag/v0.1.0

data/README.md

@@ -2,17 +2,23 @@
 
 Database seed storage system for Rails applications. Extract database records and generate seed files organized by database with automatic dependency resolution. Like a grainery stores grain, this gem stores and organizes your database seeds.
 
+> **Note:** This gem was developed with assistance from [Claude](https://claude.ai), Anthropic's AI assistant. Claude helped with code generation, documentation, and testing strategies throughout the development process.
+
+> **⚠️ Development Status:** This gem is in active development and does not yet have a comprehensive test suite. While the core functionality has been tested manually, automated tests are planned for future releases. Use with caution in production environments.
+
 ## Features
 
 - ✅ Automatic database detection
 - ✅ Dependency-aware loading (topological sort)
 - ✅ Multi-database support
+- ✅ Database schema dumping for all related databases
 - ✅ Configurable per project
 - ✅ Preserves custom seeds
 - ✅ One seed file per table
 - ✅ Clean separation of concerns
 - ✅ Supports SQL Server, MySQL, PostgreSQL
 - ✅ Test database management tasks
+- ✅ Rails 6.1 - 8.x support
 
 ## Installation
 
@@ -36,27 +42,48 @@ bundle install
 rake grainery:init_config
 ```
 
-This auto-detects all databases and creates `config/grainery.yml`.
+This auto-detects:
+- All databases and model base classes
+- Anonymizable fields in your database schema (email, phone, SSN, Greek documents, etc.)
+- Creates `config/grainery.yml` with detected configuration
 
 ### 2. Harvest Data
 
 ```bash
-# Harvest with limit (100 records per table)
+# Harvest with limit (100 records per table) + schema dump + anonymization
 rake grainery:generate
 
-# Harvest ALL records (use with caution)
+# Harvest ALL records + schema dump + anonymization (use with caution)
 rake grainery:generate_all
+
+# Harvest data only (no schema dump) + anonymization
+rake grainery:generate_data_only
+
+# Harvest without anonymization (raw production data - use with extreme caution!)
+rake grainery:generate_raw
 ```
 
+**Note:** By default, sensitive fields are anonymized using Faker. Configure anonymization in `config/grainery.yml`.
+
 ### 3. Load Seeds
 
 ```bash
+# Load seeds only (blocked in production)
 rake grainery:load
+
+# Load schemas + seeds (blocked in production)
+rake grainery:load_with_schema
+
+# Override production protection (use with extreme caution!)
+GRAINERY_ALLOW_PRODUCTION=true rake grainery:load
 ```
 
+**Note:** Loading tasks are blocked in production by default to prevent accidental data loss.
+
 This loads:
-1. Harvested seeds (in dependency order)
-2. Custom seeds from `db/seeds.rb` (last)
+1. Database schemas (if using `load_with_schema`)
+2. Harvested seeds (in dependency order)
+3. Custom seeds from `db/seeds.rb` (last)
 
 ## Directory Structure
 
@@ -65,12 +92,15 @@ db/
 ├── grainery/                   # Harvested seeds (auto-generated)
 │   ├── load_order.txt         # Load order respecting dependencies
 │   ├── primary/               # Primary database
+│   │   ├── schema.rb          # Database schema dump
 │   │   ├── users.rb
 │   │   ├── posts.rb
 │   │   └── comments.rb
 │   ├── other/                 # Other database
+│   │   ├── schema.rb          # Database schema dump
 │   │   └── projects.rb
 │   └── banking/               # Banking database
+│       ├── schema.rb          # Database schema dump
 │       └── employees.rb
 └── seeds.rb                   # Custom seeds (loaded last)
 ```
@@ -97,6 +127,51 @@ database_connections:
 
 # Lookup tables (harvest all records)
 lookup_tables: []
+
+# Field anonymization (column_name => faker_method)
+# Set to empty hash {} to disable anonymization
+anonymize_fields:
+  email: email
+  first_name: first_name
+  last_name: last_name
+  name: name
+  phone: phone_number
+  phone_number: phone_number
+  address: address
+  street_address: street_address
+  city: city
+  state: state
+  zip: zip_code
+  zip_code: zip_code
+  postal_code: zip_code
+  ssn: ssn
+  credit_card: credit_card_number
+  password: password
+  token: token
+  api_key: api_key
+  secret: secret
+  iban: iban
+  vat_number: greek_vat
+  afm: greek_vat
+  amka: greek_amka
+  social_security_number: greek_amka
+  ssn_greek: greek_amka
+  personal_number: greek_personal_number
+  personal_id: greek_personal_number
+  afm_extended: greek_personal_number
+  ada: greek_ada
+  diavgeia_id: greek_ada
+  decision_number: greek_ada
+  adam: greek_adam
+  adam_number: greek_adam
+  procurement_id: greek_adam
+  date_of_birth: date_of_birth
+  birth_date: date_of_birth
+  dob: date_of_birth
+  birthdate: date_of_birth
+  identity_number: identity_number
+  id_number: identity_number
+  national_id: identity_number
 ```
 
 ## Available Rake Tasks
@@ -107,15 +182,24 @@ lookup_tables: []
 # Initialize configuration
 rake grainery:init_config
 
-# Harvest data (with limit)
+# Harvest data (with limit) + schema dump + anonymization
 rake grainery:generate
 
-# Harvest ALL records
+# Harvest ALL records + schema dump + anonymization
 rake grainery:generate_all
 
+# Harvest data only (no schema dump) + anonymization
+rake grainery:generate_data_only
+
+# Harvest without anonymization (raw production data)
+rake grainery:generate_raw
+
 # Load harvested + custom seeds
 rake grainery:load
 
+# Load schemas + seeds + custom seeds
+rake grainery:load_with_schema
+
 # Clean grainery directory
 rake grainery:clean
 ```
@@ -177,7 +261,33 @@ lookup_tables:
   - categories
 ```
 
-## Seed File Format
+## File Formats
+
+### Schema File Format
+
+Each database gets a schema dump:
+
+```ruby
+# Schema dump for primary database
+# Generated: 2025-10-01 10:30:00
+# Adapter: postgresql
+
+ActiveRecord::Schema.define do
+
+  create_table "users", force: :cascade do |t|
+    t.string "email", null: false
+    t.string "name"
+    t.boolean "active", default: true
+    t.datetime "created_at", null: false
+    t.datetime "updated_at", null: false
+  end
+
+  add_index "users", ["email"], unique: true
+
+end
+```
+
+### Seed File Format
 
 Each table gets its own seed file:
 
@@ -221,33 +331,225 @@ Setting.create!(key: 'app_name', value: 'My App')
 
 ### Development
 ```bash
-# Harvest production-like data for development
+# Harvest production-like data for development with schemas
 rake grainery:generate
-rake grainery:load
+rake grainery:load_with_schema
 ```
 
 ### Testing
 ```bash
-# Create test fixtures
+# Create test fixtures with schemas
 rake grainery:generate
-# In test setup, load specific seeds as needed
+# In test setup, load schemas and seeds
+rake grainery:load_with_schema
 ```
 
 ### Staging
 ```bash
-# Harvest production data (anonymized)
+# Harvest production data (anonymized) with schemas
 rake grainery:generate_all
 # Deploy to staging
-# Load on staging server
-rake grainery:load
+# Load on staging server with full schema
+rake grainery:load_with_schema
+```
+
+### Cross-Database Migration
+```bash
+# Export from one database system
+rake grainery:generate_all  # Captures schema + data
+
+# Import to another database system
+rake grainery:load_with_schema  # Recreates schema + loads data
 ```
 
 ## Safety Features
 
-1. **Separate Directories**: Harvested seeds never touch `db/seeds.rb`
-2. **Dependency Order**: Foreign keys respected automatically
-3. **Custom Preservation**: Your `db/seeds.rb` always loads last
-4. **Clean Command**: `rake grainery:clean` removes only harvested files
+1. **Production Environment Protection**: Destructive tasks (load, load_with_schema, test:db:*) are blocked in production
+   - Requires explicit `GRAINERY_ALLOW_PRODUCTION=true` environment variable to override
+   - Includes 5-second countdown when override is used
+2. **Separate Directories**: Harvested seeds never touch `db/seeds.rb`
+3. **Dependency Order**: Foreign keys respected automatically
+4. **Custom Preservation**: Your `db/seeds.rb` always loads last
+5. **Clean Command**: `rake grainery:clean` removes only harvested files
+6. **Optional Schema Loading**: Schemas only load when explicitly requested
+7. **Per-Database Schemas**: Each database gets isolated schema file
+
+### Production Safety Matrix
+
+**Safe Operations (Read-Only):**
+- ✅ `rake grainery:generate` - Harvests data, no modifications
+- ✅ `rake grainery:generate_all` - Harvests all data, no modifications
+- ✅ `rake grainery:generate_data_only` - Harvests data only, no modifications
+- ✅ `rake grainery:init_config` - Creates config file only
+- ✅ `rake grainery:clean` - Deletes harvested files only (not database data)
+
+**Destructive Operations (Blocked by Default):**
+- ❌ `rake grainery:load` - Inserts data into database
+- ❌ `rake grainery:load_with_schema` - Modifies schema AND inserts data
+- ❌ `rake test:db:*` - All test database operations
+
+**Recommendation:**
+- Harvesting in production is safe and useful for creating staging/development fixtures
+- Loading in production should be tested thoroughly in staging first due to lack of automated test coverage
+- Always review generated files before loading into any environment
+
+## Data Anonymization
+
+✅ **Built-in Anonymization:** Grainery automatically anonymizes sensitive fields using the Faker gem during harvest.
+
+### Automatic Detection
+
+When you run `rake grainery:init_config`, Grainery automatically:
+1. Scans all database tables and columns
+2. Detects fields that should be anonymized based on naming patterns
+3. Adds them to `config/grainery.yml` with appropriate anonymization methods
+
+Detected patterns include: `email`, `phone`, `address`, `ssn`, `password`, `token`, Greek documents (`afm`, `amka`, `ada`, `adam`), dates of birth, and more.
+
+### How It Works
+
+When harvesting, Grainery automatically replaces sensitive field values with fake data:
+
+```ruby
+# Original production data:
+{ email: "john.doe@company.com", name: "John Doe", phone: "555-1234" }
+
+# Anonymized in seed files:
+{ email: "jane_smith@example.org", name: "Sarah Johnson", phone: "555-987-6543" }
+```
+
+### Configuration
+
+The `config/grainery.yml` file is automatically populated with detected fields during initialization. You can customize it as needed:
+
+```yaml
+anonymize_fields:
+  # Global field configuration (applies to all tables)
+  email: email                    # Uses Faker::Internet.email
+  first_name: first_name          # Uses Faker::Name.first_name
+  last_name: last_name            # Uses Faker::Name.last_name
+  name: name                      # Uses Faker::Name.name
+  phone: phone_number             # Uses Faker::PhoneNumber.phone_number
+  ssn: ssn                        # Uses Faker::IDNumber.valid
+
+  # Table-specific configuration (when same field appears in multiple tables)
+  users.address: address          # Only anonymize address in users table
+  companies.address: skip         # Don't anonymize address in companies table
+
+  # Database.table-specific configuration (most specific)
+  primary.users.email: email      # Only for users table in primary database
+  other.contacts.email: email     # Only for contacts table in other database
+```
+
+**Scoping Priority:**
+1. `database.table.field` (highest priority - most specific)
+2. `table.field` (medium priority - table-specific)
+3. `field` (lowest priority - global)
+
+When a field name appears in multiple tables, Grainery automatically uses scoped names during detection.
+
+### Disabling Anonymization
+
+**Option 1: Disable completely**
+
+```yaml
+# Set to empty hash
+anonymize_fields: {}
+```
+
+**Option 2: Use raw generation task**
+
+```bash
+rake grainery:generate_raw  # Harvests without anonymization
+```
+
+**Option 3: Skip specific fields**
+
+To keep real values for specific fields while anonymizing others, set them to `skip`:
+
+```yaml
+anonymize_fields:
+  email: email           # Will be anonymized
+  name: name             # Will be anonymized
+  company_name: skip     # Will keep real value (not anonymized)
+  department: skip       # Will keep real value (not anonymized)
+  phone: phone_number    # Will be anonymized
+```
+
+This is useful when you need to preserve certain non-sensitive reference data while still protecting personal information.
+
+### Supported Faker Methods
+
+**Personal Information:**
+- `email` - Fake email addresses
+- `first_name`, `last_name`, `name` - Fake names
+- `phone_number` - Fake phone numbers
+- `address`, `street_address` - Fake addresses
+- `city`, `state`, `zip_code`, `postal_code` - Fake location data
+- `date_of_birth` - Fake date of birth preserving approximate age (±2 years, minimum age 18 to preserve adulthood)
+
+**Financial & Identity:**
+- `ssn` - Fake social security numbers
+- `credit_card_number` - Fake credit card numbers
+- `iban` - Fake Greek IBAN (27 characters: GR + check digits + bank code + account number, auto-truncates to column size)
+- `greek_vat` - Fake Greek VAT number (AFM - 9 digits, adjusts to column size)
+- `greek_amka` - Fake Greek AMKA/Social Security Number (11 digits: DDMMYY + 5 digits, adjusts to column size)
+- `greek_personal_number` - Fake Greek Personal Number (12 characters: 2 digits + letter + 9-digit AFM, e.g., "12A123456789", adjusts to column size)
+- `greek_ada` - Fake Greek ADA/Diavgeia Decision Number (15 characters: 4 Greek letters + 2 digits + 4 Greek letters + dash + 1 digit + 2 Greek letters, e.g., "ΨΜΦΡ69ΟΤΝΡ-9ΤΟ", adjusts to column size)
+- `greek_adam` - Fake Greek ADAM/Public Procurement Publicity identifier (14-15 characters: 2 digits + PROC or REQ + 9 digits, e.g., "24REQ187755230" or "23PROC456789012", adjusts to column size)
+- `identity_number` - Fake identity number (alphanumeric format, adjusts to column size)
+
+**Security:**
+- `password` - Fake passwords (auto-truncates to column size)
+- `token` - Random alphanumeric strings (defaults to 32 characters, adjusts to column size)
+- `api_key` - Random alphanumeric strings (defaults to 40 characters, adjusts to column size)
+- `secret` - Random alphanumeric strings (defaults to 64 characters, adjusts to column size)
+
+### Custom Field Mapping
+
+Add your own field mappings to anonymize custom columns:
+
+```yaml
+anonymize_fields:
+  # Global mappings
+  employee_id: ssn
+  mobile: phone_number
+  home_address: address
+  work_email: email
+  tax_id: ssn
+  bank_account: iban
+  tin: greek_vat
+  social_insurance: greek_amka
+  citizen_id: greek_personal_number
+  passport_number: identity_number
+  diavgeia_decision: greek_ada
+  procurement_number: greek_adam
+  birth_date: date_of_birth
+
+  # Scoped examples for duplicate fields
+  users.status: skip              # Don't anonymize status in users
+  orders.status: skip             # Don't anonymize status in orders
+  primary.employees.department: skip  # Department in primary.employees
+  other.staff.department: skip        # Department in other.staff
+
+  # Skip anonymization for non-sensitive fields
+  company_name: skip
+  department: skip
+  job_title: skip
+```
+
+### Important Notes
+
+- Anonymization happens **during harvest**, not during load
+- Generated seed files contain anonymized data
+- Original production data is never modified
+- Safe to commit anonymized seed files to version control
+- Lookup tables are not anonymized (reference data)
+- Anonymization can be disabled per-harvest using `generate_raw` task
+- **Respects database constraints**: Fake values are automatically truncated to match column size limits
+- **Type-aware**: String fields respect their maximum length, numeric fields maintain their data type
+- **Selective anonymization**: Use `skip` to preserve real values for specific fields while anonymizing others
+- **Scoped configuration**: When the same field name appears in multiple tables, use `table.field` or `database.table.field` notation for table-specific or database-specific anonymization
 
 ## Best Practices