better_structure_sql 0.1.0 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4e66ec3fc122539bebd5d502233e08b258246a16f8567dbc5fb88bcdb4b98486
-  data.tar.gz: 0d7bdeaaa4b1e356d3f506fbeefd1e4e20c2e306b469ebe9184e2ab80b224b2e
+  metadata.gz: d309fdf24a779a29028226f4e0926d28087b9045fdff1b9625522bdac430f169
+  data.tar.gz: db6dbe765d3d385a1b8867d3ad1f54825f7cbeed887ab17b7e02a06f63d5ef48
 SHA512:
-  metadata.gz: 357c7013b0db9e2aed20714750783fc5b386cf21bb6bff014b37a613c6aaf3b48e52eabb965b9f75e7a71af5ca136cc1a0b604108d7bb1e4910ad631c36b5bf5
-  data.tar.gz: cb8046cbfd1b37a4ccdcde40b85dc5461b2b73f179536f88bd4376fcd91fa0ea865edbf7b8936e3524fa138719c5656aa64b0cffdc21f7dbc1fef9b4a14514c8
+  metadata.gz: 6d2f15acedf160011639a0653dce7c639cf6e825c4e9a9638bf76e3262ee02d1cd76fc6726b3ff79c03a6054953f2310d6f61fd92af87974face17888574640c
+  data.tar.gz: c559251fe213ffabf576ab234ded58970dd54905813db0e6d3c9ee45a951a9435feac53d05eb93c4758df40829b6e91af509eebe647a72349eb036e12df52283

data/CHANGELOG.md

@@ -13,6 +13,53 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+## [0.2.1] - 2025-11-20
+
+### Changed
+- Improved schema version web UI with content hash display (first 8 characters)
+- Increased view limit for better visibility of stored schema versions
+
+## [0.2.0] - 2025-11-20
+
+### Added
+- **Database comments support** - COMMENT ON statements for PostgreSQL and MySQL
+  - Comments on tables, columns, indexes, views, functions, and triggers
+  - CommentGenerator for generating COMMENT ON SQL statements
+  - CommentIntrospector for querying pg_description and information_schema
+  - Configurable via `include_comments` option (enabled by default)
+  - 10_comments directory in multi-file output with load order after triggers
+  - Full test coverage for PostgreSQL and MySQL comment introspection
+- **Hash-based schema version deduplication** - Automatic duplicate detection using MD5 content hashing
+  - `content_hash` column (VARCHAR 32) stores MD5 hexdigest of schema content
+  - Automatic skip when schema unchanged between storage attempts
+  - StoreResult value object for skip/store state communication
+  - Filesystem cleanup: delete multi-file directories after ZIP storage
+  - Enhanced rake task output showing skip reason or stored confirmation with hash
+  - `db:schema:versions` task now displays first 8 characters of content hash
+  - Content size and line count automatic tracking
+  - Streaming file reads (4MB chunks) for memory-efficient hash calculation
+  - Integration with retention management (cleanup only on actual storage)
+  - 29 new tests for deduplication workflow (all 355 tests passing)
+
+### Changed
+- **BREAKING**: Schema versions table requires `content_hash` column
+  - Run `rails generate better_structure_sql:migration` to add column
+  - Existing versions backfilled with calculated MD5 hash during migration
+- `SchemaVersions.store_current` now returns `StoreResult` instead of `SchemaVersion`
+  - Use `result.stored?` or `result.skipped?` to check operation type
+  - Access version via `result.version` when stored
+- `SchemaVersions.store` now requires `content_hash:` parameter
+- `SchemaVersion.latest` changed from scope to class method (returns record, not Relation)
+- Multi-file directories automatically cleaned up after ZIP archive creation
+- Integration apps configured to use multi-file schema dumps by default
+- Migration numbering scheme changed to support comments directory (20 directories)
+
+### Fixed
+- MySQL compatibility: Removed IF NOT EXISTS from index and type creation
+- Hash calculation now excludes manifest.json to avoid circular dependencies
+- FileWriter properly maps comments directory to correct load order
+- CodeBlock component supports both code prop and children for flexibility
+
 ## [0.1.0] - 2025-11-20
 
 ### Added

data/README.md

@@ -16,7 +16,7 @@
 
 </div>
 
-> **⚠️ Beta Notice**: This gem is currently in beta (version 0.1.0). APIs may change between releases until v1.0. We welcome feedback and contributions!
+> **⚠️ Beta Notice**: Version 0.2.1 is feature-complete and production-ready for **PostgreSQL**. Multi-database support (MySQL, SQLite) is implemented but considered experimental. APIs are stable but may see minor refinements before v1.0. We welcome feedback and contributions!
 
 ## ✨ Why BetterStructureSql?
 
@@ -67,7 +67,7 @@ Rails' database dump tools (`pg_dump`, `mysqldump`, etc.) create noisy `structur
 | **Materialized Views** | ✅ | ❌ | ❌ |
 | **Functions** | ✅ plpgsql, sql | ✅ Stored procedures | ❌ |
 | **Triggers** | ✅ BEFORE/AFTER/INSTEAD OF | ✅ BEFORE/AFTER | ✅ BEFORE/AFTER |
-| **Partitioned Tables** | ✅ RANGE/LIST/HASH | ❌ | ❌ |
+| **Partitioned Tables** | 🚧 Planned | ❌ | ❌ |
 | **Domains** | ✅ | ❌ | ❌ |
 
 ### Getting Started by Database
@@ -95,15 +95,16 @@ Rails' database dump tools (`pg_dump`, `mysqldump`, etc.) create noisy `structur
   - Custom types and enums (PostgreSQL, MySQL SET/ENUM)
 
 ### Multi-File Schema Output (Optional)
-- **Massive schema support** - Handle tens of thousands of tables effortlessly
+- **Massive schema support** - Designed to handle tens of thousands of database objects
 - **Directory-based output** - Split schema across organized, numbered directories
-- **Smart chunking** - 500 LOC per file with intelligent overflow handling
+- **Smart chunking** - 500 LOC per file (configurable) with intelligent overflow handling
 - **Better git diffs** - See only changed files, not entire schema
 - **ZIP downloads** - Download complete directory structure as archive
-- **Easy navigation** - Find tables quickly in `4_tables/`, triggers in `9_triggers/`, etc.
+- **Easy navigation** - Find tables quickly in `05_tables/`, triggers in `09_triggers/`, etc.
 
 ### Schema Versioning (Optional)
 - Store schema versions in database with metadata
+- **Hash-based deduplication** - Automatically skip storing when schema unchanged
 - Track database type and version, format type (SQL/Ruby), creation timestamp
 - ZIP archive storage for multi-file schemas
 - Configurable retention policy (keep last N versions)
@@ -141,9 +142,11 @@ Rails' database dump tools (`pg_dump`, `mysqldump`, etc.) create noisy `structur
 ```ruby
 # Gemfile
 gem 'better_structure_sql'
-gem 'pg'  # or 'mysql2' or 'sqlite3'
+gem 'pg'  # For PostgreSQL (or 'mysql2' for MySQL, or 'sqlite3' for SQLite)
 ```
 
+**Database adapter is auto-detected** from your `ActiveRecord::Base.connection.adapter_name`. No manual configuration needed!
+
 ```bash
 bundle install
 rails generate better_structure_sql:install
@@ -152,6 +155,128 @@ rails db:schema:dump_better
 
 **🎉 Your `db/structure.sql` is now clean and maintainable!**
 
+## 📦 Schema Versioning with Deduplication
+
+BetterStructureSql automatically tracks schema evolution by storing versions in your database. Hash-based deduplication ensures only meaningful schema changes are recorded.
+
+### How It Works
+
+When you run `rails db:schema:store`, the gem:
+
+1. **Reads** your current schema files (single or multi-file)
+2. **Calculates** MD5 hash of the complete schema content
+3. **Compares** with the most recent stored version's hash
+4. **Skips storage** if hash matches (no changes detected) ✨
+5. **Creates new version** if hash differs (schema changed)
+
+### Quick Example
+
+```bash
+# After migrations, dump and store schema
+rails db:migrate
+rails db:schema:dump_better
+rails db:schema:store
+
+# First run (no previous version)
+# =>
+# Stored schema version #1
+#   Format: sql
+#   Mode: single_file
+#   PostgreSQL: 15.4
+#   Size: 45.2 KB
+#   Hash: a3f5c9d2e8b1f4a6c7e9d3f1b5a8c2e4
+#   Total versions: 1
+
+# Second run (no schema changes)
+# =>
+# No schema changes detected
+#   Current schema matches version #1
+#   Hash: a3f5c9d2e8b1f4a6c7e9d3f1b5a8c2e4
+#   No new version stored
+#   Total versions: 1
+
+# After adding a table
+rails db:migrate  # Adds new table
+rails db:schema:dump_better
+rails db:schema:store
+
+# =>
+# Stored schema version #2
+#   Format: sql
+#   Mode: single_file
+#   PostgreSQL: 15.4
+#   Size: 48.7 KB
+#   Hash: b7e2d1c4f9a6c3e5d8b2f1a4c9e7d3b6
+#   Total versions: 2
+```
+
+### Production Workflow
+
+Perfect for deployment automation:
+
+```ruby
+# config/deploy.rb or GitHub Actions
+namespace :deploy do
+  task :update_schema do
+    # Run migrations (may be zero)
+    # Rails automatically dumps schema after migrations
+    execute :rake, 'db:migrate'
+
+    # Store schema version only if changed (automatic deduplication)
+    execute :rake, 'db:schema:store'
+  end
+end
+```
+
+**Benefits in Production:**
+- ✅ Deploys without migrations don't create duplicate versions
+- ✅ Developers see clean schema evolution timeline
+- ✅ Storage efficient (no duplicate content)
+- ✅ Clear audit trail of actual schema changes
+
+### Viewing Stored Versions
+
+```bash
+# List all versions with hashes
+rails db:schema:versions
+
+Schema Versions (3 total)
+
+ID   | Format | Mode        | Files | PostgreSQL | Hash     | Created             | Size
+-----|--------|-------------|-------|------------|----------|---------------------|-------
+3    | sql    | multi_file  | 47    | 15.4       | a3f5c9d2 | 2025-01-20 14:30:15 | 125 KB
+2    | sql    | single_file | -     | 15.4       | b7e2d1c4 | 2025-01-19 10:15:42 | 98 KB
+1    | sql    | single_file | -     | 15.3       | c9f8a3b2 | 2025-01-18 08:45:30 | 85 KB
+```
+
+### Configuration
+
+```ruby
+# config/initializers/better_structure_sql.rb
+BetterStructureSql.configure do |config|
+  # Enable schema versioning
+  config.enable_schema_versions = true
+
+  # Retain 10 most recent unique versions (0 = unlimited)
+  config.schema_versions_limit = 10
+end
+```
+
+### Web UI Access
+
+Developers can view stored schema versions via the web UI without database access:
+
+```ruby
+# config/routes.rb
+authenticate :user, ->(user) { user.admin? } do
+  mount BetterStructureSql::Engine, at: '/schema_versions'
+end
+```
+
+Navigate to `/schema_versions` to browse versions, view formatted schema, and download raw SQL files.
+
+📖 See [Schema Versioning Documentation](docs/schema_versions.md) for complete details.
+
 ## Docker Development Environment 🐳
 
 Get started with a fully configured development environment in seconds:
@@ -284,14 +409,21 @@ BetterStructureSql.configure do |config|
   config.enable_schema_versions = true
   config.schema_versions_limit = 10  # Keep last 10 versions (0 = unlimited)
 
-  # Customize output
+  # Customize output (feature toggles)
   config.include_extensions = true
   config.include_functions = true
   config.include_triggers = true
   config.include_views = true
-
-  # Search path
+  config.include_materialized_views = true  # PostgreSQL only
+  config.include_domains = true             # PostgreSQL only
+  config.include_sequences = true           # PostgreSQL only
+  config.include_custom_types = true        # PostgreSQL ENUM, MySQL ENUM/SET
+  # config.include_rules = false            # Not yet implemented
+  # config.include_comments = false         # Not yet implemented
+
+  # Search path and schema filtering
   config.search_path = '"$user", public'
+  config.schemas = ['public']               # Which schemas to dump
 end
 ```
 
@@ -314,11 +446,20 @@ BetterStructureSql.configure do |config|
   config.enable_schema_versions = true
   config.schema_versions_limit = 10
 
-  # Feature toggles
+  # Feature toggles (same as single-file mode)
   config.include_extensions = true
   config.include_functions = true
   config.include_triggers = true
   config.include_views = true
+  config.include_materialized_views = true
+  config.include_domains = true
+  config.include_sequences = true
+  config.include_custom_types = true
+
+  # Formatting options
+  config.indent_size = 2                    # SQL indentation (default: 2)
+  config.add_section_spacing = true         # Add blank lines between sections
+  config.sort_tables = true                 # Sort tables alphabetically
 end
 ```
 
@@ -330,34 +471,67 @@ When using `config.output_path = 'db/schema'`, your schema is organized by type
 db/schema/
 ├── _header.sql              # SET statements and search path
 ├── _manifest.json           # Metadata and load order
-├── 1_extensions/
+├── 01_extensions/
 │   └── 000001.sql
-├── 2_types/
+├── 02_types/
 │   └── 000001.sql
-├── 3_sequences/
+├── 03_functions/
 │   └── 000001.sql
-├── 4_tables/
+├── 04_sequences/
+│   └── 000001.sql
+├── 05_tables/
 │   ├── 000001.sql          # ~500 lines per file
 │   ├── 000002.sql
 │   └── 000003.sql
-├── 5_indexes/
+├── 06_indexes/
+│   └── 000001.sql
+├── 07_foreign_keys/
 │   └── 000001.sql
-├── 6_foreign_keys/
+├── 08_views/
 │   └── 000001.sql
-├── 7_views/
+├── 09_triggers/
 │   └── 000001.sql
-├── 8_functions/
+├── 10_comments/
 │   └── 000001.sql
-└── 9_triggers/
+└── 20_migrations/
     └── 000001.sql
 ```
 
 **Benefits for Large Schemas**:
 - ✅ Memory efficient - incremental file writing
 - ✅ Git friendly - only changed files in diffs
-- ✅ Easy navigation - find specific tables/triggers quickly
+- ✅ Easy navigation - find specific tables in `05_tables/`, triggers in `09_triggers/`, etc.
 - ✅ ZIP downloads - complete directory as single archive
 - ✅ Scalable - handles 50,000+ database objects
+- ✅ AI-friendly - 500-line chunks work better with LLM context windows
+
+**Manifest File (_manifest.json)**:
+
+The manifest tracks metadata and provides load order information:
+
+```json
+{
+  "version": "1.0",
+  "total_files": 11,
+  "total_lines": 2345,
+  "max_lines_per_file": 500,
+  "directories": {
+    "01_extensions": { "files": 1, "lines": 3 },
+    "02_types": { "files": 1, "lines": 13 },
+    "03_functions": { "files": 1, "lines": 332 },
+    "04_sequences": { "files": 1, "lines": 289 },
+    "05_tables": { "files": 2, "lines": 979 },
+    "06_indexes": { "files": 1, "lines": 397 },
+    "07_foreign_keys": { "files": 1, "lines": 67 },
+    "08_views": { "files": 1, "lines": 217 },
+    "09_triggers": { "files": 1, "lines": 35 },
+    "10_comments": { "files": 1, "lines": 9 },
+    "20_migrations": { "files": 1, "lines": 13 }
+  }
+}
+```
+
+This example shows a real schema with 2,345 lines split across 11 files. The `05_tables` directory has 2 files because the tables exceed the 500-line limit.
 
 ## 📝 Usage & Rake Tasks
 
@@ -394,15 +568,17 @@ rails db:schema:versions
 
 Output example:
 ```
-Total versions: 5
+Total versions: 3
 
 ID     Format  Mode          Files   PostgreSQL      Created              Size
 -----------------------------------------------------------------------------------------------
-5      sql     multi_file    47      15.3            2025-01-15 10:30:22  156.42 KB
-4      sql     single_file   1       15.3            2025-01-14 15:20:10  89.21 KB
-3      sql     single_file   1       15.2            2025-01-13 09:45:33  87.03 KB
+3      sql     multi_file    12      15.3            2025-01-15 10:30:22  56.42 KB
+2      sql     single_file   1       15.3            2025-01-14 15:20:10  45.21 KB
+1      sql     single_file   1       15.2            2025-01-13 09:45:33  44.03 KB
 ```
 
+The multi-file mode example shows 12 files across 10 directories (extensions, types, functions, sequences, tables, indexes, foreign_keys, views, triggers, migrations) stored as a ZIP archive.
+
 **Restore from Version**
 ```bash
 # Restore database from a specific version
@@ -436,10 +612,12 @@ end
 ```
 
 Access at `http://localhost:3000/schema_versions` to:
-- View list of all stored schema versions
-- Browse formatted schema content with syntax highlighting
-- Download raw SQL/Ruby schema files
+- View list of up to 100 most recent schema versions (pagination-ready)
+- Browse formatted schema content with syntax highlighting (for files <200KB)
+- Download raw SQL/Ruby schema files as text
 - Download ZIP archives for multi-file schemas
+- View manifest metadata for multi-file schemas
+- Stream large files efficiently (>2MB) without memory issues
 - Compare database versions and formats
 
 **Authentication Examples**:
@@ -503,10 +681,13 @@ fi
 |-----------|---------|-------|
 | **Ruby** | 2.7+ | Tested up to Ruby 3.4.7 |
 | **Rails** | 7.0+ | Works with Rails 8.1.1+ |
-| **Database Adapter** | | Choose one: |
-| `pg` | ≥ 1.0 | For PostgreSQL 12+ |
-| `mysql2` | ≥ 0.5 | For MySQL 8.0+ |
-| `sqlite3` | ≥ 1.4 | For SQLite 3.35+ |
+| **rubyzip** | ≥ 2.0.0 | Required for ZIP archive support |
+| **Database Adapter** | | |
+| `pg` | ≥ 1.0 | **Required dependency**. Works with PostgreSQL 12+ |
+| `mysql2` | ≥ 0.5 | Optional. For MySQL 8.0+ (experimental) |
+| `sqlite3` | ≥ 1.4 | Optional. For SQLite 3.35+ (experimental) |
+
+**Note**: The gem currently requires the `pg` gem as a dependency. Multi-database support (MySQL, SQLite) is implemented but requires manual gem installation. Future versions may make database adapters optional.
 
 ## Migration Guides
 
@@ -533,6 +714,34 @@ BetterStructureSql supports **both** `schema.rb` and `structure.sql` formats, al
 
 ---
 
+## 📊 Project Stats
+
+**Codebase Metrics** (as of v0.1.0):
+- **47 Ruby files** in `lib/` (~5,296 total lines)
+- **25 test files** in `spec/` (~3,022 lines)
+- **8 adapter files** (PostgreSQL, MySQL, SQLite, Registry, Configs)
+- **13 SQL generators** (Tables, Indexes, Functions, Triggers, Views, etc.)
+- **9 introspection modules** (Extensions, Types, Tables, Indexes, Foreign Keys, etc.)
+- **3 integration apps** (PostgreSQL, MySQL, SQLite) with Docker support
+- **React documentation site** deployed to GitHub Pages
+
+**Test Coverage**: Comprehensive RSpec test suite with unit and integration tests across all major components.
+
+**Real-World Example**: The integration app generates a multi-file schema with:
+- 11 SQL files across 10 directories
+- 2,345 total lines of SQL
+- Complete PostgreSQL feature coverage (extensions, types, functions, triggers, materialized views)
+
+**Production Status**:
+- ✅ **PostgreSQL**: Fully implemented and tested (primary focus)
+- ✅ **Multi-file output**: Complete with ZIP storage and streaming
+- ✅ **Schema versioning**: Full CRUD with web UI
+- ✅ **Rails integration**: Drop-in replacement for default tasks
+- 🧪 **MySQL**: Adapter implemented, integration app available (experimental)
+- 🧪 **SQLite**: Adapter implemented, basic testing (experimental)
+
+---
+
 ## 🤝 Contributing
 
 We welcome contributions! Bug reports and pull requests are welcome on [GitHub](https://github.com/sebyx07/better_structure_sql).

data/app/controllers/better_structure_sql/schema_versions_controller.rb

@@ -11,8 +11,9 @@ module BetterStructureSql
   class SchemaVersionsController < ApplicationController
     # Maximum file size to load into memory (2MB)
     MAX_MEMORY_SIZE = 2.megabytes
-    # Maximum file size to display in browser (200KB)
-    MAX_DISPLAY_SIZE = 200.kilobytes
+    # Maximum file size to display in browser (1MB)
+    # Large enough for most schemas but keeps browser responsive
+    MAX_DISPLAY_SIZE = 1.megabyte
 
     # Lists stored schema versions with pagination
     #
@@ -25,7 +26,7 @@ module BetterStructureSql
       # Load only metadata for listing (no content or zip_archive)
       @schema_versions = SchemaVersion
                          .select(:id, :pg_version, :format_type, :output_mode, :created_at,
-                                 :content_size, :file_count)
+                                 :content_size, :file_count, :content_hash)
                          .order(created_at: :desc)
                          .limit(100)
     end
@@ -43,7 +44,7 @@ module BetterStructureSql
       # Load metadata first
       @schema_version = SchemaVersion
                         .select(:id, :pg_version, :format_type, :output_mode, :created_at,
-                                :content_size, :line_count, :file_count)
+                                :content_size, :line_count, :file_count, :content_hash)
                         .find(params[:id])
 
       # Only load content for small single-file versions

data/app/views/better_structure_sql/schema_versions/index.html.erb

@@ -27,6 +27,7 @@
             <thead class="table-light">
               <tr>
                 <th scope="col" class="text-center" style="width: 80px;">ID</th>
+                <th scope="col" style="width: 120px;">Hash</th>
                 <th scope="col" style="width: 120px;">Format</th>
                 <th scope="col" style="width: 150px;">Mode</th>
                 <th scope="col" style="width: 180px;">PostgreSQL</th>
@@ -42,6 +43,11 @@
                   <td class="text-center fw-bold text-muted">
                     #<%= version.id %>
                   </td>
+                  <td>
+                    <code class="text-muted small" title="<%= version.content_hash %>">
+                      <%= version.content_hash[0..7] %>
+                    </code>
+                  </td>
                   <td>
                     <%= format_type_badge(version.format_type) %>
                   </td>

data/app/views/better_structure_sql/schema_versions/show.html.erb

@@ -93,6 +93,18 @@
             </p>
           </div>
         </div>
+
+        <div class="row mt-3">
+          <div class="col-md-12">
+            <h6 class="text-muted mb-1">
+              <i class="bi bi-fingerprint"></i>
+              Content Hash (MD5)
+            </h6>
+            <p class="mb-0">
+              <code class="text-muted"><%= @schema_version.content_hash %></code>
+            </p>
+          </div>
+        </div>
       </div>
     </div>
 
@@ -153,7 +165,7 @@
             <i class="bi bi-exclamation-triangle"></i>
             <strong>File too large to display</strong>
             <p class="mb-0 mt-2">
-              This schema file is too large to display in the browser (limit: 200 KB).
+              This schema file is too large to display in the browser (limit: 1 MB).
               Please use the "Download" button above to download and view it locally.
             </p>
           </div>

data/lib/better_structure_sql/adapters/base_adapter.rb

@@ -112,6 +112,17 @@ module BetterStructureSql
         raise NotImplementedError, "#{self.class} must implement #fetch_triggers"
       end
 
+      # Fetch comments on database objects
+      #
+      # @param connection [ActiveRecord::ConnectionAdapters::AbstractAdapter] Database connection
+      # @return [Hash] Hash with object types as keys (:tables, :columns, :indexes, etc.)
+      #   Each value is a hash mapping object identifier to comment text
+      #   Example: { tables: { 'users' => 'User accounts', ... }, columns: { 'users.email' => 'Email address', ... } }
+      # @raise [NotImplementedError] If not implemented by subclass
+      def fetch_comments(connection)
+        raise NotImplementedError, "#{self.class} must implement #fetch_comments"
+      end
+
       # Capability methods - indicate feature support
 
       # Indicates whether this database supports extensions (like PostgreSQL extensions)
@@ -163,6 +174,13 @@ module BetterStructureSql
         false
       end
 
+      # Indicates whether this database supports comments on database objects
+      #
+      # @return [Boolean] True if comments are supported, false otherwise
+      def supports_comments?
+        false
+      end
+
       # Version detection
 
       # Detect the database version