tape_deck 0.1.0.beta1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e3e52134a29b1bcec60211e70c2632b5bcf6e948977738bfc3c05419517223a9
-  data.tar.gz: 377b07a9f6ead12778bda7553b314bf29f800fcdba8d182f284a67289014c2cf
+  metadata.gz: 8c9c9d5d8179c5cbf64fcd6538a311c79be5bda549359d7f6a6f7e3673271462
+  data.tar.gz: 1fa65375ca49b761250aeb13a81d62acf9b309790d14214656b2abd6c96f265b
 SHA512:
-  metadata.gz: b5ae3af854d5d773d37b5cba103ec38721a5e6d7ccc8a77120a24c1e305a0362f945a6e61e86bbbd918fc3ce1fb9a4b8028d72a01a03b8c5dc55a84c657be032
-  data.tar.gz: 538704dde3a6796d74365ba092156239ee4bb34a1c7478147338dfdd7410f4d6aa3e164223ef8f08c8297d2dcbb571b195c0dd3ad73a0b0cf44da58432c372c2
+  metadata.gz: '01794027c3c706220a2c931ff28c820782b6ef20c340347d55edbfc27deef83ec88f006959f6b2ede1a482b6e90dc8e6b18d6b926090a5a29f39c4e71de18737'
+  data.tar.gz: e8e54acee961f60c499f0a599952ebd68dd9af4c2a2a06530589a4e514ce6a14610cfd923ca4d8bd41f3ae18dca8109d89394ed997560fba5304ce0986ce6a3d

data/.devcontainer/compose.yml

@@ -2,19 +2,22 @@ name: tape_deck
 
 services:
   app:
-    image: ruby:4.0
+    image: ${RUBY_IMAGE:-ruby:4.0}
     command: sleep infinity
     working_dir: /workspaces/tape_deck
     volumes:
       - ..:/workspaces/tape_deck
       - bundle:/usr/local/bundle
     environment:
-      BUNDLE_WITH: mysql
+      BUNDLE_WITH: mysql:postgres
       TZ: UTC
       TAPE_DECK_MYSQL_URL: ${TAPE_DECK_MYSQL_URL:-mysql2://root:tape_deck@mysql}
+      TAPE_DECK_POSTGRES_URL: ${TAPE_DECK_POSTGRES_URL:-postgres://postgres:tape_deck@postgres}
     depends_on:
       mysql:
         condition: service_healthy
+      postgres:
+        condition: service_healthy
 
   mysql:
     image: ${MYSQL_IMAGE:-mysql:8.4}
@@ -29,5 +32,18 @@ services:
       timeout: 5s
       retries: 30
 
+  postgres:
+    image: ${POSTGRES_IMAGE:-postgres:18}
+    environment:
+      POSTGRES_PASSWORD: tape_deck
+    tmpfs:
+      # Covers the data directory of all image versions (18 moved it into a version subdirectory)
+      - /var/lib/postgresql
+    healthcheck:
+      test: ["CMD", "pg_isready", "-U", "postgres"]
+      interval: 2s
+      timeout: 5s
+      retries: 30
+
 volumes:
   bundle:

data/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changes
 
+## 0.1.0
+
+- Breaking: tapes recorded with 0.1.0.beta1 need re-recording. Their information section is now called `tape_deck`, and ids of new rows are stored as placeholders.
+- Breaking: generated ids of inserted rows are stored as placeholders like `new shop.orders 1`, numbered per table in creation order, and so are declared foreign keys to them. Tapes no longer depend on where the database's counters stand. `keep_new_ids` keeps real ids.
+  - `number_by_content` numbers a table's new rows by content, when creation order doesn't matter, like for an import
+  - `references_to` adds references across databases and ones without a foreign key
+  - `generated_keys` adds generated keys that aren't auto increment, like UUIDs
+- Add: PostgreSQL support, and SQLite as a supported database. Database specific code lives in adapters, picked by Sequel's database type. Other databases Sequel supports read every table on every snapshot.
+- Add: opt-in views with `include_views`
+- Add: tapes store the TapeDeck version and the new-row numbering rules. Verifying uses the tape's numbering rules, not the current configuration.
+- Add: the RSpec hook fails the suite when tagged examples would share a tape, like the same description in a loop
+- Fix: the order of rows on a tape no longer depends on Ruby's `inspect` format, which changes between Ruby versions. Verifying also ignores the row order on the tape.
+- Change: values of unknown types, like from Sequel's `pg_json` or `pg_array` extensions, become plain Hashes, Arrays or Strings
+- Internal housekeeping
+  - API docs on GitHub Pages, for `main` and every release
+  - CI tests Ruby 3.2 to 4.0 and head, MySQL 8.4 and MariaDB 11.4 with `mysql2`, `trilogy` and `ruby-mysql`, and PostgreSQL 15 and 18
+  - Gemfile.lock is no longer tracked
+
 ## 0.1.0.beta1
 
 - First beta. The API and the tape format may still change.

data/DEVELOPMENT.md

@@ -8,8 +8,9 @@
 | Piece | Job |
 | --- | --- |
 | `Configuration` | Plain settings: `tape_library_dir`, `databases`, `default_record_mode`, `on_schema_change`, exclude patterns, RSpec hook |
-| `Tape` | File-like. Knows its name, path and mode (`recording?` / `verifying?`), can `read` and `write(diff, recorded:)`. Never touches a database |
+| `Tape` | File-like. Knows its name, path and mode (`recording?` / `verifying?`), can `read` and `write(diff, tape_deck:)`. Never touches a database |
 | `Recorder` | Observes databases: `snapshot` and `diff`. Knows nothing about tapes or configuration |
+| `Adapters` | The database specific parts of a snapshot: checksums to skip unchanged tables, and turning column values into plain YAML values. Picked by `db.database_type`, `Adapters::Base` for other databases |
 | `Filter` | Narrows a diff to what one use of a tape wants compared (`only:` / `ignore:`). `@api private` |
 | `Session` | One run of one tape: "before" snapshot on creation, `finish` captures the diff and records or verifies. `@api private` |
 
@@ -21,16 +22,24 @@ Decisions made on purpose. Please don't undo them without discussing first:
 - **Record modes** are `:once`, `:verify`, `:overwrite`. Renamed from VCR's `:once`, `:none`, `:all`.
 - **Vocabulary.** "record", "verify", "discard", "capture". Not "playback", not "compare" as a mode name.
 - **Errors** live directly under `TapeDeck` (`Error`, `TapeLoadingError`, `IncompatibleTapeError`, `UnexpectedDiffError`), not in an `Errors` module.
+- **Tape names come from example descriptions, never from line numbers or example ids.** Those change when examples are added above, which would silently orphan tapes. Instead, the RSpec hook fails the suite in `before(:suite)` when tagged examples share a tape file (`Configuration.check_rspec_tape_names`), using `RSpec.world.all_examples`, which is internal but long stable.
 - **Manual `insert_tape` / `eject_tape` work like `File.open` without a block**: no automatic tear down. The caller discards a tape whose action failed, and ejects outer tapes when an inner eject raises. `use_tape` and the RSpec tag do both.
 - **Options are keywords.** Unknown keywords raise.
-- **A tape stores what was recorded**: the exclude patterns (once, they are global), the watched databases, and the resolved names (`<table>`, `<table>.<column>`) the patterns left out of each.
+- **A tape stores everything that shapes its content**, in its `tape_deck` section: the TapeDeck version that recorded it, every recording rule (exclude, include, numbering), the watched databases, and the resolved names the exclude and include rules matched in each. A tape must be understandable and verifiable on its own. Per-use options (`only:`, `ignore:`) and behavior settings (record mode, `on_schema_change`) are not stored, they don't shape the content.
   - A tape must cover the run that verifies it: the watched databases are a subset of the recorded ones, and everything the tape left out is still excluded under the current rules. What is excluded now but on the tape is dropped from both sides before comparing.
   - Schema changes: the recorded patterns, applied to the current schema, must resolve to the recorded names. A difference means a matching table or column was added or removed. `on_schema_change` decides between error, warning and nothing.
+- **New rows get placeholders**, `"new <database>.<table> <n>"`, numbered per table in creation order (sorted auto increment ids). Only single, auto increment primary keys count as generated, so natural keys like a SKU stay visible. Declared single column foreign keys get the same placeholder, in inserted rows and in updated values. Ids of rows that existed before stay literal. `NewRows` runs on both sides, after the diff is captured and before it is recorded or verified. Verifying uses the numbering rules stored on the tape, not the configuration, because numbering only changes how ids are written. (Exclude and include rules are different: they decide which data is on the tape, so verifying uses the current ones and checks they fit the tape.)
+  - `references_to` adds references by column pattern, across databases too, and wins over a declared foreign key. `generated_keys` adds primary keys that aren't auto increment. Keys that aren't integers have no creation order, so their tables are always numbered by content.
+  - `number_by_content` tables are sorted by their JSON without the key, after the tables they reference, so references are placeholders already. A reference to a row without a placeholder yet (self references, cycles) counts as `"new <database>.<table>"` only, because its id isn't stable. Ties are broken by creation order.
 - **Primary key columns are never excluded**, even if a pattern matches, because rows are matched by them.
-- **Snapshots reuse unchanged tables.** `CHECKSUM TABLE` runs before and after. A table whose checksum is unchanged reuses the first snapshot object, and `diff` skips it by identity (`equal?`).
+- **Snapshots reuse unchanged tables.** The adapter computes a checksum per table before and after: `CHECKSUM TABLE` on MySQL, the row count plus a sum of row hashes on PostgreSQL. A table whose checksum is unchanged reuses the first snapshot object, and `diff` skips it by identity (`equal?`). Without a checksum (SQLite, other databases), a table is read on every snapshot.
 - **Tables without a primary key** are compared as a multiset of rows (inserted / deleted only).
+- **Row order never depends on `inspect`**, whose format changes between Ruby versions (3.4 changed `Hash#inspect`). Rows are sorted by their JSON (`TapeDeck::Rows`), and verifying sorts both sides before comparing, so the order on a tape is for reading only.
+- **Unknown value types become plain values.** Values from Sequel extensions like `pg_json` or `pg_array` are hash-like or array-like objects, which `YAML.safe_load` can't read back. `Adapters::Base#serialize` turns them into plain Hashes and Arrays, and any other unknown type into a String.
+- **PostgreSQL sees the search path only**, like `db.tables`, so names stay `"<database>.<table>"` for every database.
+- **Views are opt-in** (`include_views`), because table diffs already cover their data. Included views are always read in full (no checksum tells whether a view changed) and compared as a multiset. The tape stores the include patterns and the resolved views per database. A view no longer included is not compared, one included by changed rules raises `IncompatibleTapeError`, one added by a schema change is reported like any schema change and not compared.
 - **Binary values**: valid UTF-8 becomes a plain UTF-8 string, anything else is hex up to 64 bytes and a `sha256:` digest above that. Always a plain `String`, never a subclass, so `YAML.safe_load` can read the tape.
-- **TIME columns** are stored as a time of day, recognized by their database type (`Recorder::TIME_DB_TYPE`), because Sequel types `time` as `:time` but not `time(6)`. `mysql2` returns them as a `Time` on 2000-01-01, which would otherwise be stored as a timestamp.
+- **TIME columns** are stored as a time of day, recognized by their database type (`Adapters::Base::TIME_DB_TYPE`), because Sequel types `time` as `:time` but not `time(6)`. `mysql2` returns them as a `Time` on 2000-01-01, which would otherwise be stored as a timestamp.
 
 ## Code style
 
@@ -42,28 +51,31 @@ Decisions made on purpose. Please don't undo them without discussing first:
 
 ## Testing
 
-- `bundle exec rake spec` runs everything: the fake specs, then the MySQL specs once per adapter if `TAPE_DECK_MYSQL_URL` is set
-- `bundle exec rake spec:fake` needs no server. In-memory SQLite through Sequel, `spec/support/fake_mysql.rb` answers `CHECKSUM TABLE`, everything else is real Sequel.
-- `bundle exec rake spec:mysql2` / `spec:trilogy` run `spec/integration/mysql_spec.rb` (tagged `:mysql`) with that adapter
+- `bundle exec rake spec` runs everything: the SQLite specs, then the MySQL specs once per adapter if `TAPE_DECK_MYSQL_URL` is set, and the PostgreSQL specs if `TAPE_DECK_POSTGRES_URL` is set
+- `bundle exec rake spec:sqlite` needs no server. In-memory SQLite through Sequel, using the real SQLite adapter.
+- `bundle exec rake spec:mysql2` / `spec:trilogy` / `spec:ruby-mysql` run `spec/integration/mysql_spec.rb` (tagged `:mysql`) with that gem, through Sequel's `mysql2`, `trilogy` and `mysql` adapters
+- `bundle exec rake spec:postgres` runs `spec/integration/postgres_spec.rb` (tagged `:postgres`)
 
-`TAPE_DECK_MYSQL_URL` is a server URL without a database, e.g. `mysql2://root:tape_deck@mysql`. The adapter tasks replace its scheme with their own. The MySQL specs create and drop `tape_deck_shop` and `tape_deck_billing`.
+Both server specs include the shared examples in `spec/support/server_examples.rb`, and add their own value types. The adapters' checksum queries are also covered by the SQLite run, through Sequel's mock databases (`spec/tape_deck/adapters_spec.rb`).
 
-The devcontainer (`.devcontainer/`) has Ruby and MySQL 8.4, with `TAPE_DECK_MYSQL_URL` already set. Without VS Code:
+`TAPE_DECK_MYSQL_URL` and `TAPE_DECK_POSTGRES_URL` are server URLs without a database, e.g. `mysql2://root:tape_deck@mysql` and `postgres://postgres:tape_deck@postgres`. The MySQL tasks replace the scheme with their Sequel adapter's (`mysql://` for ruby-mysql). The server specs create and drop `tape_deck_shop` and `tape_deck_billing`.
+
+The devcontainer (`.devcontainer/`) has Ruby, MySQL 8.4 and PostgreSQL 18, with both URLs already set. Without VS Code:
 
     docker compose -f .devcontainer/compose.yml run --rm app sh -c "bundle install && bundle exec rake spec"
 
-The server is MySQL 8.4 by default. Set `MYSQL_IMAGE` for another one, e.g. `MYSQL_IMAGE=mariadb:11.4`.
+Set `MYSQL_IMAGE` for another MySQL server, e.g. `MYSQL_IMAGE=mariadb:11.4`, `POSTGRES_IMAGE` for another PostgreSQL, e.g. `POSTGRES_IMAGE=postgres:15`, and `RUBY_IMAGE` for another Ruby, e.g. `RUBY_IMAGE=ruby:3.2`.
 
-CI runs the default task, plus `spec:mysql2` and `spec:trilogy` in the devcontainer setup, against MySQL 8.4 and MariaDB 11.4.
+CI runs the default task on every supported Ruby and on Ruby head. In the devcontainer setup, on the oldest and newest supported Ruby, it runs `spec:mysql2`, `spec:trilogy` and `spec:ruby-mysql` against MySQL 8.4 and MariaDB 11.4, and `spec:postgres` against PostgreSQL 15 and 18.
 
-The adapters live in the optional `mysql` Gemfile group (`BUNDLE_WITH=mysql`), because `mysql2` needs the MySQL client library to install.
+The database adapters live in the optional `mysql` and `postgres` Gemfile groups (`BUNDLE_WITH=mysql:postgres`), because `mysql2` and `pg` may need client libraries to install.
 
 ## Docs
 
 - `bundle exec yard doc` builds the docs into `doc/`, `bundle exec yard server -r` serves them while you edit
-- `bundle exec rake docs:pages` builds `main` and every `v*` tag into `_site/docs/<ref>`, from their commits, not the working tree
+- `bundle exec rake docs:pages` builds `main` and every `v*` tag into `_site/docs/<ref>`, from their commits, not the working tree. It empties `_site` first.
 
-The Docs workflow runs `docs:pages` on every push to `main` and every version tag, and publishes `_site` to <https://fnordfish.github.io/tape_deck/docs/> (`main/`, `v0.1.0/`, ...). It needs GitHub Pages enabled with "GitHub Actions" as the source (Settings, Pages).
+The Docs workflow runs `docs:pages` on every version tag, and on pushes to `main` that change a file the docs are built from (the `paths` in `.github/workflows/pages.yml`, keep it in sync with `.yardopts`), and force-pushes `_site` as a single commit to the `gh-pages` branch, published at <https://fnordfish.github.io/tape_deck/docs/> (`main/`, `v0.1.0/`, ...). `latest/` redirects to the newest release, or the newest prerelease until there is one. GitHub Pages must serve that branch (Settings, Pages, Source: "Deploy from a branch", `gh-pages`, `/ (root)`). The branch is not deployed with `actions/deploy-pages`, because that deploys a commit only once, and a release pushes `main` and the tag for the same commit.
 
 ## Releasing
 
@@ -76,11 +88,10 @@ Releases go to rubygems.org and to the `@fnordfish` namespace on [gem.coop](http
 
 ## Tested against
 
-MySQL 8.4 and MariaDB 11.4, each with the `mysql2` and `trilogy` adapters. CI runs all four. Other servers and versions may work, but are untested.
+Ruby 3.2 to 4.0. On Ruby 3.2 and 4.0: MySQL 8.4 and MariaDB 11.4, each with the `mysql2`, `trilogy` and `ruby-mysql` gems, and PostgreSQL 15 and 18. SQLite with every supported Ruby. Other servers and versions may work, but are untested.
 
 ## Roadmap and open questions
 
-- Replace new auto increment ids with stable placeholders, so tapes don't depend on the starting state
-- Views are not snapshotted (`db.tables` skips them)
 - Binary data that happens to be valid UTF-8 (short hashes, UUIDs) is shown as text
-- `CHECKSUM TABLE` scans the whole table. Fine for test databases, worth measuring for large ones.
+- Checksums scan the whole table (`CHECKSUM TABLE`, the PostgreSQL checksum query). Fine for test databases, worth measuring for large ones.
+- PostgreSQL schemas other than the search path

data/README.md

@@ -2,34 +2,31 @@
 
 Records what an action changes in your databases, and checks later runs against that recording.
 
-[![Gem Version](https://img.shields.io/gem/v/tape_deck?include_prereleases)][gem]
+[![Gem Version](https://img.shields.io/gem/v/tape_deck)][gem]
 [![Build Status](https://github.com/fnordfish/tape_deck/actions/workflows/main.yml/badge.svg)][ci]
 [![API Docs](https://img.shields.io/badge/docs-API-blue)][docs]
 
 **[API documentation][docs]**
 
-> [!NOTE]
-> TapeDeck is in beta. The API and the tape format may still change, which can mean re-recording your tapes.
-
 You run a command or request against a black-box project. TapeDeck snapshots all configured databases before and after, and stores the row-level diff as a YAML file, a "tape". On later runs the same action must cause the same diff, so you notice when a project changes its behaviour after an update.
 
 API, naming and flow are modelled on [VCR](https://github.com/vcr/vcr), with one big difference: a tape is never played back. The action always runs for real, and the tape is only what the result is compared against.
 
-Works with [Sequel](https://sequel.jeremyevans.net) and MySQL or MariaDB (it uses `CHECKSUM TABLE` to skip unchanged tables). Tested with MySQL 8.4 and MariaDB 11.4, using the `mysql2` or `trilogy` adapter.
+Works with [Sequel](https://sequel.jeremyevans.net) and MySQL, MariaDB, PostgreSQL or SQLite. Needs Ruby 3.2 or newer.
 
 ## Installation
 
-Add this line to your application's Gemfile. While TapeDeck is in beta, the version requirement is needed, because Bundler skips prerelease versions without it:
+Add this line to your application's Gemfile:
 
 ```ruby
-gem "tape_deck", "~> 0.1.0.beta1", group: :test
+gem "tape_deck", "~> 0.1.0", group: :test
 ```
 
 Or get it from [gem.coop](https://gem.coop):
 
 ```ruby
 source "https://gem.coop/@fnordfish" do
-  gem "tape_deck", "~> 0.1.0.beta1", group: :test
+  gem "tape_deck", "~> 0.1.0", group: :test
 end
 ```
 
@@ -59,7 +56,8 @@ end
 The first run writes `spec/db_tapes/checkout/creates_order.yml`:
 
 ```yaml
-recorded:
+tape_deck:
+  version: 0.1.0
   exclude_tables: [shop.sessions, "*.schema_migrations"]
   exclude_columns: ["*.created_at", "*.updated_at"]
   databases:
@@ -88,7 +86,7 @@ diff:
           - 9
 ```
 
-`recorded` holds the exclude rules, the databases the tape watched, and what the rules left out of each.
+`tape_deck` holds everything that shaped the tape: the TapeDeck version, the recording rules, the databases the tape watched, and what the rules left out of each.
 
 Every later run verifies against it and raises `TapeDeck::UnexpectedDiffError` with a diff when the changes differ.
 
@@ -102,6 +100,15 @@ Every later run verifies against it and raises `TapeDeck::UnexpectedDiffError` w
 
 After `configure_rspec_metadata!`, tag examples with `:tape_deck`. The tape is named after the example's description, unless you pass `tape_name:`. A failing example neither records nor verifies its tape.
 
+Every example needs its own tape name. Before the first example runs, TapeDeck stops the suite if two tagged examples would share a tape, which easily happens in a loop:
+
+```ruby
+%w[paid shipped].each do |status|
+  it "creates an order", :tape_deck do ... end              # both would use "Checkout/creates an order"
+  it "creates an order (#{status})", :tape_deck do ... end  # each gets its own tape
+end
+```
+
 ```ruby
 RSpec.describe "Checkout" do
   it "creates an order", :tape_deck do
@@ -125,15 +132,73 @@ Names are `"<database>.<table>"` and `"<database>.<table>.<column>"`, matched as
 * `exclude_tables` / `exclude_columns` (configuration): never recorded. Meant for noise like timestamps. Excluding more later is fine, existing tapes just stop comparing it. Excluding less means re-recording, because the tapes lack that data (`TapeDeck::IncompatibleTapeError`).
 * `only:` / `ignore:` (per use of a tape): recorded, but not compared. Can change without re-recording.
 
-Since a tape stores its exclude rules, verifying also notices schema changes: when the rules now match tables or columns that were added or removed since recording, it raises `TapeDeck::IncompatibleTapeError`. Set `c.on_schema_change = :warn` to only print a warning, or `:ignore`.
+Since a tape stores its rules, verifying also notices schema changes: when the rules now match tables, columns or views that were added or removed since recording, it raises `TapeDeck::IncompatibleTapeError`. Set `c.on_schema_change = :warn` to only print a warning, or `:ignore`.
 
 Primary key columns are never excluded, because rows are matched by them. Tables without a primary key are compared as a multiset of rows.
 
+### New rows
+
+Generated ids depend on where the database's counters stand, and those move with every insert, also with ones a test rolled back. So tapes store placeholders instead, numbered per table in the order the rows were created:
+
+```yaml
+orders:
+  inserted:
+  - id: new shop.orders 1
+items:
+  inserted:
+  - id: new shop.items 1
+    order_id: new shop.orders 1
+```
+
+Declared foreign keys to new rows get the same placeholder. Ids of rows that existed before stay as they are. To keep the real ids of a table, use `c.keep_new_ids "shop.audit_log"`.
+
+Numbering in creation order means that creating the same rows in a different order fails verification. That's what you want if the order matters, for example when a CSV import must keep the file's order. If it doesn't, number those tables by content instead:
+
+```ruby
+c.number_by_content "shop.imported_products", "shop.imported_prices"
+```
+
+Rows are then sorted by their content, and referenced tables are numbered first, so a row's references count as part of its content. Rows with identical content are numbered in creation order.
+
+References across databases, and columns without a foreign key, need to be named. Keys that are generated, but not auto increment, like UUIDs, too:
+
+```ruby
+c.references_to "shop.orders", "billing.invoices.order_id", "*.last_order_id"
+c.generated_keys "shop.tokens.id"
+```
+
+UUIDs have no creation order, so those rows are always numbered by content.
+
+These numbering rules are stored on each tape, and verifying uses the tape's rules, not your current configuration. Changing them only affects tapes you record afterwards.
+
+### Views
+
+Views are left out, because the changes of the tables behind them are on the tape anyway. Include them to also notice a changed view definition:
+
+```ruby
+c.include_views "shop.order_totals", "billing.*"
+```
+
+Included views are read in full on every snapshot, and compared as a multiset of rows, because they have no primary key. `exclude_columns` applies to their columns.
+
+### Databases
+
+| Database | Tested with | Unchanged tables |
+| --- | --- | --- |
+| MySQL, MariaDB | MySQL 8.4 and MariaDB 11.4, `mysql2`, `trilogy` and `ruby-mysql` gems | skipped, using `CHECKSUM TABLE` |
+| PostgreSQL | PostgreSQL 15 and 18, `pg` adapter | skipped, using a checksum computed on the server |
+| SQLite | `sqlite3` adapter | read again on every snapshot |
+
+Telling unchanged tables apart still scans them on the server, but saves reading their rows. Other databases Sequel supports should work too, and read every table on every snapshot.
+
+PostgreSQL tables are seen through the search path, like Sequel's `db.tables` does, usually the `public` schema. Tables in other schemas are not recorded.
+
 ### Things to know
 
-* Your databases need the same starting state on every run. Auto increment ids and existing rows end up on the tape.
+* Rows that exist before the action end up on the tape when it changes them, so your databases need the same starting rows on every run. Where the id counters stand doesn't matter.
 * Anything the action does asynchronously must finish inside the block, or only part of it is captured.
 * When verifying, a tape watches the databases it was recorded with. `databases:` can narrow that down, but can't add a database the tape has no recording of.
+* Until 1.0, a minor version (0.2, 0.3, ...) may change the tape format. The CHANGELOG says when tapes need re-recording.
 
 The [API docs][docs] have the full story.
 
@@ -141,7 +206,7 @@ The [API docs][docs] have the full story.
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
-The default specs run against in-memory SQLite databases that fake MySQL's `CHECKSUM TABLE`, so you don't need a MySQL server. For the real thing, open the devcontainer, it comes with MySQL.
+The default specs run against in-memory SQLite databases, so you don't need a database server. For MySQL, MariaDB and PostgreSQL, open the devcontainer, it comes with both servers.
 
 Please also see [DEVELOPMENT.md](DEVELOPMENT.md) for design decisions and open questions.
 
@@ -155,4 +220,4 @@ The gem is available as open source under the terms of the [MIT License](https:/
 
 [gem]: https://rubygems.org/gems/tape_deck
 [ci]: https://github.com/fnordfish/tape_deck/actions/workflows/main.yml
-[docs]: https://fnordfish.github.io/tape_deck/docs/main/
+[docs]: https://fnordfish.github.io/tape_deck/docs/latest/

data/Rakefile

@@ -3,32 +3,46 @@
 require "bundler/gem_tasks"
 require "rspec/core/rake_task"
 
-MYSQL_ADAPTERS = %w[mysql2 trilogy].freeze
+# Gem -> the Sequel adapter that uses it, which is also the scheme of its database URLs
+MYSQL_ADAPTERS = {"mysql2" => "mysql2", "trilogy" => "trilogy", "ruby-mysql" => "mysql"}.freeze
 
 namespace :spec do
-  desc "Run the specs that need no MySQL server"
-  RSpec::Core::RakeTask.new(:fake) do |t|
-    t.rspec_opts = "--tag ~mysql"
+  desc "Run the specs that need no database server, on SQLite"
+  RSpec::Core::RakeTask.new(:sqlite) do |t|
+    # Quoted, because the command runs through a shell, and where a "mysql" user exists (GitHub's runners), ~mysql
+    # expands to its home directory
+    t.rspec_opts = "--tag '~mysql' --tag '~postgres'"
   end
 
-  MYSQL_ADAPTERS.each do |adapter|
-    desc "Run the MySQL specs with #{adapter}, needs TAPE_DECK_MYSQL_URL"
-    task adapter do
+  MYSQL_ADAPTERS.each do |gem_name, adapter|
+    desc "Run the MySQL specs with #{gem_name}, needs TAPE_DECK_MYSQL_URL"
+    task gem_name do
       server_url = ENV.fetch("TAPE_DECK_MYSQL_URL") { abort "Set TAPE_DECK_MYSQL_URL, see DEVELOPMENT.md" }
       adapter_url = server_url.sub(%r{\A\w+://}, "#{adapter}://")
       sh({"TAPE_DECK_MYSQL_URL" => adapter_url}, "rspec", "--tag", "mysql")
     end
   end
+
+  desc "Run the PostgreSQL specs, needs TAPE_DECK_POSTGRES_URL"
+  task :postgres do
+    ENV.fetch("TAPE_DECK_POSTGRES_URL") { abort "Set TAPE_DECK_POSTGRES_URL, see DEVELOPMENT.md" }
+    sh "rspec", "--tag", "postgres"
+  end
 end
 
-desc "Run all specs, the MySQL ones with every adapter if TAPE_DECK_MYSQL_URL is set"
+desc "Run all specs, the database server ones if their URL is set (MySQL with every adapter)"
 task :spec do
-  Rake::Task["spec:fake"].invoke
+  Rake::Task["spec:sqlite"].invoke
   if ENV["TAPE_DECK_MYSQL_URL"]
-    MYSQL_ADAPTERS.each { Rake::Task["spec:#{_1}"].invoke }
+    MYSQL_ADAPTERS.each_key { Rake::Task["spec:#{_1}"].invoke }
   else
     warn "Skipping the MySQL specs, TAPE_DECK_MYSQL_URL is not set"
   end
+  if ENV["TAPE_DECK_POSTGRES_URL"]
+    Rake::Task["spec:postgres"].invoke
+  else
+    warn "Skipping the PostgreSQL specs, TAPE_DECK_POSTGRES_URL is not set"
+  end
 end
 
 require "standard/rake"
@@ -51,7 +65,15 @@ end
 namespace :docs do
   desc "Build the YARD docs of main and every version tag into _site, like GitHub Pages publishes them"
   task :pages do
-    refs = ["main", *`git tag --list "v*" --sort=-v:refname`.split]
+    # Start from scratch, also after an interrupted run left worktrees behind
+    rm_rf %w[_site tmp/pages]
+    sh "git", "worktree", "prune"
+
+    version = ->(tag) { Gem::Version.new(tag.delete_prefix("v")) }
+    tags = `git tag --list "v*"`.split.select { Gem::Version.correct?(_1.delete_prefix("v")) }
+    tags = tags.sort_by(&version).reverse
+    refs = ["main", *tags]
+
     refs.each do |ref|
       # Without a local main branch (e.g. CI building a tag), use the remote one
       commit = ref
@@ -59,22 +81,35 @@ namespace :docs do
       worktree = File.expand_path("tmp/pages/#{ref}")
       output = File.expand_path("_site/docs/#{ref}")
 
-      sh "git", "worktree", "add", "--detach", "--force", worktree, commit
+      sh "git", "worktree", "add", "--detach", worktree, commit
       begin
-        Dir.chdir(worktree) { sh "yard", "doc", "--no-save", "--no-stats", "--output-dir", output }
+        Dir.chdir(worktree) do
+          sh "yard", "doc", "--no-save", "--no-stats", "--title", "TapeDeck #{ref}", "--output-dir", output
+        end
       ensure
         sh "git", "worktree", "remove", "--force", worktree
       end
     end
 
-    links = refs.map { %(<li><a href="#{_1}/">#{_1}</a></li>) }.join("\n")
+    # The newest release, or the newest prerelease until there is one
+    latest = tags.find { !version.call(_1).prerelease? } || tags.first || "main"
+    links = refs.map do |ref|
+      link = %(<a href="#{ref}/">#{ref}</a>)
+      link += " (latest)" if ref == latest
+      "<li>#{link}</li>"
+    end
     File.write("_site/docs/index.html", <<~HTML)
       <!DOCTYPE html>
       <html><head><meta charset="utf-8"><title>TapeDeck docs</title></head>
       <body><h1>TapeDeck docs</h1><ul>
-      #{links}
+      #{links.join("\n")}
       </ul></body></html>
     HTML
+    mkdir_p "_site/docs/latest"
+    File.write("_site/docs/latest/index.html", <<~HTML)
+      <!DOCTYPE html>
+      <html><head><meta charset="utf-8"><meta http-equiv="refresh" content="0; url=../#{latest}/"></head></html>
+    HTML
     File.write("_site/index.html", <<~HTML)
       <!DOCTYPE html>
       <html><head><meta charset="utf-8"><meta http-equiv="refresh" content="0; url=docs/"></head></html>

data/lib/tape_deck/adapters/base.rb

@@ -0,0 +1,96 @@
+# frozen_string_literal: true
+
+module TapeDeck
+  module Adapters
+    # Works with any database Sequel supports. Subclasses add what their database does better.
+    class Base
+      # Database types of TIME columns, with or without fractional seconds. Sequel only types +time+ as +:time+, not
+      # +time(6)+.
+      TIME_DB_TYPE = /\Atime\b/i
+
+      # @return [Sequel::Database]
+      attr_reader :db
+
+      # @param db [Sequel::Database]
+      def initialize(db)
+        @db = db
+      end
+
+      # Checksums tell which tables changed, so unchanged ones are not read again. A table without a checksum is read
+      # on every snapshot, which is what this default does for all of them.
+      #
+      # @param tables [Array<String>]
+      # @return [Hash{String => Object, nil}] table -> checksum, nil if unknown
+      def checksums(tables)
+        tables.to_h { [_1, nil] }
+      end
+
+      # Turns a column value into one that +YAML.safe_load+ reads back unchanged.
+      #
+      # @param value [Object] as returned by Sequel
+      # @param column [Hash] column info from +db.schema+
+      # @return [String, Integer, Float, Boolean, nil, Array, Hash]
+      def serialize(value, column = {})
+        case value
+        when String then maybe_utf8(value)
+        when Integer, Float, true, false, nil then value
+        when BigDecimal then value.to_s("F")
+        when Sequel::SQLTime then time_of_day(value)
+        when Time then timestamp(value, column)
+        when Date then value.iso8601
+        else structured(value)
+        end
+      end
+
+      private
+
+      # Values of types not handled above, e.g. from Sequel extensions like +pg_json+ or +pg_array+. Hash-like and
+      # array-like ones become plain Hashes and Arrays, everything else a String.
+      def structured(value)
+        if value.respond_to?(:to_hash)
+          value.to_hash.to_h { |key, v| [key.to_s, serialize(v)] }
+        elsif value.respond_to?(:to_ary)
+          value.to_ary.map { serialize(_1) }
+        else
+          value.to_s
+        end
+      end
+
+      # mysql2 returns TIME columns as a +Time+ on 2000-01-01, not as +Sequel::SQLTime+. Those keep their local time
+      # of day, because converting them to UTC would shift the hour.
+      def timestamp(value, column)
+        if TIME_DB_TYPE.match?(column[:db_type].to_s)
+          time_of_day(value)
+        else
+          value.utc.iso8601(6)
+        end
+      end
+
+      def time_of_day(value) = value.strftime("%H:%M:%S.%6N")
+
+      # Small binaries encode as hex. Large ones are shown as a digest so the YAML stays small.
+      def bin(value)
+        if value.bytesize <= 64
+          "0x#{value.unpack1("H*")}"
+        else
+          "sha256:#{Digest::SHA256.hexdigest(value)}"
+        end
+      end
+
+      # Binary columns arrive as ASCII-8BIT strings or +Sequel::SQL::Blob+, depending on the adapter. Ones that hold
+      # valid UTF-8 become plain UTF-8 strings, because YAML would otherwise store them as base64 (+!binary+).
+      # Everything else goes through {#bin}. Always returns a plain +String+, never a subclass, so YAML can read it.
+      def maybe_utf8(value)
+        binary = value.encoding == Encoding::BINARY || value.is_a?(Sequel::SQL::Blob)
+        return String.new(value) unless binary
+
+        utf8 = String.new(value, encoding: Encoding::UTF_8)
+        if utf8.valid_encoding?
+          utf8
+        else
+          bin(value)
+        end
+      end
+    end
+  end
+end

data/lib/tape_deck/adapters/mysql.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module TapeDeck
+  module Adapters
+    # MySQL and MariaDB. Uses +CHECKSUM TABLE+, so unchanged tables are not read again.
+    class MySQL < Base
+      # @param tables [Array<String>]
+      # @return [Hash{String => Integer, nil}] table -> checksum
+      def checksums(tables)
+        return {} if tables.empty?
+
+        list = tables.map { db.literal(Sequel.identifier(_1)) }.join(", ")
+        db.fetch("CHECKSUM TABLE #{list}").to_h do |r|
+          _db_name, table = r[:Table].split(".", 2) # MySQL reports "<db>.<table>"
+          [table, r[:Checksum]]
+        end
+      end
+    end
+  end
+end

data/lib/tape_deck/adapters/postgres.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module TapeDeck
+  module Adapters
+    # PostgreSQL. Sees the tables and views in the search path, like +Sequel::Database#tables+.
+    #
+    # PostgreSQL has no +CHECKSUM TABLE+. This computes one on the server: the row count, and the sum of 64 bits of
+    # each row's MD5. Like +CHECKSUM TABLE+ it scans the table, but no rows are sent. The order of rows doesn't matter,
+    # and the sum can't overflow, because it is a +numeric+.
+    class Postgres < Base
+      # Table alias in the checksum query. A short alias would mean a column instead, if the table had one with that
+      # name.
+      ROW = "tape_deck_row"
+
+      # @param tables [Array<String>]
+      # @return [Hash{String => String}] table -> checksum
+      def checksums(tables)
+        return {} if tables.empty?
+
+        queries = tables.map do |table|
+          "SELECT #{db.literal(table)} AS name, " \
+            "count(*) || ':' || coalesce(sum(('x' || substr(md5(#{ROW}::text), 1, 16))::bit(64)::bigint), 0) AS sum " \
+            "FROM #{db.literal(Sequel.identifier(table))} AS #{ROW}"
+        end
+        db.fetch(queries.join(" UNION ALL ")).to_h { |r| [r[:name], r[:sum]] }
+      end
+    end
+  end
+end

data/lib/tape_deck/adapters/sqlite.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module TapeDeck
+  module Adapters
+    # SQLite. Has no cheap way to tell whether a table changed, so every table is read on every snapshot. For a local
+    # database file that is usually fast enough.
+    class SQLite < Base
+    end
+  end
+end