iostreams 2.1.0 → 2.2.0

data/docs/upgrading.md

@@ -3,19 +3,411 @@ layout: default
 title: Upgrading
 heading: Upgrading IOStreams
 description: >-
-  What changes when upgrading IOStreams to v2.1 or v2.0, and the security
-  settings to review in your application when upgrading.
+  What changes when upgrading IOStreams to v2.2, v2.1 or v2.0, how to prepare
+  for v3.0, and the security settings to review in your application when upgrading.
 ---
 
 This page covers the changes that may need updates to your application when upgrading IOStreams,
 and the security issues to check. For every change in each release, see the
 [CHANGELOG](https://github.com/reidmorrison/iostreams/blob/main/CHANGELOG.md).
 
+## Upgrading to v2.2
+
+v2.2 prepares applications for v3.0, and changes no existing behavior, apart from its deprecation warnings and the
+[changes in behavior](#changes-in-behavior) below. It adds the methods that replace those that v3.0 deprecates or
+removes, and reports a deprecation warning when the application uses one of those, so that it can move to the
+replacement while it still runs v2.2, and then upgrade to v3.0 with fewer changes:
+
+* `#raw` reads and writes the data as it is stored, in place of `stream(:none)`, see [Raw](streams#raw).
+* `#encoding` sets the encoding of the text, in place of `option(:encode, ...)` and `stream(:encode, ...)`, see
+  [Character encoding](extensions#character-encoding).
+* `#schema` describes the rows and records of a file, as v3.0 does: the type of each field, the columns that the file
+  must and can have, and the position of each field of a fixed width file, in place of `#format_options` and the
+  column options of each call, see [Schema](formats#schema). A schema in v2.2 reads and writes as one in v3.0 does,
+  apart from the differences that v3.0 makes to every file, such as its encoding, see
+  [Schemas in IOStreams 2.2 and 3.0](formats#schemas-in-iostreams-22-and-30).
+* `IOStreams.tabular_formats` returns the formats that `#format` accepts.
+
+What is deprecated still works as before. Apart from the [changes in behavior](#changes-in-behavior) below, the only
+change to existing behavior is the deprecation warnings themselves, which Ruby shows only while `Warning[:deprecated]` is on: it is off by default, so production stays
+quiet, but Minitest and test-unit turn it on, so most test runs show them on `$stderr`, including for the calls
+that another gem makes, such as Rocket Job 7.0, see [Seeing deprecation warnings](#seeing-deprecation-warnings).
+
+To prepare for v3.0: upgrade to v2.2, run your tests, which show the deprecation warnings, see below, and move each
+use to what replaces it. Then read the changes that v3.0 makes in its own
+[upgrading guide](https://github.com/reidmorrison/iostreams/blob/main/docs/upgrading.md#upgrading-to-v30), since it
+also changes behavior that cannot be reported as a deprecation, such as reading text as UTF-8 by default.
+
+### Seeing deprecation warnings
+
+IOStreams reports the use of a deprecated method or option as a Ruby deprecation warning, the same way that Ruby
+reports its own, which names the line of your application that used it, and what to use instead:
+
+~~~
+app/jobs/export_job.rb:12: warning: IOStreams: stream(:none) is deprecated, use #raw to read or write the data as it is stored, or #file_name to read a file that is not named for its format
+~~~
+
+Ruby only shows deprecation warnings while `Warning[:deprecated]` is true, which it is not by default, so production
+stays quiet without any configuration:
+
+| Where | Deprecation warnings |
+|-------|----------------------|
+| Tests run with Minitest or test-unit | On, since both turn them on |
+| Tests run with RSpec | Off, unless run with `rspec --warnings`, or `Warning[:deprecated] = true` is set in `spec_helper.rb` |
+| Development | Off, unless `RUBYOPT=-W:deprecated` is set in the environment, or `Warning[:deprecated] = true` in code that only runs in development, such as `config/environments/development.rb` |
+| Production | Off |
+
+Rails settings such as `config.active_support.deprecation` only apply to the deprecation warnings of Rails, not to
+those that Ruby reports.
+
+Ruby writes each warning to `$stderr`. To send them elsewhere, such as to your log, or to raise them in your tests so
+that each use is found, extend `Warning` with your own `warn`:
+
+~~~ruby
+module IOStreamsDeprecations
+  def warn(message, *, **)
+    return super unless message.include?("IOStreams: ")
+
+    Rails.logger.warn(message.chomp)
+  end
+end
+Warning.extend(IOStreamsDeprecations)
+~~~
+
+### Deprecated, and what replaces it
+
+Each of these still works as before in v2.2, and reports a deprecation warning:
+
+| Deprecated | Use instead |
+|------------|-------------|
+| `stream(:none)` | `#raw` to read or write the data as it is stored, or `#file_name` to read a file that is not named for its format, such as `file_name("data.csv")` for a CSV file named `data.txt` |
+| `option(:encode, ...)` and `option_or_stream(:encode, ...)` | `#encoding`, with the same options and the same results, see [Character encoding](extensions#character-encoding) |
+| `stream(:encode, ...)` | `#encoding`, with the same options. Like any stream set with `#stream`, `stream(:encode, ...)` stops the streams being taken from the file name, while `#encoding` keeps them, so `IOStreams.path("data.csv.gz").encoding("BINARY")` reads the decompressed data. To read or write the data as it is stored, as `stream(:encode, encoding: "BINARY")` does for such a file name, use `#raw`, and to read a file that is not named for its format, set the file name that names its format with `#file_name`, then `#encoding` |
+| `embedded_within:` with `each(:array)` or `each(:hash)` | `#format`, since the format of the file decides where the lines of its rows and records are split, such as `format(:psv)` for values that are not quoted |
+| `#format_options`, and `format_options:` supplied to each call | `#schema`, which gives the `width` of each field of a fixed width file, see [Moving a layout to a schema](#moving-a-layout-to-a-schema) |
+| `columns:`, `allowed_columns:`, `required_columns:`, `skip_unknown:` and `cleanse_header:`, even when they are nil, since v3.0 raises `ArgumentError` for them whatever their value | `#schema`, which describes the columns of the file on its path, see [Moving the column options to a schema](#moving-the-column-options-to-a-schema) |
+
+`IOStreams.enforce_column_restrictions=` is deprecated too, see below, since it only applies to the column options.
+v2.2 still follows it, so keep setting it to true, as [v2.1 recommends](#column-restrictions-on-uploaded-files),
+until the column options are replaced with a schema, which applies to every input, and then remove it.
+
+v3.0 removes these, or raises `ArgumentError` for them, so in v2.2 each still works as before, and reports a
+deprecation warning:
+
+| Deprecated | Use instead |
+|------------|-------------|
+| `format:`, `file_name:`, `original_file_name:` or `default_format:` supplied to `#each`, `#reader`, `#writer` or a copy, such as `each(:hash, format: :json)`, even when they are nil | `#format`, which sets the format on the path, so that every reader and writer of the path uses it: `format(:json).each(:hash)`. A file name supplied to a call only chooses the format, when the path has none, and not the streams, as `#file_name` does, so set the format that it names instead, such as `format(:json)` for `file_name: "data.json"`. `default_format:` only applies when the path has no format, so set it only then: `path.format(:json) unless path.format`. The rows of a spreadsheet are read as CSV, so set `format(:csv)` on its path |
+| `allowed_columns:`, `required_columns:` or `skip_unknown:` supplied to `#writer` or a copy, which ignores them, even when they are nil | A schema, which describes the columns that are written, such as with `fieldsMatch: "subset"` to skip the other keys of each record |
+| Options supplied to `#writer` when the stream is a line, row or record writer that is already open, such as the one that `writer(:line)` yields, handed to `IOStreams.stream`, which writes with its own settings, and ignores them, even when they are nil, other than a nil `delimiter:`, and the streams, encoding, format and schema set on such a stream, other than the format and schema of the rows or records written to a line writer, which apply | Set them where that writer is opened, or remove them |
+| Options supplied to the `:stream` mode, the default of `#reader` and `#writer`, to a copy without a `mode:`, or to a copy with `convert: false`, including `mode:` | Remove them, since they are ignored |
+| The Zip writer's `zip_file_name`, even when it is nil | `entry_file_name` |
+| An option of `.bz2` that neither reading nor writing takes, which BZip2 ignores | Remove it |
+| An option that neither the reader nor the writer of a stream takes, such as a misspelled one, set with `#option`, `#stream` or `#option_or_stream`, also when the file name does not include the stream, such as `IOStreams.path("data.csv").option(:pgp, recipent: "partner@example.org")`, which is ignored | Correct or remove it, since v3.0 raises `ArgumentError` for it when it is set |
+| A file name ending in `.encode` | `#encoding`, which applies the encode stream whatever the file name, so the file need not be renamed |
+| `#path=` | `#join` or `#directory`, which return a new path without the streams, options, format and schema set on the path, which `#path=` keeps, so set them again, or `IOStreams.path` with the new name |
+| `#builder=` | `IOStreams.path`, to start again without the streams and options set with `#stream` or `#option` |
+| A url part or S3 argument that a path ignores, reported when the path is created: an S3 url with a fragment, which an unencoded `#` starts, a port, a user name or password, or without a bucket, an SFTP url with a query or a fragment, an SFTP or HTTP url without a host, and `region:`, `access_key_id:` or `secret_access_key:` with an `Aws::S3::Client` as `client:` | Percent-encode a `#` in a key or name as `%23`, and a `?` in an SFTP name as `%3F`; supply the endpoint of a service that implements the S3 API with `client: {endpoint: "https://host:port"}`, S3 credentials with `access_key_id:` and `secret_access_key:`, and SFTP ssh options with `ssh_options:`; name the bucket or host; and supply the region and credentials to `Aws::S3::Client.new`, or `client:` as a Hash of its options |
+| An option of an S3 path, or a name in the query of its url, that no S3 request takes, such as a misspelled one, or that the path sets itself, such as `key:` or `bucket:`, which it overrides, reported when the path is created | Correct or remove it, and percent-encode a `?` in a key as `%3F`. The options of the S3 requests that a path makes, such as `acl:`, are still accepted |
+| `IOStreams.enforce_column_restrictions=` and `IOStreams.enforce_column_restrictions?` | A schema, whose `fieldsMatch` and `constraints` apply to every input, including the keys of each JSON record, see [Moving the column options to a schema](#moving-the-column-options-to-a-schema). The setting only applies to the column options, so keep it until they are replaced with a schema, and then remove it |
+| The methods of a Struct, other than `reader_class` and `writer_class`, on the extension that `IOStreams.extensions` returns for a built-in extension, such as `IOStreams.extensions[:gz].to_a`, and the setters of every registered extension, such as `writer_class=`, also once it is no longer registered, such as the one that `IOStreams.deregister_extension` returns | `reader_class` and `writer_class`, which the format that v3.0 returns for a built-in extension, such as `IOStreams::Gzip`, answers too, and `IOStreams.register_extension` to change a registered extension, which v3.0 freezes |
+| `IOStreams.register_extension(:encode, ...)`, in any case | Register the reader and writer under another name, and apply them with `#stream`, and set the encoding of the text with `#encoding`, since v3.0 builds in the encode stream |
+| `IOStreams.deregister_extension(:encode)`, after which `#encoding` raises `ArgumentError` | Remove it: v3.0 applies the encode stream only where `#encoding` sets it, and does not deregister it |
+| `IOStreams.register_extension(:none, ...)`, in any case | Register the stream under another name, since v3.0 reserves `:none` |
+| A name that ends with a line break, such as `"xz\n"`, supplied to `IOStreams.register_extension` or `IOStreams.deregister_extension` | The name without the line break, since v3.0 raises `ArgumentError` for it |
+| A name that is not in lower case, such as `:XZ`, supplied to `IOStreams.register_extension` or `IOStreams.deregister_extension` | The name in lower case, such as `:xz`, as v3.0 registers it, which matches the file names that end in `.xz`, in any case, where `:XZ` matched none |
+| Ruby 3.2, reported once, as IOStreams is loaded, so only while deprecation warnings are on as it is loaded, such as with `RUBYOPT=-W:deprecated` | Ruby 3.3 or later, which v3.0 requires: it cannot be installed on Ruby 3.2 |
+| `#io_stream` | The IO that you supply to `IOStreams.stream`, which the stream reads and writes |
+| Changing a root path in place, such as `IOStreams.root.stream(:gz)` or `IOStreams.add_root(:default, "/data").option(...)`, which changes the root for the whole process. v3.0 keeps its own frozen copy of the root, so changing a path no longer changes the root: changing the path that `IOStreams.add_root` returns, which is that copy, raises `FrozenError`, a path supplied to it is not the root, and `IOStreams.root`, `IOStreams.roots` and `IOStreams.join` without elements return copies of it | A copy of the root, such as `IOStreams.root.dup.stream(:gz)`, or a path joined to it, such as `IOStreams.join("a.csv.gz").option(:gz, level: 9)`, which never has the streams, options, encoding, format or schema of the root, and for a named root, such as `:uploads`, `IOStreams.root(:uploads).dup` or `IOStreams.join("a.csv.gz", root: :uploads)` |
+| `#move_to` onto an existing directory, which moves the file into it, and returns the directory | The file within the directory, such as `path.move_to(IOStreams.path("/archive").join(path.basename))`, which returns the file |
+| A pattern supplied to `#each_child` or `#children` that is not a String, such as a `Pathname` | Its String, such as `pattern.to_s` |
+| A format other than `:csv` set on a spreadsheet, such as `format(:psv)` on an `.xlsx` path, or named by its file name, such as `report.psv.xlsx`, when its lines, rows or records are read. v3.0 raises `ArgumentError` for a format that is set, and reads the rows as CSV whatever the file name | `format(:csv)`, which reads the rows as CSV in both versions, or no format on a path whose file name names none, since a spreadsheet is read as CSV |
+| `delimiter:` supplied to `#each`, `#reader` or a copy of the rows and records of a spreadsheet, and an encoding set with `#encoding` when they are read, which v3.0 raises `ArgumentError` for, since they are its cells rather than lines of text | Neither: remove them. With a schema, v2.2 reads the text of the cells as UTF-8, as v3.0 does. Its lines, `each(:line)`, still take both |
+| A method of the stream that a reader block receives for `.gz`, `.zip`, `.bz2` or `.enc`, or for PGP within another stream or from an IO that is not a local file, without `verify_first: true`, other than `#read`, `#readpartial`, `#gets`, `#readline`, `#readlines`, `#each_line`, `#each` and `#eof?`, such as `#rewind`, `#pos`, `#lineno` or `#map` of a `Zlib::GzipReader`. Only the calls that the application makes itself are reported, including through a `SimpleDelegator`, `DelegateClass` or `Forwardable` that wraps the stream | Those methods, or `each(:line)`. Open the reader again to read the data from the start. Where the stream must be a `File`, copy the data into a local file first, such as with `IOStreams.temp_file` and `#copy_to`, and read that file |
+| A method of the `File` that a reader block receives for an S3 or HTTP(S) path without streams, or read with `#raw`, other than those of an IO that read, such as `#rewind`, `#size` or `#path` | The same, copying the object into a local file with `copy_to(temp, convert: false)` |
+| A method of the IO that a writer block receives when writing PGP, other than `#write`, `#<<`, `#print`, `#puts`, `#printf` and `#flush`, such as `#fileno` | Those methods. Where it must be an IO, write a local file first, such as with `IOStreams.temp_file`, and copy it with `#copy_to` |
+| A method of the `File` that a writer block receives for an S3 path without streams, other than `#write`, `#<<`, `#print`, `#puts`, `#printf` and `#flush`, such as `#path` or `#size` | Those methods. Where it must be a `File`, write a local file first, such as with `IOStreams.temp_file`, and copy it to the S3 path with `#copy_to` |
+| `#encoding` or `#cleaner` of the stream that a reader or writer block receives when an encoding is set | The settings of the path: `path.encoding`, or `Encoding.find(path.encoding)` for an `Encoding`, and `path.setting(:encode)[:cleaner]` |
+
+v3.0 changes the result of these calls, so in v2.2 each still gives the result that it gave before, and reports a
+deprecation warning that names another way to write it, which gives the same result in v2.2 and v3.0:
+
+| Deprecated | Use instead |
+|------------|-------------|
+| `IOStreams::Pgp.export(email: nil)` without a `key_id:`, and `IOStreams::Pgp.set_trust` without an `email:` or `key_id:`, which raise `IOStreams::Pgp::Failure`, where v3.0 raises `ArgumentError` | The email or key id of the key |
+| The rows and records of a file named `.text`, such as `ids.text`, which are read and written as CSV, where v3.0 reads and writes them in its `:text` format, as the text of each line | `format(:csv)`, or the format of an extension before `.text`, such as `format(:json)` for `people.json.text` |
+| `each(:array)` and `reader(:array)` of JSON without a schema, which yield each record as a Hash, where v3.0 yields each row as an Array, after a header row of the keys of the first record | `each(:hash)` to read the records, or a schema, with which v2.2 reads the rows as v3.0 does |
+| Records read without a schema, by `each(:hash)`, `reader(:hash)` or a copy in the `:hash` mode, from a header row whose columns are the same key, such as `a,a,b`, or `A` and `a`, which are cleansed to the same key, and rows written to JSON under an `IOStreams::Row::HeaderRow` whose columns are the same key, each record of which holds the value of only the last of them, where v3.0 raises `IOStreams::Errors::InvalidHeader` | Columns of names of their own, or `each(:array)`, which reads every column, in both |
+| A copy without a `mode:`, by `#copy_from` or `#copy_to`, between two formats, such as `people.csv` into `people.json` or `people.psv`, which copies the text of the source as it is, where v3.0 converts its rows into the format of the target, as the `:array` mode does | `mode: :array` to convert the rows, or `mode: :hash` when either file is JSON, whose rows are records, or `mode: :line` to copy the text as it is |
+| The rows and records of a spreadsheet read without a type for each column, by `#each`, `#reader` or a copy in the `:array` or `:hash` mode: without a schema, or with a field of type `any`, or with `extraColumns: "keep"`, which read the text of each cell, such as `"1001.0"` without a schema, or `"1001"` with one, where v3.0 reads its value, as Excel typed it, such as the `Float` `1001.0`, a `Date` or `true` | A schema that gives each column that is read a type, such as `{name: "id", type: "integer"}`, which reads `1001` in both, `{name: "zip", type: "string"}` for the text of a column such as `01234`, or of a number, such as `2134`, which both read as the number that the cell holds, rather than `2134.0`, `{name: "ok", type: "boolean"}`, `{name: "at", type: "datetime"}` for a cell of a date and time, which both read at the time of its clock in UTC, or `{name: "opens", type: "string"}` for a cell of a time of day, which both read as the time of its clock, such as `09:30:00` |
+| A `Time` or `DateTime` that is not at midnight, by its own clock, such as `Time.now`, converted by a `date` field of a schema, which drops its time of day, where v3.0 raises `IOStreams::Errors::InvalidValue` for it, while it takes one at midnight as its date. It is reported once for each field | A `Date`, converted in the time zone that the time belongs to, such as `time.to_date`, or the type `datetime` for the field, to keep its time |
+| An SFTP or HTTP(S) url without a path, such as `sftp://host` or `https://host?a=1`, to which v3.0 adds a `/`, which changes its `#to_s` and `#path`, and the directory that SFTP `#each_child` lists, from the login directory to `/` | The url with the `/`, such as `sftp://host/` or `https://host/?a=1` |
+| A name that starts with `/` joined to an S3 bucket, such as `IOStreams.path("s3://bucket").join("/exports/a.csv")`, which is the key `/exports/a.csv`, where v3.0 joins it as the key `exports/a.csv` | The name without the leading `/`, or for an object whose key starts with `/`, its url, such as `IOStreams.path("s3://bucket//exports/a.csv")` |
+| A local file name of `~`, or starting with `~/`, which is within a directory called `~` in the current directory, where v3.0 reads it within the home directory | `File.expand_path("~/data/a.csv")` for the home directory, or `./~/data/a.csv` for the directory called `~` |
+| A `file://` url, such as `file:///data/a.csv` or `file://~/a.csv`, which is a file name relative to the current directory, where v3.0 reads it as the local file that it names, such as `/data/a.csv`, or within the home directory | The name of the file, such as `/data/a.csv`, or `File.expand_path("~/a.csv")` for the home directory |
+| A `Pathname` supplied to `IOStreams.new`, `#copy_from`, `#copy_to` or `#move_to`, which read and write it as an IO, where v3.0 takes it as the name of a file, applying the streams of its file name, or to `IOStreams.stream`, which raises `ArgumentError` for it in v3.0 | `pathname.to_s` or `IOStreams.path(pathname)`, and `#raw` on the path to read or write the data as it is stored |
+| Reading or writing the path of `IOStreams.temp_file` when its extension names streams, such as `.csv.gz`, which reads and writes the data as it is stored, where v3.0 applies those streams | `#raw` on the path, to read and write the data as it is stored |
+| An option set for another extension of the format of the file, such as `option(:pgp, ...)` for a `.gpg` file, `option(:gz, ...)` for `.gzip`, or `option(:xlsx, ...)` for `.xlsm`, which is ignored, where v3.0 applies it | The option of the extension in the file name, such as `option(:gpg, ...)` |
+
+### Moving a layout to a schema
+
+`#format_options` is deprecated. It still works, and still reads and writes the file with its layout, as before, so
+no change is needed yet. A schema gives the type of each field of every format, and can be held in a JSON file, see
+[Schema](formats#schema). Each column of a layout is a field of the schema:
+
+| Layout | Schema |
+|--------|--------|
+| `key:` | `name` |
+| `size:` | `width` |
+| `size: :remainder` | `width: "remainder"` |
+| A column without a `key:`, a filler | `start` on the field after it, or the `width` of the schema at the end of the line |
+| `type: :string` | `type: "string"`, with `truncate: true` unless the layout has `truncate: false` |
+| `type: :integer` | `type: "integer"` |
+| `type: :float` and `decimals:` | `type: "number"`, with `floatNumber: true` and `decimals:`, without `decimals` for a column with `size: :remainder`, which writes the digits that the value has |
+| A record written with keys that the layout does not have, which are skipped | `fieldsMatch: "subset"` |
+
+~~~ruby
+# Before (deprecated)
+path.format(:fixed).format_options(
+  layout: [
+    {size: 23, key: "name"},
+    {size: 2},
+    {size: 5, key: "zip", type: :integer}
+  ]
+)
+
+# After
+path.format(:fixed).schema(
+  fieldsMatch: "subset",
+  fields:      [
+    {name: "name", type: "string",  width: 23, truncate: true},
+    {name: "zip",  type: "integer", width: 5,  start: 26}
+  ]
+)
+~~~
+
+`IOStreams::Schema.from_layout` returns the schema of a layout as a Hash, such as
+`IOStreams::Schema.from_layout(layout: layout)`, with `truncate: false` too when the layout has it. The columns of the
+layout can have String keys, such as a layout read from JSON. A column after the column with `size: :remainder` is
+left out, since the layout never reads it. It is only for moving off a layout, since v3.0 removes it, with
+`#format_options`: convert each layout with it once, such as in a console, and save the schema that it returns, such
+as in a JSON file, or as a Hash in the code, which `#schema` then sets, rather than call it each time a file is read
+or written, which breaks once the application upgrades to v3.0. It reports no deprecation warning.
+
+The keys of the records read are the names of the fields, so a layout with Symbol keys, such as `key: :name`, reads
+Symbol keys, as does a schema whose field is named with that Symbol. A schema saved as a JSON file, or in a database
+such as MongoDB, names its fields with Strings, so when every key of the layout is a Symbol, `from_layout` adds
+`keyType: "symbol"`, which keeps the keys of its records Symbols, see
+[Keys of the records read](formats#keys-of-the-records-read). Without `fieldsMatch: "subset"`, a schema has
+`"exact"`, so that writing a record with a key that is not a field raises `IOStreams::Errors::InvalidHeader`, rather
+than skip it as a layout does. A schema and a layout cannot both be set on a path: reading or writing it raises
+`ArgumentError`.
+
+The schema reads and writes the file as v3.0 does, which is not always as the layout does, so check that it reads
+the files of the application before it replaces the layout. What reads differently:
+
+* The `width` of each field counts the characters of the line, which is read as UTF-8, while the layout counts its
+  bytes. So a line with a character that is not ASCII, such as `é`, raises `IOStreams::Errors::InvalidLineLength`
+  when the program that wrote it counted bytes, and text that is not valid UTF-8, such as Latin-1, raises
+  `IOStreams::Errors::InvalidEncoding`. To count bytes, set the encoding of the file: `encoding("ISO-8859-1")` for
+  Latin-1, whose values are then ISO-8859-1 strings, or the name of another single-byte code page, or
+  `encoding("BINARY")` to read binary strings, as the layout does, such as for UTF-8 text whose sizes count bytes. See
+  [Schemas in IOStreams 2.2 and 3.0](formats#schemas-in-iostreams-22-and-30).
+* An `integer` or `number` field that holds no number, such as `N/A` or `1,234.50`, raises
+  `IOStreams::Errors::InvalidValue`, where the layout reads it as `0`, or as the number that its leading digits make,
+  such as `12` for `12-` and `1.0` for `1,234.50`. A fraction in an `integer` field, such as `12.5`, raises too, where
+  the layout truncates it, while a whole number with a decimal point, such as `12.0`, is still read as `12`.
+* A `number` field reads `NaN` and `Inf`, which the layout writes for a `Float` that is not finite, as `Float::NAN`
+  and `Float::INFINITY`, where the layout reads them as `0.0`.
+* A `number` field with `decimals` rounds the number read to them, such as `12.35` for `12.346` with `decimals: 2`,
+  where the layout reads all of its digits.
+* A line whose fields are all blank, even where a filler holds text, is a blank row, `nil`, which a copy skips, where
+  the layout reads a record of blank values, such as `{name: "", zip: nil}`.
+* A `string` field of only spaces is `nil`, a missing value, where the layout reads empty text, `""`, see
+  [Types](formats#types).
+* `each(:array)` yields the names of the fields first, as an `IOStreams::Row::HeaderRow`, and then an Array of the
+  values of each line, where the layout yields a Hash for each line, as `each(:hash)` does.
+
+What writes differently:
+
+* An `integer` or `number` field raises `IOStreams::Errors::InvalidValue` for a value that is not a number, such as
+  `"abc"`, or text of only spaces, such as `"  "`, and an `integer` field for a fraction, such as `12.7`, where the
+  layout writes zeros, or truncates the fraction.
+* A `string` field writes a `BigDecimal` as `12.5`, and a `Time` in ISO 8601, such as `2026-10-13T09:30:00Z`, where
+  the layout writes their `to_s`, such as `0.125e2` and `2026-10-13 09:30:00 UTC`. It raises
+  `IOStreams::Errors::InvalidValue` for a `Hash` or an `Array`, where the layout writes their `to_s`: to write their
+  JSON text, such as `{"a":1}`, give the field the type `object` or `array`.
+* A `number` field with `decimals` writes a negative number that rounds to zero, such as `-0.001` with `decimals: 2`,
+  as zero, without a minus sign, where the layout writes one, such as `-0000.00`.
+* A `number` field that takes the remainder of the line, without `decimals`, writes a whole number without a decimal
+  point, such as `12`, where the layout writes `12.0`.
+* A `number` field writes infinity as `INF`, where the layout writes `Inf`, and a `string` field writes a `Float` that
+  is not finite as `NaN`, `INF` or `-INF`, where the layout writes `NaN`, `Infinity` or `-Infinity`.
+* A filler is written as spaces, also where the layout gives it a `type:` of `:integer` or `:float`, which writes
+  zeros.
+* A record whose keys are Strings, such as `"name"`, is written to fields named with Symbols, such as `:name`, and
+  the other way around, where the layout writes spaces for each of its columns, or zeros for a number.
+* A column after the column with `size: :remainder` is not written, since it is not in the schema.
+
+### Moving the column options to a schema
+
+`columns:`, `cleanse_header:`, `allowed_columns:`, `required_columns:` and `skip_unknown:` are deprecated. They still
+work without a schema, so no change is needed yet. A [schema](formats#schema) describes the columns of the file on its
+path instead, so that every reader and writer of the path, including a copy, checks them the same way. Its fields
+name the keys of the records read, and each column of the header row is matched to a field whatever its case, spaces
+and punctuation, or to one of the `aliases` of a field, see [Matching columns to fields](formats#matching-columns-to-fields).
+So the keys no longer depend on how the cleansing rule changes a column, such as `Amount ($)` and `Amount (€)`, which
+are both `amount_`.
+
+To keep the keys of the records that the code reads, name each field as that key. The names that `allowed_columns:`
+and `required_columns:` list already are, since they were compared to the columns once cleansed, such as `first_name`
+for the column `First Name`, so use them as the names of the fields: the schema matches each column to the field
+named as the column was cleansed. With `cleanse_header: false`, they are the columns as they are in the file, so use
+them as they are.
+
+With a schema, `columns:`, `allowed_columns:` and `required_columns:` raise `ArgumentError`, unless they are nil, and
+so does `cleanse_header:`, even when it is nil. `skip_unknown:` only applies to `allowed_columns:`, so it is accepted.
+A nil one, and `skip_unknown:`, have no effect with a schema, and report a deprecation warning, since v3.0 raises
+`ArgumentError` for them, so remove them. The property of a schema that replaces each one:
+
+| Option | With a schema |
+|--------|---------------|
+| `columns:` | The fields are the columns, in order: with `header: false` to read a file of a format with a header row, such as CSV, `fieldsMatch: "partial"` to read JSON, and `fieldsMatch: "subset"` to write. A `nil` column, which was not read, is a field with `skip: true`, see [Field properties](formats#field-properties) |
+| `allowed_columns:` with `skip_unknown: false` | `fieldsMatch: "superset"`, which accepts a file without some of the fields, as `allowed_columns:` does, or `"equal"` when `required_columns:` lists every allowed column |
+| `allowed_columns:` with `skip_unknown: true` | `fieldsMatch: "partial"`, which accepts a file without some of the fields, as `allowed_columns:` does, or `"subset"` when `required_columns:` lists every allowed column |
+| `required_columns:` | `constraints: {required: true}` with `allowBlank: true` on each field, which needs the column but not a value, as `required_columns:` did, or a `fieldsMatch` that needs every field. Without `allowBlank`, a blank value raises too |
+| `required_columns:` without `allowed_columns:`, which also read the other columns | Also `fieldsMatch: "subset"`, `extraColumns: "keep"` and `keyType: "cleansed"`, which keep the other columns, keyed as `cleanse_header` cleansed them, see [Keeping the other columns](formats#keeping-the-other-columns) |
+| `cleanse_header:` | The keys of each record are the names of the fields, which the columns are matched to, or with `keyType: "cleansed"` the names once cleansed, as `cleanse_header: true` cleansed the columns, see [Keys of the records read](formats#keys-of-the-records-read) |
+| `cleanse_header: false` without the other options, which read every column as it is written | `fieldsMatch: "subset"`, `extraColumns: "keep"` and no fields |
+
+~~~ruby
+# Before (deprecated)
+path.each(:hash, allowed_columns: %w[first_name zip_code], required_columns: %w[zip_code]) do |record|
+  p record
+end
+
+# After
+path.schema(
+  fieldsMatch: "partial",
+  fields:      [
+    {name: "first_name"},
+    {name: "zip_code", constraints: {required: true}, allowBlank: true}
+  ]
+).each(:hash) do |record|
+  p record
+end
+
+# Both read "First Name,Zip Code,Notes\nJack,12345,x\nJill,,y\n" as:
+# {"first_name" => "Jack", "zip_code" => "12345"}
+# {"first_name" => "Jill", "zip_code" => nil}
+~~~
+
+Reading a CSV file without a header row:
+
+~~~ruby
+# Before (deprecated)
+IOStreams.path("no_header.csv").each(:hash, columns: ["name", "address", "zip"]) do |record|
+  p record
+end
+
+# After
+path = IOStreams.path("no_header.csv").schema(
+  header: false,
+  fields: [
+    {name: "name"},
+    {name: "address"},
+    {name: "zip"}
+  ]
+)
+path.each(:hash) do |record|
+  p record
+end
+~~~
+
+Writing only some of the keys of each record, in a fixed order:
+
+~~~ruby
+# Before (deprecated)
+path = IOStreams.path("sample.csv")
+path.writer(:hash, columns: ["name", "zip"]) do |io|
+  io << {"name" => "Jack Jones", "address" => "Somewhere", "zip" => 12345}
+end
+
+# After
+path = IOStreams.path("sample.csv").schema(
+  fieldsMatch: "subset",
+  fields:      [
+    {name: "name"},
+    {name: "zip"}
+  ]
+)
+path.writer(:hash) do |io|
+  io << {"name" => "Jack Jones", "address" => "Somewhere", "zip" => 12345}
+end
+~~~
+
+With a schema, `writer(:array)` writes the names of the fields as the header row, and takes each row as the values of
+the fields, in order, as `columns:` did.
+
+A schema reads the same records as the options it replaces, named as above, as it does in v3.0, except that:
+
+* A column is matched to a field whatever its case, spaces and punctuation, so a column that `allowed_columns:` did
+  not list once cleansed, such as `FirstName` for `first_name`, is read.
+* Two columns that are the same field, such as `Email` and `email`, or `Amount ($)` and `Amount (€)` for the field
+  `amount_`, raise `IOStreams::Errors::InvalidHeader`, where the value of the last one was kept.
+* The keys of each record are in the order of the fields, rather than of the columns of the file.
+* `each(:array)` yields a header row of the names of the fields, an `IOStreams::Row::HeaderRow`, and each row holds
+  only the values of the fields, where the column options left the header row and the rows as they are in the file.
+* A value after the last column, which is not blank, raises `IOStreams::Errors::InvalidHeader` unless `fieldsMatch`
+  is `"subset"` or `"partial"`, where it was dropped.
+* The text read is UTF-8, see [Schemas in IOStreams 2.2 and 3.0](formats#schemas-in-iostreams-22-and-30).
+* Empty text, such as a quoted `""` in CSV or `""` in JSON, is a missing value, `nil`, where the column options read
+  it as it is, see [Types](formats#types). Writing works the same way: empty text is written as a missing value, an
+  empty value in CSV and `null` in JSON, where the column options wrote `""`.
+* `IOStreams.enforce_column_restrictions` does not apply, since a schema applies to every input. So a schema applies
+  its fields to what the column options were not applied to, while `enforce_column_restrictions` is false, the
+  default: the keys of JSON records, the `columns:` of a file without a header row, and a header row read with
+  `cleanse_header: false`. One schema for the files of every format, such as uploads whose format is detected from
+  their file names, applies to each of them, where `allowed_columns:` restricted the columns of a CSV file but not the
+  keys of a JSON one. The names given to `columns:` were not cleansed, while a field named as one is matched to its
+  column loosely.
+* A name that the cleansing of v2.1 named without its letters that are not ASCII, such as `prnom` for the column
+  `Prénom`, which an application may have kept in `allowed_columns:`, is not the field of that column, since a schema
+  cleanses it to `prénom`, which IOStreams 3.0 also does without a schema. With `fieldsMatch: "subset"` or `"partial"`
+  the column is skipped without an error, so name the field `prénom`, or keep the name and give the field the column
+  as an alias, such as `{name: "prnom", aliases: ["Prénom"]}`.
+* A `Time` written to a field without a type is written in ISO 8601, such as `2026-10-13T09:30:00.123456Z`, where
+  `columns:` wrote its `to_s`, `2026-10-13 09:30:00 UTC`. A `datetime` field with
+  `format: "%Y-%m-%d %H:%M:%S UTC"` writes the text that `to_s` wrote.
+* A key of a record written that is the field only once compared loosely, such as `LastName` for `last_name`, is
+  written, where `columns:` wrote an empty value, and so is a Symbol key for a field named with a String, such as
+  `:name` for `"name"`, where a layout wrote spaces.
+
+Every use of the options has a schema form, see the table above, including `required_columns:` without
+`allowed_columns:` and `cleanse_header: false` on its own, which read every column, with `extraColumns: "keep"`, and
+`columns:` with `nil` for a column to skip, with `skip: true`. The options still work, so keep them until the schema
+has been checked against the files of the application, since it reads some of them differently, see above.
+
+### Changes in behavior
+
+- A failed copy no longer deletes a target that it did not write. `#copy_from`, `#copy_to` and `#move_to` open the
+  source before the target, as v3.0 does, and an error raised before the target is opened, such as for a source that
+  does not exist, or a mode that a copy does not convert with, leaves an existing target as it was, where v2.1 deleted
+  it, and a failed copy to S3 keeps the existing object. A copy that fails part way still leaves no part of the file,
+  see [When a copy fails](copy_files#when-a-copy-fails).
+- A mode that a copy does not convert with, such as `copy_from(source, mode: :stream)`, raises `ArgumentError` before
+  the source or the target is opened, with a message that starts with v2.1's, `Invalid mode: :stream`, and then lists
+  the modes.
+- A copy of a path, made with `dup`, has its own streams and options, so that changing it does not change the path
+  that it was copied from, and a copy with `convert: false` no longer changes the streams of the source or the
+  target. A copy with `convert: false` from a source with options set with `#option` copies it, where v2.1 raised
+  `ArgumentError`.
+
 ## Upgrading to v2.1
 
 v2.1 is a security release, and is backward compatible except for `IOStreams::Pgp.delete_keys`,
 described below. Most applications upgrade without any code changes. The changes that would break
-existing code are postponed to v3.0, and log a warning in v2.1. See [Coming in v3.0](#coming-in-v30).
+existing code are postponed to v3.0, see [Coming in v3.0](#coming-in-v30), which says which of them
+log a warning.
 
 ### Changes that may need code changes
 
@@ -99,7 +491,7 @@ IOStreams.path("s3://my-bucket-name/uploads").join(untrusted_name)
 
 A username and password supplied in an HTTP or SFTP url remain part of it, so `#to_s` returns
 them, as does any log or error message that includes the path. Supply them with the `username:`
-and `password:` arguments instead:
+and `password:` arguments instead, and log `#display_name`, which never includes them:
 
 ~~~ruby
 # Returns the password from #to_s
@@ -160,41 +552,96 @@ IOStreams.enforce_column_restrictions = true
 ~~~
 
 See [Column restrictions apply to every input](#column-restrictions-apply-to-every-input) for what
-it changes.
+it changes. v2.2 deprecates this setting, with the column options that it applies to: a schema applies to every input
+instead, see [Moving the column options to a schema](#moving-the-column-options-to-a-schema).
 
 ## Coming in v3.0
 
-These changes are postponed to v3.0 since they could break existing code. Each one logs a warning
-in v2.1 via `IOStreams.logger` when it would change the result, so check your logs for them.
+These changes are postponed to v3.0 since they could break existing code. v2.1 and v2.2 log a warning
+via `IOStreams.logger` when the column restrictions or the BZip2 options would change the result, so
+check your logs for them, and v2.2 reports PGP `export` without an email or key id as deprecated.
+v3.0 makes many more changes, see its
+[upgrading guide](https://github.com/reidmorrison/iostreams/blob/main/docs/upgrading.md#upgrading-to-v30), and
+[Upgrading to v2.2](#upgrading-to-v22) for the deprecations that v2.2 reports to prepare for it. v3.0 also changes
+what the encode stream reads and writes, which v2.2 cannot report, see
+[What `#encoding` reads and writes in v3.0](#what-encoding-reads-and-writes-in-v30).
 
 ### Column restrictions apply to every input
 
-When reading records, `allowed_columns`, `required_columns` and `skip_unknown` will apply to every
-input. In v2.1 they are ignored for JSON and `:hash` input, when `columns:` is supplied, and with
-`cleanse_header: false`.
+v3.0 removes the column options of each call, `columns:`, `allowed_columns:`, `required_columns:`,
+`skip_unknown:` and `cleanse_header:`, and `IOStreams.enforce_column_restrictions=` with them, which
+v2.2 deprecates. A schema describes the columns of the file instead, and its `fieldsMatch` and
+`constraints` apply to every input: a header row, the columns of a file without one, and the keys of
+each JSON record. In v2.1 and v2.2 the column options are ignored for JSON and `:hash` input, when
+`columns:` is supplied, and with `cleanse_header: false`, unless `IOStreams.enforce_column_restrictions`
+is true, so renaming an upload from `.csv` to `.json` bypasses them.
 
-When either `allowed_columns` or `required_columns` is set, JSON keys will be cleansed the same way
-as a header row, for example `"Name"` becomes `"name"`. Unknown keys will be skipped, or raise
-`IOStreams::Errors::InvalidHeader` when `skip_unknown: false`, and a record that is missing a
-required column will raise `IOStreams::Errors::InvalidHeader`.
-
-To prepare: set `IOStreams.enforce_column_restrictions = true`, and check that the JSON files you
-read with these options have the expected keys. See [Header options](formats#header-options).
+To prepare: describe the columns with a schema, see
+[Moving the column options to a schema](#moving-the-column-options-to-a-schema), and check that it reads
+the files of the application as the column options did. Until then, keep
+`IOStreams.enforce_column_restrictions = true` where the column options restrict what an upload can set,
+since v2.2 still applies it, see [Header options](formats#header-options).
 
 ### BZip2 options are strict
 
-The BZip2 reader and writer ignore any option they do not accept, and log a warning. In v3.0 it will
-raise `ArgumentError`, like every other stream. They accept `autoclose`, `first_only` and `small` when
+The BZip2 reader and writer ignore any option they do not accept, and log a warning. In v3.0 an option
+that neither of them takes, such as a misspelled one, raises `ArgumentError` when it is set, like that of
+every other stream, while an option of the other direction, such as `block_size` when reading, is
+accepted, and ignored, as v2.1 and v2.2 do. They accept `autoclose`, `first_only` and `small` when
 reading, and `autoclose`, `block_size` and `work_factor` when writing.
 
-To prepare: remove the option, or use a separate path for reading and for writing. See
-[Reading and writing need separate options](streams#reading-and-writing-need-separate-options).
+To prepare: remove an option that neither of them takes, which v2.2 reports as deprecated when it is
+set, see [Deprecated, and what replaces it](#deprecated-and-what-replaces-it).
 
 ### PGP `export` without an email or key id
 
 `IOStreams::Pgp.export(email: nil)` without a `key_id:` raises `IOStreams::Pgp::Failure`, as it did
-before v2.1. In v3.0 it will raise `ArgumentError`. Calling it without either argument already raises
-`ArgumentError`.
+before v2.1, and so does `IOStreams::Pgp.set_trust` without an `email:` or `key_id:`, since gpg rejects
+the empty user id that it is asked for. In v3.0 they raise `ArgumentError`, so v2.2 reports them as
+deprecated. Calling `export` without either argument already raises `ArgumentError`.
+
+To prepare: supply the email or key id of the key, or rescue `ArgumentError` as well as
+`IOStreams::Pgp::Failure`.
+
+### What `#encoding` reads and writes in v3.0
+
+`#encoding` sets the encode stream with the same options, and the same results, as `option(:encode, ...)`,
+which it replaces. v3.0 changes what that stream reads and writes, so these results of `#encoding` change
+on upgrading, while the call stays the same, which v2.2 cannot report as deprecated. Each says what gives
+the same result in v2.2 and v3.0, where something does:
+
+- **`replace:` keeps the valid characters.** v2.2 takes the data read as binary, so `replace:` replaces
+  each byte that is not ASCII, even of a valid character: `encoding("UTF-8", replace: "")` reads
+  `José,\xFF` as `Jos,`, while v3.0 takes the data as text in the encoding, and only replaces what is not
+  valid in it, `José,`. So does `encoding("Windows-1252", replace: "")` for Windows-1252 text. To keep
+  only ASCII on both, use `encoding("US-ASCII", replace: "")`. To keep the valid UTF-8 characters on
+  both, read with `encoding("BINARY")`, and replace the others yourself, such as with
+  `line.force_encoding("UTF-8").scrub("")`.
+- **Text that is not valid in the encoding raises `IOStreams::Errors::InvalidEncoding`**, an
+  `Encoding::UndefinedConversionError`, on every read, naming its line. v2.2 raises `ArgumentError` for
+  lines, rows and records, and `Encoding::UndefinedConversionError` for `#read`, and a write raises
+  `Encoding::UndefinedConversionError` on both. To prepare, rescue
+  `ArgumentError, Encoding::UndefinedConversionError`.
+- **A byte order mark is removed** from the start of the data, so the first line, or the first value of
+  the first row, no longer starts with `"\uFEFF"`, which v2.2 keeps with `encoding("UTF-8")`. To get the
+  same on both, remove it yourself, such as with `line.delete_prefix("\uFEFF")`.
+- **`read(length)` returns whole characters**, at least one, so that a character is not split between
+  reads. v2.2 returns `length` bytes, which can end within a character, in the same String for each
+  read. To read the same bytes on both, read without `#encoding`, or with `encoding("BINARY")`.
+- **Text written to a `StringIO` keeps the encoding that is set.** v3.0 writes the bytes of the
+  encoding, such as of Windows-1252 for `IOStreams.stream(io).encoding("Windows-1252")`, while in v2.2 a
+  `StringIO` of a UTF-8 String, such as `StringIO.new`, converts them back to UTF-8. Supply a `StringIO`
+  of a binary String, `StringIO.new("".b)`, which holds the bytes of the encoding on both.
+- **`#read` of the whole file takes the data as text in the encoding**, as reading lines does. v2.2
+  converts it into the encoding instead: from binary for most files, so that `encoding("UTF-8").read`
+  raises `Encoding::UndefinedConversionError` for any character that is not ASCII, and from
+  `Encoding.default_external` for a `.gz` or `.enc` file, so that `encoding("Windows-1252").read` converts
+  UTF-8 text into Windows-1252, and raises for Windows-1252 text. To get the same on both, read the lines,
+  rows or records, or `#read` without `#encoding`, and call `force_encoding` on the String, such as
+  `read.force_encoding("Windows-1252")`.
+
+See the [upgrading guide](https://github.com/reidmorrison/iostreams/blob/main/docs/upgrading.md#text-and-encodings)
+of v3.0 for the other changes to text, such as reading it as UTF-8 by default.
 
 ## Upgrading to v2.0