iostreams 1.11.0 → 2.1.0

data/docs/tutorial.md

@@ -0,0 +1,483 @@
+---
+layout: default
+title: Tutorial
+heading: File / Data Streaming with Ruby
+description: >-
+  A step by step introduction to streaming files with IOStreams, from counting
+  the lines in a gzip file to reading tabular data out of S3.
+---
+
+If all files were small, they could just be loaded into memory in their entirety. 
+However, multi Gigabytes, or even Terabytes in size, loading them into memory is not feasible.
+ 
+In linux it is common to use pipes to stream data between processes. 
+For example:
+
+~~~
+# Count the number of lines in a file that has been compressed with gzip
+cat abc.gz | gunzip -c | wc -l
+~~~
+
+For large files it is critical to be able to read and write these files as streams. Ruby has support
+for reading and writing files using streams, but has no built-in way of passing one stream through
+another to support for example compressing the data, encrypting it and then finally writing the result
+to a file. Several streaming implementations exist for languages such as `C++` and `Java` to chain
+together several streams, `IOStreams` offers similar features for Ruby.
+
+~~~ruby
+# Read the first 1024 characters from a compressed file:
+path = IOStreams.path("hello.gz")
+path.reader do |io|
+  data = io.read(1024)
+  puts "Read: #{data}"
+end
+~~~
+
+The true power of streams is shown when many streams are chained together to achieve the end
+result, without holding the entire file in memory, or ideally without needing to create
+any temporary files to process the stream.
+
+~~~ruby
+# Create a file that is compressed with GZip and then encrypted with Symmetric Encryption:
+path = IOStreams.path("hello.gz.enc")
+path.writer do |io|
+  io << "Hello World"
+  io << "and some more"
+end
+~~~
+
+The power of the above example applies when the data being written starts to exceed hundreds of megabytes,
+or even gigabytes.
+
+By looking at the file name supplied above, IOStreams is able to determine which streams to apply
+to the data being read or written. For example:
+* `hello.zip` => Compressed using Zip
+* `hello.zip.enc` => Compressed using Zip and then encrypted using Symmetric Encryption
+* `hello.gz.enc` => Compressed using GZip and then encrypted using Symmetric Encryption
+
+The objective is that all of these streaming processes are performed used streaming
+so that only the current portion of the file is loaded into memory as it moves
+through the entire file.
+
+## Step by Step 
+
+Install IOStreams gem:
+~~~
+gem install iostreams --no-doc
+~~~
+
+If you want to follow the AWS S3 examples below install the AWS S3 gem:
+~~~
+gem install aws-sdk-s3 --no-doc
+~~~
+
+See [Configuring the AWS SDK for Ruby](https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html)
+to configure the Ruby AWS library.
+
+Open a ruby interactive console:
+
+~~~
+irb
+~~~
+ 
+Load iostreams:
+
+~~~ruby
+require "iostreams"
+~~~
+ 
+Reference a file path to hold CSV data and then for fun lets also compress it with GZip: 
+~~~ruby
+path = IOStreams.path("sample/example.csv.gz")
+# => #<IOStreams::Paths::File:sample/example.csv.gz pipeline={:gz=>{}}> 
+~~~
+
+The path and file name does not exist yet:
+~~~ruby
+path.exist?
+# => false 
+~~~
+
+If the path `sample` does not exist, it is created automatically during the first write. 
+Write CSV data to the file, compressing to GZip as we go.
+~~~ruby
+path.writer do |io|
+  io << "name,login\n"
+  io << "Jack Jones,jjones\n"
+  io << "Jill Smith,jsmith\n"
+end
+~~~
+
+To verify the data written above, read the entire file:
+~~~ruby
+path.read
+# => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n"
+~~~
+
+It would be much easier if we could write the CSV data as hashes and let IOStreams deal
+with all the details on how to create properly formatted CSV data: 
+~~~ruby
+path.writer(:hash) do |io|
+  io << {name: "Jack Jones", login: "jjones"}
+  io << {name: "Jill Smith", login: "jsmith"}
+end 
+~~~
+
+Verify the data written by reading the entire file:
+~~~ruby
+path.read
+# => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n"
+~~~
+
+Now lets read the file one line at a time:
+~~~ruby
+path.each do |line|
+  puts line
+end
+~~~
+Output:
+~~~
+name,login
+Jack Jones,jjones
+Jill Smith,jsmith
+~~~
+
+But who wants to do CSV parsing by hand, lets get IOStreams to do that for us by passing `:array` to `each`:
+~~~ruby
+path.each(:array) do |array|
+  p array
+end
+~~~
+Output:
+~~~ruby
+["name", "login"]
+["Jack Jones", "jjones"]
+["Jill Smith", "jsmith"]
+~~~
+
+That was better, but we really want a hash back where IOStreams takes care of the CSV header:
+~~~ruby
+path.each(:hash) do |hash|
+  p hash
+end
+~~~
+Output:
+~~~ruby
+{"name"=>"Jack Jones", "login"=>"jjones"}
+{"name"=>"Jill Smith", "login"=>"jsmith"}
+~~~
+
+As the file gets larger and we reach millions of rows the above code does not have to change at all. 
+And memory utilization stays about the same since each block is read in, decompressed, and parsed 
+from CSV one block at a time. The garbage collector can then free the released blocks from memory. 
+
+Now lets read a zip file hosted on an HTTP Web Server, displaying the first row as a hash: 
+
+~~~ruby
+IOStreams.
+  path("https://www5.fdic.gov/idasp/Offices2.zip").
+  option(:zip, entry_file_name: "OFFICES2_ALL.CSV").
+  each(:hash) do |row|
+    p row
+    # Just show the first line for this tutorial
+    break
+  end
+~~~
+Output:
+~~~ruby
+{"address"=>"1 Lincoln St. Fl 1", "bkclass"=>"SM", "cbsa"=>"Boston-Cambridge-Newton, MA-NH", "cbsa_div"=>"Boston, MA", "cbsa_div_flg"=>"1", "cbsa_div_no"=>"14454", "cbsa_metro"=>"14460", "cbsa_metro_flg"=>"1", "cbsa_metro_name"=>"Boston-Cambridge-Newton, MA-NH", "cbsa_micro_flg"=>"0", "cbsa_no"=>"14460", "cert"=>"14", "city"=>"Boston", "county"=>"Suffolk", "csa"=>"Boston-Worcester-Providence, MA-RI-NH-CT", "csa_flg"=>"1", "csa_no"=>"148", "estymd"=>"1792-01-01", "fi_uninum"=>"6", "mainoff"=>"1", "name"=>"State Street Bank And Trust Company", "offname"=>"State Street Bank And Trust Company", "offnum"=>nil, "rundate"=>"2020-05-14", "servtype"=>"11", "stalp"=>"MA", "stcnty"=>"25025", "stname"=>"Massachusetts", "uninum"=>"6", "zip"=>"2111"}
+~~~
+
+Noticed that it took a while to return the first line?
+
+That is because `zip` requires the entire file to be downloaded before it can decompress anything
+in the file. And HTTP uses a push protocol when reading files, so it is downloaded automatically
+into a temp file behind the scenes so that we can read it as if it was a local file.
+
+## Same Code - Varying File Types
+
+Lets define a method to write data to a file.
+~~~ruby
+def write_lines(file_name)
+  path = IOStreams.path(file_name)
+  path.writer do |io|
+    io << "name,login\n"
+    io << "Jack Jones,jjones\n"
+    io << "Jill Smith,jsmith\n"
+  end
+end
+~~~
+
+Create some sample files to work with
+~~~ruby
+write_lines("sample/example.csv")
+write_lines("sample/example.csv.gz")
+~~~
+
+For PGP files we also need to specify the recipient that can decrypt the file.
+~~~ruby
+path = IOStreams.path("sample/example.csv.pgp")
+path.option(:pgp, recipient: "receiver@example.org")
+write_lines(path)
+~~~
+
+`IOStreams.path` takes a string as its argument, it can also accept an existing instance of `IOStreams`.
+That allows the same method to accept the pgp recipient without having to pass the pgp specific recipient information
+as an argument to the method.
+
+Consider a simple method to display the contents of a file a line at a time 
+prefixed with the line number within the file:
+~~~ruby
+def show_lines(file_name)
+  line_number = 1
+  path = IOStreams.path(file_name)
+  path.each(:line) do |line|
+    puts "[#{line_number}] #{line}"
+    line_number += 1
+  end
+end
+~~~
+
+Lets read all of the files created above with the new `show_lines` method:
+
+~~~ruby
+show_lines("sample/example.csv")
+show_lines("sample/example.csv.gz")
+show_lines("sample/example.csv.pgp")
+~~~
+
+Noticed how they all returned the exact same output, even though the first file was plain text, the second was
+compressed with Gzip and the third was encrypted with PGP. They all returned:
+~~~
+[1] name,login
+[2] Jack Jones,jjones
+[3] Jill Smith,jsmith
+~~~
+
+Now a program can be developed using IOStreams and then without any code changes is able read plain text, compressed,
+or encrypted files.
+
+## Same Code - Any File Storage
+
+Using the unchanged `write_lines` and `show_lines` methods above, lets use them to read and write from S3.
+
+But how is that possible since our program / methods above were only tested against local files?
+
+Create the same sample files to work with, but this time on AWS S3 in a bucket name `my-iostreams-bucket`
+~~~ruby
+write_lines("s3://my-iostreams-bucket/sample/example.csv")
+write_lines("s3://my-iostreams-bucket/sample/example.csv.gz")
+~~~
+
+For PGP files we also need to specify the recipient that can decrypt the file.
+~~~ruby
+path = IOStreams.path("s3://my-iostreams-bucket/sample/example.csv.pgp")
+path.option(:pgp, recipient: "receiver@example.org")
+write_lines(path)
+~~~
+
+The only change to switch to S3 storage was to prefix the file name passed in with `s3://my-iostreams-bucket/`.
+
+Lets read all of the files created above with the new `show_lines` method:
+
+~~~ruby
+show_lines("s3://my-iostreams-bucket/sample/example.csv")
+show_lines("s3://my-iostreams-bucket/sample/example.csv.gz")
+show_lines("s3://my-iostreams-bucket/sample/example.csv.pgp")
+~~~
+
+Noticed how they all returned the exact same output, even though the first file was plain text, the second was
+compressed with Gzip and the third was encrypted with PGP. They all returned:
+~~~
+[1] name,login
+[2] Jack Jones,jjones
+[3] Jill Smith,jsmith
+~~~
+
+Now a program can be developed using IOStreams and then without any code changes is able to read and write across
+multiple storage locations.
+
+### Configure storage once, not in every call
+
+In the examples above we still wrote the `s3://my-iostreams-bucket/` prefix into every call, which
+means the code itself decides where files are stored. The real goal is for the application code to
+say nothing at all about storage, so the same code can run unchanged against local files in
+development and S3 in production.
+
+Roots make that possible. Configure the storage location once in a startup initializer:
+~~~ruby
+# config/initializers/iostreams.rb (development)
+IOStreams.add_root(:default, "tmp/files")
+~~~
+
+Then use `IOStreams.join` instead of `IOStreams.path`, and never mention the storage location again:
+~~~ruby
+def write_lines(file_name)
+  path = IOStreams.join(file_name)
+  path.writer do |io|
+    io << "name,login\n"
+    io << "Jack Jones,jjones\n"
+    io << "Jill Smith,jsmith\n"
+  end
+end
+
+write_lines("sample/example.csv")
+write_lines("sample/example.csv.gz")
+~~~
+
+To run the exact same code against S3 in production, change only the initializer:
+~~~ruby
+# config/initializers/iostreams.rb (production)
+IOStreams.add_root(:default, "s3://my-iostreams-bucket/files")
+~~~
+
+The application code does not change at all. See [Configuring IOStreams](config) for multiple roots,
+for example separate locations for downloads, uploads, and reports.
+
+## Tabular Files
+
+Tabular files are any files that start with a header row and then follows with rows of data with each row
+on a separate line.
+
+For example "example.csv"
+~~~
+name,login
+Jack Jones,jjones
+Jill Smith,jsmith
+~~~
+
+The first line contains the header: `name,login`
+Each subsequent line contains the data delimited by a special character such as `,` in the same order as the header.
+
+Another example is PSV (Pipe Separated Files)
+~~~
+name|login
+Jack Jones|jjones
+Jill Smith|jsmith
+~~~
+
+Of course these are simple examples and there are lots of rules on how to embed or escape the row or column delimiters.
+
+### Reading Tabular Files
+
+When reading these files, IOStreams can handle the complexity of the files format and always return the data as a
+`hash`, or `array`.
+
+Lets create another method along the lines of `show_lines` above:
+~~~ruby
+def show_rows(file_name)
+  line_number = 1
+  path = IOStreams.path(file_name)
+  path.each(:hash) do |row|
+    puts "[#{line_number}] #{row.inspect}"
+    line_number += 1
+  end
+end
+~~~
+
+The key difference is that `:hash` is being passed into `each` instead of `:line`.
+
+Using the sample files created above:
+~~~ruby
+show_rows("sample/example.csv")
+~~~
+
+Outputs:
+~~~
+[1] {"name"=>"Jack Jones", "login"=>"jjones"}
+[2] {"name"=>"Jill Smith", "login"=>"jsmith"}
+~~~
+
+Notice how only 2 rows are returned, since the header row is not actual data, it is just the definition of the
+rows that follow.
+
+The same method works without changes regardless of where the file was stored, or whether it was encrypted or
+compressed.
+~~~ruby
+show_rows("s3://my-iostreams-bucket/sample/example.csv")
+show_rows("s3://my-iostreams-bucket/sample/example.csv.gz")
+show_rows("s3://my-iostreams-bucket/sample/example.csv.pgp")
+~~~
+
+### Writing Tabular Files
+
+Lets define a new method that uses a tabular api to write the data.
+~~~ruby
+def write_tabular(file_name)
+  path = IOStreams.path(file_name)
+  path.writer(:hash) do |io|
+    io << {"name"=>"Jack Jones", "login"=>"jjones"}
+    io << {"name"=>"Jill Smith", "login"=>"jsmith"}
+  end
+end
+~~~
+
+The key difference is that `:hash` is being passed into `writer` to indicate that it will receive hashes instead of
+raw data.
+
+Lets create a sample file, and then read it to compare its contents to the raw writer above.
+~~~ruby
+write_tabular("sample/example.csv")
+IOStreams.path("sample/example.csv").read
+# => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n"
+~~~
+
+Note how the output file is identical to the one created above. 
+Using `writer(:hash)` makes it easier to develop the application without regard for:
+- The order of columns
+- Missing columns
+- Specialized escaping of values to handle row or column delimiters
+
+Note: The first row written determines the column names as well as the order of the elements to be written.
+To supply the header columns up front, to set the order or to filter out which columns should be written to the
+target file, pass them to the writer: `path.writer(:hash, columns: ["name", "login"])`.
+
+Now lets write the same data into a JSON file, then read it to see what it looks like:
+~~~ruby
+write_tabular("sample/example.json")
+puts IOStreams.path("sample/example.json").read
+# => "{\"name\":\"Jack Jones\",\"login\":\"jjones\"}\n{\"name\":\"Jill Smith\",\"login\":\"jsmith\"}\n"
+~~~
+
+Using the same `show_rows` method above to display the file line by line
+~~~ruby
+show_rows("sample/example.json")
+~~~
+
+Outputs the same data even though the file is now json instead of the previous file that was csv:
+~~~
+[1] {"name"=>"Jack Jones", "login"=>"jjones"}
+[2] {"name"=>"Jill Smith", "login"=>"jsmith"}
+~~~
+
+The same method works without changes regardless of where the file was stored, or whether it was encrypted or
+compressed, or whether the format was csv or json.
+~~~ruby
+show_rows("sample/example.csv")
+show_rows("sample/example.json")
+show_rows("s3://my-iostreams-bucket/sample/example.csv")
+show_rows("s3://my-iostreams-bucket/sample/example.json.gz")
+show_rows("s3://my-iostreams-bucket/sample/example.json.pgp")
+~~~
+
+## Conclusion
+
+IOStreams makes it possible to write an application to a common api so that
+* the file can be accessed anywhere ( at least a local file, AWS S3, HTTP(S) and SFTP for now).
+* the application does not care if or how the file was compressed.
+* the application does not care if or how the file was encrypted.
+* the actual file storage mechanism can be determined at runtime, or per environment.
+* it is transparent whether the application receives an Excel Spreadsheet, CSV, or PSV formatted file.
+  It just works with hashes when desired.  
+
+IOStreams is an incredibly powerful streaming library that makes changes to file formats, compression, encryption, 
+or storage mechanism transparent to the application.
+
+## Next Steps
+
+Move onto the [IOStreams Path Documentation](path) to see how to create paths that support AWS S3, HTTP(S), or SFTP.
+
+Or jump straight into the [IOStreams Streams Documentation](streams) for detailed information on working with streams.
+
+Read the [IOStreams PGP Documentation](pgp) for a tutorial on how to work with PGP Encrypted files.
+

data/docs/upgrading.md

@@ -0,0 +1,217 @@
+---
+layout: default
+title: Upgrading
+heading: Upgrading IOStreams
+description: >-
+  What changes when upgrading IOStreams to v2.1 or v2.0, and the security
+  settings to review in your application when upgrading.
+---
+
+This page covers the changes that may need updates to your application when upgrading IOStreams,
+and the security issues to check. For every change in each release, see the
+[CHANGELOG](https://github.com/reidmorrison/iostreams/blob/main/CHANGELOG.md).
+
+## Upgrading to v2.1
+
+v2.1 is a security release, and is backward compatible except for `IOStreams::Pgp.delete_keys`,
+described below. Most applications upgrade without any code changes. The changes that would break
+existing code are postponed to v3.0, and log a warning in v2.1. See [Coming in v3.0](#coming-in-v30).
+
+### Changes that may need code changes
+
+#### PGP `delete_keys` requires an email or key id
+
+`IOStreams::Pgp.delete_keys` raises `ArgumentError` unless an `email:` or `key_id:` is supplied.
+Previously, on GnuPG 2.1 and later, calling it without either deleted every key in the keyring, and
+with `private: true`, every secret key. This is the only breaking change in v2.1, since that could
+not be undone. To delete several keys, call it once for each key, for example for each key returned
+by `IOStreams::Pgp.list_keys`.
+
+#### SFTP `#each_child` requires a known host key
+
+`#each_child` on an SFTP path now verifies the server's host key, the same way reading and writing
+already did. Previously it trusted a host key the first time it was seen. If listing files now
+fails, supply the server's host key with the `HostKey` ssh option, or add it to the `known_hosts`
+file of the user running the application. See [SFTP](path#sftp-sftp).
+
+`#each_child` now also supports the `HostKey`, `IdentityKey` and `IdentityFile` ssh options, and
+some others. Previously any ssh option made it raise `ArgumentError`. Any ssh option it does not
+support raises `ArgumentError`.
+
+#### Other changes in behavior
+
+- SFTP `#each_child` lists the files in the path's directory. Previously it ignored the path and
+  listed the login directory, so `IOStreams.path("sftp://host/data/in").each_child("*.csv")`
+  listed `~/*.csv`, returning paths such as `/a.csv`. A url without a path, such as `sftp://host`,
+  still lists the login directory.
+- `Path#join` and `IOStreams.join` only return an element unchanged when it is the path itself or
+  is inside it. Previously any element that started with the same characters was returned as-is,
+  so `IOStreams.path("s3://bucket/reports").join("reports_2024.csv")` gave
+  `s3://bucket/reports_2024.csv`, instead of `s3://bucket/reports/reports_2024.csv`.
+- When writing PSV or fixed width files, a line break within a value is replaced with a space, so
+  that a value can no longer start a separate record.
+- Internal temp files are created so that only the current user can read them.
+- When reading lines, newlines within quoted values are kept within the line when the tabular format
+  quotes its values, such as CSV. Previously this depended on whether the file name contained `.csv`,
+  so `.format(:psv)` on a `.csv` file still joined quoted lines, and `.format(:csv)` on a stream
+  without a file name did not.
+- An `ArgumentError` for an option that a stream does not accept now says which direction the option
+  belongs to, or lists the valid options. Previously it was Ruby's `unknown keyword` error.
+
+### Security checklist
+
+When upgrading, review how your application uses IOStreams for the following documented security
+issues. Each one depends on how the application is written, so upgrading alone does not fix it.
+
+#### File names from untrusted input
+
+If a user, a job, or any other untrusted input can supply a file name, it can also supply `..`
+or an absolute path, and read or overwrite any file that the process can access. `IOStreams.join`
+and root paths do not prevent this.
+
+New in v2.1, add **allowed paths** in an initializer, so that accessing any other path raises
+`IOStreams::Errors::AccessDenied`:
+
+~~~ruby
+IOStreams.add_allowed_path("/var/my_app/uploads")
+IOStreams.add_allowed_path("s3://my-app-bucket-name/export")
+~~~
+
+Once any allowed path is added, every path IOStreams accesses in the process must be within one of
+them, so add every location that your application reads or writes. See
+[Restricting access with allowed paths](path#restricting-access-with-allowed-paths).
+
+#### Untrusted names in S3 urls
+
+Any query string in an S3 url is added to the S3 request parameters. Do not interpolate an
+untrusted file name into the url, since a name such as `file.csv?acl=public-read` would set
+request parameters. Join it onto the path instead. See [AWS S3](path#aws-s3-s3).
+
+~~~ruby
+# Unsafe
+IOStreams.path("s3://my-bucket-name/uploads/#{untrusted_name}")
+
+# Safe
+IOStreams.path("s3://my-bucket-name/uploads").join(untrusted_name)
+~~~
+
+#### Credentials in HTTP and SFTP urls
+
+A username and password supplied in an HTTP or SFTP url remain part of it, so `#to_s` returns
+them, as does any log or error message that includes the path. Supply them with the `username:`
+and `password:` arguments instead:
+
+~~~ruby
+# Returns the password from #to_s
+IOStreams.path("sftp://jack:secret@sftp.example.org/file.csv")
+
+# Does not
+IOStreams.path("sftp://sftp.example.org/file.csv", username: "jack", password: "secret")
+~~~
+
+#### Untrusted HTTP urls
+
+When any part of an HTTP url comes from untrusted input, an attacker can point it at internal
+services or cloud metadata endpoints (Server Side Request Forgery), either directly or with a
+redirect. Use `allow_hosts`, `http_redirect_count` and `maximum_file_size`, or allowed paths, which
+also check every redirect. See [Security: untrusted URLs (SSRF)](path#security-untrusted-urls-ssrf).
+
+#### Processing PGP files before they are verified
+
+By default IOStreams passes the decrypted contents of a PGP file to your block as gpg decrypts it.
+gpg can only check the file's integrity and signature once it has read the whole file, so a
+tampered file is only rejected after your block has processed its contents. Either do not commit
+any side effects until the block returns, for example by using a database transaction, or supply
+the `verify_first` option, new in v2.1. See
+[Verifying a file before processing it](pgp#verifying-a-file-before-processing-it).
+
+~~~ruby
+path.option(:pgp, passphrase: "receiver_passphrase", verify_first: true)
+~~~
+
+#### Importing and trusting PGP keys
+
+`import_and_trust` and the writer's `import_and_trust_key` option trust a key at the Ultimate
+level by default, so only use them with keys received from a verified, trusted source. When a key
+cannot be fully verified, supply a lower `trust_level:` to `import_and_trust`, or a lower
+`import_and_trust_level` option to the writer. See
+[Trust level](pgp#trust-level).
+
+In v2.1, `import_and_trust_key` encrypts to the imported key's fingerprint on GnuPG 2.1 and later.
+Previously it encrypted to the key's email address, so if your keyring holds another key with the
+same email address, that key could have been used instead. Check your keyring for duplicate email
+addresses.
+
+#### PGP files without integrity protection
+
+Only enable `ignore_mdc_error` for files from a trusted source, since without MDC the decrypted
+contents are not protected against tampering. See
+[Reading legacy files without MDC integrity protection](pgp#reading-legacy-files-without-mdc-integrity-protection).
+
+#### Column restrictions on uploaded files
+
+By default, `allowed_columns` and `required_columns` are ignored for JSON input. Since the format
+is usually inferred from the file name, renaming an upload from `.csv` to `.json` bypasses them.
+If your application relies on them to restrict which columns an upload can set, apply them to every
+input in an initializer, new in v2.1, and check whether any uploads were affected:
+
+~~~ruby
+IOStreams.enforce_column_restrictions = true
+~~~
+
+See [Column restrictions apply to every input](#column-restrictions-apply-to-every-input) for what
+it changes.
+
+## Coming in v3.0
+
+These changes are postponed to v3.0 since they could break existing code. Each one logs a warning
+in v2.1 via `IOStreams.logger` when it would change the result, so check your logs for them.
+
+### Column restrictions apply to every input
+
+When reading records, `allowed_columns`, `required_columns` and `skip_unknown` will apply to every
+input. In v2.1 they are ignored for JSON and `:hash` input, when `columns:` is supplied, and with
+`cleanse_header: false`.
+
+When either `allowed_columns` or `required_columns` is set, JSON keys will be cleansed the same way
+as a header row, for example `"Name"` becomes `"name"`. Unknown keys will be skipped, or raise
+`IOStreams::Errors::InvalidHeader` when `skip_unknown: false`, and a record that is missing a
+required column will raise `IOStreams::Errors::InvalidHeader`.
+
+To prepare: set `IOStreams.enforce_column_restrictions = true`, and check that the JSON files you
+read with these options have the expected keys. See [Header options](formats#header-options).
+
+### BZip2 options are strict
+
+The BZip2 reader and writer ignore any option they do not accept, and log a warning. In v3.0 it will
+raise `ArgumentError`, like every other stream. They accept `autoclose`, `first_only` and `small` when
+reading, and `autoclose`, `block_size` and `work_factor` when writing.
+
+To prepare: remove the option, or use a separate path for reading and for writing. See
+[Reading and writing need separate options](streams#reading-and-writing-need-separate-options).
+
+### PGP `export` without an email or key id
+
+`IOStreams::Pgp.export(email: nil)` without a `key_id:` raises `IOStreams::Pgp::Failure`, as it did
+before v2.1. In v3.0 it will raise `ArgumentError`. Calling it without either argument already raises
+`ArgumentError`.
+
+## Upgrading to v2.0
+
+v2.0 is a major release with breaking changes:
+
+- **Ruby 3.2 or later is required.**
+- **Writing Zip files requires the `zip_kit` gem.** The retired `zip_tricks` gem has been replaced
+  by its successor, `zip_kit`. If your application writes Zip files, replace `gem "zip_tricks"` with
+  `gem "zip_kit"` in your Gemfile. Reading Zip files is unaffected.
+- **The deprecated pre-v1.6 API has been removed**, including the `IOStreams::Deprecated` mix-in.
+  Move any code still using it to the `IOStreams.path` and `IOStreams.stream` API.
+- **The deprecated PGP writer `compression:` option has been removed.** Use `compress:` instead,
+  available since v1.11.0.
+- **`IOStreams::Pgp.logger` and `IOStreams::Pgp.logger=` have been removed.** Logging is configured
+  for the whole library with `IOStreams.logger=`. See [logger](config#logger).
+
+## Upgrading from before v1.6
+
+The pre-v1.6 API was deprecated in v1.6 and removed in v2.0. Upgrade to v1.11 first, move all code to
+the `IOStreams.path` and `IOStreams.stream` API, and then upgrade to v2.