iostreams 1.11.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dfb827d5403c211fcdcbefbdb79e8d28f0a29f8d5982cc5de0eb832cf72d60bc
-  data.tar.gz: 710894a9919d7d3935867f67dd7a7bdfae041c29c3c5700dd80e0cccd7b7777c
+  metadata.gz: d74053d56262767ee38729e8fc3810416b340ec7e1a318c37a95579f392a92d6
+  data.tar.gz: 7e6e2fa0088656a52a8fd9c45c78b96ed8b2c324e9f9379a091fcfdd3939dde9
 SHA512:
-  metadata.gz: d246b2d041bbedf6de44340c6db298cfabf71ad14180c7a4743c83c3a60deca81b4b1f996ce6cf27fc8674e708f6276432929870ab7b4160669d96b4aeae1f53
-  data.tar.gz: 3abfbfba9845a6fe135612a5048f8f3f2cf961907898f1ae4759cb2fd5093460cff0093536764f9b5c7600c313dc1b2043c7795a6fde660be093daa38d9f1d86
+  metadata.gz: a492f9f89787eae3675fcf0aef1a1fbc9f67e461a0c1d0c9422888fa4cc66277c55d39248041b79a50de1b0d321a51a013445379de93404b0f85105d48914962
+  data.tar.gz: 2e6303a89a55bbbda4a9c73a19bdd6780823633afa4391d2fe3071ba8ca85e360a08214e76ed2e869182d8ed94388ed0454769b5d6dc41e540abf704d1c8a794

data/README.md

@@ -1,8 +1,9 @@
 # IOStreams
 [![Gem Version](https://img.shields.io/gem/v/iostreams.svg)](https://rubygems.org/gems/iostreams) [![Downloads](https://img.shields.io/gem/dt/iostreams.svg)](https://rubygems.org/gems/iostreams) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg)
 
-IOStreams is an incredibly powerful streaming library that makes changes to file formats, compression, encryption, 
-or storage mechanism transparent to the application.
+IOStreams is a streaming library for Ruby that makes compression, encryption, file format, and storage
+location transparent to your code. Read and write files of any size, one block at a time, whether they
+are gzip, zip, or PGP encrypted, and whether they live on local disk, AWS S3, SFTP, or are fetched over HTTP.
 
 ## Project Status
 
@@ -10,26 +11,26 @@ Production Ready, heavily used in production environments, many as part of Rocke
 
 ## Documentation
 
-Start with the [IOStreams tutorial](https://iostreams.rocketjob.io/tutorial) to get a great introduction to IOStreams.
+Start with the [IOStreams tutorial](https://iostreams.reidmorrison.com/tutorial) to get a great introduction to IOStreams.
 
-Next, checkout the remaining [IOStreams documentation](https://iostreams.rocketjob.io/)
+Next, checkout the remaining [IOStreams documentation](https://iostreams.reidmorrison.com/)
 
-## Upgrading to v1.6
+See the [CHANGELOG](CHANGELOG.md) for the release history and notable changes.
 
-The old, deprecated api's are no longer loaded by default with v1.6. To add back the deprecated api support, add
-the following line to your code:
+## Upgrading
 
-~~~ruby
-IOStreams.include(IOStreams::Deprecated)
-~~~
-
-It is important to move any of the old deprecated apis over to the new api, since they will be removed in a future
-release.
+See [Upgrading IOStreams](https://iostreams.reidmorrison.com/upgrading) for the changes that may need
+updates to your application, and the security settings to review, when upgrading.
 
 ## Versioning
 
 This project adheres to [Semantic Versioning](http://semver.org/).
 
+## Contributing
+
+Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on documentation
+updates, code changes, the project architecture, and the code of conduct.
+
 ## Author
 
 [Reid Morrison](https://github.com/reidmorrison)

data/Rakefile

@@ -1,10 +1,12 @@
 require "rake/testtask"
 require_relative "lib/io_streams/version"
 
+desc "Build the iostreams gem"
 task :gem do
   system "gem build iostreams.gemspec"
 end
 
+desc "Build and publish the iostreams gem, then tag and push the release"
 task publish: :gem do
   system "git tag -a v#{IOStreams::VERSION} -m 'Tagging #{IOStreams::VERSION}'"
   system "git push --tags"
@@ -12,6 +14,56 @@ task publish: :gem do
   system "rm iostreams-#{IOStreams::VERSION}.gem"
 end
 
+desc "Start an IRB console with the gem loaded"
+task :console do
+  exec "irb -I lib -r iostreams"
+end
+
+desc "Generate docs/llms-full.txt from docs/llms.txt and the doc pages it links to"
+task :llms_full do
+  require "uri"
+
+  docs_dir  = File.join(__dir__, "docs")
+  llms_path = File.join(docs_dir, "llms.txt")
+  out_path  = File.join(docs_dir, "llms-full.txt")
+
+  llms_txt = File.read(llms_path)
+  header   = llms_txt[/\A.*?(?=\n## Docs)/m]&.strip
+  raise "Could not find intro text before '## Docs' in #{llms_path}" unless header
+
+  docs_section = llms_txt[/^## Docs\n(.*?)(?=\n## |\z)/m, 1]
+  raise "Could not find '## Docs' section in #{llms_path}" unless docs_section
+
+  sections = docs_section.each_line.filter_map do |line|
+    next unless line =~ /^- \[(?<title>[^\]]+)\]\((?<url>[^)]+)\)/
+
+    title = $~[:title]
+    path  = URI.parse($~[:url]).path.sub(%r{\A/}, "")
+    path  = "index" if path.empty?
+    file  = File.join(docs_dir, "#{path}.md")
+    raise "Missing doc file for #{$~[:url]}: #{file}" unless File.exist?(file)
+
+    raw = File.read(file)
+
+    # The page heading lives in front matter, which the shared docs theme
+    # renders as the page's h1. Front matter is stripped below, so lift the
+    # heading back out and re-emit it. `heading` wins over `title` where a page
+    # sets both, which is the same precedence the theme uses. Falls back to the
+    # link text in llms.txt for a page that sets neither.
+    front_matter = raw[/\A---\n(.*?)\n---\n/m, 1].to_s
+    heading      = %w[heading title].
+                   filter_map { |key| front_matter[/^#{key}:[ \t]*(.+)$/, 1] }.
+                   first.to_s.strip.delete_prefix('"').delete_suffix('"')
+    heading      = title if heading.empty?
+
+    body = raw.sub(/\A---\n.*?\n---\n/m, "").strip
+    "# #{heading}\n\n#{body}"
+  end
+
+  File.write(out_path, "#{header}\n\n#{sections.join("\n\n---\n\n")}\n")
+  puts "Wrote #{out_path}"
+end
+
 Rake::TestTask.new(:test) do |t|
   t.pattern = "test/**/*_test.rb"
   t.verbose = true

data/docs/CLAUDE.md

@@ -0,0 +1,9 @@
+# Documentation site
+
+User-facing documentation is a Jekyll site under [docs/](docs/), published to iostreams.reidmorrison.com. The markdown pages are what matter when reading or updating documentation.
+
+**The look and feel is not in this repo.** `docs/_config.yml` sets `remote_theme: reidmorrison/rm-docs-theme@v1`, and the layout, stylesheet, sidebar and syntax highlighting all come from there. This repo holds only its content: the markdown pages and `docs/images`. **Do not add a `docs/_layouts`, `docs/stylesheets` or `docs/javascripts` directory**; they were deleted deliberately, because six gem repos each carried a near-identical copy of the same theme and the copies had drifted. A styling change belongs in `rm-docs-theme`, where it reaches every doc site at once. `v1` is a moving major tag, so theme fixes arrive on the next build; breaking changes go to `v2` and are opted into by editing the pin. `jekyll-remote-theme` must stay in `plugins`: GitHub Pages enables it on its own, but a local build does not, and without it every page silently renders with no layout. Preview against a local theme checkout with `~/src/rm-docs-theme/bin/preview ~/src/iostreams/docs`.
+
+**A page's title lives in its front matter**, not in a `#` heading at the top of the markdown; the theme renders it as the page's `h1`. `title` is the browser title and the default heading, `heading` overrides the h1 where the two should differ, and `description` is the page's meta description. `index.md` sets `heading` only, so the home page keeps the tuned SEO `<title>` from `_config.yml`. Adding or renaming a page means editing the `nav` block in `docs/_config.yml` and `docs/llms.txt`.
+
+The site also serves two files for AI assistants: [docs/llms.txt](docs/llms.txt), a hand-maintained index of the docs pages, and `docs/llms-full.txt`, all pages concatenated, regenerated with `bundle exec rake llms_full`. **After editing any `docs/*.md` page, re-run `bundle exec rake llms_full`** and commit the result; never edit `llms-full.txt` by hand. That task reads the page heading out of the front matter, so a page that sets neither `heading` nor `title` falls back to its link text in `llms.txt`.

data/docs/config.md

@@ -0,0 +1,157 @@
+---
+layout: default
+title: Configuring IOStreams
+description: >-
+  Named roots via IOStreams.add_root so the same code targets different storage
+  per environment, plus the temp directory and logger settings.
+---
+
+## add_root
+
+Roots allow paths to reference a particular root directory, so that all path names are appended to that root.
+Their primary purpose is to allow the exact same code to run in production and development, yet use completely
+different data sources in each. For example, in production a root can point to an S3 bucket, while in
+development it points to the local file system.
+
+Roots are configured via an initializer at startup. `IOStreams.join` then joins the supplied path
+elements onto the named root, using the `:default` root whenever a root is not supplied.
+
+Set the default root for this environment in an initializer:
+~~~ruby
+IOStreams.add_root(:default, "/var/my_app/files")
+~~~
+
+Now the default root path is available:
+~~~ruby
+IOStreams.root
+# => #<IOStreams::Paths::File:/var/my_app/files pipeline={}>
+ 
+IOStreams.root.to_s
+# => "/var/my_app/files"
+~~~
+
+Comparing the final path using `path` and then `join` that uses a root path:
+~~~ruby
+IOStreams.path("/var/my_app/files", "my_test_file.txt").to_s
+# => "/var/my_app/files/my_test_file.txt"
+
+IOStreams.join("my_test_file.txt").to_s
+# => "/var/my_app/files/my_test_file.txt"
+~~~
+
+
+Using `path`:
+~~~ruby
+IOStreams.path("/var/my_app/files", "my_test_file.txt").write("Hello World")
+~~~
+
+With the default root path configured the above code can be simplified by using `join` since it resolves to the same path.
+~~~ruby
+IOStreams.join("my_test_file.txt").write("Hello World")
+~~~
+
+Multiple roots can be setup, for example one for input files, another for output files, another for
+reports, etc. During development the roots can all point to a common location, while in production
+they could be completely different S3 buckets.
+
+For example add special paths for `downloads` and `uploads`.
+~~~ruby
+IOStreams.add_root(:downloads, "/var/my_app/downloads")
+IOStreams.add_root(:uploads, "/var/my_app/uploads")
+~~~
+
+An example that writes a file into the `/var/my_app/downloads` directory:
+~~~ruby
+IOStreams.join("my_test_file.txt", root: :downloads).write("Hello World")
+~~~
+
+The other benefit is that the root paths used in an application are externalized from the code base. That way the
+roots can be changed to different locations depending on the environment.
+
+We can also change the storage mechanism by changing the root:
+~~~ruby
+IOStreams.add_root(:downloads, "s3://my-app-bucket-name/downloads")
+IOStreams.add_root(:uploads, "s3://my-app-bucket-name/uploads")
+~~~
+
+Now the application will write to S3 and the code does not change at all.
+~~~ruby
+IOStreams.join("my_test_file.txt", root: :downloads).write("Hello World")
+~~~
+
+To use or query a configured root path:
+~~~ruby
+IOStreams.root(:downloads).to_s
+# => "s3://my-app-bucket-name/downloads"
+~~~
+
+## temp_dir
+
+When working with large files the standard temp file system location can be too small to handle downloading large
+files. For example to decrypt a pgp file from S3, because GnuPG is not streaming capable and only operates on local files.
+
+By default IOStreams looks up the location to store temp files in the following order:
+* `ENV['TMPDIR']`
+* `ENV['TMP']`
+* `ENV['TEMP']`
+* `Etc.systmpdir`
+* `/tmp` (if it exists)
+* Otherwise `.`
+
+To explicity set the temp file location the following config option can be used:
+
+~~~ruby
+IOStreams.temp_dir = "/var/really_big_temp"
+~~~
+
+### temp_file
+
+To work with a temporary file directly, `IOStreams.temp_file` yields a path inside `temp_dir`
+and deletes the file when the block completes:
+
+~~~ruby
+IOStreams.temp_file("export", ".csv") do |path|
+  path.write("Hello World")
+  # ... use the temp file ...
+end
+# The temp file has been deleted.
+~~~
+
+The first argument is a base file name to include in the generated temp file name, and the
+optional second argument is the file extension.
+
+## logger
+
+IOStreams can log debug information, such as the external commands it runs for PGP and SFTP.
+
+When [Semantic Logger](https://logger.reidmorrison.com) is loaded it is detected automatically, and IOStreams
+logs to it without any additional configuration.
+
+To use a different logger, or to log when Semantic Logger is not present, assign any logger that
+responds to the standard logging methods:
+
+~~~ruby
+require "logger"
+IOStreams.logger = Logger.new($stdout)
+~~~
+
+To disable logging entirely, set the logger to `nil`:
+
+~~~ruby
+IOStreams.logger = nil
+~~~
+
+IOStreams also logs warnings for behavior that will change in the next major version. See
+[Coming in v3.0](upgrading#coming-in-v30).
+
+## enforce_column_restrictions
+
+Applies `allowed_columns`, `required_columns` and `skip_unknown` to every input when reading
+records, including JSON records, so that renaming an uploaded file from `.csv` to `.json` cannot
+bypass them. Set it in an initializer:
+
+~~~ruby
+IOStreams.enforce_column_restrictions = true
+~~~
+
+Default: false. It will default to true in v3.0. See [Header options](formats#header-options).

data/docs/copy_files.md

@@ -0,0 +1,75 @@
+---
+layout: default
+title: Copying Between Files
+description: >-
+  Using IOStreams.copy to move a file between storage locations and to compress,
+  decompress, encrypt or decrypt it on the way.
+---
+
+File copying can be used to:
+* copy from one storage location to another.
+* create a decrypted / encrypted copy of an existing file.
+* create a decompressed / compressed copy of an existing file.
+
+## Examples
+
+Decompress `example.csv.gz` into `example.csv`:
+
+~~~ruby
+source = IOStreams.path("example.csv.gz")
+IOStreams.path("example.csv").copy_from(source)
+~~~
+
+Decrypt a file encrypted with Symmetric Encryption:
+
+~~~ruby
+source = IOStreams.path("example.csv.enc")
+IOStreams.path("example.csv").copy_from(source)
+~~~
+
+Encrypt a file using PGP encryption so that it can only be read by `receiver@example.org`.
+
+~~~ruby
+source = IOStreams.path("example.csv")
+target = IOStreams.path("example.csv.pgp")
+target.option(:pgp, recipient: "receiver@example.org")
+target.copy_from(source)
+~~~
+
+When the file name does not have file extensions that would allow IOStreams to infer what streams to apply,
+the streams can be explicitly set using `stream`.
+
+In this example, the file `CUSTOMER_DATA` has no extensions, so `stream(:enc)` tells IOStreams
+that its contents were encrypted with Symmetric Encryption. The decrypted contents are then
+PGP encrypted and written to `xyz.csv.pgp` using the pgp key for `receiver@example.org`.
+
+~~~ruby
+input = IOStreams.path("CUSTOMER_DATA").stream(:enc)
+IOStreams.path("xyz.csv.pgp").option(:pgp, recipient: "receiver@example.org").copy_from(input)
+~~~
+
+To copy a file _without_ performing any conversions (ignore file extensions), set `convert` to `false`:
+
+~~~ruby
+input = IOStreams.path("sample.json.zip")
+IOStreams.path("sample.copy").copy_from(input, convert: false)
+~~~
+
+Custom stream conversions can be applied to both the source and the target in a single copy.
+Here the source is read as binary and the target is PGP encrypted:
+
+~~~ruby
+source = IOStreams.path("source_file").stream(:encode, encoding: "BINARY")
+IOStreams.path("target_file.pgp").option(:pgp, passphrase: "hello").copy_from(source)
+~~~
+
+To convert the contents row by row, or record by record, during the copy, supply `mode`.
+For example, copy a CSV file into JSON, parsing and rendering each record:
+
+~~~ruby
+source = IOStreams.path("source_file.csv")
+IOStreams.path("target_file.json").copy_from(source, mode: :hash)
+~~~
+
+Notes:
+* `mode` accepts `:line`, `:array`, or `:hash`, and only applies when `convert` is `true`.

data/docs/extensions.md

@@ -0,0 +1,111 @@
+---
+layout: default
+title: File Extensions
+description: >-
+  How the extensions in a file name, such as .csv.gz.pgp, decide which streams
+  IOStreams applies when reading or writing, and how to register your own.
+---
+
+IOStreams uses the extensions in the file name to determine which streams to apply when
+reading or writing a file. Multiple extensions are applied in order, so `sample.csv.gz.pgp`
+is first decrypted with PGP and then decompressed with GZip when read.
+
+Supported extensions:
+
+| Extension        | Stream               | Read | Write | Required gem / program            |
+|:-----------------|:---------------------|:-----|:------|:----------------------------------|
+| `.bz2`           | BZip2                | Yes  | Yes   | `bzip2-ffi`                        |
+| `.enc`           | Symmetric Encryption | Yes  | Yes   | `symmetric-encryption`             |
+| `.gz`, `.gzip`   | GZip                 | Yes  | Yes   | None (Ruby standard library)       |
+| `.zip`           | Zip                  | Yes  | Yes   | `rubyzip` (read), `zip_kit` (write). On JRuby the built-in Java zip support is used for reading. |
+| `.pgp`, `.gpg`   | PGP                  | Yes  | Yes   | GnuPG command line program (`gpg`) |
+| `.xlsx`, `.xlsm` | Excel Spreadsheet    | Yes  | No    | `creek`                            |
+
+The gems above are soft dependencies: IOStreams does not require them for installation,
+they only need to be added to the `Gemfile` when the corresponding extension is used.
+
+## Compression options
+
+GZip accepts a compression `level` when writing, from `0` (no compression) to `9` (best compression):
+
+~~~ruby
+IOStreams.path("sample.csv.gz").option(:gz, level: 9).write(data)
+~~~
+
+BZip2 passes its options through to `bzip2-ffi`: `block_size` (`1` to `9`) and `work_factor` (`0` to `250`)
+when writing, and `small` and `first_only` when reading:
+
+~~~ruby
+IOStreams.path("sample.csv.bz2").option(:bz2, block_size: 9).write(data)
+IOStreams.path("sample.csv.bz2").option(:bz2, small: true).read
+~~~
+
+Options are strict, so an option that a stream does not accept raises an `ArgumentError`.
+See [Streams](streams#reading-and-writing-need-separate-options). The exception is BZip2, which
+ignores any other option and logs a warning. In v3.0 it will raise an `ArgumentError` too.
+
+## Reading an Excel Spreadsheet
+
+Each row in the spreadsheet is converted into a CSV line, so the regular `:line`, `:array`,
+and `:hash` modes apply:
+
+~~~ruby
+IOStreams.path("spreadsheet.xlsx").each(:hash) do |record|
+  p record
+end
+~~~
+
+Notes:
+* Since the underlying `creek` gem operates on files, when reading from a stream (for example S3 or HTTP)
+  the contents are first downloaded into a temp file.
+* Writing xlsx files is not supported.
+
+## Character encoding
+
+The special `:encode` stream converts the character encoding of the data being read or written.
+It is applied with `option` or `stream` rather than a file name extension:
+
+~~~ruby
+IOStreams.path("sample.csv.gz").
+  option(:encode, encoding: "UTF-8", cleaner: :printable, replace: "").
+  each do |line|
+    puts line
+  end
+~~~
+
+Options:
+
+* `encoding: [String|Encoding]`
+  The target encoding, for example `"UTF-8"`, `"US-ASCII"`, or `"ASCII-8BIT"`.
+  Default: `"UTF-8"`
+
+* `replace: [String]`
+  The character to replace with when a character cannot be converted to the target encoding.
+  Default: nil (raise `Encoding::UndefinedConversionError` on invalid characters)
+
+* `cleaner: [nil|Symbol|Proc]`
+  Cleanse the data. Built-in rules:
+  * `:printable` removes all non-printable characters except `\r` and `\n`.
+  * `:replace_non_printable` replaces all non-printable characters except `\r` and `\n`
+    with the `replace` value, or an empty string when `replace` is nil.
+  A Proc can also be supplied to perform custom cleansing; it is called with the data
+  and the `replace` value after every read or write.
+  Default: nil
+
+## Registering a custom extension
+
+To add a new extension, supply its reader and writer classes. Both must implement `.open`
+that yields a stream implementing `#read` or `#write` respectively. See any of the streams
+under `lib/io_streams` for examples.
+
+~~~ruby
+IOStreams.register_extension(:xls, MyXls::Reader, MyXls::Writer)
+~~~
+
+Similarly, to support a new storage location, supply a Path class for its URI scheme.
+See [IOStreams::Paths::S3](https://github.com/reidmorrison/iostreams/blob/main/lib/io_streams/paths/s3.rb)
+for an example of what is required.
+
+~~~ruby
+IOStreams.register_scheme(:gcs, MyGoogleCloudStoragePath)
+~~~

data/docs/formats.md

@@ -0,0 +1,188 @@
+---
+layout: default
+title: File Formats
+description: >-
+  Converting rows and records to and from CSV, PSV, JSON and fixed width files,
+  including format inference, format options and header handling.
+---
+
+When reading or writing rows (`:array`) or records (`:hash`), IOStreams converts each line
+to or from the file's tabular format. The following formats are supported:
+
+* `:csv`   Comma Separated Values
+* `:psv`   Pipe Separated Values
+* `:json`  One JSON document per line
+* `:fixed` Fixed width columns
+* `:array` Each line is already an array of values
+* `:hash`  Each line is already a hash
+
+PSV has no way to escape values, so when writing PSV, a `|` within a value is replaced with `:`
+and a line break with a space, so that a value cannot add columns or records.
+
+## Format inference
+
+The format is inferred from the file name when it contains a recognized extension:
+
+~~~ruby
+IOStreams.path("sample.csv").each(:hash) { |record| p record }
+IOStreams.path("sample.json").each(:hash) { |record| p record }
+IOStreams.path("sample.psv").each(:hash) { |record| p record }
+~~~
+
+The format extension can appear anywhere in the file name, so `sample.csv.gz` and
+`sample.json.pgp` are recognized as CSV and JSON respectively.
+
+When the file name does not contain a recognized format extension, the format defaults to `:csv`.
+
+## Specifying the format
+
+When the file name cannot be used to infer the format, set it explicitly with `format`:
+
+~~~ruby
+path = IOStreams.path("sample_data")
+path.format(:json)
+path.each(:hash) { |record| p record }
+~~~
+
+`format` can be chained with the other path methods:
+
+~~~ruby
+IOStreams.path("sample_data").format(:json).each(:hash) { |record| p record }
+~~~
+
+## Format options
+
+Format specific options are supplied with `format_options`. They are passed to the parser
+for the chosen format. The `:fixed` format requires its file layout to be supplied this way,
+as shown in the next section. The other formats do not currently take any options.
+
+## Fixed width files
+
+Fixed width files have no delimiters; each column is identified by its position within the line.
+Since the layout cannot be inferred from the file, supply it using `format_options`:
+
+~~~ruby
+path = IOStreams.path("sample_data")
+path.format(:fixed)
+path.format_options(
+  layout: [
+    {size: 23, key: "name"},
+    {size: 40, key: "address"},
+    {size: 5,  key: "zip"}
+  ]
+)
+path.each(:hash) { |record| p record }
+~~~
+
+Writing a fixed width file uses the same layout to render each record:
+
+~~~ruby
+path = IOStreams.path("sample_data")
+path.format(:fixed)
+path.format_options(
+  layout: [
+    {size: 23, key: "name"},
+    {size: 40, key: "address"},
+    {size: 5,  key: "zip"}
+  ]
+)
+path.writer(:hash) do |io|
+  io << {"name" => "Jack Jones", "address" => "Somewhere", "zip" => 12345}
+end
+~~~
+
+Note: The keys in the hashes being written must match the layout `:key` values exactly,
+including whether they are strings or symbols.
+
+Layout column definitions:
+
+* `:size` The number of characters this column occupies.
+  The last column may use a size of `:remainder` to take the rest of the line as its value.
+* `:key` The name for this column. Leave out the key to ignore the column during parsing,
+  and to space fill when rendering.
+* `:type` `:string` (default), `:integer`, or `:float`.
+  Strings are left justified and space padded, numbers are right justified and zero padded.
+  When writing, line breaks within a string are replaced with a space so that a value cannot
+  add records.
+  Raises `IOStreams::Errors::ValueTooLong` when an `:integer` or `:float` value cannot be
+  rendered in `size` characters.
+* `:decimals` For `:float` columns, the number of decimal places to render.
+  Default: 2
+
+In addition to `layout`, the `:fixed` format takes one more option:
+
+* `truncate: [true|false]`
+  Whether to truncate string values that are longer than their column `:size` when writing.
+  When false, a string value that is too long raises `IOStreams::Errors::ValueTooLong`
+  instead of being truncated. Numeric values are never truncated.
+  Default: true
+
+## Header options
+
+When reading or writing records (`:hash`), the following options control the header row:
+
+* `columns: [Array<String>]`
+  When reading, supplies the header columns for files that do not include a header row.
+  When writing, sets the columns to write, including their order. Keys not listed in
+  `columns` are ignored during writes.
+
+* `cleanse_header: [true|false]`
+  Whether to cleanse the column names read from the header row.
+  Column names are stripped of leading and trailing whitespace, lowercased, and spaces
+  and dashes are converted to underscores, so the header `" First Name "` becomes `"first_name"`.
+  Default: true
+
+* `allowed_columns: [Array<String>]`
+  List of columns to allow. Any other columns are ignored when `skip_unknown` is true,
+  otherwise an `IOStreams::Errors::InvalidHeader` exception is raised.
+  Default: nil (allow all columns)
+
+* `required_columns: [Array<String>]`
+  List of columns that must be present, otherwise an exception is raised.
+
+* `skip_unknown: [true|false]`
+  When true, any columns not present in `allowed_columns` are skipped entirely as if they
+  were not in the file at all. When false, an unknown column raises
+  `IOStreams::Errors::InvalidHeader`.
+  Default: true
+
+By default, when reading records, `allowed_columns`, `required_columns` and `skip_unknown` only
+apply to a header row read from the file, and only when `cleanse_header` is true. They are ignored
+for JSON and `:hash` input, when `columns` are supplied, and with `cleanse_header: false`, and a
+warning is logged when applying them would change the records read.
+
+Since the format is usually inferred from the file name, renaming an uploaded file from `.csv` to
+`.json` bypasses them. When they restrict which columns an upload can set, apply them to every input
+in an initializer:
+
+~~~ruby
+IOStreams.enforce_column_restrictions = true
+~~~
+
+They then also apply to the supplied `columns`, to a header row read with `cleanse_header: false`,
+and, for formats without a header row such as JSON, to the keys of each record. When either
+`allowed_columns` or `required_columns` is set, JSON keys are cleansed the same way as a header row,
+unless `cleanse_header: false` is supplied. This will be the default in v3.0.
+
+Example, reading a headerless CSV file:
+
+~~~ruby
+path = IOStreams.path("no_header.csv")
+path.each(:hash, columns: ["name", "address", "zip"]) do |record|
+  p record
+end
+~~~
+
+Example, writing only specific columns in a fixed order:
+
+~~~ruby
+path = IOStreams.path("sample.csv")
+path.writer(:hash, columns: ["name", "zip"]) do |io|
+  io << {"name" => "Jack Jones", "address" => "Somewhere", "zip" => 12345}
+end
+path.read
+# => "name,zip\nJack Jones,12345\n"
+~~~
+
+Note: Column names are converted to strings, and the keys in the hashes being written may
+be strings or symbols.