iostreams 1.11.0 → 2.1.0

data/docs/streams.md

@@ -0,0 +1,337 @@
+---
+layout: default
+title: Streams
+description: >-
+  Reading and writing a path a block, line, row or record at a time, and the
+  pipeline of compression, encryption and format streams applied to it.
+---
+
+Once you have a [path](path), you read from and write to it with a small, consistent set of methods.
+Reading and writing always happen a block, line, or record at a time, so memory use stays low no
+matter how large the file is. Choose how each chunk is delivered by passing a mode: the default
+streams raw data, `:line` yields one line at a time, `:array` yields each row as an array, and
+`:hash` yields each record as a hash keyed by the header row.
+
+Read 128 characters at a time from the file:
+~~~ruby
+IOStreams.path("example.csv").reader do |io|
+  while (data = io.read(128))
+    p data 
+  end
+end
+~~~
+
+Read one line at a time from the file:
+~~~ruby
+IOStreams.path("example.csv").each do |line|
+  puts line
+end
+~~~
+
+By default the line delimiter is auto-detected from the file, handling both Windows (`\r\n`)
+and Linux (`\n`) line endings. To break the file up by something other than its line endings,
+supply `delimiter`:
+~~~ruby
+IOStreams.path("example.txt").each(:line, delimiter: "|") do |line|
+  puts line
+end
+~~~
+
+When a line can contain embedded newlines, such as a CSV field wrapped in double quotes that
+spans multiple lines, supply `embedded_within` so those newlines are not treated as line endings:
+~~~ruby
+IOStreams.path("example.csv").each(:line, embedded_within: '"') do |line|
+  puts line
+end
+~~~
+
+Notes:
+* Newlines embedded within quoted fields are kept on the same line automatically when the resolved
+  tabular format quotes its fields, such as CSV (detected from a `.csv` file name or set explicitly via
+  `.format(:csv)`). `embedded_within` only needs to be supplied for quoted formats that are not
+  auto-detected, or to override the quote character.
+* A file that is named `.csv` but is actually pipe-delimited can avoid quote parsing by declaring its
+  real format with `.format(:psv)`, or by passing `embedded_within: nil` to disable it explicitly:
+~~~ruby
+IOStreams.path("pipe_delimited.csv").format(:psv).each(:line) do |line|
+  puts line
+end
+~~~
+
+Display each row from the csv file as an array:
+~~~ruby
+IOStreams.path("example.csv").each(:array) do |array|
+  p array
+end
+~~~
+
+Display each row from the csv file as a hash, where the first line in the CSV file is the header:
+~~~ruby
+IOStreams.path("example.csv").each(:hash) do |hash|
+  p hash
+end
+~~~
+
+Write data to the file.
+~~~ruby
+IOStreams.path("abc.txt").writer do |io|
+  io << "This"
+  io << " is "
+  io << " one line\n"
+end
+~~~
+
+Write lines to the file. By adding `:line` to `writer`, each write appends a new line character. 
+~~~ruby
+IOStreams.path("example.csv").writer(:line) do |file|
+  file << "these"
+  file << "are"
+  file << "all"
+  file << "separate"
+  file << "lines"
+end
+~~~
+
+Write an array (row) at a time to the file.
+Each array is converted to csv before being compressed with zip.
+
+~~~ruby
+IOStreams.path("example.csv").writer(:array) do |io|
+  io << ["name", "address", "zip_code"]
+  io << ["Jack", "There", "1234"]
+  io << ["Joe", "Over There somewhere", 1234]
+end
+~~~
+
+Write a hash (record) at a time to the file.
+Each hash is converted to csv before being compressed with zip.
+The header row is extracted from the first hash write that is performed. 
+
+~~~ruby
+IOStreams.path("example.csv").writer(:hash) do |stream|
+  stream << {name: "Jack", address: "There", zip_code: 1234}
+  stream << {zip_code: 1234, address: "Over There somewhere", name: "Joe"}
+end
+~~~
+
+Notes
+* Any additional keys supplied during subsequent write operations will be ignored
+  since the header row has already been written to the file. 
+* The order of the header and values is determined by the order of the keys supplied 
+  during the first write.
+* The order of keys in the subsequent writes does not matter. 
+
+Stream into an in-memory buffer, useful for testing. 
+The original filename still needs to be supplied so that the streaming pipeline can still be inferred.
+
+~~~ruby
+io = StringIO.new
+IOStreams.stream(io).file_name("example.csv.gz").writer(:hash) do |stream|
+  stream << {name: "Jack", address: "There", zip_code: 1234}
+  stream << {name: "Joe", zip_code: 1234, address: "Over There somewhere"}
+end
+puts io.string
+~~~
+
+Read a CSV file and write the output to an encrypted file in JSON format.
+
+~~~ruby
+IOStreams.path("sample.json.enc").writer(:hash) do |output|
+  IOStreams.path("sample.csv").each(:hash) do |record|
+    output << record
+  end
+end
+~~~
+
+Read a zip file hosted on a HTTP Web Server, returning each row as a hash:
+~~~ruby
+IOStreams.
+  path("https://www5.fdic.gov/idasp/Offices2.zip").
+  option(:zip, entry_file_name: "OFFICES2_ALL.CSV").
+  each(:hash) do |row|
+    p row
+  end
+~~~
+
+Notes:
+* By default IOStreams will read the first file in the zip file. 
+* To choose a specific file name within the zip file, supply: `entry_file_name` 
+
+## Notes
+
+* Reading a Zip file requires the entire file to be available locally, so reading from a
+  stream (for example S3 or HTTP) downloads it into a temp file first.
+  Writing Zip is fully streamed, no temp file is required.
+* When writing, `entry_file_name` sets the name of the file entry within the zip file.
+  It defaults to the file name without the `.zip` extension, so writing to
+  `example.csv.zip` creates an entry named `example.csv`.
+* Gzip is still recommended over Zip for very large files, since Zip files can only
+  be read via a local file.
+
+## Pipeline
+
+If the file is compressed, the pipeline will infer the necessary streams that need to be applied to it:
+
+~~~ruby
+path = IOStreams.path("somewhere/example.csv.gz")
+# => #<IOStreams::Paths::File:somewhere/example.csv.gz pipeline={:gz=>{}}>
+ 
+path.pipeline
+# => {:gz=>{}} 
+~~~
+
+The `pipeline` above includes `:gz` to indicate that the file should compressed / decompressed with GZip.
+
+#### Option
+
+Each path supports several options which can be supplied using the `option` method. 
+
+Set the options for a stream in the pipeline for this file. Each stream can only be applied once and is uniquely
+identified by its symbolic name.
+
+To see the pipeline of streams that IOStreams would infer: 
+~~~ruby
+IOStreams.path("example.pgp").pipeline
+# => {:pgp=>{}}
+ 
+IOStreams.path("example.gz").pipeline
+# => {:gz=>{}}
+ 
+IOStreams.path("example.gz.pgp").pipeline
+# => {:gz=>{}, :pgp=>{}}
+~~~
+
+If the relevant stream is not found for this file it is ignored.
+For example, if the file does not have a pgp extension then the pgp option is ignored.
+~~~ruby
+IOStreams.path("example.csv.gz").
+  option(:pgp, passphrase: "receiver_passphrase").
+  read
+~~~
+
+This is great way to pass in stream specific options for when they are required, and to still support
+paths that do not use that stream. For example, the same code can support pgp encrypted, Symmetric Encryption encrypted,
+and plain text files.
+~~~ruby
+IOStreams.path("example.csv.enc").
+  option(:pgp, passphrase: "receiver_passphrase").
+  read
+~~~
+
+To see the what value was previously set for a particular option:
+~~~ruby 
+path = IOStreams.path("example.pgp")
+path.option(:pgp, passphrase: "receiver_passphrase")
+
+path.setting(:pgp)
+# => {:passphrase=>"receiver_passphrase"}
+~~~
+
+#### Reading and writing need separate options
+
+The options for a stream are passed to its reader when reading and to its writer when writing,
+and the two directions usually accept different options. For example, the PGP writer needs
+the `recipient` to encrypt for, while the PGP reader needs the `passphrase` for the private key.
+
+Options are strict: an option that does not apply to the direction being used raises an
+`ArgumentError` that names the direction it belongs to, rather than being silently ignored.
+So configure one path for writing and a separate path for reading:
+
+~~~ruby
+IOStreams.path("example.csv.pgp").
+  option(:pgp, recipient: "receiver@example.org").
+  write("name,login\nJack Jones,jjones\n")
+
+IOStreams.path("example.csv.pgp").
+  option(:pgp, passphrase: "receiver_passphrase").
+  read
+~~~
+
+Reading with the writer's options fails before any data is read:
+~~~ruby
+IOStreams.path("example.csv.pgp").
+  option(:pgp, recipient: "receiver@example.org").
+  read
+# ArgumentError: :recipient only applies when writing a :pgp stream and cannot be used when reading.
+#   Configure a separate path or stream without it for reading.
+~~~
+
+An option that neither direction accepts, such as a misspelled one, raises an `ArgumentError`
+that lists the valid options. The exception is BZip2, which ignores options it does not accept and
+logs a warning instead. In v3.0 it will raise an `ArgumentError` too.
+
+Options for a stream that is not in the pipeline are still ignored, as described above,
+since they are not passed to any reader or writer.
+
+#### Stream
+
+The `stream` method stops IOStreams from inferring the streams for this path and only uses the specified streams.
+
+For example when using a filename that does not have the necessary file extensions. 
+In this case the file was compressed with Zip, so tell IOStreams to unzip it:  
+~~~ruby 
+path = IOStreams.path("tempfile2527")
+path.stream(:zip)
+path.read
+~~~
+
+The above example could also be written as:
+~~~ruby 
+IOStreams.path("tempfile2527").
+  stream(:zip).
+  read
+~~~
+
+Multiple streams can also be specified:
+
+~~~ruby 
+IOStreams.path("tempfile2527").
+  stream(:zip).
+  stream(:pgp, passphrase: "receiver_passphrase").
+  read
+~~~
+
+Now that IOStreams is not inferring the pipeline from the filename, we can still see the above streams: 
+~~~ruby 
+IOStreams.path("tempfile2527").
+  stream(:zip).
+  stream(:pgp, passphrase: "receiver_passphrase").
+  pipeline
+# => {:zip=>{}, :pgp=>{:passphrase=>"receiver_passphrase"}}
+~~~
+
+
+In this example the file contains JSON data that was compressed with Zip, and since we want to read each row as a hash:
+~~~ruby 
+IOStreams.path("tempfile2527").
+  stream(:zip).
+  each(:hash, format: :json) do |row|
+    p row
+  end
+~~~
+
+Alternatively if the original file name is available it can also be supplied allowing IOStreams to infer the above streams:
+~~~ruby 
+IOStreams.path("tempfile2527").
+  file_name("file.json.zip").
+  each(:hash) do |row|
+    p row
+  end
+~~~
+
+To see the what value was previously set for a particular stream:
+~~~ruby 
+path = IOStreams.path("tempfile2527")
+path.stream(:zip)
+path.stream(:pgp, passphrase: "receiver_passphrase")
+
+path.setting(:pgp)
+# => {:passphrase=>"receiver_passphrase"}
+~~~
+
+To ensure no streams are inferred or applied use stream `:none` 
+~~~ruby 
+path = IOStreams.path("file.zip")
+path.stream(:none)
+path.read
+~~~