flat_kit 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: fa0b52b3ab12e9cf86492571258daf5d98dde967c25d5aad65b74a6003900fa2
+  data.tar.gz: fcd1abff76e54854d86b131aa6773d411f4d54acbf2e48b541c69c72dd2c1149
+SHA512:
+  metadata.gz: 072e966f06fe03de16b686f469f0e3ac84aa2cc93910cc61210875e6d9cbf0bb3cc24bb1cf4f0863b90f2d02110abcdb0f88bbfa72777f2c19b2cce30c847e45
+  data.tar.gz: f5f67f8818e5cce8da7b547a3ec652d6dc288f9b66a957a62519878d819c893285ea3e81e7a32286de0d614412416f7ba38d8c28e7807f12add9ec00b3241e92

data/CONTRIBUTING.md

@@ -0,0 +1,46 @@
+# Hi there!
+
+I see you are interested in contributing. That is wonderful. I love
+contributions.
+
+I guarantee that there are bugs in this software. And I guarantee that there is
+a feature you want that is not in here yet. As such, any and all bugs reports
+are gratefully accepted, bugfixes even more so. Helping out with bugs is the
+easiest way to contribute.
+
+
+## The Quick Version
+
+* Have a [GitHub Account][].
+* Search the [GitHub Issues][] and see if your issue already present. If so
+  add your comments, :thumbsup:, etc.
+* Issue not there? Not a problem, open up a [new issue][].
+    * **Bug reports** please be as detailed as possible. Include:
+        * full ruby engine and version: `ruby -e 'puts RUBY_DESCRIPTION'`
+        * operating system and version
+        * version of flat_kit `ruby -e "require 'flat_kit'; puts FlatKit::VERSION"`
+        * as much detail about the bug as possible so I can replicate it. Feel free
+          to link in a [gist][]
+    * **New Feature**
+        * What the new feature should do.
+        * What benefit the new feature brings to the project.
+* Fork the [repo][].
+* Create a new branch for your issue: `git checkout -b issue/my-issue`
+* Lovingly craft your contribution:
+    * `rake develop` to bootstrap development. (skip if you already have bundler installed)
+    * `bundle install` to install dependencies.
+    * `rake test` to run tests
+* Make sure that `rake test` passes. It's important, I said it twice.
+* Add yourself to the contributors section below.
+* Submit your [pull request][].
+
+# Contributors
+
+* [Jeremy Hinegardner](https://github.com/copiousfreetime)
+
+[GitHub Account]: https://github.com/signup/free "GitHub Signup"
+[GitHub Issues]:  https://github.com/copiousfreetime/flat_kit/issues "FlatKit Issues"
+[new issue]:      https://github.com/copiousfreetime/flat_kit/issues/new "New FlatKit Issue"
+[gist]:           https://gist.github.com/ "New Gist"
+[repo]:           https://github.com/copiousfreetime/flat_kit "FlatKit Repo"
+[pull request]:   https://help.github.com/articles/using-pull-requests "Using Pull Requests"

data/HISTORY.md

@@ -0,0 +1,5 @@
+# FlatKit Changelog
+## Version 0.1.0
+
+* Initial Release - Yeah!
+

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 Jeremy Hinegardner
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/Manifest.txt

@@ -0,0 +1,66 @@
+CONTRIBUTING.md
+HISTORY.md
+LICENSE.txt
+Manifest.txt
+README.md
+Rakefile
+bin/fk
+lib/flat_kit.rb
+lib/flat_kit/cli.rb
+lib/flat_kit/command.rb
+lib/flat_kit/command/cat.rb
+lib/flat_kit/command/merge.rb
+lib/flat_kit/command/sort.rb
+lib/flat_kit/descendant_tracker.rb
+lib/flat_kit/error.rb
+lib/flat_kit/format.rb
+lib/flat_kit/input.rb
+lib/flat_kit/input/file.rb
+lib/flat_kit/input/io.rb
+lib/flat_kit/internal_node.rb
+lib/flat_kit/jsonl.rb
+lib/flat_kit/jsonl/format.rb
+lib/flat_kit/jsonl/reader.rb
+lib/flat_kit/jsonl/record.rb
+lib/flat_kit/jsonl/writer.rb
+lib/flat_kit/leaf_node.rb
+lib/flat_kit/logger.rb
+lib/flat_kit/merge.rb
+lib/flat_kit/merge_tree.rb
+lib/flat_kit/output.rb
+lib/flat_kit/output/file.rb
+lib/flat_kit/output/io.rb
+lib/flat_kit/reader.rb
+lib/flat_kit/record.rb
+lib/flat_kit/sentinel_internal_node.rb
+lib/flat_kit/sentinel_leaf_node.rb
+lib/flat_kit/sort.rb
+lib/flat_kit/writer.rb
+lib/flat_kit/xsv.rb
+lib/flat_kit/xsv/format.rb
+lib/flat_kit/xsv/reader.rb
+lib/flat_kit/xsv/record.rb
+lib/flat_kit/xsv/writer.rb
+tasks/default.rake
+tasks/extension.rake
+tasks/man.rake
+tasks/this.rb
+test/device_dataset.rb
+test/input/test_file.rb
+test/input/test_io.rb
+test/jsonl/test_format.rb
+test/jsonl/test_reader.rb
+test/jsonl/test_record.rb
+test/jsonl/test_writer.rb
+test/output/test_file.rb
+test/output/test_io.rb
+test/test_conversions.rb
+test/test_format.rb
+test/test_helper.rb
+test/test_merge.rb
+test/test_merge_tree.rb
+test/test_version.rb
+test/xsv/test_format.rb
+test/xsv/test_reader.rb
+test/xsv/test_record.rb
+test/xsv/test_writer.rb

data/README.md

@@ -0,0 +1,80 @@
+## FlatKit
+
+* [Homepage](https://github.com/copiousfreetime/flat_kit/)
+* [Github Project](https://github.com/copiousfreetime/flat_kit)
+
+## DESCRIPTION
+
+ A library and commandline program for reading, writing, indexing,
+ sorting, and merging CSV, TSV, JSON and other flat-file formats.
+
+## FEATURES
+
+  * Currently supporting CSV/TSV/XSV and JSON formats
+  * Transparently handles gzipped compressed input or output
+  * Sort records based upon the named keys of your choice
+  * Efficent singles pass merge of any number of sorted input files into a
+    single output file.
+  * Both a commandline tool and a ruby library to utilize in your own programs
+
+
+```
+Usage:
+  fk <command> [<args>...]
+  fk [options]
+
+Options:
+  -v, --verbose    Force debug. Output lots of informtion to standard error
+  --list           List all the commands
+  --log=<s>        Set the logger output location (default: <stderr>)
+  -h, --help       Show help message
+  --version        Print version and exit
+
+Commands:
+
+  cat      Concatenate files together that have the same structure.
+  merge    Merge sorted files together that have the same structure.
+  sort     Sort a given file by a set of fields.
+```
+
+## EXAMPLES
+
+**Convert input csv files into a single output json file**
+
+    fk cat files/*csv -o output.json
+
+**Sort an input json file and output it as a compressed csv**
+
+    fk sort --key year,month,day input.json -o output.csv.gz
+
+**Merge an entire directory of sorted record compressed csv files into a compress json file**
+
+    fk merge --key category,timestamp sorted/*.csv.gz -o sorted.json.gz
+
+## CREDITS / REFERENCES
+
+* https://github.com/BurntSushi/xsv - lots of concepts and ideas pulled from xsv
+
+## MIT LICENSE
+
+MIT License
+
+Copyright (c) 2021 Jeremy Hinegardner
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/Rakefile

@@ -0,0 +1,20 @@
+# vim: syntax=ruby
+load 'tasks/this.rb'
+
+This.name     = "flat_kit"
+This.author   = "Jeremy Hinegardner"
+This.email    = "jeremy@copiousfreetime.org"
+This.homepage = "http://github.com/copiousfreetime/#{ This.name }"
+
+This.ruby_gemspec do |spec|
+  spec.add_dependency('oj', '~> 3.0')
+  spec.add_dependency('optimist', '~> 3.0')
+
+  spec.add_development_dependency( 'rake'     , '~> 13.0')
+  spec.add_development_dependency( 'minitest' , '~> 5.11' )
+  spec.add_development_dependency( 'minitest-focus' , '~> 1.2' )
+  spec.add_development_dependency( 'rdoc'     , '~> 6.3' )
+  spec.add_development_dependency( 'simplecov', '~> 0.21')
+end
+
+load 'tasks/default.rake'

data/bin/fk

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+
+require 'flat_kit'
+FlatKit::Cli.new.run(argv: ARGV, env: ENV)

data/lib/flat_kit.rb

@@ -0,0 +1,23 @@
+module FlatKit
+  VERSION = "0.1.0"
+end
+require 'flat_kit/error'
+require 'flat_kit/logger'
+require 'flat_kit/descendant_tracker'
+require 'flat_kit/format'
+require 'flat_kit/record'
+require 'flat_kit/reader'
+require 'flat_kit/writer'
+require 'flat_kit/input'
+require 'flat_kit/output'
+require 'flat_kit/cli'
+require 'flat_kit/xsv'
+require 'flat_kit/jsonl'
+require 'flat_kit/merge'
+require 'flat_kit/sort'
+
+require 'flat_kit/merge_tree'
+require 'flat_kit/internal_node'
+require 'flat_kit/sentinel_internal_node'
+require 'flat_kit/sentinel_leaf_node'
+require 'flat_kit/leaf_node'

data/lib/flat_kit/cli.rb

@@ -0,0 +1,80 @@
+require 'optimist'
+require_relative '../flat_kit/command'
+
+module FlatKit
+  class Cli
+    attr_reader :options
+
+    def initialize
+      @parser = nil
+    end
+
+    def parser
+      @parser ||= ::Optimist::Parser.new do
+        version ::FlatKit::VERSION
+
+        banner "fk v#{self.version}"
+
+        banner <<~USAGE
+
+        Usage:
+          fk <command> [<args>...]
+          fk [options]
+        USAGE
+
+        banner <<~OPTIONS
+
+        Options:
+
+        OPTIONS
+
+        opt :verbose, "Force debug. Output lots of informtion to standard error", default: false
+        opt :list, "List all the commands", default: false, short: :none
+        opt :log, "Set the logger output location", default: "<stderr>", short: :none
+        opt :help, "Show help message", short: :h
+        opt :version, "Print version and exit", short: :none
+
+        stop_on FlatKit::Command.names
+        banner Cli.commands_banner
+      end
+    end
+
+    def self.commands_banner
+      sorted_commands = FlatKit::Command.children.sort_by{ |c| c.name }
+      left_width = sorted_commands.map { |c| c.name.length }.sort.last
+      banner = StringIO.new
+      banner.puts
+      banner.puts "Commands:"
+      banner.puts
+
+      sorted_commands.each do |command|
+        banner.puts "  #{command.name.ljust(left_width)}    #{command.description}"
+      end
+      banner.string
+    end
+
+    def run(argv: ARGV, env: ENV)
+      opts = ::Optimist::with_standard_exception_handling(parser) do
+        parser.parse(argv)
+      end
+
+      if opts[:log_given] then
+        ::FlatKit.log_to(opts[:log])
+      end
+
+      if opts[:verbose] then
+        ::FlatKit.logger.level = :debug
+      else
+        ::FlatKit.logger.level = :info
+      end
+
+      ::FlatKit.logger.debug opts
+      ::FlatKit.logger.debug argv
+
+      command_name  = argv.shift
+      command_klass = FlatKit::Command.for(command_name)
+      command       = command_klass.new(argv: argv, logger: ::FlatKit.logger, env: env)
+      command.call
+    end
+  end
+end

data/lib/flat_kit/command.rb

@@ -0,0 +1,53 @@
+module FlatKit
+  class Command
+    extend DescendantTracker
+
+    attr_reader :argv
+    attr_reader :env
+    attr_reader :logger
+    attr_reader :opts
+    attr_reader :readers
+    attr_reader :writer
+
+    def self.name
+      raise NotImplementedError, "#{self.class} must implement #{self.class}.name"
+    end
+
+    def self.description
+      raise NotImplementedError, "#{self.class} must implement #{self.class}.description"
+    end
+
+    def self.parser
+      raise NotImplementedError, "#{self.class} must implement #{self.class}.parser"
+    end
+
+    def self.names
+      children.map { |c| c.name }
+    end
+
+    def self.for(name)
+      children.find do |child_klass|
+        child_klass.name == name
+      end
+    end
+
+    def initialize(argv:, logger: ::FlatKit.logger, env: ENV)
+      @argv = argv
+      @env = env
+      @logger = logger
+      parse
+    end
+
+    def parse
+      raise NotImplementedError, "#{self.class} must implement #{self.class}#parse"
+    end
+
+    def call
+      raise NotImplementedError, "#{self.class} must implement #{self.class}.description"
+    end
+  end
+end
+
+require 'flat_kit/command/cat'
+require 'flat_kit/command/merge'
+require 'flat_kit/command/sort'

data/lib/flat_kit/command/cat.rb

@@ -0,0 +1,93 @@
+module FlatKit
+  class Command
+    # TODO: Implement the --flatten commandline switch
+    class Cat < ::FlatKit::Command
+      def self.name
+        "cat"
+      end
+
+      def self.description
+        "Concatenate files together that have the same structure."
+      end
+
+      def self.parser
+        ::Optimist::Parser.new do
+          banner "#{Cat.description}"
+          banner ""
+
+          banner <<~BANNER
+          Concatenates files that have the same field structure together into
+          a single file. The files can be of different formats, but must have
+          the same fields and names.
+
+          This is probably most easily usable as a way to convert CSV to JSON
+          and vice versa.
+
+          The flatfile type(s) will be automatically determined by the file name.
+          If the inputs or output is not a file, but from stdin or stdout then
+          the input and output types must be specified.
+
+          NOTE: If converting from JSON to CSV and the input JSON does not have
+                every possible field on ever record, then the output csv iwll
+                be corrupted.
+
+                In this case the input json should be fed through 'flatten' first
+                or use the '--flatten' flag which will require an additional pass
+                through the input to gather all the fields
+          BANNER
+
+          banner <<~USAGE
+
+          Usage:
+            fk cat file1.csv file2.csv > combinded.csv
+            fk cat --output-format json file1.csv
+            fk cat file1.csv.gzip -o file2.json.gzip
+            fk cat file1.csv.gzip --output-format json | gzip -c > file1.jsonl.gz
+
+          USAGE
+
+          banner <<~OPTIONS
+
+          Options:
+
+          OPTIONS
+
+          opt :output, "Send the output to the given path instead of standard out.", default: "<stdout>"
+          opt :input_format, "Input format, csv or json", default: "auto", short: :none
+          opt :output_format, "Output format, csv or json", default: "auto", short: :none
+        end
+      end
+
+      def parse
+        parser = self.class.parser
+        ::Optimist::with_standard_exception_handling(parser) do
+          begin
+            @opts = parser.parse(argv)
+            paths = parser.leftovers
+
+            @readers = ::FlatKit::Reader.create_readers_from_paths(paths: paths, fallback: opts[:input_format])
+            @writer  = ::FlatKit::Writer.create_writer_from_path(path: opts[:output], fallback: opts[:output_format],
+                                                                 reader_format: @readers.first.format_name)
+          rescue ::FlatKit::Error => e
+            raise ::Optimist::CommandlineError, e.message
+          end
+        end
+      end
+
+      def call
+        total = 0
+        readers.each do |r|
+          logger.info "cat #{r.source} to #{writer.destination}"
+          r.each do |record|
+            writer.write(record)
+            total += 1
+          end
+          logger.info "read #{r.count} records from #{r.source}"
+        end
+        writer.close
+        logger.debug "processed #{writer.count} records"
+        logger.debug "read #{total} records"
+      end
+    end
+  end
+end