estratto 1.0.0 → 1.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f4425ecb691c561000266f1717d1bcab1aa9c198
-  data.tar.gz: f3526413190af262cf089112ae34682f1465894b
+  metadata.gz: 768680440af76aff2f08d4a72dad782fbfa17c70
+  data.tar.gz: 68974ef5b81a15b4fd57814f21025b93a5cd2ac8
 SHA512:
-  metadata.gz: fd3154c347e94acd858b7a92e88817d00f3e55618ac8191786a5b5606212fb519f9180e071dcffec9e776d15c8bb88dedbe709a3fcedd514098d74ae96718ce6
-  data.tar.gz: be0cdd97b6a300d68bd30c6b4090c20c2274c2b4a160ca7d925c49c32a30704c144cf6de82e88aa8cd0f2a068cc22b83c71f4ab84e5282daffd11728f5f903c7
+  metadata.gz: 0bff7f6b4c314824bd827e8a61a8459f2b9af8e2c8511fb319b244beb5d7d1034102293fda7843a71ed2c5a990885be763ae3507d79b2264d52d3d0d3eca4597
+  data.tar.gz: 31f0f3e48656b4359972033f5b188525dc34bacf78128f9d73a4ecbe990c9c6925cbbba8535d9e80fb4373eb248f6c32a28cf913e8bdf654ddfaaf2535516386

data/Gemfile.lock

@@ -1,11 +1,13 @@
 PATH
   remote: .
   specs:
-    estratto (1.0.0)
+    estratto (1.0.1)
+      charlock_holmes
 
 GEM
   remote: https://rubygems.org/
   specs:
+    charlock_holmes (0.7.6)
     coderay (1.1.2)
     coveralls (0.8.22)
       json (>= 1.8, < 3)

data/README.md

@@ -1,10 +1,18 @@
 # Estratto
 
+[![Gem Version](https://badge.fury.io/rb/estratto.svg)](https://badge.fury.io/rb/estratto)
 [![Build Status](https://travis-ci.com/Rynaro/estratto.svg?branch=master)](https://travis-ci.com/Rynaro/estratto)
 [![Coverage Status](https://coveralls.io/repos/github/Rynaro/estratto/badge.svg?branch=master)](https://coveralls.io/github/Rynaro/estratto?branch=master)
 [![Maintainability](https://api.codeclimate.com/v1/badges/46532b90e850401fce72/maintainability)](https://codeclimate.com/github/Rynaro/estratto/maintainability)
 
-### TODO: Write a super README
+[![Waffle.io - Columns and their card count](https://badge.waffle.io/Rynaro/estratto.svg?columns=all)](https://waffle.io/Rynaro/estratto)
+
+> Estratto is a easy to handle parser based on YAML templating engine. Creating a easy interface for developers, and non developers to extract data from fixed width files
+
+## Motivation
+
+In various scenarios the data processment is a crucial step of a integration with partner systems, or data storage. But the task to create parsing and import data from these text format is boring, and causing code duplication in every code project.
+This project borns to help developers to reduce the time spent in this task, or creating a total delegation scenario to other team responsabilities.
 
 ## Installation
 
@@ -24,7 +32,245 @@ Or install it yourself as:
 
 ## Usage
 
-TODO: Comming soon
+**Estratto** works with simple input of _data to parse file_ and a _yaml layout equivalent_.
+
+Example of a default call for parsing:
+
+```ruby
+Estratto::Document.process(file: 'path/to/data.txt', layout: 'path/to/layout.yml')
+```
+
+### Layout specifications
+
+Fixed width files is sometimes ~always~ painful for human reading, and the layout manual comes in a very useful pdf or spreasheet format.
+
+Here, we'll try to made things fun again, or less painful. :joy:
+
+The base layout for YAML file is:
+
+```yaml
+layout:
+  name: 'jojo stand users'
+  multi-register: true
+  prefix: 0..1
+  registers:
+    - register: '01'
+      fields:
+        - name: name
+          range: 2..45
+          type: String
+        - name: stand
+          range: 46..75
+          type: String
+```
+
+And the output will be a array of hashes reflection of your columns:
+
+```ruby
+[
+    {
+        name: 'Jotaro Kujo',
+        stand: 'Star Platinum'
+    },
+    {
+        name: 'Giorno Giovanna',
+        stand: 'Golden Experience Requiem'
+    },
+    {
+        name: 'Jobin Higashikata',
+        stand: 'Speed King'
+    }
+]
+```
+
+The structure follows the strict directive
+```yaml
+layout:
+    (base configuration)
+    registers:
+        (layouts)
+```
+
+Actually **Estratto** supports these types of fixed width  layouts:
+
+- Batch prefix based registers
+- Mono layout based registers _(development)_
+
+
+### UTF-8 Conversion
+
+Estratto makes use of [CharlockHolmes](https://github.com/brianmario/charlock_holmes) gem to detect the file content encoding and convert it to UTF-8.
+This approach prevents invalid characters from being present in the output.
+
+### Type Coercion
+
+**Estratto** supports type coercion, with some perks called _formats_, on layout file.
+
+Data type supported to handle in **Estratto**
+
+- String
+- Integer
+- Float
+- DateTime
+- Date
+
+Default data type in fields is `String`, if no one type is setted in field list register.
+
+Registers fields list always respect this base structure:
+
+```yaml
+  fields:
+    - name: name
+      range: 2..12
+      type: String
+      formats:
+        strip: true
+```
+
+`name` is your field identification of field, this value will be your symbol in hashed parsed data
+
+`range` is where data is inside the file. (First index is 0)
+
+`type` data type to be coerced
+
+`formats` receives a specific configuration for data type. Here we can format Strings, and adjust precision for unformatted Float data.
+
+### Formats
+
+Formats is the resource for deal with some "surprises" that this type of file can provide to us. Like, super large string fields that has a huge blank space, DateTime with suspicious formatting, or Float without any decimal point, but the manual description shows _"Decimal(15, 2)"_
+
+#### String
+
+##### strip
+
+Works like common ruby String strip method
+
+```yaml
+strip: true
+```
+
+Output example:
+
+```ruby
+#raw_data
+'Hierophant Green         '
+# with strip clause
+'Hierophant Green' 
+```
+
+#### Integer
+
+Simple integer values converter. Useful in cases that you need to deal with ids.
+
+Actually we don't have any formats for Integer. :)
+
+```ruby
+#raw_data
+'000123'
+# coerced
+123 
+#raw_data
+'123'
+# coerced
+123 
+#raw_data
+'a'
+# coerced
+0 
+```
+
+#### Float
+
+Float is one of most important types here. The fixed width files always respect the _non logical_ format to deliver information.
+
+##### precision
+
+```yaml
+precision: <integer>
+```
+
+Examples:
+
+```yaml
+precision: 2
+```
+
+```ruby
+#raw data
+'12345'
+# with precision
+123.45
+```
+
+```yaml
+precision: 3
+```
+
+```ruby
+#raw data
+'12345'
+# with precision
+12.345
+```
+
+##### comma_format
+
+```yaml
+comma_format: <boolean>
+```
+
+Examples:
+
+```yaml
+comma_format: true
+```
+
+```ruby
+#raw data
+'123,45'
+# with comma formats
+123.45
+```
+
+
+#### DateTime and Date
+
+The `DateTime` and `Date` has the same formats attributes. But the difference, one shows DateTime format, and other always respect Date output
+
+
+##### format
+
+```yaml
+format: <ruby strptime format pattern>
+```
+
+Examples
+
+```yaml
+format: '%Y%m%d'
+```
+
+```ruby
+#raw data
+'20180101'
+# with comma formats
+#<DateTime: 2018-01-01T00:00:00+00:00 ...>
+```
+
+```yaml
+format: '%d/%m/%Y'
+```
+
+```ruby
+#raw data
+'01/01/2018'
+# with comma formats
+#<DateTime: 2018-01-01T00:00:00+00:00 ...>
+```
+
+## Tests
+
+Simple `rake spec`
 
 ## Development
 

data/estratto.gemspec

@@ -20,6 +20,8 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
+  spec.add_dependency  "charlock_holmes"
+
   spec.add_development_dependency "bundler", "~> 1.17"
   spec.add_development_dependency "rake", "~> 10.0"
   spec.add_development_dependency "rspec", "~> 3.0"

data/lib/estratto/content.rb

@@ -0,0 +1,11 @@
+require_relative 'encoder'
+
+module Estratto
+  class Content
+    def self.for(file_path)
+      content = File.read(file_path)
+      encoded_content = Encoder.new(content).encode
+      encoded_content.split("\n")
+    end
+  end
+end

data/lib/estratto/encoder.rb

@@ -0,0 +1,21 @@
+require 'charlock_holmes'
+
+module Estratto
+  class Encoder
+    attr_reader :content
+
+    def initialize(content)
+      @content = content
+    end
+
+    def encode
+      CharlockHolmes::Converter.convert(content, encoding, 'UTF-8')
+    end
+
+    private
+
+    def encoding
+      CharlockHolmes::EncodingDetector.detect(content)[:encoding]
+    end
+  end
+end

data/lib/estratto/parser.rb

@@ -1,4 +1,5 @@
 require_relative 'register'
+require_relative 'content'
 
 module Estratto
   class Parser
@@ -10,16 +11,15 @@ module Estratto
     end
 
     def perform
-      @data ||= raw_data.map do |line|
+      @data ||= raw_content.map do |line|
         register_layout = layout.register_fields_for(line[layout.prefix_range])
         next if register_layout.nil?
         Register.new(line, register_layout).refine
       end.compact
     end
 
-    def raw_data
-      @raw_data ||= File.open(file_path, 'r')
+    def raw_content
+      @raw_data = Content.for(file_path)
     end
-
   end
 end

data/lib/estratto/version.rb

@@ -1,3 +1,3 @@
 module Estratto
-  VERSION = "1.0.0"
+  VERSION = "1.0.1"
 end

metadata

@@ -1,15 +1,29 @@
 --- !ruby/object:Gem::Specification
 name: estratto
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.0.1
 platform: ruby
 authors:
 - Henrique A. Lavezzo
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2019-01-08 00:00:00.000000000 Z
+date: 2019-01-21 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: charlock_holmes
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -85,6 +99,7 @@ files:
 - bin/setup
 - estratto.gemspec
 - lib/estratto.rb
+- lib/estratto/content.rb
 - lib/estratto/data/base.rb
 - lib/estratto/data/coercer.rb
 - lib/estratto/data/date.rb
@@ -93,6 +108,7 @@ files:
 - lib/estratto/data/integer.rb
 - lib/estratto/data/string.rb
 - lib/estratto/document.rb
+- lib/estratto/encoder.rb
 - lib/estratto/helpers/range.rb
 - lib/estratto/layout/base.rb
 - lib/estratto/layout/factory.rb