csv_plus_plus 0.1.0 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0a93c3ee93f2320dc717ee330c55cd3f300cfa4d155a4502b9a433cdafdcc389
-  data.tar.gz: aa95dca43a374d57f2217030e0aefef680341549613b0388cb5df1122cb5f815
+  metadata.gz: d0fb79a1f7a00d60f77289518fb9c6ccac6b059024d5cf66a3f083ec92e77def
+  data.tar.gz: a35796b982b01171636e27b1d1a9d19d6d414249b0e4634947d5a754cb240146
 SHA512:
-  metadata.gz: 19a827c9a5fc8f0f9fe9d58c4e6b85de821c9e09627465673c2813c2690fadc886c7d8f24e3da5402783b27347a79f7832b13c82abe4e204026d12839d6f4691
-  data.tar.gz: b157b531d9ae9e2fda9c93ed466976deacfe639b1fbc1527b60c706be9d97792946341a767f00c32b6d35cbc6e388711980767d9ef6b5c20b4a2f44ebe7fb121
+  metadata.gz: 45dd804f7889d65ac5f5c63ccc131d774e7ea24097bfe9449af63a48345ca0b9f16ea9e84c17a73151c9ee8d9d90ce4b01d67b7ced817e831e66781d2e7141e0
+  data.tar.gz: 21cda263af6d05d5ee396a9d00b4c1c78f1a043d91787dcf0d822e1adb6a97a2326da836cdc3bbc3c78e43e5c342c1bdb9da966884f1d9d8df364e574f334ded

data/CHANGELOG.md

@@ -1,8 +1,23 @@
+## v0.1.2
+
+- var=... modifier which allows binding a variable to a cell
+- Improved error handling and messages
+- Moving in a direction that allows for the context-dependent aspects of modifiers
+- Fixes a bug with creating a new excel spreadsheet
+- Docs & tests
+
+## v0.1.1
+
+- Better support for the various infix operators (+,-,/,*,^,%,=,<,etc)
+  * Previously we were converting them to their prefix equivalent (multiply, minus, concat, etc) but excel doesn't support most of those.  So we keep them infix
+  * Didn't support some infix operators (^, %, </>/<=/>=/<>)
+  * Proper support for operator precedence
+- When in verbose mode, print a summary of compiled functions and variables
+- docs & tests
 
 ## v0.1.0
 
 - revamp of builtin functions
-
 - docs & tests
 
 ## v0.0.5

data/README.md

@@ -9,101 +9,57 @@ A tool that allows you to programatically author spreadsheets in your favorite t
 
 A `csvpp` file consists of a (optional) code section and a CSV section separated by `---`.  In the code section you can define variables and functions that can be used in the CSV below it.  For example:
 
+###### **`mystocks.csvpp`**
 ```
 fees := 0.50 # my broker charges $0.50 a trade
 
-price := cellref(C)
-quantity := cellref(D)
+price := celladjacent(C)
+quantity := celladjacent(D)
 
 def profit() (price * quantity) - fees
 
 ---
 ![[format=bold/align=center]]Date,Ticker,Price,Quantity,Total,Fees
-![[expand]],[[format=bold]],,,"=PROFIT()",$$fees
+![[expand]],[[format=bold]],,,"=profit()",$$fees
 ```
 
-## Variables
-
-Variables can be defined in the code section by giving a name (a combination of letters, numbers and underscores ) the expression `:=` and followed with a value.
-
-### Built-in Variables
-
-* `$$rownum` - The current row number.  The first row of the spreadsheet starts at 1.  Can be used anywhere and it's value will evaluate to the current row being processed.
-
-## Functions
-
-### Built-in Functions
-* `cellref(CELL)` - Returns a reference to the `CELL` relative to the current row.  If the current `$$rownum` is `2`, then `CELLREF("C")` returns  a reference to cell `C2`.
-
-## Modifiers
-
-Modifiers can change the formatting of a cell or row, apply validation, change alignment, etc. All of the normal rules of CSV apply, with the addition that each cell can have modifiers (specified in `[[`/`]]` for cells and `![[`/`]]` for rows):
+And can be compiled into a `.xlsx` file by:
 
 ```
-foo,[[...]]bar,baz
+$ csv++ -n 'My Stock Tracker' -o mystocks.xlsx mystocks.csvpp
 ```
 
-specifying formatting or various other modifiers to the cell.  Additionally a row can start with:
 
-```
-![[...]]foo,bar,baz
-```
+See the [Language Reference](./docs/LANGUAGE_REFERENCE.md) for a full explanation of features.
 
-which will apply that modifier to all cells in the row.
+## Installing
 
-### Examples
-
-* Align the second cell left, align the last cell to the center and make it bold and italicized:
-
-```
-Date,[[align=left]]Amount,Quantity,[[align=center/format=bold italic]]Price
-```
-
-* Underline and center-align an entire row:
-
-```
-![[align=center/format=underline]]Date,Amount,Quantity,Price
-```
-
-* A header for the first row, then some formulas that repeat for each row for the rest of the spreadsheet:
+Just install it via rubygems (homebrew and debian packages are in the works):
 
-```
-![[align=center/format=bold]]Date,Price,Quantity,Profit
-![[expand=1:]],,,"=MULTIPLY(cellref(B), cellref(C))"
-```
+`$ gem install csv_plus_plus`
 
-## Setup (Google Sheets)
+or if you want the very latest changes, clone the repository and run:
 
-Just install it via rubygems (homebrew and debian packages are in the works):
+`$ rake gem:install`
 
-`$ gem install csv_plus_plus`
+### [Setting Up Google Sheets](./docs/README_GOOGLE_SHEETS.md)
 
-### Publishing to Google Sheets
+## Examples
 
-* Go to the [GCP developers console](https://console.cloud.google.com/projectselector2/apis/credentials?pli=1&supportedpurview=project), create a service account and export keys for it to `~/.config/gcloud/application_default_credentials.json`
-* "Share" the spreadsheet with the email associated with the service account
+Take a look at the [examples](./examples/) directory for a bunch of example `.csvpp` files.
 
 ## CLI Arguments
 
 ```
 Usage: csv++ [options]
+    -h, --help                       Show help information
     -b, --backup                     Create a backup of the spreadsheet before applying changes.
-    -g, --google-sheet-id SHEET_ID   The id of the sheet - you can extract this from the URL: https://docs.google.com/spreadsheets/d/< ... SHEET_ID ... >/edit#gid=0
     -c, --create                     Create the sheet if it doesn't exist.  It will use --sheet-name if specified
+    -g, --google-sheet-id SHEET_ID   The id of the sheet - you can extract this from the URL: https://docs.google.com/spreadsheets/d/< ... SHEET_ID ... >/edit#gid=0
     -k, --key-values KEY_VALUES      A comma-separated list of key=values which will be made available to the template
     -n, --sheet-name SHEET_NAME      The name of the sheet to apply the template to
+    -o, --output OUTPUT_FILE         The file to write to (must be .csv, .ods, .xls)
     -v, --verbose                    Enable verbose output
     -x, --offset-columns OFFSET      Apply the template offset by OFFSET cells
     -y, --offset-rows OFFSET         Apply the template offset by OFFSET rows
-    -h, --help                       Show help information
-```
-
-## Usage Examples
-
-```
-# apply my_taxes_template.csvpp to an existing Google Sheet with name "Taxes 2022"
-$ csv++ --sheet-name "Taxes 2022" --sheet-id "[...]" my_taxes_template.csvpp
-
-# take input from stdin, supply a variable ($$rate = 1) and apply to the "Stocks" spreadsheet
-$ cat stocks.csvpp | csv++ -k "rate=1" -n "Stocks" -i "[...]"
 ```

data/lib/csv_plus_plus/benchmarked_compiler.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+module CSVPlusPlus
+  # Extend a +Compiler+ class and add benchmark timings
+  #
+  # @attr_reader timings [Array<Benchmark::Tms>] +Benchmark+ timings that have been accumulated by each step of
+  #   compilation
+  # @attr_reader benchmark [Benchmark] A +Benchmark+ instance
+  module BenchmarkedCompiler
+    attr_reader :benchmark, :timings
+
+    # Wrap a +Compiler+ with our instance methods that add benchmarks
+    def self.with_benchmarks(compiler, &block)
+      ::Benchmark.benchmark(::Benchmark::CAPTION, 25, ::Benchmark::FORMAT, '> Total') do |x|
+        # compiler = new(options:, runtime:, benchmark: x)
+        compiler.extend(self)
+        compiler.benchmark = x
+
+        block.call(compiler)
+
+        [compiler.timings.reduce(:+)]
+      end
+    end
+
+    # @param benchmark [Benchmark] A +Benchmark+ instance
+    def benchmark=(benchmark)
+      @benchmark = benchmark
+      @timings = []
+    end
+
+    # Time the Compiler#outputting! stage
+    def outputting!
+      time_stage('Writing the spreadsheet') { super }
+    end
+
+    protected
+
+    def parse_code_section!
+      time_stage('Parsing code section') { super }
+    end
+
+    def parse_csv_section!
+      time_stage('Parsing CSV section') { super }
+    end
+
+    def expanding
+      time_stage('Expanding rows') { super }
+    end
+
+    def resolve_all_cells!(template)
+      time_stage('Resolving each cell') { super(template) }
+    end
+
+    private
+
+    def time_stage(stage, &block)
+      ret = nil
+      @timings << @benchmark.report(stage) { ret = block.call }
+      ret
+    end
+  end
+end

data/lib/csv_plus_plus/can_define_references.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+module CSVPlusPlus
+  # Methods for classes that need to manage +@variables+ and +@functions+
+  module CanDefineReferences
+    # Define a (or re-define an existing) variable
+    #
+    # @param id [String, Symbol] The identifier for the variable
+    # @param entity [Entity] The value (entity) the variable holds
+    def def_variable(id, entity)
+      variables[id.to_sym] = entity
+    end
+
+    # Define (or re-define existing) variables
+    #
+    # @param variables [Hash<Symbol, Variable>] Variables to define
+    def def_variables(vars)
+      vars.each { |id, entity| def_variable(id, entity) }
+    end
+
+    # Define a (or re-define an existing) function
+    #
+    # @param id [String, Symbol] The identifier for the function
+    # @param entity [Entities::Function] The defined function
+    def def_function(id, entity)
+      functions[id.to_sym] = entity
+    end
+
+    # Is the variable defined?
+    #
+    # @param var_id [Symbol, String] The identifier of the variable
+    #
+    # @return [boolean]
+    def defined_variable?(var_id)
+      variables.key?(var_id.to_sym)
+    end
+
+    # Is the function defined?
+    #
+    # @param fn_id [Symbol, String] The identifier of the function
+    #
+    # @return [boolean]
+    def defined_function?(fn_id)
+      functions.key?(fn_id.to_sym)
+    end
+
+    # Provide a summary of the functions and variables compiled (to show in verbose mode)
+    #
+    # @return [String]
+    def verbose_summary
+      <<~SUMMARY
+        # Code Section Summary
+
+        ## Resolved Variables
+
+        #{variable_summary}
+
+        ## Functions
+
+        #{function_summary}
+      SUMMARY
+    end
+
+    private
+
+    def variables
+      @variables ||= {}
+    end
+
+    def functions
+      @functions ||= {}
+    end
+
+    def variable_summary
+      return '(no variables defined)' if variables.empty?
+
+      variables.map { |k, v| "#{k} := #{v}" }
+               .join("\n")
+    end
+
+    def function_summary
+      return '(no functions defined)' if functions.empty?
+
+      functions.map { |k, f| "#{k}: #{f}" }
+               .join("\n")
+    end
+  end
+end

data/lib/csv_plus_plus/can_resolve_references.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+module CSVPlusPlus
+  # Methods for classes that need to resolve references
+  module CanResolveReferences
+    # TODO
+  end
+end

data/lib/csv_plus_plus/cell.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
-require_relative './language/cell_value.tab'
-require_relative './modifier'
+require_relative 'modifier'
+require_relative 'parser/cell_value.tab'
 
 module CSVPlusPlus
   # A cell of a template
@@ -23,7 +23,7 @@ module CSVPlusPlus
     # @return [Cell]
     def self.parse(value, runtime:, modifier:)
       new(value:, row_index: runtime.row_index, index: runtime.cell_index, modifier:).tap do |c|
-        c.ast = ::CSVPlusPlus::Language::CellValueParser.new.parse(value, runtime)
+        c.ast = ::CSVPlusPlus::Parser::CellValue.new.parse(value, runtime)
       end
     end
 

data/lib/csv_plus_plus/cli.rb

@@ -31,14 +31,16 @@ module CSVPlusPlus
     #
     # @param error [CSVPlusPlus::Error, Google::Apis::ClientError, StandardError]
     def handle_error(error)
+      # make sure that we're on a newline (verbose mode might be in the middle of printing a benchmark)
+      puts("\n\n") if @options.verbose
+
       case error
-      when ::CSVPlusPlus::Error
+      when ::CSVPlusPlus::Error::Error
         handle_internal_error(error)
       when ::Google::Apis::ClientError
         handle_google_error(error)
       else
-        # TODO: more if verbose?
-        warn(error.message)
+        unhandled_error(error)
       end
     end
 
@@ -48,18 +50,33 @@ module CSVPlusPlus
       option_parser.parse!
       validate_options
     rescue ::OptionParser::InvalidOption => e
-      raise(::CSVPlusPlus::Error, e.message)
+      raise(::CSVPlusPlus::Error::Error, e.message)
     end
 
-    # @return [String]
+    # @return [::String]
     def to_s
       "CLI(options: #{options})"
     end
 
     private
 
+    # An error was thrown that we weren't planning on
+    def unhandled_error(error)
+      warn(
+        <<~ERROR_MESSAGE)
+          An unexpected error was encountered.  Please try running again with --verbose and
+          reporting the error at: https://github.com/patrickomatic/csv-plus-plus/issues/new'
+        ERROR_MESSAGE
+
+      return unless @options.verbose
+
+      warn(error.full_message)
+      warn("Cause: #{error.cause}") if error.cause
+    end
+
     def handle_internal_error(error)
-      if error.is_a?(::CSVPlusPlus::Language::SyntaxError)
+      case error
+      when ::CSVPlusPlus::Error::SyntaxError
         warn(@options.verbose ? error.to_verbose_trace : error.to_trace)
       else
         warn(error.message)
@@ -78,7 +95,7 @@ module CSVPlusPlus
       return if error_message.nil?
 
       puts(option_parser)
-      raise(::CSVPlusPlus::Error, error_message)
+      raise(::CSVPlusPlus::Error::Error, error_message)
     end
 
     def option_parser

data/lib/csv_plus_plus/color.rb

@@ -9,13 +9,19 @@ module CSVPlusPlus
   class Color
     attr_reader :red_hex, :green_hex, :blue_hex
 
-    # create an instance from a string like "#FFF" or "#FFFFFF"
+    HEX_STRING_REGEXP = /^#?([0-9a-f]{1,2})([0-9a-f]{1,2})([0-9a-f]{1,2})/i
+    public_constant :HEX_STRING_REGEXP
+
+    # @return [boolean]
+    def self.valid_hex_string?(hex_string)
+      !(hex_string.strip =~ ::CSVPlusPlus::Color::HEX_STRING_REGEXP).nil?
+    end
+
+    # Create an instance from a string like "#FFF" or "#FFFFFF"
     #
     # @param hex_string [String] The hex string input to parse
     def initialize(hex_string)
-      @red_hex, @green_hex, @blue_hex = hex_string
-                                        .gsub(/^#?/, '')
-                                        .match(/([0-9a-f]{1,2})([0-9a-f]{1,2})([0-9a-f]{1,2})/i)
+      @red_hex, @green_hex, @blue_hex = hex_string.strip.match(::CSVPlusPlus::Color::HEX_STRING_REGEXP)
                                         &.captures
                                         &.map { |s| s.length == 1 ? s + s : s }
     end
@@ -43,12 +49,12 @@ module CSVPlusPlus
 
     # Create a hex representation of the color (without a '#')
     #
-    # @return [String]
+    # @return [::String]
     def to_hex
       [@red_hex, @green_hex, @blue_hex].join
     end
 
-    # @return [String]
+    # @return [::String]
     def to_s
       "Color(r: #{@red_hex}, g: #{@green_hex}, b: #{@blue_hex})"
     end

data/lib/csv_plus_plus/compiler.rb

@@ -0,0 +1,156 @@
+# frozen_string_literal: true
+
+require_relative 'benchmarked_compiler'
+require_relative 'entities'
+require_relative 'parser/code_section.tab'
+require_relative 'runtime'
+require_relative 'scope'
+
+module CSVPlusPlus
+  # Encapsulates the parsing and building of objects (+Template+ -> +Row+ -> +Cell+). Variable resolution is delegated
+  # to the +Scope+
+  #
+  # @attr_reader options [Options] The +Options+ to compile with
+  # @attr_reader runtime [Runtime] The runtime execution
+  # @attr_reader scope [Scope] +Scope+ for variable resolution
+  class Compiler
+    attr_reader :options, :runtime, :scope
+
+    # Create a compiler and make sure it gets cleaned up
+    #
+    # @param runtime [Runtime] The initial +Runtime+ for the compiler
+    # @param options [Options]
+    def self.with_compiler(runtime:, options:, &block)
+      compiler = new(options:, runtime:)
+      if options.verbose
+        ::CSVPlusPlus::BenchmarkedCompiler.with_benchmarks(compiler) do |c|
+          block.call(c)
+        end
+      else
+        yield(compiler)
+      end
+    ensure
+      runtime.cleanup!
+    end
+
+    # @param runtime [Runtime]
+    # @param options [Options]
+    # @param scope [Scope, nil]
+    def initialize(runtime:, options:, scope: nil)
+      @options = options
+      @runtime = runtime
+      @scope = scope || ::CSVPlusPlus::Scope.new(runtime:)
+
+      # TODO: infer a type
+      # allow user-supplied key/values to override anything global or from the code section
+      @scope.def_variables(
+        options.key_values.transform_values { |v| ::CSVPlusPlus::Entities::String.new(v.to_s) }
+      )
+    end
+
+    # Write the compiled results
+    def outputting!
+      @runtime.start_at_csv!
+      yield
+    end
+
+    # Compile a template and return a +::CSVPlusPlus::Template+ instance ready to be written with a +Writer+
+    #
+    # @return [Template]
+    def compile_template
+      parse_code_section!
+      rows = parse_csv_section!
+
+      ::CSVPlusPlus::Template.new(rows:, scope: @scope).tap do |t|
+        t.validate_infinite_expands(@runtime)
+        expanding { t.expand_rows! }
+        resolve_all_cells!(t)
+      end
+    end
+
+    # @return [String]
+    def to_s
+      "Compiler(options: #{@options}, runtime: #{@runtime}, scope: #{@scope})"
+    end
+
+    protected
+
+    # Parses the input file and returns a +CodeSection+
+    #
+    # @return [CodeSection]
+    def parse_code_section!
+      @runtime.start!
+
+      # TODO: this flow can probably be refactored, it used to have more needs back when we had to
+      # parse and save the code_section
+      parsing_code_section do |input|
+        csv_section = ::CSVPlusPlus::Parser::CodeSection.new(@scope).parse(input, @runtime)
+        # TODO: call scope.resolve_static_variables?? or maybe it doesn't matter
+
+        # return the csv_section to the caller because they're gonna re-write input with it
+        next csv_section
+      end
+      # @scope.code_section
+    end
+
+    # Parse the CSV section and return an array of +Row+s
+    #
+    # @return [Array<Row>]
+    def parse_csv_section!
+      @runtime.start_at_csv!
+      @runtime.map_rows(::CSV.new(runtime.input)) do |csv_row|
+        parse_row(csv_row)
+      end
+    ensure
+      # we're done with the file and everything is in memory
+      @runtime.cleanup!
+    end
+
+    # Iterates through each cell of each row and resolves it's variable and function references.
+    #
+    # @param template [Template]
+    # @return [Array<Entity>]
+    def resolve_all_cells!(template)
+      @runtime.start_at_csv!
+      @runtime.map_rows(template.rows, cells_too: true) do |cell|
+        cell.ast = @scope.resolve_cell_value if cell.ast
+      end
+    end
+
+    # Expanding rows
+    def expanding
+      @runtime.start_at_csv!
+      yield
+    end
+
+    private
+
+    def parsing_code_section
+      csv_section = yield(@runtime.input.read)
+      @runtime.rewrite_input!(csv_section)
+    end
+
+    # Using the current +@runtime+ and the given +csv_row+ parse it into a +Row+ of +Cell+s
+    # +csv_row+ should have already been run through a CSV parser and is an array of strings
+    #
+    # @param csv_row [Array<Array<String>>]
+    # @return [Row]
+    def parse_row(csv_row)
+      row_modifier = ::CSVPlusPlus::ValidatedModifier.new(row_level: true)
+
+      cells = @runtime.map_row(csv_row) { |value, _cell_index| parse_cell(value, row_modifier) }
+
+      ::CSVPlusPlus::Row.new(@runtime.row_index, cells, row_modifier)
+    end
+
+    def parse_cell(value, row_modifier)
+      cell_modifier = ::CSVPlusPlus::ValidatedModifier.new
+      parsed_value = ::CSVPlusPlus::Parser::Modifier.new(cell_modifier:, row_modifier:, scope: @scope).parse(
+        value,
+        @runtime
+      )
+
+      ::CSVPlusPlus::Cell.parse(parsed_value, runtime:, modifier: cell_modifier)
+    end
+  end
+end