proptax 0.0.1

data/lib/proptax/generators/report/default.Rmd

@@ -0,0 +1,292 @@
+---
+title: 2018 Property Assessment Analysis
+author: Prepared by reports@taxreformyyc.com
+geometry: margin=1.5cm
+---
+
+```{r loadLibraries, echo=FALSE, message=FALSE}
+# Load the required libraries
+library(knitr)
+library(scales)
+library(formattable)
+library(ggplot2)
+```
+
+```{r defineConstants, echo=FALSE}
+# Constants
+address <- "<%= address %>"
+myAssessedValue <- <%= assessed_value %>
+csvFile <- "<%= csv_file %>"
+
+# Get the house number and street name
+m <- regexpr("^\\d+", address, perl=TRUE)
+myHouseNumber <- regmatches(address, m)
+#myHouseNumber <- gsub("[^\\d]+", "", address, perl=TRUE)
+myStreetName <- gsub(".*[\\d]", "", address, perl=TRUE)
+
+# How do this factors impact an assessment?
+unknownImpact <- c('Taxation.Status', 'Assessment.Class',
+                   'Property.Type', 'Property.Use', 'Valuation.Approach',
+                   'Market.Adjustment', 'Community', 'Market.Area', 'Sub.Neighbourhood.Code..SNC.',
+                   'Sub.Market.Area', 'Influences', 'Land.Use.Designation', 'Building.Count',
+                   'Building.Type.Structure', 'Year.of.Construction', 'Quality', 'Basement.Suite',
+                   'Walkout.Basement', 'Garage.Type', 'Fireplace.Count',
+                   'Constructed.On.Original.Foundation', 'Modified.For.Disabled',
+                   'Old.House.On.New.Foundation', 'Basementless', 'Penthouse')
+
+# These factors are informational and don't affect the assessment
+noImpact <- c('Roll.Number', 'Location.Address')
+```
+
+```{r loadData, echo=FALSE}
+# Load data
+data <- read.csv(csvFile, header=TRUE)
+```
+
+```{r getMetaData, echo=FALSE}
+streetAddresses <- data[,noImpact[2]]
+
+# Remove the street names from the addresses
+#houseNumbers <- as.numeric(gsub("[^\\d]+", "", streetAddresses, perl=TRUE))
+m <- regexpr("^\\d+", streetAddresses, perl=TRUE)
+houseNumbers <- as.numeric(regmatches(streetAddresses, m))
+```
+
+\center
+`r paste(min(houseNumbers), "-", max(houseNumbers), myStreetName, sep=" ")`
+
+\flushleft
+
+# Synopsis
+
+This analysis pertains to the property located at **`r address`**.
+It documents the treatment of the assessment data provided by the City of
+Calgary for the pertinent property and those adjacent on the street
+(i.e., buildings `r paste(min(houseNumbers), "-", max(houseNumbers), sep=" ")`)
+The data under investigation was obtained from
+[assessmentsearch.calgary.ca](https://assessmentsearch.calgary.ca) and
+is presented alongside this document in a consolidated CSV file.
+
+# Approach Overview
+
+The data investigated in this analysis consists of properties constrained to
+one street on one city block. No properties were omitted to ensure the integrity
+of this report and its conclusion.
+
+Given that there are many factors contained in the data whose precise impact on
+assessed values are unknown, commonalities are identified and omitted from 
+consideration, as all such factors should have identical impact on the
+final assessment.
+
+Factors that vary are identified and presented for consideration, as
+transparency and integrity is of the utmost importance. Again, the precise
+impact of these factors on the final assessment is unknown, as the weights
+assigned by the City of Calgary are not divulged in the assessment data they
+provide. 
+
+Having acknowledged the factors that cannot easily be quantified, the focus
+turns to the properties' lot sizes, total developed area, and assessed values.
+By visualizing the relationship between these quantifiable factors, the 
+pertinent property's assessed value is contrasted with those of neighbouring
+properties. The conclusion of this analysis is drawn from the underlying data.
+
+# Identify common factors
+
+Many of the factors that impact the assessments are identical. This data can
+safely be removed from consideration, as the impact on the assessed values
+should be the same for all the properties under investigation.
+
+```{r commonFactors, echo=FALSE}
+# Identify common factors
+headers <- c()
+identical <- c()
+displayHeaders <- c()
+for (col in colnames(data)) {
+  values = data[,col][!duplicated(data[,col])]
+  if (length(values) == 1) {
+    headers <- append(headers, col)
+    displayHeaders <-append(displayHeaders, gsub("\\.", " ", col, perl=TRUE))
+    identical <- append(identical, toString(values))
+  }
+}
+commonFactors <- data.frame(headers, displayHeaders, identical)
+```
+
+Here, **`r length(commonFactors$headers)`** common factors can safely be 
+removed from the data set:
+
+`r kable(data.frame(commonFactors$displayHeaders, commonFactors$identical),
+         col.names=c('Factors', 'Identical Values'), align=c('l', 'r'))`
+
+```{r removeCommonFactors, echo=FALSE}
+# Remove common factors
+data <- data[,!(names(data) %in% commonFactors$headers)]
+```
+
+
+# Identify unknown and non-impacting factors
+
+Of the **`r length(colnames(data))`** remaining columns, some cannot be
+quantified. Others certainly impact the assessed value of a property, but the
+assessment data provided by the City of Calgary does not reveal to what extent.
+
+## Non-impacting factors
+
+```{r noImpactDisplayHeaders, echo=FALSE}
+# Remove the dots from the header name
+noImpactDisplayHeaders <- gsub("\\.", " ", noImpact, perl=TRUE)
+```
+
+These factors cannot be quantified and are administrative in purpose:
+
+`r kable(noImpactDisplayHeaders, col.names=c('Non-Impacting Factors'))`
+
+```{r removeIrrelevantFactors, echo=FALSE}
+data <- data[,!(names(data) %in% noImpact)]
+```
+
+These are removed and the remaining **`r length(colnames(data))`** columns are
+carried forward.
+
+## Unknown factors
+
+The impact these remaining columns have on assessment values is unknown:
+
+```{r identifyUnknowns, echo=FALSE}
+# Identify unknown factors
+unknownFactors <- names(data[,names(data) %in% unknownImpact])
+```
+
+`r kable(gsub("\\.", " ", unknownFactors), col.names=c('Unknown Factors'))`
+
+The variability within these unknown columns is presented here in the interest
+of transparency:
+
+```{r consolidateUnknownFactors, echo=FALSE}
+consolidatedUnknownFactors <- data[,(names(data) %in% c(unknownFactors))]
+rownames(consolidatedUnknownFactors) <- houseNumbers
+```
+
+`r kable(consolidatedUnknownFactors[order(as.numeric(row.names(consolidatedUnknownFactors))),],
+         align=c(rep('c', length(unknownFactors))),
+         row.names=TRUE,
+         col.names=gsub("\\.", " ", unknownFactors))`
+
+These undoubtedly have an impact on the valuation, but their precise weighting
+and significance are not presented in the assessment data provided by the City
+of Calgary. As such, they are removed from the dataset.
+
+```{r removeUnknownFactors, echo=FALSE}
+# Remove unknown factors from data set
+data <- data[,!(names(data) %in% unknownFactors)]
+```
+
+The remaining **`r length(colnames(data))`** columns contain the following data:
+
+```{r addRowNamesToData, echo=FALSE}
+# Add row names to data
+rownames(data) <- houseNumbers
+```
+
+`r kable(data[order(as.numeric(row.names(data))),], row.names=TRUE, col.names=gsub("\\.", " ", colnames(data)))`
+
+# Visualization
+
+The raw data presented above is summarized in Figure 1. It
+illustrates the disparity between the assessed property values. The pertinent
+property is coloured red.
+
+The blue line running through the graph is _best fit_ for the visualized model.
+It serves as a predictor, or indicator, as to where the properties in question
+should be positioned.
+
+The pertinent property's overassessment is determined by measuring the distance
+between the red point and the blue line.
+
+
+```{r adjustValues, echo=FALSE}
+# Sum each property's lot size and total developed space
+areaTotals <- rowSums(data[,-1])
+
+# Isolate all assessed values
+assessedValues <- data[,1]
+
+# Plot the best fit regression line
+reg <- lm(assessedValues~areaTotals)
+
+# Plot distances between points and the regression line
+assessedDifferences <- residuals(reg)
+adjustedValues <- predict(reg)
+
+# Reconcile adjusted property values
+adjustedProperties <- data.frame(houseNumbers, adjustedValues, assessedDifferences)
+```
+
+```{r generatePlot, fig.cap=paste(myHouseNumber, "Overassessment", " "), fig.width=12, echo=FALSE}
+plot.title <- paste(min(houseNumbers), "-", max(houseNumbers), myStreetName, sep=" ")
+plot.subtitle = 'Current Assessed Property Values'
+ggplot(data, aes(x=areaTotals, y=assessedValues)) +
+    # Plot title
+    ggtitle(bquote(atop(bold(.(plot.title)), atop(italic(.(plot.subtitle)), "")))) +
+    theme(plot.title=element_text(size=24, hjust = 0.5)) +
+
+    # Axis labels
+    ylab("Assessed House Values on your Street") +
+    xlab("Total House and Lot Size (Sq. Feet)") +
+    theme(axis.title.x=element_text(size=18, face="bold", margin=margin(20,0,0,0)),
+          axis.title.y=element_text(size=18, face="bold", margin=margin(0,20,0,0))) +
+
+    # Axis tick labels
+    scale_x_continuous(labels=comma) +
+<% if opts.ylimit? %>
+    scale_y_continuous(labels=dollar, breaks=pretty_breaks(n=10),
+                       limits=c(min(assessedValues)-<%= opts.ylimit %>, max(assessedValues)+<%= opts.ylimit %>)) +
+<% else %>
+    scale_y_continuous(labels=dollar, breaks=pretty_breaks(n=10)) +
+<% end %>
+    # Scatter plot points
+    geom_point(shape=ifelse(assessedValues==myAssessedValue, 16, 1),
+               size=ifelse(assessedValues==myAssessedValue, 5, 4),
+               colour=ifelse(assessedValues==myAssessedValue, "red", "blue")) +
+
+    # Point labels
+    geom_text(aes(label=houseNumbers), hjust=0.5, vjust=-2, size=5) +
+    geom_text(aes(label=paste("$", accounting(assessedValues, format="d"), sep="")),
+              hjust=0.5, vjust=-1.2) +
+
+    # Best fit line
+    geom_smooth(method=lm) #+ 
+```
+
+# Conclusion 
+
+The data investigated in this analysis describes the factors considered in
+assessing the properties located at 
+`r paste(min(houseNumbers), "-", max(houseNumbers), myStreetName, sep=" ")`.
+It was collected and provided by the City of Calgary. This report set out to
+quantify the disparity between the pertinent property (i.e., `r address`) and
+the neighbouring properties on its street.
+
+Common factors were identified and eliminated from the analysis. Similarly,
+varying factors were identified, catalogued, and removed from consideration. The
+City of Calgary's property reports do not describe how these characteristic
+features are quantified and weighted in determining a property's assessed
+value. As such, they could not be included.
+
+The conclusions that follow are drawn from comparing lot sizes, total developed
+area, and assessed values. The underlying data and the overall approach have
+been presented in full.
+
+## Overvaluation: `r currency(adjustedProperties[houseNumbers==myHouseNumber,]$assessedDifferences)`
+
+This investigation compared the assessed values with lot sizes and developed
+square footage. It has revealed that that the pertinent property is overvalued
+by 
+**`r currency(adjustedProperties[houseNumbers==myHouseNumber,]$assessedDifferences)`**.
+
+## Corrected Assessed Value: `r currency(adjustedProperties[houseNumbers==myHouseNumber,]$adjustedValues)`
+
+In order to bring the pertinent property's assessed value in line with those of
+its immediate neighbours, it must be reassessed at
+**`r currency(adjustedProperties[houseNumbers==myHouseNumber,]$adjustedValues)`**.
+

data/lib/proptax/generators/report.rb

@@ -0,0 +1,31 @@
+require 'thor/group'
+require 'csv'
+module Proptax
+  module Generators
+    class Report < Thor::Group
+      include Thor::Actions
+      attr_accessor :address, :assessed_value, :y_axis_limits
+
+      argument :csv_file, :type => :string
+      argument :opts
+
+      def create_report_dir
+        empty_directory('reports')
+        FileUtils.cp(csv_file, "reports")
+      end
+
+      def copy_report_template
+        CSV.foreach(csv_file, headers: true) do |row|
+          self.address = row['Location Address']
+          self.assessed_value = row['Current Assessed Value']
+          file_name = address.gsub(/\s/, '_')
+          template("#{opts.template}.Rmd", "reports/#{file_name}.Rmd")
+        end
+      end
+
+      def self.source_root
+        File.dirname(__FILE__) + "/report"
+      end
+    end
+  end
+end

data/lib/proptax/version.rb

@@ -0,0 +1,3 @@
+module Proptax
+  VERSION = "0.0.1"
+end

data/lib/proptax.rb

@@ -0,0 +1,153 @@
+require "proptax/version"
+
+module Proptax
+  class Consolidator
+
+  require 'csv'
+  require 'pathname'
+
+    #
+    # Headers for publicly available assessment data for 2018.
+    #
+    Headers = ['Current Assessed Value', 'Roll Number', 'Location Address', 'Taxation Status',
+               'Assessment Class', 'Property Type', 'Property Use', 'Valuation Approach',
+               'Market Adjustment', 'Community', 'Market Area', 'Sub Neighbourhood Code (SNC)',
+               'Sub Market Area', 'Influences', 'Land Use Designation', 'Assessable Land Area',
+               'Building Count', 'Building Type/Structure', 'Year of Construction',
+               'Quality', 'Total Living Area Above Grade', 'Living Area Below Grade', 'Basement Suite',
+               'Walkout Basement', 'Garage Type', 'Garage Area', 'Fireplace Count',
+               'Constructed On Original Foundation', 'Modified For Disabled', 'Old House On New Foundation',
+               'Basementless', 'Penthouse']
+
+    #
+    # Keys that can be assigned multiple values
+    #
+    MultiFieldHeaders = ['Influences']
+
+    #
+    # Section headers for publicly available assessment data for 2018.
+    #
+    Sections = ['Assessment Details', 'Assessment Approach', 'Location Details', 'Land Details', 'Building Details']
+  
+    #
+    # Convert a directory containing 2018 PDF assessment reports to text and
+    # write the relevant CSV information to stdout
+    #
+    # @param string - path to directory containing PDFs
+    #
+    # @return nil
+    #
+    def self.process(dir_name)
+      # `strip!': can't modify frozen String (RuntimeError)
+      #dir_name.strip!
+      dir_name = dir_name.strip
+      if dir_name.empty?
+        puts "No directory specified"
+        return
+      end
+
+      begin
+        path = Pathname.new(dir_name).realpath
+        puts Headers.to_csv if Dir.exists?(path)
+        Dir.glob("#{path}/*.{pdf,PDF}") do |pdf|
+          data = `pdftotext "#{pdf}" -`
+          puts parse(data).to_csv
+        end
+      rescue 
+        puts "Directory #{dir_name} does not exist"
+      end
+    end
+  
+    #
+    # Take text-converted 2018 assessment reports and extract the relevant data
+    # into an array.
+    # 
+    # @param string - text converted 2018 assessment report
+    #
+    # @return array
+    #
+    def self.parse(text)
+      csv_hash = {}
+      current_header = nil
+      text.split(/\n/).each do |line|
+        # Replace non-breaking spaces with space and strip
+        line = line.gsub(/\u00a0/, ' ').strip
+  
+        # Remove soft hyphens
+        line.gsub!(/\u00AD/, '')
+        ## This is untested
+        line.gsub!(/â\\200\\224/, '-')
+
+        # Remove trailing colon
+        line.gsub!(/:+$/, "");
+  
+        next if line.empty?
+  
+        if current_header
+          if Sections.include? line
+            current_header = nil 
+            next
+          end
+
+          if MultiFieldHeaders.include? current_header
+            if csv_hash[current_header].nil?
+              csv_hash[current_header] = clean(line, current_header)
+            else
+              csv_hash[current_header] += "/#{clean(line, current_header)}"
+            end
+          else
+            csv_hash[current_header] = clean(line, current_header)
+            current_header = nil
+          end
+        elsif Headers.include? line
+          current_header = line
+        #
+        # Need a rasterized report (a la, Windows 7)
+        #
+        # For tesseract-extracted text
+#        else
+#          Headers.each do |header|
+#            current_header = line[Regexp.new("^#{Regexp.escape(header)}")]
+#            break if current_header
+#          end
+#          if current_header
+#            line = line.gsub(Regexp.new("^#{Regexp.escape(current_header)}"), '').strip
+#            csv_hash[current_header] = clean(line, current_header)
+#            current_header = nil
+#          end
+        end
+      end
+      Headers.map { |header| csv_hash[header] || '0' }
+    end
+  
+    #
+    # Return square footage only when given spacial data
+    #
+    # @param string - the text data to be cleaned
+    # @param string - the header associated with the text data
+    #
+    # @return string
+    #
+    def self.clean(text, header)
+      case header
+      when 'Current Assessed Value'
+        text.gsub!(/,/, '')
+      when 'Assessable Land Area',
+           'Total Living Area Above Grade',
+           'Living Area Below Grade',
+           'Garage Area'
+        text = text.split(' ')[0].gsub(/,/, '')
+      when 'Basement Suite',
+           'Walkout Basement',
+           'Constructed On Original Foundation',
+           'Modified For Disabled',
+           'Old House On New Foundation',
+           'Basementless',
+           'Penthouse',
+           'Market Adjustment'
+        text = text == 'Yes' ? 'T' : 'F'
+      end
+      text
+    end
+  end
+end

data/proptax.gemspec

@@ -0,0 +1,38 @@
+# coding: utf-8
+lib = File.expand_path("../lib", __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require "proptax/version"
+
+Gem::Specification.new do |spec|
+  spec.name          = "proptax"
+  spec.version       = Proptax::VERSION
+  spec.authors       = ["Daniel Bidulock"]
+  spec.email         = ["daniel@capitolhill.ca"]
+
+  spec.summary       = %q{Automatically process and visualize residential property data collected by the City of Calgary in 2018.}
+  spec.description   = %q{This software produces reports from property data collected and provided by the City of Calgary. The versions used in previous years have won three victories before Calgary's Assessment Review Board, the quasi-judicial body that rules on property tax appeals.}
+  spec.homepage      = "https://github.com/TaxReformYYC/report-generator-2018"
+
+  # Prevent pushing this gem to RubyGems.org. To allow pushes either set the 'allowed_push_host'
+  # to allow pushing to a single host or delete this section to allow pushing to any host.
+#  if spec.respond_to?(:metadata)
+#    spec.metadata["allowed_push_host"] = "TODO: Set to 'http://mygemserver.com'"
+#  else
+#    raise "RubyGems 2.0 or newer is required to protect against " \
+#      "public gem pushes."
+#  end
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.15"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.0"
+  spec.add_development_dependency "aruba", "~> 0.14.3"
+
+  spec.add_dependency "thor"
+end

metadata

@@ -0,0 +1,136 @@
+--- !ruby/object:Gem::Specification
+name: proptax
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Daniel Bidulock
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2018-01-12 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.15'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.15'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: aruba
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.14.3
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.14.3
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This software produces reports from property data collected and provided
+  by the City of Calgary. The versions used in previous years have won three victories
+  before Calgary's Assessment Review Board, the quasi-judicial body that rules on
+  property tax appeals.
+email:
+- daniel@capitolhill.ca
+executables:
+- proptax
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- ".travis.yml"
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- exe/proptax
+- lib/R/filter_csv.R
+- lib/proptax.rb
+- lib/proptax/cli.rb
+- lib/proptax/generators/report.rb
+- lib/proptax/generators/report/cherry-picked.Rmd
+- lib/proptax/generators/report/default.Rmd
+- lib/proptax/version.rb
+- proptax.gemspec
+homepage: https://github.com/TaxReformYYC/report-generator-2018
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.6.8
+signing_key: 
+specification_version: 4
+summary: Automatically process and visualize residential property data collected by
+  the City of Calgary in 2018.
+test_files: []