spatial_stats 0.1.1 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 56905c84e53f041ed8acc1437d7d3ee1312ff9515f3e11e08bd31120e7ed0c26
-  data.tar.gz: 2879c58f356b430285d55717cba5fc27f239097c35067032bd9d4407688c5086
+  metadata.gz: 8ec49f792086ad98d8c07da5dde57041085df82d491b168bf66aebac1fab9cb4
+  data.tar.gz: 134adbb6d387487c87b1fee3c4529e56c1228dd4dd44d01f2822dadb1aaa9955
 SHA512:
-  metadata.gz: 7fd244b8cc798e4806d6c380bd3e1e261da3f8f509ed87f4f920f9eb706cb8d94150e1a915cc3f73417196ae8d91fc29579d6604f05c38cbd4c1a82b6cb34e48
-  data.tar.gz: 205d0850f885b0e0d673d51976d0d69d7357bd2a2f0382aaa0140f096fd5164db1cce7f1fc0c31cb204b97153f75cb1a18ff9d4120c905d2f73c4a9b247659cb
+  metadata.gz: e02e5a207e8dae161aac12fd55f7325634a15dbd2485b521125e7ae41276b022eb7fca9ecfca051d5aded322d07d715bc1788e83940a2c4bab84a4bd5ae6e022
+  data.tar.gz: 3932e7019b3989c85476632aa740d3f48ce7b7467be2a78daefb45d4038c6ba7a21cd1d3d6704331b6021ea22e5314faa1bdeee24c91d50bbd90a7c5aa5edeba

data/README.md

@@ -1,6 +1,8 @@
+[![Build Status](https://travis-ci.com/keithdoggett/spatial_stats.svg?branch=master)](https://travis-ci.com/keithdoggett/spatial_stats)
+
 # SpatialStats
 
-Short description and motivation.
+SpatialStats is an ActiveRecord plugin that utilizes PostGIS and Ruby to compute weights/statistics of spatial data sets in Rails Apps.
 
 ## Installation
 
@@ -24,13 +26,178 @@ $ gem install spatial_stats
 
 ## Usage
 
-How to use my plugin.
+### Weights
+
+Weights define the spatial relation between members of a dataset. Contiguous operations are supported for `polygons` and `multipolygons`, and distant operations are supported for `points`.
+
+To compute weights, you need an `ActiveRecord::Relation` scope and a geometry field. From there, you can pick what type of weight operation to compute (`knn`, `queen neighbors`, etc.).
+
+#### Compute Queen Weights
+
+```ruby
+# County table has the following fields: avg_income: float, geom: multipolygon.
+scope = County.all
+geom_field = :geom
+weights = SpatialStats::Weights::Contiguous.queen(scope, geom_field)
+# => #<SpatialStats::Weights::WeightsMatrix>
+```
+
+#### Compute KNN of Centroids
+
+The field being queried does not have to be defined in the schema, but could be computed during the query for scope.
+
+This example finds the inverse distance weighted, 5 nearest neighbors for the centroid of each county.
+
+```ruby
+scope = County.all.select("*, st_centroid(geom) as geom")
+weights = SpatialStats::Weights::Distant.idw_knn(scope, :geom, 5)
+# => #<SpatialStats::Weights::WeightsMatrix>
+```
+
+#### Define WeightsMatrix without Query
+
+Weight matrices can be defined by a hash that describes each key's neighbor and weight.
+
+Note: Currently, the keys must be numeric.
+
+Example: Define WeightsMatrix and get the matrix in row_standardized format.
+
+```ruby
+weights = {
+    1 => [{ j_id: 2, weight: 1 }, { j_id: 4, weight: 1 }],
+    2 => [{ j_id: 1, weight: 1 }],
+    3 => [{ j_id: 4, weight: 1 }],
+    4 => [{ j_id: 1, weight: 1 }, { j_id: 3, weight: 1 }]
+}
+keys = weights.keys
+wm = SpatialStats::Weights::WeightsMatrix.new(keys, weights)
+#  => #<SpatialStats::Weights::WeightsMatrix:0x0000561e205677c0 @keys=[1, 2, 3, 4], @weights={1=>[{:j_id=>2, :weight=>1}, {:j_id=>4, :weight=>1}], 2=>[{:j_id=>1, :weight=>1}], 3=>[{:j_id=>4, :weight=>1}], 4=>[{:j_id=>1, :weight=>1}, {:j_id=>3, :weight=>1}]}, @n=4>
+
+wm.standardized
+#  => Numo::DFloat#shape=[4,4]
+#[[0, 0.5, 0, 0.5],
+# [1, 0, 0, 0],
+# [0, 0, 0, 1],
+# [0.5, 0, 0.5, 0]]
+```
+
+### Lagged Variables
+
+Spatially lagged variables can be computed with a 2-D n x n `Numo::NArray` and 1-D vector (`Array` or `Numo::NArray`).
+
+#### Compute a Lagged Variable
+
+```ruby
+w = Numo::DFloat[[0, 0.5, 0, 0.5],
+                 [1, 0, 0, 0],
+                 [0, 0, 0, 1],
+                 [0.5, 0, 0.5, 0]]
+vec = [1, 2, 3, 4]
+lagged_var = SpatialStats::Utils::Lag.neighbor_sum(w, vec)
+# => [3.0, 1.0, 4.0, 2.0]
+```
+
+### Global Stats
+
+Global stats compute a value for the dataset, like how clustered the observations are within the region.
+
+Most `stat` classes take three parameters: `scope`, `data_field`, and `weights`. All `stat` classes have the `stat` method that will compute the target statistic. These are also aliased with the common name of the statistic, such as `i` for `Moran` or `c` for `Geary`.
+
+#### Compute Moran's I
+
+```ruby
+scope = County.all
+weights = SpatialStats::Weights::Contiguous.rook(scope, :geom)
+moran = SpatialStats::Global::Moran.new(scope, :avg_income, weights)
+# => <SpatialStats::Global::Moran>
+
+moran.stat
+# => 0.834
+
+moran.i
+# => 0.834
+```
+
+#### Compute Moran's I Z-Score
+
+```ruby
+scope = County.all
+weights = SpatialStats::Weights::Contiguous.rook(scope, :geom)
+moran = SpatialStats::Global::Moran.new(scope, :avg_income, weights)
+# => <SpatialStats::Global::Moran>
+
+moran.z_score
+# => 3.2
+```
+
+#### Run a Permutation Test on Moran's I
+
+All stat classes have the `mc` method which takes `permutations` and `seed` as its parameters. `mc` runs a permutation test on the class and returns the psuedo p-value.
+
+```ruby
+scope = County.all
+weights = SpatialStats::Weights::Contiguous.rook(scope, :geom)
+moran = SpatialStats::Global::Moran.new(scope, :avg_income, weights)
+# => <SpatialStats::Global::Moran>
+
+moran.mc(999, 123_456)
+# => 0.003
+```
+
+### Local Stats
+
+Local stats compute a value each observation in the dataset, like how similar its neighbors are to itself. Local stats operate similarly to global stats, except that almost every operation will return an array of length `n` where `n` is the number of observations in the dataset.
+
+Most `stat` classes take three parameters: `scope`, `data_field`, and `weights`. All `stat` classes have the `stat` method that will compute the target statistic. These are also aliased with the common name of the statistic, such as `i` for `Moran` or `c` for `Geary`.
+
+#### Compute Moran's I
+
+```ruby
+scope = County.all
+weights = SpatialStats::Weights::Contiguous.rook(scope, :geom)
+moran = SpatialStats::Local::Moran.new(scope, :avg_income, weights)
+# => <SpatialStats::Local::Moran>
+
+moran.stat
+# => [0.888, 0.675, 0.2345, -0.987, -0.42, ...]
+
+moran.i
+# => [0.888, 0.675, 0.2345, -0.987, -0.42, ...]
+```
+
+#### Compute Moran's I Z-Scores
+
+Note: Many classes do not have a variance or expectation method implemented and this will raise a `NotImplementedError`.
+
+```ruby
+scope = County.all
+weights = SpatialStats::Weights::Contiguous.rook(scope, :geom)
+moran = SpatialStats::Local::Moran.new(scope, :avg_income, weights)
+# => <SpatialStats::Local::Moran>
+
+moran.z_score
+# => # => [0.65, 1.23, 0.42, 3.45, -0.34, ...]
+```
+
+#### Run a Permutation Test on Moran's I
+
+All stat classes have the `mc` method which takes `permutations` and `seed` as its parameters. `mc` runs a permutation test on the class and returns the psuedo p-values.
+
+```ruby
+scope = County.all
+weights = SpatialStats::Weights::Contiguous.rook(scope, :geom)
+moran = SpatialStats::Local::Moran.new(scope, :avg_income, weights)
+# => <SpatialStats::Local::Moran>
+
+moran.mc(999, 123_456)
+# => [0.24, 0.13, 0.53, 0.023, 0.65, ...]
+```
 
 ## Contributing
 
 Once cloned, run the following commands to setup the test database.
 
-```sh
+```bash
 cd ./spatial_stats
 bundle install
 cd test/dummy
@@ -40,7 +207,7 @@ rake db:migrate
 
 If you are getting an error, you may need to set the following environment variables.
 
-```
+```bash
 $PGUSER # default "postgres"
 $PGPASSWORD # default ""
 $PGHOST # default "127.0.0.1"
@@ -50,7 +217,7 @@ $PGDATABASE # default "spatial_stats_test"
 
 If the dummy app is setup correctly, run the following:
 
-```
+```bash
 cd ../..
 rake
 ```
@@ -59,7 +226,7 @@ This will run the tests. If they all pass, then your environment is setup correc
 
 Note: It is recommended to have GEOS installed and linked to RGeo. You can test this by running the following:
 
-```
+```bash
 cd test/dummy
 rails c
 
@@ -71,8 +238,11 @@ RGeo::Geos.supported?
 
 - ~~Memoize expensive functions within classes~~
 - ~~Make star a parameter to getis-ord class~~
-- Add examples to docs
-- Create RDocs
+- ~~Add examples/usage to docs~~
+- ~~Create RDocs~~
+- Refactor Global Moran and BVMoran
+- Support non-numeric keys in WeightsMatrix/General refactor
+- Write SparseMatrix C ext
 
 ## Future Work
 
@@ -80,16 +250,22 @@ RGeo::Geos.supported?
 
 - ~~Refactor stats to inherit an abstract class.~~
 - Change WeightsMatrix class and Stat classes to utilize sparse matrix methods.
+- Split into two separate gems spatial_stats and spatial_stats-activerecord
 
 #### Weights
 
-- Add Kernel based weighting.
+- Add Kernel based weighting
 
 #### Utils
 
 - Rate smoothing
 - Bayes smoothing
 
+### Global
+
+- Geary class
+- GetisOrd class
+
 #### Local
 
 - Join Count Statistic

data/lib/spatial_stats.rb

@@ -9,10 +9,13 @@ require 'spatial_stats/queries'
 require 'spatial_stats/utils'
 require 'spatial_stats/weights'
 
+##
+# SpatialStats is an ActiveRecord/PostGIS gem that provides descriptive spatial
+# stats to your application.
 module SpatialStats
-  def self.included(klass)
-    puts 'here', klass
-    # klass.extend(SpatialStats::Queries::Weights)
-  end
+  # def self.included(klass)
+  #   puts 'here', klass
+  #   # klass.extend(SpatialStats::Queries::Weights)
+  # end
   # Your code goes here...
 end

data/lib/spatial_stats/enumerable_ext.rb

@@ -1,6 +1,18 @@
 # frozen_string_literal: true
 
+##
+# Extension to the Enumerable module
 module Enumerable
+  ##
+  # Standardize works with a numeric array and transforms each value so that
+  # the mean is 0 and the variance is 1.
+  # Formula is (x - mean)/stdev
+  #
+  # @example
+  #   [1,2,3].standardize
+  #   [-1.0, 0.0, 1.0]
+  #
+  # @return [Array] the standardized array
   def standardize
     # standardize is (variable - mean)/stdev
     m = mean
@@ -8,10 +20,27 @@ module Enumerable
     map { |v| (v - m) / std }
   end
 
+  ##
+  # Mean works with a numeric array and returns the arithmetic mean.
+  #
+  # @example
+  #   [1,2,3].mean
+  #   2.0
+  #
+  # @return [Float] the arithmetic mean
   def mean
     sum / size.to_f
   end
 
+  ##
+  # Sample Variance works with a numeric array and returns the variance.
+  # Formula for variance is (x - mean)**2/(n - 1)
+  #
+  # @example
+  #   [1,2,3].sample_variance
+  #   1.0
+  #
+  # @return [Float] the sample variance
   def sample_variance
     m = mean
     numerator = sum { |v| (v - m)**2 }

data/lib/spatial_stats/global.rb

@@ -3,3 +3,18 @@
 require 'spatial_stats/global/stat'
 require 'spatial_stats/global/bivariate_moran'
 require 'spatial_stats/global/moran'
+
+module SpatialStats
+  ##
+  # The Global module provides functionality for global spatial statistics.
+  # Global spatial statistics describe the entire dataset with one value,
+  # like how clustered the observations are across the entire dataset.
+  #
+  # All global classes define a +stat+ method that returns the described
+  # statistic and an +mc+ method that runs a permutation test determine a
+  # pseudo p-value for the statistic. Some also define +variance+ and
+  # +z_score+  methods that can be used to calculate p-values if the
+  # distribution is known.
+  module Global
+  end
+end

data/lib/spatial_stats/global/bivariate_moran.rb

@@ -1,9 +1,20 @@
 # frozen_string_literal: true
 
-# https://geodacenter.github.io/workbook/5b_global_adv/lab5b.html
 module SpatialStats
   module Global
+    ##
+    # BivariateMoran computes the correlation between a variable x and
+    # a spatially lagged variable y.
     class BivariateMoran < Stat
+      ##
+      # A new instance of BivariateMoran
+      #
+      # @param [ActiveRecord::Relation] scope
+      # @param [Symbol, String] x_field to query from scope
+      # @param [Symbol, String] y_field to query from scope
+      # @param [WeightsMatrix] weights to define relationship between observations in scope
+      #
+      # @return [BivariateMoran]
       def initialize(scope, x_field, y_field, weights)
         @scope = scope
         @x_field = x_field
@@ -12,7 +23,12 @@ module SpatialStats
       end
       attr_writer :x, :y
 
-      def i
+      ##
+      # Computes the global spatial correlation of x against spatially lagged
+      # y.
+      #
+      # @return [Float]
+      def stat
         w = @weights.standardized
         y_lag = SpatialStats::Utils::Lag.neighbor_sum(w, y)
         numerator = 0
@@ -23,13 +39,22 @@ module SpatialStats
         denominator = x.sum { |xi| xi**2 }
         numerator / denominator
       end
+      alias i stat
 
+      ##
+      # The expected value of +#stat+.
+      # @see https://en.wikipedia.org/wiki/Moran%27s_I#Expected_value
+      # @return [Float]
       def expectation
         -1.0 / (@weights.n - 1)
       end
 
+      ##
+      # The variance of the bivariate spatial correlation.
+      # @see https://en.wikipedia.org/wiki/Moran%27s_I#Expected_value
+      #
+      # @return [Float]
       def variance
-        # https://en.wikipedia.org/wiki/Moran%27s_I#Expected_value
         n = @weights.n
         wij = @weights.full
         w = wij.sum
@@ -47,16 +72,35 @@ module SpatialStats
         var_left - var_right
       end
 
+      ##
+      # Permutation test to determine a pseudo p-value of the computed I stat.
+      # Shuffles y values while holding x constant and recomputing I for each
+      # variation, then compares that I value to the computed one.
+      # The ratio of more extreme values to permutations is returned.
+      #
+      # @see https://geodacenter.github.io/glossary.html#perm
+      #
+      # @param [Integer] permutations to run. Last digit should be 9 to produce round numbers.
+      # @param [Integer] seed used in random number generator for shuffles.
+      #
+      # @return [Float]
       def mc(permutations = 99, seed = nil)
-        # call super monte carlo for multivariate
         mc_bv(permutations, seed)
       end
 
+      ##
+      # Standardized variables queried from +x_field+.
+      #
+      # @return [Array]
       def x
         @x ||= SpatialStats::Queries::Variables.query_field(@scope, @x_field)
                                                .standardize
       end
 
+      ##
+      # Standardized variables queried from +y_field+.
+      #
+      # @return [Array]
       def y
         @y ||= SpatialStats::Queries::Variables.query_field(@scope, @y_field)
                                                .standardize

data/lib/spatial_stats/global/moran.rb

@@ -2,37 +2,62 @@
 
 module SpatialStats
   module Global
+    ##
+    # Moran's I statistic computes the spatial autocorrelation of variable x.
+    # It does this by computing a spatially lagged version of itself and
+    # comparing that with each observation based on the weights matrix.
     class Moran < Stat
+      ##
+      # A new instance of Moran
+      #
+      # @param [ActiveRecord::Relation] scope
+      # @param [Symbol, String] field to query from scope
+      # @param [WeightsMatrix] weights to define relationship between observations in scope
+      #
+      # @return [Moran]
       def initialize(scope, field, weights)
         super(scope, field, weights)
       end
       attr_writer :x
 
-      def i
+      ##
+      # Computes the global spatial autocorrelation of x against a spatially
+      # lagged x.
+      #
+      # @return [Float]
+      def stat
         # compute's Moran's I. numerator is sum of zi * spatial lag of zi
         # denominator is sum of zi**2.
-        # have to use row-standardized
-        @i ||= begin
-          w = @weights.standardized
-          z_lag = SpatialStats::Utils::Lag.neighbor_sum(w, z)
-          numerator = 0
-          z.each_with_index do |zi, j|
-            row_sum = zi * z_lag[j]
-            numerator += row_sum
-          end
-
-          denominator = z.sum { |zi| zi**2 }
-          numerator / denominator
+        # have to use row-standardized weights
+        w = @weights.standardized
+        z_lag = SpatialStats::Utils::Lag.neighbor_sum(w, z)
+        numerator = 0
+        z.each_with_index do |zi, j|
+          row_sum = zi * z_lag[j]
+          numerator += row_sum
         end
+
+        denominator = z.sum { |zi| zi**2 }
+        numerator / denominator
       end
+      alias i stat
 
+      ##
+      # The expected value of +#stat+.
+      # @see https://en.wikipedia.org/wiki/Moran%27s_I#Expected_value
+      #
+      # @return [Float]
       def expectation
         # -1/(n-1)
         -1.0 / (@weights.n - 1)
       end
 
+      ##
+      # The variance of the spatial correlation.
+      # @see https://en.wikipedia.org/wiki/Moran%27s_I#Expected_value
+      #
+      # @return [Float]
       def variance
-        # https://en.wikipedia.org/wiki/Moran%27s_I#Expected_value
         n = @weights.n
         wij = @weights.full
         w = wij.sum
@@ -50,18 +75,43 @@ module SpatialStats
         var_left - var_right
       end
 
+      ##
+      # Permutation test to determine a pseudo p-value of the computed I stat.
+      # Shuffles x values recomputes I for each variation, then compares that I
+      # value to the computed one. The ratio of more extreme values to
+      # permutations is returned.
+      #
+      # @see https://geodacenter.github.io/glossary.html#perm
+      #
+      # @param [Integer] permutations to run. Last digit should be 9 to produce round numbers.
+      # @param [Integer] seed used in random number generator for shuffles.
+      #
+      # @return [Float]
       def mc(permutations = 99, seed = nil)
         super(permutations, seed)
       end
 
+      ##
+      # Values of the +field+ queried from the +scope+
+      #
+      # @return [Array]
       def x
         @x ||= SpatialStats::Queries::Variables.query_field(@scope, @field)
       end
 
+      # TODO: remove these last 2 methods and just standardize x.
+      ##
+      # Mean of x
+      #
+      # @return [Float]
       def zbar
         x.sum / x.size
       end
 
+      ##
+      # Array of xi - zbar for i: [0:n-1]
+      #
+      # @return [Array]
       def z
         x.map { |val| val - zbar }
       end
@@ -76,12 +126,12 @@ module SpatialStats
 
       def s2_calc(n, wij)
         s2 = 0
-        (0..n - 1).each do |i|
+        (0..n - 1).each do |idx|
           left_term = 0
           right_term = 0
           (0..n - 1).each do |j|
-            left_term += wij[i, j]
-            right_term += wij[j, i]
+            left_term += wij[idx, j]
+            right_term += wij[j, idx]
           end
           s2 += (left_term + right_term)**2
         end
@@ -90,9 +140,9 @@ module SpatialStats
 
       def s1_calc(n, wij)
         s1 = 0
-        (0..n - 1).each do |i|
+        (0..n - 1).each do |idx|
           (0..n - 1).each do |j|
-            s1 += (wij[i, j] + wij[j, i])**2
+            s1 += (wij[idx, j] + wij[j, idx])**2
           end
         end
         s1 / 2