citysdk 1.0.1 → 1.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 8fd2afe55971d61fbecb78c89d4a828a93adc22f
-  data.tar.gz: fcf46e34bf468cad4b75325172587dfab8bc2a63
+  metadata.gz: 470ce38f410d78f4caf61b12f9a26a7e42f8fab9
+  data.tar.gz: 67fb900ec6bdf7a09db87fb81e4f808fe037b676
 SHA512:
-  metadata.gz: cdc4b57532d758e9f794d836db960f95c6868a88281f129c14884b555654e540fd6cbeb79641d93ec80ec9f428ac69ae4f870a14d8b3ac3b4bb39bbf3e1633e3
-  data.tar.gz: fc6897d281e7beed34b6e35578177691979f559d3d79beee149c045c253ba7543f2f4768217ba860f35c34cfb12fca24a7a660ab85e7eb3081a26511bee16006
+  metadata.gz: 761fde18d53a8a0f1cfceb9b687841ec324a98c70283b5b7ee4a33ed813305392949f37d69ae9bcb6e0d6a3eea7051aa37d916450cc3acda14eb77a9222ce8c6
+  data.tar.gz: 8eb62b0eafccb689ea4036631676954dd72275da1ca409e8d1a5e1cde7315325cd5dcacbd730db33d9339c922e4d44be705033851e4b87a4792600a53d0f3b4f

data/README.md

@@ -1,12 +1,12 @@
 # CitySDK GEM
 
-The CitySDK gem encapulates the CitySDK LD-API, and offers high-level file import functionalities.
-The CitySDK LD-API is part of an (open)data distribution platform developed in the EU CitySDK program by [Waag Society](http://waag.org).
-Find the platform itself on [github](https://github.com/waagsociety/citysdk-ld), background is [here](http://dev.citysdk.waag.org).
+The CitySDK gem encapulates the CitySDK LD API, and offers high-level file import functionalities.
+The CitySDK LD API is part of an open data distribution platform developed in the EU CitySDK program by [Waag Society](http://waag.org).
+Find the platform itself on [GitHub](https://github.com/waagsociety/citysdk-ld), background is [here](http://dev.citysdk.waag.org).
 
-In order to best get an overview of the way to use the GEM to import files into the CitySDK LD-API, have a look at the 'admr' importer for the top-level administrative regions in the Netherlands.
-The data consists of three ESRI shape files; the importer is well commented, explaining most of the possibilities of the GEM. 
-Please download the importer [here](https://github.com/waagsociety/citysdk-amsterdam/tree/master/importers/admr)
+In order to best get an overview of the way to use the gem to import files into the CitySDK LD API, have a look at the [`admr` importer](https://github.com/waagsociety/citysdk-amsterdam/tree/master/importers/admr) for the top-level administrative regions in the Netherlands. The data consists of three ESRI Shapefiles; the importer is well-commented, explaining most of the possibilities of the gem.
+
+The `bin` directory contains a preliminary version of the CitySDK LD API CLI tool. __😁 Warning, unfinished 😁!__
 
 ## Installation
 
@@ -18,12 +18,11 @@ If you're using Bundler, add the gem to Gemfile.
 
     gem 'citysdk'
 
-Then run `bundle install`.
-
+Then, run `bundle install`.
 
 ## Usage
-The GEM can be used on two different levels, either simply as a wrapper around the API, or as a means to read, convert and import various types of data files.
-The gem exposes it's functionality through three different objects:
+
+The gem can be used on two different levels, either simply as a wrapper around the API, or as a means to read, convert and import various types of data files. The gem exposes its functionality through three different objects:
 
 - CitySDK::API
 - CitySDK::FileReader
@@ -33,38 +32,29 @@ The FileReader can be used stand-alone to read, edit and save data files, the Im
 
 The FileReader has support for CSV, Shape and (Geo)Json files. XML is currently not supported; we recommend [OpenRefine](https://github.com/OpenRefine/OpenRefine/wiki/Downloads) to convert these to either CSV or JSON.
 
-
 ### CitySDK::API Usage
 
-
-CALL                                  | Description
+Call                                   | Description
 |:-------------------------------------|:-------------------------------------------------------
 `@api = CitySDK::API.new('<endpoint IP>')` | Establish a link to the particular endpoint
 `@api.authenticate('<name>','<password>')` | For reading this is not necessary, for writing and deleting you need to authenticate. The authentication times out if not used (is reset by writing to the API). This call takes an optional  block for immediate automatic release when the block has been called.
-`@api.get path` | Simply issue a 'GET' against the API; the results are returned as a ruby Hash.
-`@api.put path,data` | PUT a resource; see [Documentation](https://github.com/waagsociety/citysdk-ld/wiki/Objects)
-`@api.post path,data` | POST a resource; see [Documentation](https://github.com/waagsociety/citysdk-ld/wiki/Objects)
-`@api.patch path,data` | PATCH a resource; see [Documentation](https://github.com/waagsociety/citysdk-ld/wiki/Objects)
+`@api.get path` | Simply issue a 'GET' against the API; the results are returned as a Ruby Hash.
+`@api.put path,data` | PUT a resource; see [documentation](https://github.com/waagsociety/citysdk-ld/wiki/Objects)
+`@api.post path,data` | POST a resource; see [documentation](https://github.com/waagsociety/citysdk-ld/wiki/Objects)
+`@api.patch path,data` | PATCH a resource; see [documentation](https://github.com/waagsociety/citysdk-ld/wiki/Objects)
 `@api.delete path` | Issue a DELETE to the API; path points to the resource to delete. Requires authentication.
-`@api.release` | This call will flush any buffered object that were scheduled to be created or updated. See also batch_size, below.
+`@api.release` | This call will flush any buffered object that were scheduled to be created or updated. See also `batch_size`, below.
 `@api.set_layer(layername)` | Set the the layer for subsequent 'create_object' calls. User must be authenticate for this layer in order to sucesfully create objects.
 `@api.create_object(objhash)` | Adds an object to the database. See also batch_size, below.
 `@api.layers` | Shortcut for `get '/layers'`
 `@api.owners` | Shortcut for `get '/owners'`
 `@api.objects(layer=nil)` | Shortcut for `get '/objects'`. When a layer name is supplied, returns only objects from with data on this layer, and the layerdata itself.
 `@api.next` | When more results are available, returns the next page.
-`@api.format = <format>` | Specify the output format. Currently supported are (Geo)JSON and (Geo)JSON-LD.
-`@api.page_size = <page_size>` | Specify the number of features returned; default is 25.
+`@api.format = <format>` | Specify the output format. Currently supported are `json-ld` and `json`.
+`@api.per_page = <per_page>` | Specify the number of features returned; default is 25.
 `@api.batch_size = <n>` | When adding objects thorugh 'create_object', they are buffered until '<n>' objects are available, then a single call is issuesd to the API.
 `@api.last_result` | Returns a Hash with the last HTTP status and the headers returned from the last call.
 
-
-
-
-
-
-
-
 ## Contributing
 
 1. Fork it

data/lib/citysdk/api.rb

@@ -1,16 +1,8 @@
 require 'json'
 require 'faraday'
 
-
-
 module CitySDK
 
-  def log(m)
-    File.open(File.expand_path('~/csdk.log'), "a")  do |f|
-      f.write(m + "\n")
-    end
-  end
-  
   class HostException < ::Exception
   end
 
@@ -18,35 +10,30 @@ module CitySDK
     attr_reader   :last_result
     attr_reader   :error
     attr_accessor :batch_size
-    attr_accessor :page_size
+    attr_accessor :per_page
     attr_accessor :format
 
-    @@create_tpl =  {
-      type: "FeatureCollection",
-      features: []
-    }
-
-    def initialize(host, port=nil)
+    def initialize(host)
       @error = '';
       @layer = '';
       @batch_size = 1000;
-      @page_size = 25;
+      @per_page = 25;
       @format = 'jsonld'
       @updated = @created = 0;
-      set_host(host,port)
+      set_host(host)
     end
-    
-    def authenticate(n,p)
-      @name = n;
-      @passw = p;
 
-      resp = @connection.get '/session', { :name => @name, :password => @passw }
-      if resp.status.between?(200, 299) 
-        resp = CitySDK::parseJson(resp.body)
+    def authenticate(name, password)
+      @name = name;
+      @password = password;
+
+      resp = @connection.get '/session', { name: @name, password: @password }
+      if resp.status.between?(200, 299)
+        resp = CitySDK::parse_json(resp.body)
         if (resp.class == Hash) and resp[:session_key]
           @connection.headers['X-Auth'] = resp[:session_key]
         else
-          raise Exception.new("Invalid credentials")
+          raise Exception.new('Invalid credentials')
         end
       else
         raise Exception.new(resp.body)
@@ -61,55 +48,41 @@ module CitySDK
       true
     end
 
-    def set_host(host,port=nil)
-      @host = host
-      @port = port
-      
-      @host.gsub!(/^http(s)?:\/\//,'')
-
-      if port.nil?
-        if host =~ /^(.*):(\d+)$/
-          @port = $2
-          @host = $1
-        else
-          @port = 80
-        end
-      end
-      
-      if !($nohttps or @host =~ /.+\.dev/ or @host == 'localhost' or @host == '127.0.0.1' or @host == '0.0.0.0')
-        @connection = Faraday.new :url => "https://#{@host}", :ssl => {:verify => false }
+    def set_host(host)
+      if host.index('https') == 0
+        @connection = Faraday.new url: host, ssl: { verify: false }
       else
-        @connection = Faraday.new :url => "http://#{@host}:#{@port}"
+        @connection = Faraday.new url: host
       end
+
       @connection.headers = {
-        :user_agent => 'CitySDK_API GEM ' + CitySDK::VERSION,
+        :user_agent => 'CitySDK LD API GEM ' + CitySDK::VERSION,
         :content_type => 'application/json'
       }
-      begin 
+      begin
         get('/')
       rescue Exception => e
-        raise CitySDK::Exception.new("Trouble connecting to api @ #{host}")
+        raise CitySDK::Exception.new("Trouble connecting to API @ #{host}")
       end
-      @create = @@create_tpl
+      @create =  {
+        type: "FeatureCollection",
+        features: []
+      }
     end
-    
-    def addFormat(path)
+
+    def add_format(path)
       if path !~ /format/
         path = path + ((path =~ /\?/) ? "&" : "?") + "format=#{@format}"
       end
       return path if path =~ /per_page/
-      path + "&per_page=#{@page_size}"
-    end
-
-    def set_createTemplate(ctpl) 
-      ctpl[:features] = []
-      @create = @@create_tpl = ctpl
+      path + "&per_page=#{@per_page}"
     end
 
     def set_layer(l)
+      create_flush
       @layer = l
     end
-  
+
     def next
       @next ? get(@next) : "{}"
     end
@@ -130,85 +103,72 @@ module CitySDK
       @create[:features] << n
       create_flush if @create[:features].length >= @batch_size
     end
-  
+
     def authorized?
-     !! @connection.headers['X-Auth'] 
+     !! @connection.headers['X-Auth']
     end
 
     def release
-      create_flush # send any remaining entries in the create buffer
+      # send any remaining entries in the create buffer
+      create_flush
       if authorized?
         resp = @connection.delete('/session')
         if resp.status.between?(200, 299)
           @connection.headers.delete('X-Auth')
         else
-          @error = CitySDK::parseJson(resp.body)[:error]
+          @error = CitySDK::parse_json(resp.body)[:error]
           raise HostException.new(@error)
         end
       end
       return {created: @created}
     end
-  
+
     def delete(path)
-      if authorized? 
-        resp = @connection.delete(addFormat(path))
-        if resp.status.between?(200, 299)
-          @last_result = { status: resp.status, headers: resp.headers }
-          return (resp.body and resp.body !~ /\s*/) ? CitySDK::parseJson(resp.body) : ''
-        end
-        @error = CitySDK::parseJson(resp.body)[:error]
-        raise HostException.new(@error)
-      end
-      raise CitySDK::Exception.new("DELETE needs authorization.")
+      write :delete, path
     end
-  
-    def post(path,data)
-      if authorized? 
-        resp = @connection.post(addFormat(path),data.to_json)
-        @last_result = { status: resp.status, headers: resp.headers }
-        return CitySDK::parseJson(resp.body) if resp.status.between?(200, 299)
-        @error = resp.body # CitySDK::parseJson(resp.body)[:error]
 
-        File.open(File.expand_path("~/post_error_data.json"),"w") do |fd|
-          fd.write(JSON.pretty_generate({error: @error, data: data}))
-        end
+    def post(path, data)
+      write :post, path, data
+    end
 
-        raise HostException.new(@error)
-      end
-      raise CitySDK::Exception.new("POST needs authorization.")
+    def patch(path, data)
+      write :patch, path, data
     end
 
-    def patch(path,data)
-      if authorized? 
-        resp = @connection.patch(addFormat(path),data.to_json)
-        @last_result = { status: resp.status, headers: resp.headers }
-        return CitySDK::parseJson(resp.body) if resp.status.between?(200, 299)
-        @error = CitySDK::parseJson(resp.body)[:error]
-        raise HostException.new(@error)
-      end
-      raise CitySDK::Exception.new("PATCH needs authorization.")
+    def put(path, data)
+      write :put, path, data
     end
 
-    def put(path,data)
-      if authorized? 
-        resp = @connection.put(addFormat(path),data.to_json)
+    def write(method, path, data = nil)
+      if authorized?
+        payload = data ? data.to_json : nil
+        resp = @connection.send(method, path, payload)
         @last_result = { status: resp.status, headers: resp.headers }
-        return CitySDK::parseJson(resp.body) if resp.status.between?(200, 299)
-        @error = CitySDK::parseJson(resp.body)[:error] || {status: resp.status}
+
+        if resp.status == 401 and @name and @password
+          # API was authenticated before, so probably timed out. Try again!
+          if authenticate(@name, @password)
+            resp = @connection.send(method, path, payload)
+            @last_result = { status: resp.status, headers: resp.headers }
+          end
+        end
+
+        return CitySDK::parse_json(resp.body) if resp.status.between?(200, 299)
+        @error = CitySDK::parse_json(resp.body)[:error] || {status: resp.status}
         raise HostException.new(@error)
       end
-      raise CitySDK::Exception.new("PUT needs authorization.")
+      raise CitySDK::Exception.new("#{method.upcase} needs authorization")
     end
 
     def get(path)
-      resp = @connection.get(addFormat(path))
+      resp = @connection.get(add_format(path))
       @next = (resp.headers['Link'] =~ /^<(.+)>;\s*rel="next"/) ? $1 : nil
       @last_result = { status: resp.status, headers: resp.headers }
-      return CitySDK::parseJson(resp.body) if resp.status.between?(200, 299)
-      @error = CitySDK::parseJson(resp.body)[:error]
+      return CitySDK::parse_json(resp.body) if resp.status.between?(200, 299)
+      @error = CitySDK::parse_json(resp.body)[:error]
       raise HostException.new(@error)
     end
-    
+
     def create_flush
       if @create[:features].length > 0
         tally post("/layers/#{@layer}/objects",@create)
@@ -219,8 +179,7 @@ module CitySDK
     def tally(res)
       @created += res.length
     end
-    
+
   end
 
 end
-

data/lib/citysdk/file_reader.rb

@@ -8,48 +8,48 @@ require 'tmpdir'
 module CitySDK
 
   class FileReader
-    
+
     RE_Y = /lat|(y.*coord)|(y.*pos.*)|(y.*loc(atie|ation)?)/i
     RE_X = /lon|lng|(x.*coord)|(x.*pos.*)|(x.*loc(atie|ation)?)/i
     RE_GEO = /^(geom(etry)?|location|locatie|coords|coordinates)$/i
     RE_NAME = /(title|titel|naam|name)/i
     RE_A_NAME = /^(naam|name|title|titel)$/i
-    
+
     attr_reader :file, :content,:params
 
     def initialize(pars)
       @params = pars
       file_path = File.expand_path(@params[:file_path])
       if File.extname(file_path) == '.csdk'
-        readCsdk(file_path)
+        read_csdk(file_path)
       else
         ext = @params[:originalfile] ? File.extname(@params[:originalfile]) : File.extname(file_path)
         case ext
           when /\.zip/i
-            readZip(file_path)
+            read_zip(file_path)
           when /\.(geo)?json/i
-            readJSON(file_path)
+            read_json(file_path)
           when /\.shp/i
-            readShape(file_path)
+            read_shapefile(file_path)
           when /\.csv|tsv/i
-            readCsv(file_path)
+            read_csv(file_path)
           when /\.csdk/i
-            readCsdk(file_path)
+            read_csdk(file_path)
           else
             raise "Unknown or unsupported file type: #{ext}."
         end
       end
-      
+
       @params[:rowcount] = @content.length
-      getFields unless @params[:fields]
-      guessName unless @params[:name]
-      guessSRID unless @params[:srid]
-      findUniqueField  unless @params[:unique_id]
-      getAddress unless @params[:hasaddress]
-      setId_Name
+      get_fields unless @params[:fields]
+      guess_name unless @params[:name]
+      guess_srid unless @params[:srid]
+      find_unique_field  unless @params[:unique_id]
+      get_address unless @params[:hasaddress]
+      set_id_name
     end
-    
-    def getAddress
+
+    def get_address
       pd = pc = hn = ad = false
       @params[:housenumber] = nil
       @params[:hasaddress] = 'unknown'
@@ -67,7 +67,7 @@ module CitySDK
 
     end
 
-    def setId_Name
+    def set_id_name
       count = 123456
       if @params[:unique_id]
         @content.each do |h|
@@ -86,8 +86,8 @@ module CitySDK
         end
       end
     end
-    
-    def findUniqueField
+
+    def find_unique_field
       fields = {}
       @params[:unique_id] = nil
       @content.each do |h|
@@ -103,10 +103,10 @@ module CitySDK
           break
         end
       end
-      
+
     end
 
-    def guessName
+    def guess_name
       @params[:name] = nil
       @params[:fields].reverse.each do |k|
         if(k.to_s =~ RE_A_NAME)
@@ -119,7 +119,7 @@ module CitySDK
       end
     end
 
-    def getFields
+    def get_fields
       @params[:fields] = []
       @params[:original_fields] = []
       @params[:alternate_fields] = {}
@@ -130,10 +130,10 @@ module CitySDK
         @params[:alternate_fields][k] = k
       end
     end
-    
-    def guessSRID
+
+    def guess_srid
       return unless @content[0][:geometry] and @content[0][:geometry].class == Hash
-      @params[:srid] = 4326 
+      @params[:srid] = 4326
       g = @content[0][:geometry][:coordinates]
       if(g)
         while g[0].is_a?(Array)
@@ -141,19 +141,19 @@ module CitySDK
         end
         lon = g[0]
         lat = g[1]
-        # if lon > -180.0 and lon < 180.0 and lat > -90.0 and lat < 90.0 
-        #   @params[:srid] = 4326 
+        # if lon > -180.0 and lon < 180.0 and lat > -90.0 and lat < 90.0
+        #   @params[:srid] = 4326
         # else
         if lon.between?(-7000.0,300000.0) and lat.between?(289000.0,629000.0)
           # Dutch new rd system
           @params[:srid] = 28992
         end
-      else 
-        
+      else
+
       end
     end
-    
-    def findColSep(f)
+
+    def find_col_sep(f)
       a = f.gets
       b = f.gets
       [";","\t","|"].each do |s|
@@ -162,7 +162,7 @@ module CitySDK
       ','
     end
 
-    def isWkbGeometry(s)
+    def is_wkb_geometry?(s)
       begin
         f = GeoRuby::SimpleFeatures::GeometryFactory::new
         p = GeoRuby::SimpleFeatures::HexEWKBParser.new(f)
@@ -174,7 +174,7 @@ module CitySDK
       nil
     end
 
-    def isWktGeometry(s)
+    def is_wkt_geometry?(s)
       begin
         f = GeoRuby::SimpleFeatures::GeometryFactory::new
         p = GeoRuby::SimpleFeatures::EWKTParser.new(f)
@@ -187,7 +187,7 @@ module CitySDK
     end
 
     GEOMETRIES = ["point", "multipoint", "linestring", "multilinestring", "polygon", "multipolygon"]
-    def isGeoJSON(s)
+    def is_geo_json?(s)
       return nil if s.class != Hash
       begin
         if GEOMETRIES.include?(s[:type].downcase)
@@ -207,8 +207,8 @@ module CitySDK
       nil
     end
 
-    def geomFromText(coords)
-      # begin 
+    def geom_from_text(coords)
+      # begin
       #   a = factory.parse_wkt(coords)
       # rescue
       # end
@@ -219,14 +219,14 @@ module CitySDK
           coor = $2.gsub('(','[').gsub(')',']')
           coor = coor.gsub(/([-+]?[0-9]*\.?[0-9]+)\s+([-+]?[0-9]*\.?[0-9]+)/) { "[#{$1},#{$2}]" }
           coor = JSON.parse(coor)
-          return { :type => type, 
+          return { :type => type,
             :coordinates => coor }
         end
       end
       {}
     end
-    
-    def findGeometry(xfield=nil, yfield=nil)
+
+    def find_geometry(xfield=nil, yfield=nil)
       unless(xfield and yfield)
         @params[:hasgeometry] = nil
         xs = true
@@ -236,35 +236,35 @@ module CitySDK
           next if k.nil?
 
           if k.to_s =~ RE_GEO
-            
-            srid,g_type = isWkbGeometry(v)
+
+            srid,g_type = is_wkb_geometry?(v)
             if(srid)
               @params[:srid] = srid
               @params[:geomtry_type] = g_type
               @content.each do |h|
-                a,b,g = isWkbGeometry(h[:properties][:data][k])
+                a,b,g = is_wkb_geometry?(h[:properties][:data][k])
                 h[:geometry] = g
                 h[:properties][:data].delete(k)
               end
               @params[:hasgeometry] = k
               return true
             end
-            
-            
-            srid,g_type = isWktGeometry(v)
+
+
+            srid,g_type = is_wkt_geometry?(v)
             if(srid)
               @params[:srid] = srid
               @params[:geomtry_type] = g_type
               @content.each do |h|
-                a,b,g = isWktGeometry(h[:properties][:data][k])
+                a,b,g = is_wkt_geometry?(h[:properties][:data][k])
                 h[:geometry] = g
                 h[:properties][:data].delete(k)
               end
               @params[:hasgeometry] = k
               return true
             end
-            
-            srid,g_type = isGeoJSON(v)
+
+            srid,g_type = is_geo_json?(v)
             if(srid)
               @params[:srid] = srid
               @params[:geomtry_type] = g_type
@@ -275,9 +275,9 @@ module CitySDK
               @params[:hasgeometry] = k
               return true
             end
-            
+
           end
-          
+
           hdc = k.to_s.downcase
           if hdc == 'longitude' or hdc == 'lon' or hdc == 'x'
             xfield=k; xs=false
@@ -285,7 +285,7 @@ module CitySDK
           if hdc == 'latitude' or hdc == 'lat' or hdc == 'y'
             yfield=k; ys=false
           end
-          xfield = k if xs and (hdc =~ RE_X) 
+          xfield = k if xs and (hdc =~ RE_X)
           yfield = k if ys and (hdc =~ RE_Y)
         end
       end
@@ -305,19 +305,18 @@ module CitySDK
         # factory = ::RGeo::Cartesian.preferred_factory()
         @params[:hasgeometry] = "[#{xfield}]"
         @content.each do |h|
-          h[:geometry] = geomFromText(h[:properties][:data][xfield])
+          h[:geometry] = geom_from_text(h[:properties][:data][xfield])
           h[:properties][:data].delete(xfield) if h[:geometry]
         end
         @params[:geomtry_type] = ''
         @params[:fields].delete(xfield) if @params[:fields]
         return true
       end
-      
-      
+
       false
     end
-    
-    def readCsv(path)
+
+    def read_csv(path)
       @file = path
       c=''
       File.open(path, "r:bom|utf-8") do |fd|
@@ -330,12 +329,12 @@ module CitySDK
       c = c.force_encoding('utf-8')
       c = c.gsub(/\r\n?/, "\n")
       @content = []
-      @params[:colsep] = findColSep(StringIO.new(c)) unless @params[:colsep]
+      @params[:colsep] = find_col_sep(StringIO.new(c)) unless @params[:colsep]
       csv = CSV.new(c, :col_sep => @params[:colsep], :headers => true, :skip_blanks =>true)
       csv.header_convert { |h| h.blank? ? '_' : h.strip.gsub(/\s+/,'_')  }
       csv.convert { |h| h ? h.strip : '' }
       index = 0
-      begin 
+      begin
         csv.each do |row|
           r = row.to_hash
           h = {}
@@ -348,17 +347,17 @@ module CitySDK
       rescue => e
         raise CitySDK::Exception.new("Read CSV; line #{index}; #{e.message}")
       end
-      findGeometry
+      find_geometry
     end
 
-    def readJSON(path)
+    def read_json(path)
       @content = []
       @file = path
       raw = ''
       File.open(path, "r:bom|utf-8") do |fd|
         raw = fd.read
       end
-      hash = CitySDK::parseJson(raw)
+      hash = CitySDK::parse_json(raw)
 
       if hash.is_a?(Hash) and hash[:type] and (hash[:type] == 'FeatureCollection')
         # GeoJSON
@@ -382,44 +381,43 @@ module CitySDK
             end
           end
         end
-        
+
         if val
           val.each do |h|
             @content << { :properties => {:data => h} }
           end
         end
-        findGeometry
+        find_geometry
       end
     end
 
-    def sridFromPrj(str)
+    def srid_from_prj(str)
       begin
         connection = Faraday.new :url => "http://prj2epsg.org"
         resp = connection.get('/search.json', {:mode => 'wkt', :terms => str})
-        if resp.status.between?(200, 299) 
-          resp = CitySDK::parseJson resp.body
+        if resp.status.between?(200, 299)
+          resp = CitySDK::parse_json resp.body
           @params[:srid] = resp[:codes][0][:code].to_i
         end
       rescue
       end
     end
 
-    def readShape(path)
-      
+    def read_shapefile(path)
+
       @content = []
       @file = path
-      
+
       prj = path.gsub(/.shp$/i,"") + '.prj'
       prj = File.exists?(prj) ? File.read(prj) : nil
-      sridFromPrj(prj) if (prj and @params[:srid].nil?)
-      
+      srid_from_prj(prj) if (prj and @params[:srid].nil?)
+
       @params[:hasgeometry] = 'ESRI Shape'
-      
-      
+
       GeoRuby::Shp4r::ShpFile.open(path) do |shp|
         shp.each do |shape|
           h = {}
-          h[:geometry] = CitySDK::parseJson(shape.geometry.to_json) #a GeoRuby SimpleFeature
+          h[:geometry] = CitySDK::parse_json(shape.geometry.to_json) #a GeoRuby SimpleFeature
           h[:properties] = {:data => {}}
           att_data = shape.data #a Hash
           shp.fields.each do |field|
@@ -431,17 +429,17 @@ module CitySDK
         end
       end
     end
-    
-    def readCsdk(path)
+
+    def read_csdk(path)
       h = Marshal.load(File.read(path))
       @params = h[:config]
       @content = h[:content]
-    end    
-    
-    def readZip(path)
-      begin 
-        Dir.mktmpdir("cdkfi_#{File.basename(path).gsub(/\A/,'')}") do |dir| 
-          raise CitySDK::Exception.new("Error unzipping #{path}.", {:originalfile => path}, __FILE__,__LINE__) if not system "unzip '#{path}' -d '#{dir}' > /dev/null 2>&1"
+    end
+
+    def read_zip(path)
+      begin
+        Dir.mktmpdir("cdkfi_#{File.basename(path).gsub(/\A/,'')}") do |dir|
+          raise CitySDK::Exception.new("Error unzipping #{path}.", {:originalfile => path}, __FILE__, __LINE__) if not system "unzip '#{path}' -d '#{dir}' > /dev/null 2>&1"
           if File.directory?(dir + '/' + File.basename(path).chomp(File.extname(path)))
             dir = dir + '/' + File.basename(path).chomp(File.extname(path) )
           end
@@ -449,21 +447,21 @@ module CitySDK
             next if f =~ /^\./
             case File.extname(f)
               when /\.(geo)?json/i
-                readJSON(dir+'/'+f)
+                read_json(dir+'/'+f)
                 return
               when /\.shp/i
-                readShape(dir+'/'+f)
+                read_shapefile(dir+'/'+f)
                 return
               when /\.csv|tsv/i
-                readCsv(dir+'/'+f)
+                read_csv(dir+'/'+f)
                 return
             end
           end
         end
       rescue Exception => e
-        raise CitySDK::Exception.new(e.message, {:originalfile => path}, __FILE__,__LINE__)
+        raise CitySDK::Exception.new(e.message, {:originalfile => path}, __FILE__, __LINE__)
       end
-      raise CitySDK::Exception.new("Could not proecess file #{path}", {:originalfile => path}, __FILE__,__LINE__)
+      raise CitySDK::Exception.new("Could not proecess file #{path}", {:originalfile => path}, __FILE__, __LINE__)
     end
 
     def write(path=nil)
@@ -480,6 +478,6 @@ module CitySDK
     end
 
   end
-  
+
 end
 

data/lib/citysdk/importer.rb

@@ -1,14 +1,14 @@
 
 module CitySDK
-  
+
   class Importer
     attr_reader :filereader, :api, :params
-    
+
     def initialize(pars, fr = nil)
       @params = pars
-      
+
       raise Exception.new("Missing :host in Importer parameters.") if @params[:host].nil?
-      raise Exception.new("Missing :layername in Importer parameters.") if @params[:layername].nil?
+      raise Exception.new("Missing :layer in Importer parameters.") if @params[:layer].nil?
       raise Exception.new("Missing :file_path in Importer parameters.") if @params[:file_path].nil?
 
       @api = CitySDK::API.new(@params[:host])
@@ -20,16 +20,16 @@ module CitySDK
 
       @params[:addresslayer] = 'bag.vbo' if @params[:addressleyer].nil?
       @params[:addressfield] = 'postcode_huisnummer' if @params[:addressfield].nil?
-      
+
       @filereader = fr || FileReader.new(@params)
     end
-    
-    
+
+
     def write(path)
       return @filereader.write(path)
     end
-    
-    def setParameter(k,v)
+
+    def set_parameterr(k,v)
       begin
         @params[(k.to_sym rescue k) || k] = v
         return true
@@ -42,7 +42,7 @@ module CitySDK
       begin
         sign_out if @signed_in
         @api.set_host(@params[:host])
-        @api.set_layer(@params[:layername])
+        @api.set_layer(@params[:layer])
         @api.authenticate(@params[:name],@params[:password])
         @signed_in = true
       rescue => e
@@ -56,65 +56,58 @@ module CitySDK
       @signed_in = false
       return @api.release
     end
-  
-    def filterFields(h)
+
+    def filter_fields(h)
       data = {}
       h.each_key do |k|
         k = (k.to_sym rescue k) || k
         j = @params[:alternate_fields][k]
-        data[j] = h[k] if @params[:fields].include?(k)
+        data[j] = h[k] if @params[:fields].include?(k) and j
       end
       data
     end
-    
-    
-    def geomToJSON(a)
-      # puts JSON.pretty_generate(a)
-      a
-    end
-    
-    def doImport(&block)
+
+    def do_import(&block)
       result = {
-        :created => 0,
-        :not_added => 0
+        created: 0,
+        not_added: 0
       }
-      
+
       failed = nil
-      
-#      if @params[:hasaddress] == 'certain'
-#        failed = addToAddress(&block)
-#      end
-      
+
       # TODO: add possibility to add node to postal code
+      # if @params[:hasaddress] == 'certain'
+      #   failed = add_to_address(&block)
+      #  end
 
       if failed == []
         result[:updated] += @filereader.content.length
         return result
       end
-      
-      if failed 
+
+      if failed
         result[:updated] += (@filereader.content.length - failed.length)
       end
-      
+
       objects = failed || @filereader.content
       count = objects.length
 
       begin
         sign_in
-        
+
         if @params[:hasgeometry]
           begin
             objects.each do |record|
-              
+
               node = {
                 type: 'Feature',
                 properties: record[:properties],
-                geometry: geomToJSON(record[:geometry]),
+                geometry: record[:geometry],
               }
-              
-              node[:properties][:data] = filterFields(record[:properties][:data])
+
+              node[:properties][:data] = filter_fields(record[:properties][:data])
               node[:crs] = { type: 'EPSG', properties: { code: @params[:srid]  } } if @params[:srid] != 4326
-              
+
               node[:properties][:title] = node[:properties][:data][@params[:title]] if @params[:title]
 
               yield(node[:properties]) if block_given?
@@ -139,7 +132,7 @@ module CitySDK
       return result
     end
 
-    def addToAddress()
+    def add_to_address()
       failed = []
       if @params[:postcode] and @params[:housenumber]
         begin
@@ -160,8 +153,8 @@ module CitySDK
               qres[:status]='nix'
             end
             if qres[:status]=='success' and qres[:results] and qres[:results][0]
-              url = '/' + qres[:results][0][:cdk_id] + '/' + @params[:layername]
-              data = filterFields(row)
+              url = '/' + qres[:results][0][:cdk_id] + '/' + @params[:layer]
+              data = filter_fields(row)
               yield({addto: qres[:results][0][:cdk_id], data: data}) if block_given?
               n = @api.put(url,{'data'=>data})
             else

data/lib/citysdk/util.rb

@@ -21,13 +21,12 @@ end
 
 
 module CitySDK
-  
+
   # for debugging purposes...
   def jsonlog(o)
     puts JSON.pretty_generate({ o.class.to_s => o })
   end
 
-
   class Exception < ::Exception
     def initialize(message,parms=nil,srcfile=nil,srcline=nil)
       if parms and srcfile and srcline
@@ -41,11 +40,11 @@ module CitySDK
     end
   end
 
-  def self.parseJson(jsonstring)
+  def self.parse_json(str)
     begin
-      return jsonstring.blank? ? {} : JSON.parse(jsonstring,symbolize_names: true)
+      return str.blank? ? {} : JSON.parse(str, symbolize_names: true)
     rescue Exception => e
-      raise CitySDK::Exception.new("#{e.message}; input: #{jsonstring}")
+      raise CitySDK::Exception.new("#{e.message}; input: #{str}")
     end
   end
 

data/lib/citysdk.rb

@@ -1,9 +1,8 @@
-require 'citysdk/util.rb'
-require 'citysdk/api.rb'
-require 'citysdk/file_reader.rb'
-require 'citysdk/importer.rb'
+require_relative './citysdk/util.rb'
+require_relative './citysdk/api.rb'
+require_relative './citysdk/file_reader.rb'
+require_relative './citysdk/importer.rb'
 
 module CitySDK
-  VERSION = "1.0.1"
+  VERSION = '1.0.2'
 end
-

data/spec/api_spec.rb

@@ -1,5 +1,4 @@
 
-
 module CitySDK
   TEST_HOST = 'localhost:9292'
   # TEST_HOST = 'test-api.citysdk.waag.org'
@@ -20,14 +19,11 @@ module CitySDK
     licence: "CC0",
     fields: []
   }
-  
+
   $api = API.new(TEST_HOST)
 
-  
-  
   describe API do
 
-
     it "can be connected to" do
       expect($api.class).to be(CitySDK::API)
     end
@@ -67,5 +63,5 @@ module CitySDK
     end
 
   end
-  
+
 end

data/spec/import_spec.rb

@@ -7,7 +7,7 @@ module CitySDK
     def newImporter(f)
       @importer = Importer.new({
         file_path: f,
-        host: TEST_HOST, 
+        host: TEST_HOST,
         layername: TEST_LAYER[:name]
       })
     end
@@ -30,9 +30,9 @@ module CitySDK
     it "can make a layer and delete" do
 
       newImporter('./spec/files/hotels.csv')
-      
-      @importer.setParameter(:name,TEST_USER)
-      @importer.setParameter(:password,TEST_PASS)
+
+      @importer.set_parameter(:name,TEST_USER)
+      @importer.set_parameter(:password,TEST_PASS)
       @importer.sign_in
       res = @importer.api.post('/layers',TEST_LAYER)
       expect( (!!res[:features] and res[:features].length == 1) ).to be(true)
@@ -43,18 +43,18 @@ module CitySDK
       res = @importer.api.get("/layers/test.rspec/objects")
       expect( (!!res[:features] and res[:features].length == 0) ).to be(true)
 
-      @importer.api.authenticate(TEST_USER,TEST_PASS) do 
+      @importer.api.authenticate(TEST_USER,TEST_PASS) do
         expect( @importer.api.delete("/layers/#{TEST_LAYER[:name]}") ).to eq({})
       end
 
     end
 
   end
-  
+
   describe "FileReader" do
 
     it "can parse json" do
-      j = CitySDK::parseJson('{ "arr" : [0,1,1,1], "hash": {"aap": "noot"}, "num": 0 }')
+      j = CitySDK::parse_json('{ "arr" : [0,1,1,1], "hash": {"aap": "noot"}, "num": 0 }')
       expect(j[:arr].length).to be(4)
       expect(j[:num].class).to be(Fixnum)
       expect(j[:hash][:aap]).to eq("noot")

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: citysdk
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 1.0.2
 platform: ruby
 authors:
 - Tom Demeyer
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-12-24 00:00:00.000000000 Z
+date: 2014-12-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: dbf
@@ -98,7 +98,7 @@ description: Encapsulates the CitySDK api.
 email:
 - tom@waag.org
 executables:
-- csdk
+- citysdk-cli
 extensions: []
 extra_rdoc_files: []
 files:
@@ -108,9 +108,8 @@ files:
 - LICENSE.txt
 - README.md
 - Rakefile
-- bin/csdk
+- bin/citysdk-cli
 - citysdk.gemspec
-- examples/simple_api.rb
 - lib/citysdk.rb
 - lib/citysdk/api.rb
 - lib/citysdk/file_reader.rb

data/examples/simple_api.rb

@@ -1,68 +0,0 @@
-#!/usr/bin/env ruby
-
-# check the wiki for detailed api usage: https://github.com/waagsociety/citysdk-ld/wiki
-
-require 'citysdk'
-include CitySDK
-
-api = API.new('api.citysdk.waag.org') # point to a hosted instance of the api.
-
-# simple GET
-# GET requests do not need authentication
-
-endpoint = api.get '/'
-puts "Enpoint is: #{endpoint[:features][0][:properties][:title]}."
-
-first10layers = api.get('/layers')
-puts "First layer: #{JSON.pretty_generate(first10layers[:features][0])}"
-
-# find out how many owners the endpoint knows
-owners = api.get('/owners?per_page=1&count')
-puts "Number of data maintaners: #{api.last_result[:headers]['x-result-count']}"
-
-# authenticate for write actions.
-unless api.authenticate('<name>','<passw>')
-  puts "Did not authenticate..."
-  exit 
-end
-  
-# make a layer
-# everybody can wite to the temporary 'test' domain: 
-
-layer = {
-  name: "test.cities",
-  owner: "citysdk",
-  title: "Cities: 🏠🏢🏣🏨🏫🏬🏭🏡",
-  description: "Cities big and small.",
-  data_sources: [ "http://fantasy.com" ],
-  authoritative: false,
-  rdf_type: "dbp:City",
-  category: "administrative",
-  subcategory: "cities",
-  licence: "CC0"
-}
-
-api.post('/layers',layer)
-
-  # add data to this layer
-  # attach to the node representing the city of Rotterdam
-object = {type: "Feature", 
-          geometry: {
-            type: 'Point',
-            coordinates: [4.4646,51.9222] }, 
-            properties: {
-              id: 'Rotterdam', 
-              title: 'Rotterdam', 
-              data: {
-                a: 1, 
-                b: 2
-              }
-            }
-         }
-  
-api.post('/layers/test.cities/objects', object)
-
-
-# don't forget to release! this will also send 'unfilled' batches to the backend.
-api.release
-