chawk 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 594fd4f014cdc8821cd8feef0648fb62c55ad867
-  data.tar.gz: 3fb8f4e6e0d0c5721abeb9f754da547f0a7b53be
+  metadata.gz: 75c272978a418036f10a0f03297b48af7f40b7f4
+  data.tar.gz: 44b65c108649fb2fd8ab58f337c919ad02e7e959
 SHA512:
-  metadata.gz: d4861cad5119b155663c60c3c70acaa0063fb022a4153b74d547442ddf450fcee2d2f3ae0b9189bfeaa51cd7b917065d54282cdc51c8a2414ffcb97dbaebcbd4
-  data.tar.gz: acb26f7f49583c10175e4ec31fe95bbfa36f9f2234688fc49c2e460f8bd0ec559ae5aff60a53bdf20d2c174ce5c77d73cd5945124f5de2c7514d638fef64bfbe
+  metadata.gz: 72f85f23a8d05e8adc1308e389796fb605867d6f9126655744e03c04ae9f9bc298a302418078f1ab3e2bba57c56d3b34e4b747a9c6212912a84fcc10c83fdf90
+  data.tar.gz: 05640ad20ba4b52f3671d7a1ffeb401f4445b9a2edc4c5bafb76794ebdf35241e5b7f36a4508aa840daa43d0279582850fcd551908716e87bd1b80997440db65

data/.pryrc

@@ -3,8 +3,20 @@ $:.unshift('lib')
 require 'active_record'
 require 'hirb'
 require 'chawk'
+require 'pp'
 
 
-Hirb.enable
-ActiveRecord::Base.logger = Logger.new(STDOUT)
-ActiveRecord::Base.establish_connection ENV["TEST_DATABASE_URL"]
+ActiveRecord::Base.establish_connection adapter: "sqlite3", database: ":memory:"
+ActiveRecord::Migration.verbose = false
+require "chawk/migration"
+CreateChawkBase.migrate :up
+File.open('./test/schema.rb', "w") do |file|
+	ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
+end
+
+agent =  Chawk::Models::Agent.first || Chawk::Models::Agent.create(:name=>"Test User")
+node1 = Chawk.node(agent,'a:b')
+node1.points.destroy_all
+
+ts = Time.now.to_f
+

data/CHANGES

@@ -6,4 +6,11 @@ Switched to ActiveRecord, removed DataMapper
 -- 0.1.17 --
 Points can now be passed as strings representing integers
 -- 0.1.18 --
-metadata is stored as json (must be passed as a hash)
+metadata is stored as json (must be passed as a hash)
+-- 0.2.0 --
+Semantic Versioning
+-- 0.3.0 --
+Values are now working with Points.
+Recalculating / self invalidating Ranges added
+Statistical analysis is functional, if not efficient.
+Finer grained security model

data/README.md

@@ -2,26 +2,17 @@
 
 
 [![Gem Version](https://badge.fury.io/rb/chawk.png)](http://badge.fury.io/rb/chawk)
-[![Build Status][BS img]][Build Status]
-[![Dependency Status][DS img]][Dependency Status]
-[![Code Climate][CC img]][Code Climate]
-
-[Build Status]: https://travis-ci.org/queuetue/chawk-gem
-[travis pull requests]: https://travis-ci.org/queuetue/chawk-gem/pull_requests
-[Dependency Status]: https://gemnasium.com/queuetue/chawk-gem
-[Code Climate]: https://codeclimate.com/github/queuetue/chawk-gem
-
-[BS img]: https://travis-ci.org/queuetue/chawk-gem.png
-[DS img]: https://gemnasium.com/queuetue/chawk-gem.png
-[CC img]: https://codeclimate.com/github/queuetue/chawk-gem.png
-[CS img]: https://coveralls.io/repos/queuetue/chawk/badge.png?branch=master
+[![Build Status](https://travis-ci.org/queuetue/chawk-gem.svg)](https://travis-ci.org/queuetue/chawk-gem)
+[![Dependency Status](http://img.shields.io/gemnasium/queuetue/chawk-gem.svg)](https://gemnasium.com/queuetue/chawk-gem)
+[![Code Climate](http://img.shields.io/codeclimate/github/queuetue/chawk-gem.svg)](https://codeclimate.com/github/queuetue/chawk-gem)
+[![License](http://img.shields.io/:license-mit-blue.svg)](http://doge.mit-license.org)
 
 ## Description
 Chawk is a database agnostic time-series database written in Ruby.
 
-It tracks both both points (Integers) and values (String) in seperate datastores, and will eventually provide statistical and aggregate tools for numeric data.
+It tracks points (Integers) and will eventually provide statistical and aggregate tools for numeric data.
 
-This is the gem that powers the soon-to-be server, chawk-server
+This is the gem that powers the server, [Chawkolate](http://www.github.com/queuetue/chawkolate "Chawkolate at Github"). 
 
 Docs at [Queuetue.com](http://queuetue.com/Chawk "queuetue.com")
 
@@ -45,76 +36,93 @@ Or install it yourself as:
 Setup
 
     require 'chawk'
-    Chawk.setup 'sqlite::memory:'
+	ActiveRecord::Base.logger = Logger.new(STDOUT)
+	ActiveRecord::Base.establish_connection adapter: "sqlite3", database: ":memory:"
+
+The first time using a new database (Like this sqlite memory one that is destroyed at program exit) you should call: 
 
-DO NOT DO THIS ON AN EXISTING DATABASE, but the first time using a new database (Like this sqlite memory one that is destroyed at program exit) you should call: 
+	require "chawk/migration"
+	CreateChawkBase.migrate :up
+	File.open('./test/schema.rb', "w") do |file|
+		ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
+	end
 
-    DataMapper.auto_upgrade!
+Or, setup activerecord and manage migrations however you usually do.  (Rails will handle this for you, using the chawk-rails gem)
 
-All Chawk data operations require an Agent.  This can be used as the main actor in your code, or can be a proxy for your own User, etc through the foreign_id property.
+Chawk has a permissions model, which provides a framework for implementors to build a robust security model around, but it does not prevent implementors from overriding it.
 
-    agent = Chawk::Models::Agent.first(name:"Steve Austin") || Chawk::Models::Agent.new(name:"Steve Austin")
+Chawk's permissions begin with the Agent.  All Chawk data operations require an Agent.  This can be used as the main actor in your implementation code, or can be a proxy for your own User, etc through the foreign_id property.
 
-All data operations are performed through an Addr object.
+    agent = Chawk::Models::Agent.where(name:"Steve Austin").first || Chawk::Models::Agent.new(name:"Steve Austin")
 
-    addr = Chawk.addr(agent,"inventory/popcorn")
+All data operations are performed through an Node object, which requires an agent.
 
-The Addr object has two store objects - values and points.  **Points** are integers and allow mathematical and statistical operations. **Values** are strings and are intended for storing informational or serialized time series data.:
+    node = Chawk.node(agent,"inventory:popcorn")
 
-    addr.values << "This is a test."
-    addr.values.last
-    =>  #<Chawk::Models::Value @id=...
-    		@observed_at=... 
-			@recorded_at=#<DateTime: ...> 
-			@meta=nil 
-			@value="This is a test." 
-			@node_id=... 
-			@agent_id=...>
+Chawk.add assumes you are requesting full permissions, but you can specifically request :read, :write, :admin, or :full, which will allow specific operations and deny others.  If a node does not exist when requested, it will be created and the current agen will be given full (read write and admin) permissions.
 
-	addr.values << ["AND","SO","IS","THIS"]
-    addr.values.last
-	 => #<Chawk::Models::Value ... @value="THIS">
-	addr.values.last(10)
-	=> [#<Chawk::Models::Value ... @value="This is a test.">, 
-		#<Chawk::Models::Value ... @value="AND">, 
-		#<Chawk::Models::Value ... @value="SO">, 
-		#<Chawk::Models::Value ... @value="IS">, 
-		#<Chawk::Models::Value ... @value="THIS">]
+    node = Chawk.node(agent,"inventory:popcorn", :read)
 
-Addr can also return ranges from the past using the range method or the last method:
+Giving (or taking) permissions from an Node can be done with the set_permissions method:
 
-	addr.values.range(Time.now-2000,Time.now-1000)
-	=> [#<Chawk::Models::Value ... @value="ROCK">, 
-		#<Chawk::Models::Value ... @value="AROUND">]
+	node.set_permissions(agent, read, write, admin)
 
-	addr.values.since(Time.now-1000)
-	=> [#<Chawk::Models::Value ... @value="THE.">, 
-		#<Chawk::Models::Value ... @value="CLOCK">]
+Setting all three to false removes the Node from the list of the agent's nodes.
 
-These same methods also work for points:
+Nodes can also be given public read and write permissions, which allow agents without relationships to the Node to manipulate it. The methods set_public_read(bool) and set_public_write(bool) set and remove these public permissions.
 
-	addr.points << [10,9,8,7,6,5]
-	addr.points.last
+The Node object stores and protects points.  Points are integers and allow mathematical and statistical operations. 
+
+	node.add_points [10,9,8,7,6,5]
+	node.points.last
 	=> #<Chawk::Models::Point ... @value=5>
-	addr.points.last(2)
+	node.points.last(2)
 	=> [#<Chawk::Models::Point ... @value=6>, #<Chawk::Models::Point ... @value=5>]
 
 Points can also use the increment and decrement operators
 
-	addr.points.last
+	node.points.last
 	=> #<Chawk::Models::Point ... @value=5>
-	addr.points + 10
-	addr.pointslast
+	node.points + 10
+	node.pointslast
 	=> #<Chawk::Models::Point ... @value=15>
 
-As well as max and min
-
-	addr.points.max
-	=> 15
-	addr + 10
-	addr.points.last
+Node can also return ranges from the past using the range method or the last method:
+
+	ts = Time.now
+	node._insert_point(0,ts-1000)
+	node._insert_point(1,ts-1000)
+	node._insert_point(2,ts-1000)
+	node._insert_point(5,ts-800)
+	node._insert_point(8,ts-200)
+	node._insert_point(9,ts-10)
+	node.points_range(ts-1001,ts).length
+	=> 6
+	node.points_range(ts-801,ts).length 
+	=>3
+	node.points_range(ts-201,ts).length 
+	=> 2
+	node.points_range(ts-11,ts).length 
 	=> 1
+	node.points_range(ts-1001,ts-999).length
+	=> 3
+
+## Chawk::Models::Range
+
+A Chawk::Models::Range object, (soon to be merged with the Chawk.range command) produces time-limited, quantized data sets prepared for viewing, with resolution to the quarter second (one beat).
+
+    range = Chawk::Models::Range.create(start_ts:1085.0,stop_ts:1140.0,beats:1,parent_node:node1)
+
+This will return all data from the Node parent_node in the range from timestamp 1085 to 1140, resampled to the quarter beat. (220 data points, no matter how many are actually present in the sample)  This will become a stable hidden node (accessable via Node.ranges) and will automatically rebuild itself if data within it's range changes.
+
+    range = Chawk::Models::Range.create(start_ts:1088.0,stop_ts:8100.0,beats:14400,parent_node:node1)
 
+This will return all data from the Node parent_node in the range from timestamp 1085 to 8100, resampled to the quarter beat. (2 data points, no matter how many are actually present in the sample)
+
+## Chawk::Models::NodeAggregator
+The NodeAggregator is a (currently expensive) object for aggregate calculations on a Node.  It's intended ot be use on a Range's data_node property, since doing aggregate math on an entire datase can be prohibitively expensive.
+
+In the future, the NodeAggregator will be replaced with a nonblocking concurrent object, paving the way for something like a distributed MapReduce solution.
 
 ## Contributing
 
@@ -124,6 +132,23 @@ As well as max and min
 4. Push to the branch => `git push origin my-new-feature`
 5. Create new Pull Request
 
+## Rights
+
+Limor Fried, also known as Ladayada of adafruit industries has suggested these rights for Internet of Things creators.
+They are published here to support fair and honest practices for data collection initiatives. [Original Link](http://www.nytimes.com/roomfordebate/2013/09/08/privacy-and-the-internet-of-things/a-bill-of-rights-for-the-internet-of-things)
+
+* Open is better than closed; this ensures portability between Internet of Things devices.
+
+* Consumers, not companies, own the data collected by Internet of Things devices.
+
+* Internet of Things devices that collect public data must share that data.
+
+* Users have the right to keep their data private.
+
+* Users can delete or back up data collected by Internet of Things devices.
+
+Chawk is designed with these ideals in mind.
+
 ## License
 
 Copyright (c) 2014 Scott Russell (queuetue@gmail.com / queuetue.com)

data/chawk.gemspec

@@ -22,10 +22,12 @@ Gem::Specification.new do |spec|
   
   spec.add_development_dependency "bundler", "~> 1.5"
   spec.add_development_dependency "rake"
-  spec.add_development_dependency('minitest', '5.3.1')
   spec.add_development_dependency('rack-test', "0.6.2")
   spec.add_development_dependency('json', "1.8.1")
   spec.add_development_dependency('simplecov', "0.8.2")
   spec.add_development_dependency('pg', "0.17.1")
   spec.add_development_dependency('sqlite3', "1.3.9")
+  # spec.add_development_dependency('pry')
+  # spec.add_development_dependency('pry-debugger')
+  # spec.add_development_dependency('pry-stack_explorer')
 end

data/lib/chawk/migration.rb

@@ -2,6 +2,27 @@
 
 class CreateChawkBase < ActiveRecord::Migration
 	def up
+		create_table "chawk_ranges", force: true do |t|
+			t.string "subkey"
+			t.integer "parent_node_id"
+			t.integer "data_node_id"
+			t.float  "start_ts"
+			t.float  "stop_ts"
+			t.integer  "beats"
+			t.integer  "default"
+			t.timestamp "expires"
+			t.timestamps
+		end
+
+		create_table "chawk_usage_records", force: true do |t|
+			t.integer "agent_id"
+			t.integer "node_id"
+			t.string  "description"
+			t.float  "time"
+			t.float  "space"
+			t.timestamps
+		end
+
 		create_table "chawk_agents", force: true do |t|
 			t.integer "foreign_id"
 			t.string  "name",       limit: 200

data/lib/chawk/version.rb

@@ -1,4 +1,4 @@
 module Chawk
 	# The current version of Chawk
-	VERSION = "0.2.0"
+	VERSION = "0.3.0"
 end

data/lib/chawk.rb

@@ -4,41 +4,61 @@ require 'models'
 
 # Chawk is a gem for storing and retrieving time seris data.
 module Chawk
-	def self.check_node_security(agent,node)
 
-		if node.public_read
-			return node
+	def self.check_node_relations_security(rel, access)
+		if (rel && (rel.read || rel.admin))
+			case access
+			when :read
+				(rel.read or rel.admin)
+			when :write
+				(rel.write or rel.write)
+			when :admin
+				rel.admin
+			when :full
+				(rel.read && rel.write && rel.admin)
+			end
+		end
+	end
+
+	def self.check_node_public_security(node, access)
+		case access
+		when :read
+			node.public_read == true
+		when :write
+			node.public_write == true
 		end
+	end
+
+	def self.check_node_security(agent,node,access=:full)
 
 		rel = node.relations.where(agent_id:agent.id).first
 
-		if (rel && (rel.read || rel.admin))
-			return node
-		else
-			raise SecurityError,"You do not have permission to access this node. #{agent} #{rel}"
-		end
+		return node if check_node_relations_security(rel,access) || check_node_public_security(node,access)
+
+		raise SecurityError,"You do not have permission to access this node. #{agent} #{rel} #{access}"
 	end
 
-	def self.find_or_create_node(agent,key)
+	def self.find_or_create_node(agent,key,access=:full)
 		#TODO also accept regex-tested string
 		raise(ArgumentError,"Key must be a string.") unless key.is_a?(String)
 
 		node = Chawk::Models::Node.where(key:key).first
 		if node
-			node = check_node_security(agent,node)
+			node = check_node_security(agent,node,access)
 		else
 			node = Chawk::Models::Node.create(key:key) if node.nil?
-			node.relations.create(agent:agent,node:node,admin:true,read:true,write:true)
-			return node
+			node.set_permissions(agent,true,true,true)
 		end
+		node.access = access
+		return node
 	end
 
 	# @param agent [Chawk::Agent] the agent whose permission will be used for this request 
-	# @param key [String] the string address this addr can be found in the database.
-	# @return [Chawk::Addr]
-	# The primary method for retrieving an Addr.  If a key does not exist, it will be created 
+	# @param key [String] the string address this node can be found in the database.
+	# @return [Chawk::Node]
+	# The primary method for retrieving an Node.  If a key does not exist, it will be created 
 	# and the current agent will be set as an admin for it.
-	def self.addr(agent,key)
+	def self.node(agent,key,access=:full)
 
 		unless key =~ /^[\w\:\$\!\@\*\[\]\~\(\)]+$/
 			raise ArgumentError, "Key can only contain [A-Za-z0-9_:$!@*[]~()] (#{key})"
@@ -52,7 +72,7 @@ module Chawk
 			raise ArgumentError, 'key must be a string.'
 		end
 
-		node = find_or_create_node(agent,key)
+		node = find_or_create_node(agent,key,access)
 
 		unless node
 			raise ArgumentError,"No node was returned."
@@ -69,8 +89,8 @@ module Chawk
 	def self.bulk_add_points(agent, data)
 		data.keys.each do |key|
 			dset = data[key]
-			daddr = addr(agent,key)
-			daddr.add_points dset
+			dnode = node(agent,key)
+			dnode.add_points dset
 		end
 	end
 

data/lib/models.rb

@@ -1,4 +1,6 @@
 require 'active_record'
+require 'node'
+require 'range'
 module Chawk
 	# Models used in Chawk.  ActiveRecord classes.
 	module Models
@@ -9,6 +11,13 @@ module Chawk
 			has_many :tags
 			has_many :agent_tags
 			has_many :relations
+			has_many :nodes, :through=>:relations
+			has_many :read_relations, ->{where("read = ? OR admin = ?", true,true)}, :class_name=>"Chawk::Models::Relation"
+			has_many :read_nodes, :through=>:read_relations, :source=>:node #, ->{where(read:true, admin:true)}
+			has_many :write_relations, ->{where("write = ? OR admin = ?", true,true)}, :class_name=>"Chawk::Models::Relation"
+			has_many :write_nodes, :through=>:write_relations, :source=>:node #, ->{where(read:true, admin:true)}
+			has_many :admin_relations, ->{where(admin:true)}, :class_name=>"Chawk::Models::Relation"
+			has_many :admin_nodes, :through=>:admin_relations, :source=>:node #, ->{where(read:true, admin:true)}
 		end
 
 		# Agent Relation classes, with permission flags.
@@ -18,155 +27,6 @@ module Chawk
 			belongs_to :node
 		end
 
-		# The Node, where most Chawk:Addr information is persisted..
-		class Node < ActiveRecord::Base
-			attr_accessor :agent
-			after_initialize :init
-			self.table_name_prefix = "chawk_"
-			belongs_to :agent
-			has_many :points
-			has_many :values
-			has_many :relations			
-
-			def init
-				@agent = nil
-			end
-
-			def _insert_point(val,ts,options={})
-				values = {value:val,observed_at:ts.to_f}
-				if options[:meta]
-					if options[:meta].is_a?(Hash)
-						values[:meta] = options[:meta].to_json
-					else
-						raise ArgumentError, "Meta must be a JSON-representable Hash. #{options[:meta].inspect}"
-					end
-				end
-				self.points.create(values)
-			end
-
-			def _insert_point_hash(item,ts,options)
-				if item['v'] && item['v'].is_a?(Integer)
-					if item['t']
-						_insert_point item['v'],item['t'], options
-					else
-						_insert_point item['v'],ts, options
-					end
-				else
-					raise ArgumentError, "Hash must have 'v' key set to proper type.. #{item.inspect}"
-				end
-			end
-
-			def _insert_point_array(item,options)
-				if item.length == 2 && item[0].is_a?(Integer)
-					_insert_point item[0],item[1], options
-				else
-					raise ArgumentError, "Array Items must be in [value,timestamp] format. #{item.inspect}"
-				end
-			end
-
-			def _insert_point_string(item,ts,options)
-				if item.length > 0 && item =~ /\A[-+]?[0-9]+/
-					_insert_point item.to_i,ts, options
-				else
-					raise ArgumentError, "String Items must represent Integer. #{item.inspect}"
-				end
-			end
-
-			def point_recognizer(item, dt, options={})
-				case 
-				when item.is_a?(Integer)
-					_insert_point item,dt, options
-				when item.is_a?(Array)
-					_insert_point_array(item, options)
-				when item.is_a?(Hash)
-					_insert_point_hash(item,dt,options)
-				when item.is_a?(String)
-					_insert_point_string(item,dt,options)
-				else
-					raise ArgumentError, "Can't recognize format of data item. #{item.inspect}"
-				end
-			end
-
-			# @param args [Object, Array of Objects]
-			# @param options [Hash] You can also pass in :meta and :timestamp 
-			# Add an item or an array of items (one at a time) to the datastore.
-			def add_points(args,options={})
-				options[:observed_at] ? dt = options[:observed_at] : dt = Time.now
-				if args.is_a?(Array)
-					args.each do |arg|
-						point_recognizer(arg, dt, options)
-					end
-				else
-					point_recognizer(args, dt, options)
-				end
-			end
-
-			def increment(value=1, options={})
-				if value.is_a?(Integer)
-					last = self.points.last
-					add_points last.value + value,options 
-				else 
-					raise ArgumentError, "Value must be an Integer"
-				end
-			end
-
-			def decrement(value=1, options={})
-				if value.is_a?(Integer)
-					increment (-1) * value, options
-				else 
-					raise ArgumentError, "Value must be an Integer"
-				end
-			end
-
-			def max()
-				points.maximum('value') || 0
-			end
-
-			def min()
-				points.minimum('value') || 0
-			end
-
-			# Returns items whose observed_at times fit within from a range.
-			# @param dt_from [Time::Time] The start time.
-			# @param dt_to [Time::Time] The end time.
-			# @return [Array of Objects] 
-			def points_range(dt_from, dt_to,options={})
-				vals = points.where("observed_at >= :dt_from AND  observed_at <= :dt_to",{dt_from:dt_from.to_f,dt_to:dt_to.to_f}, limit:1000,order:"observed_at asc, id asc")
-				return vals
-			end
-
-			# Returns items whose observed_at times fit within from a range ending now.
-			# @param dt_from [Time::Time] The start time.
-			# @return [Array of Objects] 
-			def points_since(dt_from)
-				self.points_range(dt_from,Time.now)
-			end
-
-			# Sets public read flag for this address 
-			# @param value [Boolean] true if public reading is allowed, false if it is not.
-			def set_public_read(value)
-				value = value ? true : false
-				self.public_read = value
-				save
-			end
-
-			# Sets permissions flag for this address, for a specific agent.  The existing Chawk::Relationship will be destroyed and 
-			# a new one created as specified.  Write access is not yet checked.
-			# @param agent [Chawk::Agent] the agent to give permission.
-			# @param read [Boolean] true/false can the agent read this address.
-			# @param write [Boolean] true/false can the agent write this address. (Read acces is required to write.)
-			# @param admin [Boolean] does the agent have ownership/adnim rights for this address. (Read and write are granted if admin is as well.)
-			def set_permissions(agent,read=false,write=false,admin=false)
-				relations.where(agent_id:agent.id).destroy_all
-				if read || write || admin
-					vals = {agent:agent,read:(read ? true : false),write:(write ? true : false),admin:(admin ? true : false)}
-					relations.create(vals)
-				end
-				nil
-			end
-
-		end
-
 		# The Node, where most Chawk point information is persisted..
 		class Point < ActiveRecord::Base
 			self.table_name_prefix = "chawk_"
@@ -180,4 +40,4 @@ module Chawk
 		end
 
 	end
-end
+end