statlysis 0.0.3 → 0.0.7

data/README.markdown

@@ -1,52 +1,45 @@
 Statlysis
 ===============================================
-Statistical & Analysis in Ruby DSL
+Statistical and analysis in Ruby DSL, just as simple as SQL operations in ActiveRecord.
 
-Usage
+项目来由，理念，和使用说明
 -----------------------------------------------
-### setup
+该项目起因是为eoe.cn做一套统计后台，而其构思来自于2012上半年做 [Android优亿市场数据采集分析系统](http://mvj3.github.io/2012/11/01/android_eoemarket_data_collect_and_analysis_system_summary/) 时的一些经验和心得，在2013年上半年完成了架构和大部分代码，支持ActiveRecord和Mongoid两个ORM。下半年在阳光书屋加上了对Mongoid的MapReduce支持。
 
-```ruby
-Statlysis.setup do
-  set_database :statlysis
+针对一般互联网网站的统计需求，都是把Google Analysis等分析网站的一段Javascript脚本放到网页底部，然后就可以看到网站每日详细的访问情况了。但是针对内部数据需求，比如每日注册用户量，这个一般就不可能开放给第三方去统计了，所以这就是statlysis的存在意义。
 
-  daily CodeGist
-  hourly EoeLog, :time_column => :t # support custom time_column
+做过数据分析的人都知道其中的坑，比如有些是直接拿SQL跨多个表Join统计，每次有数据需求均执行一次完整的查询，随着数据量的增大，性能问题可想而知。
 
-  [EoeLog,
-   EoeLog.where(:ui => 0), # support query scope
-   EoeLog.where(:ui => {"$ne" => 0}),
-   Mongoid[/eoe_logs_[0-9]+$/].where(:ui => {"$ne" => 0}), # support collection name regexp
-   EoeLog.where(:do => {"$in" => [DOMAINS_HASH[:blog], DOMAINS_HASH[:my]]}),
-  ].each do |s|
-    daily s, :time_column => :t
-  end
-end
-```
+下面介绍如何用statlysis进行统计分析：
 
-### access
+#### 是否要 ETL(Extract, Transform, Load) 数据清洗
+statlysis认为数据源一定要被ETL为简单几个维度的单层数据集，因为用户最后能看到和理解的也就是两三维的分析图表而已，所以从用户理解出发。
 
-```ruby
-Statlysis.daily # => return daily crons
-Statlysis.daily.run # => run daily crons
-Statlysis.daily[/name_regexp/] # => return matched daily crons
-```
+这里也需要注意如果当该数据表是可以直接支持统计分析的，但是数据量大，那么得加上相关索引，或者导入到另外的单表里(在ORM里可以在`after_save`等hooks里操作)再加索引。
 
-### process
+#### 流程
+1. 分析数据统计需求，画出包含对应维度的图表。
+2. ETL，参照 #是否要 ETL(Extract, Transform, Load) 数据清洗#
+3. 在 `Statlysis.setup { }` 代码块里配置出页面需要的数据，注意得是单表的，类似没有跨表JOIN的SQL查询。
+4. 跑统计分析，比如 `Statlysis.daily.run`。此过程可以用cron定时来驱动，或者`after_save`等数据更新来驱动。
+5. 编写用于数据需求人员查看的HTML页面，其中统计数据可以用`Statlysis.daily['code_gists'].first.stat_model`或`TimelyCodegist`来直接查询。
 
-```irb
-[23] pry(#<Statlysis::Configuration>)> Statlysis.daily['multi'].first
-```
+#### 尽量采用MongoDB来作为统计数据源
+MongoDB作为NoSQL数据库，它是为 **单collection** 里读写 **单个记录的整体** 而优化设计的，并支持MapReduce并发来加快统计过程。
 
-Features
+成功案例
 -----------------------------------------------
-* Support time column that stored as integer.
+* eoe.cn各子网站的页面访问统计，和包含多个条件的数据库表每日数据统计，详情见 [示例配置文件](https://github.com/mvj3/statlysis/blob/master/examples/eoecn.rb) ，按日期维度分。
+* 阳光书屋的学习提高班的关于做题情况的统计分析，详情见 [示例配置文件](https://github.com/mvj3/statlysis/blob/master/examples/sunshinelibrary.rb) ，按班级维度分。
 
-TODO
+Usage
 -----------------------------------------------
-* Admin interface
-* statistical query api in Ruby and HTTP
-* Interacting with Javascript charting library, e.g. Highcharts, D3.
+见上面的 [成功案例](#成功案例) 的配置文件 和 [手把手操作示例](http://mvj3.github.io/statlysis/showterm.html) 。
+
+Features
+-----------------------------------------------
+* 支持Mongoid和ActiveRecord两种ORM，其中Mongoid以MapReduce方式统计，ActiveRecord基于纯SQL操作。
+* Support time column that stored as integer.
 
 
 Statistical Process
@@ -73,34 +66,13 @@ Q: In Mongodb, why use MapReduce instead of Aggregation?
 A: The result of aggregation pipeline is a document and is subject to the BSON Document size limit, which is currently 16 megabytes, see more details at http://docs.mongodb.org/manual/core/aggregation-pipeline/#pipeline
 
 
-Copyright
+TODO
 -----------------------------------------------
-MIT. David Chen at eoe.cn.
+* Admin interface
+* statistical query api in Ruby and HTTP
+* Interacting with Javascript charting library, e.g. Highcharts, D3.
 
 
-Related
+Copyright
 -----------------------------------------------
-### Projects
-* https://github.com/paulasmuth/fnordmetric FnordMetric is a redis/ruby-based realtime Event-Tracking app
-* https://github.com/thirtysixthspan/descriptive_statistics adds methods to the Enumerable module to allow easy calculation of basic descriptive statistics for a set of data
-* https://github.com/tmcw/simple-statistics simple statistics for javascript in node and the browser
-* https://github.com/clbustos/statsample/  A suite for basic and advanced statistics on Ruby. 
-* https://github.com/SciRuby/sciruby Tools for scientific computation in Ruby/Rails
-
-### Articles
-* http://www.slideshare.net/WombatNation/logging-app-behavior-to-mongo-db
-
-### Event collector
-* https://github.com/fluent
-* https://github.com/logstash/logstash
-
-### Admin interface
-* http://three.kibana.org/ browser based analytics and search interface to Logstash and other timestamped data sets stored in ElasticSearch.
-
-
-### ETL
-* https://github.com/activewarehouse/activewarehouse-etl/ 
-* http://jisraelsen.github.io/drudgery/ ruby ETL DSL, support csv, sqlite3, ActiveRecord, without support time range
-* https://github.com/square/ETL Simply encapsulates the SQL procedures
-
-
+MIT. David Chen at eoe.cn, sunshine-library .

data/examples/eoecn.rb

@@ -0,0 +1,35 @@
+# encoding: UTF-8
+
+Statlysis.setup do
+  set_database :statlysis
+  update_time_columns :t
+  set_tablename_default_pre :st
+
+  # 统计网站总体每日访问量
+  @log_model = IS_DEVELOP ? EoeLogTest : EoeLog
+  hourly @log_model, :time_column => :t
+  daily  @log_model, :time_column => :t
+  # 统计登陆和非登陆用户访问量
+  daily  @log_model.where(:ui => 0), :time_column => :t
+  daily  @log_model.where(:ui => {"$ne" => 0}), :time_column => :t
+
+  # 统计各个子网站每日访问量
+  daily  @log_model.where(:do => {"$in" => [DOMAINS_HASH[:blog], DOMAINS_HASH[:my]]}), :time_column => :t
+  [:www, :code, :skill, :book, :edu, :news, :wiki, :salon, :android].each do |site|
+    daily  @log_model.where(:do => DOMAINS_HASH[site]), :time_column => :t
+  end
+
+  # 统计各个数据模型在不同条件下每天的变化量
+  daily CodeGist
+  [BlogPost, NewsNews, WikiPage].each do |model|
+    daily model.where("create_time > 0"), :time_column => :create_time
+    daily model.where("update_time > 0"), :time_column => :update_time
+  end
+
+  daily CommonComment.where("is_delete = 0"), :time_column => :create_time
+  daily CommonComment.where(:model => 'blog').where("is_delete = 0"), :time_column => :create_time
+  daily CommonComment.where(:model => 'code').where("is_delete = 0"), :time_column => :create_time
+
+  daily CommonMember.where("regdate > 0"), :time_column => :regdate
+
+end

data/examples/sunshinelibrary.rb

@@ -0,0 +1,41 @@
+# encoding: UTF-8
+
+Statlysis.setup do
+  set_database :local_statistic
+
+  daily UserRecord.where(item_type: "activity")
+
+  # 表关系 subject <= chapter <= lesson <= activity <= problem
+  #        room和[user, duration]等绑定
+
+  # ********
+  # **列表**
+  # ********
+  # 查询条件:     [chapter]
+  # 章节课时分析: room, lesson, level{5}, group_concat(user), count
+  # 推断其他字段: [lesson] => [chapter]
+  %w[not_done bad good1 good3 good5].each do |level|
+    always ETL::LessonLog.where(:level => level),
+           :group_by_columns => [
+             {:column_name => :room, :type => :string},
+             {:column_name => :lesson, :type => :string}
+           ],
+           :group_concat_columns => [:user]
+  end
+
+  # ********
+  # **详情**
+  # ********
+  # 查询条件:     [activity]
+  # Activity分析: room, problem, answer, group_concat(user, duration), count
+  # 推断其他字段: [problem] => [activity, lesson]
+  always ETL::ProblemLog,
+         :group_by_columns => [
+           {:column_name => :room, :type => :string},
+           {:column_name => :problem, :type => :string},
+           # statlysis.gem use column_name to create table name, so that's why no_index option exists
+           {:column_name => :answer, :type => :string, :no_index => true}
+         ],
+         :group_concat_columns => [:user, :duration]
+
+end

data/lib/statlysis.rb

@@ -31,7 +31,7 @@ module Statlysis
 
       logger.info "Start to setup Statlysis" if ENV['DEBUG']
       time_log do
-        self.config.instance_exec(&blk)
+        self.configuration.instance_exec(&blk)
       end
     end
 
@@ -44,10 +44,10 @@ module Statlysis
     end
 
     # delagate config methods to Configuration
-    def config; Configuration.instance end
+    def configuration; Configuration.instance end
     require 'active_support/core_ext/module/delegation.rb'
     Configuration::DelegateMethods.each do |sym|
-      delegate sym, :to => :config
+      delegate sym, :to => :configuration
     end
 
     attr_accessor :logger
@@ -56,9 +56,9 @@ module Statlysis
     def source_to_database_type; @_source_to_database_type ||= {} end
 
     # 代理访问 各个时间类型的 crons
-    def daily; CronSet.new(Statlysis.config.day_crons) end
-    def hourly; CronSet.new(Statlysis.config.hour_crons) end
-    def always; CronSet.new(Statlysis.config.always_crons) end
+    def daily; CronSet.new(Statlysis.configuration.day_crons) end
+    def hourly; CronSet.new(Statlysis.configuration.hour_crons) end
+    def always; CronSet.new(Statlysis.configuration.always_crons) end
 
   end
 

data/lib/statlysis/clock.rb

@@ -6,9 +6,7 @@ module Statlysis
     include Common
 
     # feature is a string
-    def initialize feature, default_time
-      raise "Please assign default_time params" if not default_time
-
+    def initialize feature, default_time = nil
       # init table & model
       cron.stat_table_name = [Statlysis.tablename_default_pre, 'clocks'].compact.join("_")
       unless Statlysis.sequel.table_exists?(cron.stat_table_name)
@@ -23,12 +21,13 @@ module Statlysis
       cron.stat_model = h[:model]
 
       # init default_time
+      default_time ||= DateTime.now
       cron.clock = cron.stat_model.find_or_create(:feature => feature)
       cron.clock.update :t => default_time if cron.current.nil?
       cron
     end
 
-    def update time
+    def update time = DateTime.now
       time = DateTime.now if time == DateTime1970
       return false if time && (time < cron.current)
       cron.clock.update :t => time

data/lib/statlysis/cron.rb

@@ -3,6 +3,7 @@
 module Statlysis
   class Cron
     attr_reader :multiple_dataset, :source_type, :time_column, :time_unit, :time_zone
+    attr_reader :clock
     include Common
 
     def initialize s, opts = {}

data/lib/statlysis/cron/timely.rb

@@ -34,6 +34,9 @@ module Statlysis
         end
       end
 
+      # record last executed time
+      clock.update
+
       return self
     end
 
@@ -90,7 +93,7 @@ module Statlysis
       _truncated_columns = _group_by_columns_index_name.dup # only String column
       _group_by_columns_index_name = _group_by_columns_index_name.unshift :t if cron.time_column?
       # TODO use https://github.com/german/redis_orm to support full string indexes
-      if !Statlysis.config.is_skip_database_index && _group_by_columns_index_name.any?
+      if !Statlysis.configuration.is_skip_database_index && _group_by_columns_index_name.any?
         mysql_per_column_length_limit_in_one_index = (1000 / 3.0 / _group_by_columns_index_name.size.to_f).to_i
         index_columns_str = _group_by_columns_index_name.map {|s| _truncated_columns.include?(s) ? "#{s.to_s}(#{mysql_per_column_length_limit_in_one_index})" : s.to_s }.join(", ")
         index_columns_str = "(#{index_columns_str})"
@@ -151,6 +154,8 @@ module Statlysis
 
     private
     def remodel
+      @clock ||= reclock
+
       n = cron.stat_table_name.to_s.singularize.camelize
       cron.stat_model = class_eval <<-MODEL, __FILE__, __LINE__+1
         class ::#{n} < Sequel::Model;
@@ -162,6 +167,11 @@ module Statlysis
       MODEL
     end
 
+    def reclock
+      # setup a clock to record the last updated
+      @clock = Clock.new "last_updated_at__#{cron.stat_table_name}"
+    end
+
   end
 end
 

data/lib/statlysis/rake.rb

@@ -7,23 +7,23 @@ namespace :statlysis do
   Statlysis::TimeUnits.each do |unit|
     desc "statistical in #{unit}"
     only_one_task "#{unit}_count" => :environment do
-      Statlysis.config.send("#{unit}_crons").map(&:run)
+      Statlysis.configuration.send("#{unit}_crons").map(&:run)
     end
   end
 
   desc "realtime process"
   only_one_task :realtime_process => :environment do
-    loop { Statlysis.config.realtime_crons.map(&:run); sleep 1 }
+    loop { Statlysis.configuration.realtime_crons.map(&:run); sleep 1 }
   end
 
   desc "similar process"
   only_one_task :similar_process => :environment do
-    Statlysis.config.similar_crons.map(&:run)
+    Statlysis.configuration.similar_crons.map(&:run)
   end
 
   desc "hotest process"
   only_one_task :hotest_process => :environment do
-    Statlysis.config.hotest_crons.map(&:run)
+    Statlysis.configuration.hotest_crons.map(&:run)
   end
 
 end

data/statlysis.gemspec

@@ -4,13 +4,13 @@ $:.push File.expand_path("../lib", __FILE__)
 
 Gem::Specification.new do |s|
   s.name          = 'statlysis'
-  s.version       = '0.0.3'
-  s.date          = '2013-12-03'
+  s.version       = '0.0.7'
+  s.date          = '2013-12-30'
   s.summary       = File.read("README.markdown").split(/===+/)[1].strip.split("\n")[0]
   s.description   = s.summary
   s.authors       = ["David Chen"]
   s.email         = 'mvjome@gmail.com'
-  s.homepage      = 'https://github.com/SunshineLibrary/statlysis'
+  s.homepage      = 'https://github.com/mvj3/statlysis'
   s.license       = 'MIT'
 
   s.files         = `git ls-files`.split("\n")

data/test/helper.rb

@@ -28,8 +28,8 @@ require 'sqlite3'
 
 # load ActiveRecord setup
 Statlysis.set_database ":memory:"
-Statlysis.config.is_skip_database_index = true
-ActiveRecord::Base.establish_connection(Statlysis.config.database_opts.merge("adapter" => "sqlite3"))
+Statlysis.configuration.is_skip_database_index = true
+ActiveRecord::Base.establish_connection(Statlysis.configuration.database_opts.merge("adapter" => "sqlite3"))
 Dir[File.expand_path("../migrate/*.rb", __FILE__).to_s].each { |f| require f }
 Dir[File.expand_path("../models/*.rb", __FILE__).to_s].each { |f| require f }
 
@@ -45,6 +45,24 @@ csv.each do |row|
 end
 
 
+
+
+
+
+
+
+
+
+(require 'pry-debugger';binding.pry) if ENV['DEBUG']
+
+
+
+
+
+
+
+
+
 Statlysis.setup do
   hourly EoeLog, :time_column => :t
 
@@ -62,5 +80,4 @@ Statlysis.setup do
   cron1 = Statlysis.daily['mul'][1]
   cron2 = Statlysis.daily['cod'][0]
   cron3 = Statlysis.always['code']['mongoid'][0]
-  require 'pry-debugger';binding.pry
 end

data/test/{test_daily_count.rb → test_generally_count.rb}

@@ -1,8 +1,9 @@
 # encoding: UTF-8
+# NOTE 以下统计数据依赖code_gists测试数据。
 
 require 'helper'
 
-class TestDailyCount < Test::Unit::TestCase
+class TestGenerallyCount < Test::Unit::TestCase
   def setup
     @output = Statlysis.daily['code_gist'].first.output
   end
@@ -20,5 +21,4 @@ class TestDailyCount < Test::Unit::TestCase
     assert_equal @output[-1][:totally_favcount_s].to_i, CodeGist.all.map(&:fav_count).reduce(:+)
   end
 
-
 end 

data/test/{test_statlysis.rb → test_manipulate_table_and_model.rb}

@@ -2,7 +2,7 @@
 
 require 'helper'
 
-class TestStatlysis < Test::Unit::TestCase
+class TestManipulateTableAndModel < Test::Unit::TestCase
   def setup
     @old_datetime = Time.zone.parse("20130105")
   end

data/test/test_mapreduce.rb

@@ -7,12 +7,18 @@ class TestMapReduce < Test::Unit::TestCase
   end
 
   def test_multiple_dimensions_output_without_time_column
+    before_time = DateTime.now
     cron = Statlysis.always['mongoid']['code'][0]
     assert_equal cron.time_column, false
     assert_equal cron.time_unit, false
     assert_equal cron.stat_table_name, 'timely_codegistmongoids_author_a'
 
     cron.run
+
+    # 测试更新最后执行时间的时钟
+    # TODO 可能移到其他地方
+    assert((cron.clock.current - before_time) > 0)
+
     assert_equal cron.output.detect {|h| h[:author] == 'mvj3' }[:c].to_i, cron.multiple_dataset.sources.first.where(:author => 'mvj3').count
   end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: statlysis
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 0.0.7
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-12-03 00:00:00.000000000 Z
+date: 2013-12-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -251,7 +251,8 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
-description: Statistical & Analysis in Ruby DSL
+description: Statistical and analysis in Ruby DSL, just as simple as SQL operations
+  in ActiveRecord.
 email: mvjome@gmail.com
 executables: []
 extensions: []
@@ -263,6 +264,8 @@ files:
 - Guardfile
 - README.markdown
 - Rakefile
+- examples/eoecn.rb
+- examples/sunshinelibrary.rb
 - lib/statlysis.rb
 - lib/statlysis/clock.rb
 - lib/statlysis/common.rb
@@ -298,12 +301,12 @@ files:
 - test/models/.gitkeep
 - test/models/code_gist.rb
 - test/models/eoe_log.rb
-- test/test_daily_count.rb
+- test/test_generally_count.rb
+- test/test_manipulate_table_and_model.rb
 - test/test_mapreduce.rb
 - test/test_single_log_in_multiple_collections.rb
-- test/test_statlysis.rb
 - test/test_timeseries.rb
-homepage: https://github.com/SunshineLibrary/statlysis
+homepage: https://github.com/mvj3/statlysis
 licenses:
 - MIT
 post_install_message: 
@@ -316,22 +319,17 @@ required_ruby_version: !ruby/object:Gem::Requirement
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: -1643509325996557122
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: -1643509325996557122
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
-summary: Statistical & Analysis in Ruby DSL
+summary: Statistical and analysis in Ruby DSL, just as simple as SQL operations in
+  ActiveRecord.
 test_files: []