movieDB 0.3.4 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: abd7277f09e27f87c05a46760b7604142a13c4f4
-  data.tar.gz: 7b7982a640eff36476dcc23dfeb737e87f3ee911
+  metadata.gz: 18eb3efc3ad17c7cb5b6134a951402964af0b5da
+  data.tar.gz: 7eeb628cab2588293f92aa458eeabb903cbfb6ec
 SHA512:
-  metadata.gz: 950a7d07348be94e30fb40e74048de44c6548f1794bf0c65ac140a6f2845733a76533c0d49e7ea5dec9f5b8397e76082f8b93580959c08ae0cf3d2e8589cca5c
-  data.tar.gz: e8293834822371dec1e2579eeff6e4f6aa9baf035c46ca8a0324d7458ca029c6c875afec8577cbb329448022207bc0c2d20ed0fec243b2c1eb11dee1b3f81863
+  metadata.gz: 09c390931150038ca9b18d52ffdd084a805b344c4272876c9ff6880ee2c4ae3da5623be445cc9b3ad6e7f192e60594fb62a682ebad4168f68d08f8e07b3629b5
+  data.tar.gz: 4b559dcc3e841b450187b87f1060039033109bdce2cd4791ae5c9628bea929db875acff15748e807f9e6444b2eeddcd03ff37b0b2b62d4b324bffdf30468d068

data/.coveralls.yml

@@ -0,0 +1,2 @@
+service_name: travis-ci
+repo_token: Uytw7gmmEhTTwfj9STT7rokiH7D0su0GD

data/.gitignore

@@ -16,6 +16,8 @@ test/tmp
 test/version_tmp
 tmp
 reports
-.coveralls.yml
 readme_first.txt
 test/.DS_Store
+coverage
+*.log
+.idea/

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -3,6 +3,11 @@ rvm:
   - 2.2.2
 env:
   - TEST_SUITE = units
+global:
+  - "CI=true"
+  - "TRAVIS=true"
+after_success:
+  - coveralls --verbose
 before_script:
   - rm -rf reports/*.8
 bundler_args: --without development production --quiet

data/Gemfile

@@ -2,7 +2,12 @@ source 'https://rubygems.org'
 
 gemspec
 
-group :documentation do
-  gem 'coveralls', :require => false
-end
+gem 'coveralls', require: false
+gem 'rspec'
+
 
+# Code coverage analysis tool for Ruby.
+group :test do
+  gem 'simplecov', :require => false
+  gem 'pullreview-coverage', require: false
+end

data/README.md

@@ -1,164 +1,308 @@
-## MovieDB
+# MovieDB
 
- MovieDB is a ruby wrapper for fetching raw Movie or TV Data from IMDb and performing a variety of statistical analysis and computation.
- The objective and usage of this tool is to help media producers make high level structured decisions based on realistic analysis of actual data.
-
- The fetched data is stored in memory using Redis and has an expiration time of 1800 seconds for all cached objects.
+ MovieDB is a multi-threaded ruby wrapper for performing advance statistical computation and high-level data analysis on Movie or TV Data from IMDb.
+ The objective and usage of this tool is to allow producers, directors, writers to make logical business decisions that will generate profitable ROI.
 
+## Badges
+  - [![PullReview stats](https://www.pullreview.com/github/keeperofthenecklace/movieDB/badges/master.svg?)](https://www.pullreview.com/github/keeperofthenecklace/movieDB/reviews/master)
+  - [![Dependency Status](https://gemnasium.com/keeperofthenecklace/movieDB.svg)](https://gemnasium.com/keeperofthenecklace/movieDB)
   - [![Join the chat at https://gitter.im/keeperofthenecklace/movieDB](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/keeperofthenecklace/movieDB?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
-  - [![Coverage Status](https://coveralls.io/repos/keeperofthenecklace/movieDB/badge.svg)](https://coveralls.io/r/keeperofthenecklace/movieDB)
+  - [![Coverage Status](https://coveralls.io/repos/keeperofthenecklace/movieDB/badge.svg?branch=master&service=github)](https://coveralls.io/github/keeperofthenecklace/movieDB?branch=master)
   - [![Code Climate](https://codeclimate.com/github/keeperofthenecklace/movieDB.png)](https://codeclimate.com/github/keeperofthenecklace/movieDB)
   - [![Gem Version](https://badge.fury.io/rb/movieDB.png)](http://badge.fury.io/rb/movieDB)
   - [![Build Status](https://secure.travis-ci.org/keeperofthenecklace/movieDB.png?branch=master)](http://travis-ci.org/keeperofthenecklace/movieDB)
 
-## Basic functions and Data Analysis:
+## Technology
+* SciRuby is used for all statistical and scientific computations.
+* Redis is used to store all data.
+* IMDb and TMDb is the source for all film / TV data.
+* BoxOfficeMojo is where we will be scraping future film / TV data.
+* Celluloid is used to build the fault-tolerant concurrent programs. Note, if you are using MRI or YARV,
+multi-threading won't work since these types of interpreters have Global Interpreter Lock (GIL).
+Fortunately, you can use JRuby or Rubinius, since they don’t have a GIL and support real parallel threading.
+
+## Requirements
+ruby-2.2.2 or higher
+
+jruby-9.0.0.0
+
+## Category
+movieDB is broken down into 3 components namely:
+
+* Statistics
+* Visualizations (Work in progress)
+* DataMining (Work in progress)
 
-* Data Analysis
-* Exploratory Data Analysis
-* Confirmatory Data Analysis
-* More to come...
+# Statistics
+
+Simple statistical analysis on numeric data.
+The corresponding computation is performed on
+both numeric and string vectors within the
+collected data.
 
 ## Installation
 
-Please make sure you have redis installed.
+Redis Installation
 
-    This tutorial doesn't cover redis installation.
+  This tutorial doesn't cover redis installation.
+  You will find that information at: http://redis.io/topics/quickstart
 
-Add this line to your application's Gemfile:
+movieDB is available through [Rubygems](http://rubygems.org/gems/movieDB) and can be installed via Gemfile.
 
-    gem 'movieDB'
+``` ruby
+gem 'movieDB'
+```
 
 And then execute:
 
-    $> bundle install
+``` ruby
+$ bundle install
+```
 
 Or install it yourself as:
 
-    $> gem install movieDB
+``` ruby
+gem install movieDB
+```
+
+## Console - loading the libraries
 
-## Require - loading the libraries
+``` bash
+$ irb
+```
 
-    $> irb
+## Require the gem
 
-    $> require 'movieDB'
+```ruby
+require 'movieDB'
+```
 
-## Usage - Fetch Raw Movie Data From IMDb
+## Initialize MovieDB (multi-thread setup)
 
-    $> imdb_ids = ["0369610", "3079380"]
+``` ruby
+m = MovieDB::Movie.pool(size: 2)
+```
+## Step Process
 
-    $> MovieDB::Movie.find_imdb_id(imdb_ids)
+Fetching and analysing movie / TV data using movieDB is a simple 2 step process.
 
-      /* YOU CAN ADD AS MANY IMDB IDs AS YOU LIKE. BUT DO NOT EXCEED THE MAXIMUM REQUEST RATE. */
+First, fetch the data from IMDb.
 
-### IMDb Data
+Next, run your choice of statistic.
 
-When IMDb data is fetched, two things happen.
+That's it! It is that simple.
 
-First, a reports folder is created in the movieDB gem folder.
+## Part 1 - Fetch Data from IMDb
 
-Next, the fetched data is written to an xls format and stored in the reports directory.
+There are 2 ways to find IMDb ids.
 
-From your terminal you can locate movieDB gem directory like this:
+* Finding specific IMDb ids
 
-    $ gem content movieDB
+* Finding random IMDb ids.
 
-If you use our above IMDb id, you should find the following xls file.
+### Fetching specific IMDb ids
 
-Feel free to open it.
+To find IMDb id for specific movies, you must go to:
 
-    $ open ../reports/imdb_JurassicWorld_Spy_.xls
+```bash
+http://www.imdb.com
+```
+Search for your movie of choice. Once you do, IMDb redirects you to the movie's page.
 
-## Usage - Analyzing Data From IMDb.
+The URL for the redirect page includes the IMDB id.
 
-    $ irb
+``` ruby
+http://www.imdb.com/title/tt0369610/
+```
+0369610 is the IMDb id.
 
-    > require 'MovieDB/data_analysis'
+### Fetching random IMDb ids (multi-thread setup)
+You can fetch IMDb ids random.
 
-    > require 'MovieDB/data_process'
+``` ruby
+r = Random.new
 
-    > MovieDB::DataProcess.send(:basic_statistic, 'imdb_JurassicWorld_Spy_.xls')
+39.times do |i|
+  m.async.fetch(sprintf '%07d', r.rand(300000))
+  sleep(4)
+end
 
-A statistical computation is performed and the results is written to movieDB gem reports folder.
+sleep(10)
+```
+Note: IMDB has a rate limit of 40 requests every 10 seconds and are limited by IP address, not API key.
+If you exceed the limit, you will receive a 429 HTTP status with a 'Retry-After' header.
+As soon your cool down period expires, you are free to continue making requests.
 
-Feel free to open it.
+Also, movieDB will throw a NameError if the randomly generated IMDb id in invalid.
 
-    $ open ../reports/Statistic_imdb_JurassicWorld_Spy.xls
+### Get Movie Data
 
-## What's Next
+``` ruby
+m.async.fetch("0369610", "3079380", "0478970")
+```
+By calling m.async, this instructs Celluloid that you would like for the given method to be called asynchronously.
+This means that rather than the caller waiting for a response of querying both IMDb and TMDb, the caller sends a
+message to the concurrent object that you'd like the given method invoked, and then the caller proceeds without waiting for a response.
+The concurrent object receiving the message will then process the method call in the background.
 
-##### More statistical computations coming soon:
+Asynchronous calls will never raise an exception, even if an exception occurs when the receiver is processing it.
 
-`:GaussNewtonAlgorithm`
+### Redis - caching objects
 
-    > Iteratively_Reweighted_Least_Squares
-    > Lack_Of_Fit_Sum_Of_Squares
-    > Least_Squares_Support_Vector_Machine
-    > Mean_Squared_Error
-    > Moving_Least_Sqares
-    > Non_Linear_Iterative_Partial_Least_Squares
-    > Non_Linear_Least_Squares
-    > Ordinary_Least_Squares
-    > Partial_Least_Squares_Regression
-    > Partition_Of_Sums_Of_Squares
-    > Proofs_Involving_Ordinary_Least_Squares
-    > Residual_Sum_Of_Squares
-    > Total_Least_Squares
-    > Total_Sum_Of_Squares
+By default, any movie fetched from IMDb is stored in redis and has an expiration time of 1800 seconds (30 minutes).
 
-`:EstimationOfDensity`
+But you can change this expiration time.
 
-    > Cluster_Weighted_Modeling
-    > Density_Estimation
-    > Discretization_Of_Continuous_Features
-    > Mean_Integrated_Squared_Error
-    > Multivariate_Kernel_Density_Estimation
-    > Variable_Kernel_Density_Estimation
+```ruby
+m.async.fetch("0369610", "3079380", expire: 86400)
+```
+Here, I set the expiration time to 86400 seconds which is equivalent to 24 hours.
 
-`:ExploratoryDataAnalysis`
+## Part 2 - Run the statistic
 
-    > Data_Reduction
-    > Table_Diagonalization
-    > Configural_Frequency_Analysis
-    > Median_Polish
-    > Stem_And_Leaf_Display
+Below, we've collected 3 specific IMDb ids to analyze.
 
-    > Data_Mining
-      > Applied_DataMining
-      > Cluster_Analysis
-      > Dimension_Reduction
-      > Applied_DataMining
+* Ant Man - 0369610
+* Jurassic World - 079380
+* Spy - 0478970
 
-    > RegressionAnalysis
-      > Choice_Modelling
+Finding the Mean value.
+```ruby
+m.mean
+```
+Below is the result generated.
 
-      > Generalized_Linear_Model
-        > Binomial_Regression
-        > Generalized_Additive_Model
-        > Linear_Probability_Model
-        > Poisson_Regression
-        > Zero_Inflated_Model
+```ruby
+                             mean
+       ant-man  576.8444444444444
+jurassic_world  512.5111111111111
+           spy 369.73333333333335
+```
+Below are more statistic methods you can invoke on your objects.
 
-      > Nonparametric_Regression
-      > Statistical_Outliers
-      > Regression_And_Curve_Fitting_Software
-      > Regression_Diagnostics
-      > Regression_Variable_Selection
-      > Regression_With_Time_Series_Structure
-      > Robust_Regression
-      > Choice_Modeling
+Feel free to try them out.
 
-    > Resampling
-      > Bootstrapping_Population
+* std
+* sum
+* count
+* max
+* min
+* product
+* standardize
+* describe
+* covariance
+* correlation
 
-    > Sensitivity_Analysis
-      > Variance_Based_Sensitivity_Analysis
-      > Elementary_Effects_Method
-      > Experimental_Uncertainty_Analysis
-      > Fourier_Amplitude_Sensitivity_Testing
-      > Hyperparameter
+### Layout and Template
 
-    > Time_Series_Analysis
-      > Frequency_Deviation
+movieDB allows you to view all your data fields in a worksheet style layout.
+
+``` ruby
+m.worksheet
+```
+A total of 45 fields are printed out. But, we've truncated the result for ease of reading.
+
+
+``` ruby
+              ant-man jurassic_w        spy
+production        177        128         40
+belongs_to          0        151          0
+plot_synop       9083          0       9629
+   company         14         18         21
+     title          7         14          3
+filming_lo        267       1037        530
+cast_chara       4094       5894       1001
+trailer_ur          0         46         45
+cast_membe       2833       3452        939
+     votes          5          6          5
+     adult          5          5          5
+also_known        928       1601       1195
+  director         15         19         13
+plot_summa        373        298        311
+ countries          7         16          7
+       ...        ...        ...        ...
+```
+
+## Filters
+
+When performing statistics on an object, movieDB by default processes all fields.
+
+Contrary, to this default approach, you now have the option of filtering what fields you want processed with the following 2 filters.
+
+* only
+* except
+
+'only' analyzes the fields you provide.
+
+'Except' is the inverse of 'only'. It analyzes all the fields you did not provide.
+
+``` ruby
+m.standardize only: [:budget, :revenue, :length, :vote_average]
+
+```
+Processes only budget, revenue, length and vote_average values.
+``` ruby
+              ant-man jurassic_w        spy
+    budget 1.49999999 -0.3616594 1.49999999
+   revenue -0.5000006 1.49304559 -0.5000013
+    length -0.4999988 -0.5656929 -0.4999976
+vote_avera -0.5000005 -0.5656931 -0.5000010
+```
+# Commands
+
+movieDB comes with commands to help you query or manipulate stored objects in redis.
+
+* HGETALL key
+Get all the fields and values in a hash of the movie
+
+``` ruby
+m.hgetall(["0369610"])
+# => {"production_companies"=>"[{\"name\"=>\"Universal Studios\", \"id\"=>13},...}
+```
+* HKEYS key
+Get all the fields in a hash of the movie
+
+``` ruby
+m.hkeys
+# => ["production_companies", "belongs_to_collection", "plot_synopsis", "company", "title",...]
+```
+
+* HVALS key
+Get all the values in a hash of the movie
+
+``` ruby
+m.hvals
+# => ["[{\"name\"=>\"Universal Studios\", \"id\"=>13}, {\"name\"=>\"Amblin Entertainment\",...]
+```
+
+* ALL_IDS key
+Get all the id of movies
+
+``` ruby
+m.all_ids
+# => ["0369610", "3079380"...]
+```
+
+* TTL key
+Gets the remaining time to live of a movie.
+
+``` ruby
+m.ttl("0369610)
+# => 120
+```
+
+* DELETE_ALL key
+deletes all movie objects stored in redis.
+
+``` ruby
+m.delete_all
+# => []
+```
+# Visualizations
+(Work in progress)
+
+# Data mining
+(Work in progress)
 
 ## Contact me
 
@@ -166,4 +310,7 @@ If you'd like to collaborate, please feel free to fork source code on github.
 
 You can also contact me at albertmck@gmail.com
 
-###### Copyright (c) 2013 Albert McKeever, released under MIT license
+## Disclaimer
+This software is provided “as is” and without any express or implied warranties, including, without limitation, the implied warranties of merchantibility and fitness for a particular purpose.
+
+###### Copyright (c) 2013 - 2015 Albert McKeever, released under MIT license