gooddata 0.6.0.pre9 → 0.6.0.pre10

data/doc/pages/tutorial/BRICKS.md

@@ -0,0 +1,257 @@
+You can run Ruby on GoodData platform. Let's have a look at the platform and walk step by step through doing the simplest possible deployment and then move to more advanced tasks.
+
+## Not reinventing the wheel
+
+The main idea is that only minority of people should be forced to write code. The others should be happy running them without understanding the details. Soon we will introduce better UI to do just that. Until then it is more programatic but if you are not scared read on.
+
+## Setting up the stage
+
+GoodData Ruby SDK stack is built so you can easily develop things locally and deploy them when you have tested them and are happy with how they work. You need to set up an environment first.
+
+### Prerequisites
+ * Git
+ * Ruby (JRuby recommended)
+ * Ruby Gems
+
+Go ahead and run
+
+    git clone https://github.com/gooddata/app_store.git
+
+We just cloned the remote repository which contains information about the used libraries and also contains some stuff that others created. We will investigate that later. Let's continue with setting things up.
+
+Run
+
+    gem install bundler
+
+This will install bundler which is a useful package installation tool. Let's use it
+
+    cd local_repo
+    bundle install --binstubs
+
+This will ensure that you have installed exactly what we have on the production machines. This should mitigate bugs caused by slightly different versions of libraries and incompatible APIs.
+
+You are ready to go.
+
+## Running your first brick
+
+The small pieces of ruby that are run on the platform are called bricks. Nobody knows where and why this name emerged but there are rumors that it is supposed to reference the fact that out of brick just laid together you can create a solid wall.
+
+If you open the repository you will see there some directories. Find directory called misc/hello_world and open it. It contains only one file. Do not open it yet.
+
+Run this in console.
+
+    bundle exec gooddata -l -Uname@gooddata.com -Pmy_pass run_ruby -p project_pid -d hello_world_brick --name "some_deploy" --remote
+
+It will take some time but after a while you should see green DONE on your console and a link for the log. Open it in your browser and you should see there something like this. On one of the lines there should be hello world. Great you just ran your first ruby brick.
+
+###Looking inside Hello World
+
+Let's see what is happening inside. Open the main.rb file in your favorite editor. You should see something like this.
+
+    require 'gooddata'
+    require 'logger'
+
+    module GoodData::Bricks
+        class HelloWorldBrick
+
+            def call(params)
+                logger = Logger.new(params[:GDC_LOGGER_FILE])
+                logger.info "Hello world"
+            end
+
+        end
+    end
+
+    GoodData::Bricks::HelloWorldBrick.new
+
+This is all. Let's dissect it. The interface is very simple. You have to provide an instance of an object that responds to a message :call (in other words does implement method call). This method takes one parameter and that is a hash map of parameters.
+
+You can see that we have implemented such a class and returning an instance of that class. The method accepts params and you can see that we immediately make use of them when grabbing logger and the writing to it. Platform does the heavy lifting on the back and you have already seen the result.
+
+## Digging deeper
+
+It might be surprising if I tell you that this is not exactly how majority of the real bricks are implemented. What we showed you is fine and this is how we started but after we implemented some bricks we found out that we are repeating ourselves a lot. So we tried to come up with something better.
+
+remark: If you know how Rack or any similar framework works for abstracting web applications you would be right at home since that is where most of the inspiration came from.
+
+We introduced three concepts.
+
+ * _Application_ - This part is responsible for doing the core of the task you are interested in. Structure of an app is pretty much what you have already seen.
+ * _Middleware_ - very similar to app. The main difference is that you can chain them together. The main similarity is that it has the same interface as an app
+ * _Pipelines_ - If you chain multiple middleweres and applications it creates a pipeline.
+
+Let's have a look how it works visually.
+
+![Example pipeline consisting of 3 middlewares](https://dl.dropboxusercontent.com/s/g5rymdmmx97hc61/middlewares.png?token_hash=AAE7qAjkOxA6tQGDk8UY17ltRu0ZG5UqwSJ_8ZtAl7ZNaA)
+
+This is a simple pipeline with 2 middlewares and one app. The arrows depict how the execution order would flow.
+
+### Executing a pipeline
+Your pipeline is executed. First middleware is called then the second and third. Then your app is called it does what it needs to and then the call goes back through the middlewares (so they can actually act twice).
+
+This probably does not seem that much useful so let's have a look at couple of examples where you might use it.
+
+Plumbing -  just the plumbing. Did you notice how we had to set up the logger in our Hello World example? It is not a lot of code but imagine that you need to do 10 things like this. It can bog you down. There are couple of middlewares that try to help you with similar stuff. It is similar to what AOP style of programming tries to do.
+
+Examples
+
+ * log in to various systems and prepeare for action
+ * Set up loggers
+
+Decorators - Imagine that you have done something great. For example computing hierarchy of people from some information. It is so much useful that you would like to let other people use it. But everybody has slightly different use case. Somebody wants to output it to web dav or s3 storage. Somebody want's to tweet about that it finished somebody might want to store this file into vertica. Implementing serialization in a separate middlewere means that you do not need to touch the actually code that.
+
+ * measuring time
+ * serializing stuff to various places
+ * letting other people know
+
+### First pipeline
+
+Ok let's create our first pipeline. Let's open misc/hello_world_pipeline_brick in your browser
+
+You will see two files. Let's check the hello_world.rb first.
+
+    module MyFirstBrick
+
+        class HelloWorldBrick
+            def call(params)
+                logger = params[:gdc_logger]
+                logger.info "Hello world"
+            end
+        end
+
+    end
+
+It looks exactly the same as in previosus case except for the logger setting up. Let's look at the other file main.rb
+
+You see that at the top we are requiring the hello_world.rb and then we are setting up the pipeline. You can see that even this basic example uses quite a bit of middleare but hopefully their names are fairly self describing. Logging will hook up a logger. GoodData logs you in to GD and hooks the library to the logger if you want to. Timinng will simply measure the execution of the app itself.
+
+Notice several things. Pipeline has exactly one app. It also has to be the last one in the pipeline. It has zero or more middlewares. When constructong a pipeline you can specify a stage either via a class or an instance. If it is a class we instatntiate it for you (without parameters). This is useful if you want to parametrize the middleware somehow.
+
+## Middlewares
+Let's have a look at some middlewares
+App was not very challenging from programming perspective (which is the goal) and you will see that middleware is not any more complicated. There are generally two types of middleares. First is a middleware that wants to act only once. It is like fire and forget. For example setting up logger. You just create it and do not care. There is a second type and that is a middleware that wants to act twice. Once before the app itself was called. Second after an app was called. Typical example might be time measuring. You want to start your clock before the app itself is called but then you have to stop them sometimes. The difference is absolutely minimal let's walk through both of them.
+
+### Logger middleware
+    require 'logger'
+
+    module GoodData::Bricks
+      class LoggerMiddleware < GoodData::Bricks::Middleware
+
+        def call(params)
+          logger = params[:gdc_logger] = params[:GDC_LOGGER_FILE].nil? ? Logger.new(STDOUT) : Logger.new(params[:GDC_LOGGER_FILE])
+          logger.info("Pipeline starts")
+
+          returning(@app.call(params)) do |result|
+            logger.info("Pipeline ending")
+          end
+        end
+
+      end
+    end
+
+### Benchmarking middleware
+
+    require 'benchmark'
+
+    module GoodData::Bricks
+
+      class BenchMiddleware < GoodData::Bricks::Middleware
+
+        def call(params)
+          puts "Starting timer"
+          result = nil
+          report = Benchmark.measure { result = @app.call(params) }
+          puts "Stopping timer"
+          pp report
+          result
+        end
+
+      end
+    end
+
+You see there are really only minimal changes. Let's walk through the couple of important points more carefully. As we stated before difference between app and a middleware is mainly in the fact that you can chain middlewares. Thus middleware has to know who is the next on in the chain and at some point it is going to call him. That is the `@app.call(params)`. Notice how we are still using the same interface. This way it is no difference if the next guy is app or middleware. The second important piece might be the returning function but that is just a way how to be more dry. Returning will take one param evaluate it store it you can do some stuff on it and then its value is returned. These things are equivalent.
+
+    returning(Person.new) do |o|
+      p.name = "Tomas"
+    end
+
+    p = Person.new
+    p.name = "Tomas"
+    p
+
+### Passing values
+
+We haven't talk much explicitly about passing values. The recommended way how to pass parameters is through the `:call(params)` methods parameters. You can see example of that for example in the logger middleware. It creates a logger and puts it into the param object. All following middlewares that are called can benefit from it. You can again see the usage in the HelloWorld.
+
+### Providing initial sets of parameters
+
+When deploying you will have to either provide parameters during execution or to the scheduler. Since we are developing locally it would be great to have similar functionality during local execution. This is exactly what `--params` parameter does.
+
+    bundle exec gooddata -v run_ruby --remote asdas/asdadas --params path_to_params_file.json
+
+The file is simple JSON file. This file is used in both local or remote execution.
+
+    {
+      "param1" : "value1",
+      "param2" : "value2"
+    }
+
+### Return parameters
+
+## Local harness
+Let's talk a little about the harness that makes it possible to test things locally. It tries to run your bricks in similar environment as it would run on the server. This mainly means that you will get the same parameters etc. The goal is to provide you environment to debug and test before you deploy.
+
+
+### Run locally
+
+The most complete command
+
+    bundle exec gooddata -l -s https://secure.gooddata.com -Uname@gooddata.com -Pmy_pass -w https://secure-di.getgooddata.com run_ruby -p rjzbt1shubkj9c8es6f75t2avke748mj -d brick_test --name "some_deploy"
+
+looks intimidating but let's break it down. Many of these can be omitted and are there in case you can override everything.
+
+    bundle exec
+
+means you are executing it with exact the same libraries as you would on the server (we are expecting that your local repo is up to date).
+
+    -l
+
+Means that HTTP communication will be logged to STDOUT.
+
+    -s https://secure.gooddata.com
+
+Means which datacenter you are connecting to. If you are not going against something special you should be able to leave this out
+
+    -w https://secure-di.getgooddata.com
+
+File staging area used for uploading the files for execution. Similar case as for he server. By default you should not be forced to use it.
+
+    -p PID
+
+Currently only execution context we have is project context. You have to specify a project PID to be abel to run your brick
+
+    -d path_to_brick
+
+This tells the tool where brick lives. It expect the directory where the main.rb lives. Do not point this directly to a file.
+
+    --name some_name
+
+Used when deployed to name the process so later you can identify it
+
+### Run remotely
+
+The only change to do when you are running things remotely is to add `--remote` to the command. Everything else should remain the same.
+
+<div class="section-nav">
+  <div class="left align-right">
+      <a href="/docs/file/doc/pages/tutorial/TEST_DRIVEN_DEVELOPMENT.md" class="prev">
+        Back
+      </a>
+  </div>
+  <div class="right align-left">
+
+      <span class="next disabled">Next</span>
+
+  </div>
+  <div class="clear"></div>
+</div>

data/doc/pages/tutorial/CREATING_A_MODEL.md

@@ -0,0 +1,79 @@
+There are several ways how to express a model and create it in GoodData. The most prominent way to do it is the visual modeler that is part of the CloudConnect package. There are clear advantages like being visual but there are also drawbacks. It is not repeatable, it is not programmable and it is not text based. Let's have a look how to create a simple model using Ruby SDK.
+
+## The model
+The model we will be creating is this
+
+## The code
+Create a file called model.rb and put this inside.
+
+    GoodData::Model::ProjectBuilder.create("gooddata-ruby test #{Time.now.to_i}") do |p|
+      p.add_date_dimension("closed_date")
+
+      p.add_dataset("users") do |d|
+        d.add_anchor("id")
+        d.add_label("name", :reference => 'id')
+      end
+
+      p.add_dataset("regions") do |d|
+        d.add_anchor("id")
+        d.add_attribute("name")
+      end
+
+      p.add_dataset("opportunities") do |d|
+        d.add_fact("amount")
+        d.add_date("closed_date", :dataset => "closed_date")
+        d.add_reference("user_id", :dataset => 'users', :reference => 'id')
+        d.add_reference("region_id", :dataset => 'regions', :reference => 'id')
+      end
+    end
+
+Hopefully the model is self descriptive and if you are not strong on terminology like label, anchor etc please refer to "Building a model with GD".
+
+## Some rules
+
+Please note several things
+
+  * we are trying to apply several conventions if you follow them it will be less typing for you but you can always override them.
+  * in all the cases where you type a name it is a string that will be used to create a technical name in gooddata also called identifier on the API. The user visible name which we call title will be inferred if not provided. The inferring process is simple. We expect you to provide name in the snake case (as is typical in ruby, this means things like close_date, opportunity_dimension etc). These will be translated int human readable strings (Close date, Opportunity dimension). If you do not like the title you can specify it directly via :title => "My own title"`
+  * the names are also used as reference names in the model. Notice how the date is using name of the close_date dimension and also the user_id reference is using reference users
+
+## Executing the model
+
+    gooddata --username joe@example.com --password my_secret_pass --token my_token  project build model.rb
+
+## Loading data
+As part of the process we allow you to load data since sometimes some initial datasets should be part of the model and not ETL. The typical usecase is for the sake of defining reports which are filtered on certain values these values have to be present.
+
+### Loading data given inline
+
+    p.upload([["id", "name"],
+              ["1", "Tomas"],
+              ["2", "Petr"]], :dataset => 'users')
+
+
+### Loading data given by filename
+
+    p.upload("/some/local_file.csv", :dataset => "users")
+
+### Loading data given by web file
+
+    p.upload("http://www.example.com/some/remote_file.csv", :dataset => "users")
+
+In all cases the file has to have headers that has the same name as the name of the particular columns (not necessarily in the same order).
+
+<div class="section-nav">
+  <div class="left align-right">
+      <a href="/docs/file/doc/pages/tutorial/YOUR_FIRST_PROJECT.md" class="prev">
+        Back
+      </a>
+  </div>
+
+  <div class="right align-left">
+
+      <a href="/docs/file/doc/pages/tutorial/CRUNCHING_NUMBERS.md" class="next">
+        Next
+      </a>
+
+  </div>
+  <div class="clear"></div>
+</div>

data/doc/pages/tutorial/CRUNCHING_NUMBERS.md

@@ -0,0 +1,233 @@
+<div markdown="1">
+    This is *true* markdown text.
+</div>
+
+MAQL is a language that is fairly similar to SQL but it is aimed towards getting the data from OLAP system. You are never forced to talk about columns and specify joins explicitly. This is great but there are some drawbacks. Same as SQL MAQL is aimed towards users more than machines which does not help for automation but there is one more caveat that make it hard to use even for humans. Probably to your surprise
+
+    SELECT SUM(Amount)
+
+Is not a MAQL statement (even though you probably have seen this on UI). The more correct (and what goes back and forth over the wire) is
+
+    SELECT SUM([/gdc/md/132131231/obj/1])
+
+GoodData UI does a great job at hiding this complexity from you but this significantly hinders the use of MAQL over and API by regular Joes. Ruby SDK tries to alleviate the situation with some tricks. It also gives you many tools to programmatically define and deal with reports and lays the foundations for test driven BI.
+
+### Jack in
+If you do not have a project best would be to create one by following our tutorial [Your first project](http://sdk.gooddata.com/gooddata-ruby/recipe/2014/01/19/your-first-project.html) so you can get predictable results.
+
+First let's look around. There are no metrics
+
+	  GoodData::Metric[:all]
+	  > []
+
+there is one fact
+
+	  GoodData::Fact[:all]
+  	> [{"link"=>"/gdc/md/ptbedvc1841r4obgptywd2mzhbwjsfyr/obj/223",
+	    "author"=>"/gdc/account/profile/4e1e8cacc4989228e0ae531b30853248",
+	    "tags"=>"",
+	    "created"=>"2014-02-18 07:44:26",
+	    "deprecated"=>"0",
+	    "summary"=>"",
+	    "title"=>"Lines changed",
+	    "category"=>"fact",
+	    "updated"=>"2014-02-18 07:44:26",
+	    "contributor"=>"/gdc/account/profile/4e1e8cacc4989228e0ae531b30853248"}]
+
+### First metric
+Let's create our first metric. There are couple of ways so I will show them one by one. Regardless of how you create the metric the result is the same so pick the one that suits your style or situation.
+
+TBD(add identifier based metric)
+
+	  m = GoodData::Metric.create("SELECT SUM([/gdc/md/ptbedvc1841r4obgptywd2mzhbwjsfyr/obj/223])")
+
+You can do it like this but obviously this is the ugly verbose way.
+
+    m = GoodData::Metric.xcreate('SELECT SUM(#"Lines changed")')
+
+Here you are using the name of the fact. Let's notice couple of things. First we are not using create any more. Method xcreate stands for eXtended notation and tries to turn it into valid MAQL. When you are specifying the fact you are doing it using #"NAME".
+
+Regardless of which way you used you have a metric definition. Metric is only locally on your computer we haven't saved it yet. Let's do it.
+
+    m.save
+    > RuntimeError: Meric needs to have title
+
+Uh ok. You have two options
+
+    m.title = "My shiny metric"
+
+or
+
+    m = GoodData::Metric.xcreate(:title => "My shiny metric", :expression => 'SELECT SUM(#"Lines changed")')
+
+Go ahead and try saving it again.
+
+    m.save
+    > #<GoodData::Metric:0x007f95b609b548 ....
+
+Great, looks good. Let's see if it worked
+
+    m.saved?
+    > true
+
+    m.uri
+    > "/gdc/md/ptbedvc1841r4obgptywd2mzhbwjsfyr/obj/292"
+
+Let's get some numbers. You can execute the metric.
+
+    m.execute
+    > 9.0
+
+Fantastic. You just created your first report over API.
+
+### More on metric execution
+
+Maybe you are wondering if you cannot just execute stuff to poke around. Well you kinda can. The API does not allow to execute a metric without it is saved (but we hope this will change soon). SDK tries to hide this from you but sometimes you can see the wiring. Let's explore. This is our well known metric
+
+    m = GoodData::Metric.xcreate('SELECT SUM(#"Lines changed")')
+
+Let's try executing it
+
+    m.execute
+    > 9
+
+It works. What happens behind the scenes is that SDK saves the metric and then deletes it again. It should mostly work
+
+    m.is_saved?
+    > false
+
+    m.uri
+    > nil
+
+But sometimes you can see some inconsistencies.
+
+    m.title
+    > "Untitled metric"
+
+This should not stop you most of the time. Just keep this in mind. Hopefully it will go completely away soon.
+
+### Defining a ReportDefinition
+
+Ok we have our metric. Now it would be interesting to see a report. The metric broken down. If you are familiar with the model you know the metric is a summation of lines changed in all commits in all products made by all developers. Let's see, how developers contributed.
+
+    GoodData::ReportDefinition.execute(:top => ["attr.devs.id"], :left => [m])
+    > [ 1  |  2  |  3 ]
+      [1.0 | 3.0 | 5.0]
+
+Again, there are a lots of ways how to achieve the same result so let's have a look at what is available right now. You can already see that you can see a metric by reference and attribute can be referenced just by a string containing an identifier. Let's pass the attribute as an object as well.
+
+    a = GoodData::Attribute.get_by_id("attr.devs.id")
+    GoodData::ReportDefinition.execute(:top => [a], :left => [m])
+
+If you studied UI well you know that Report is defined using Display Forms (or labels) not by attribute. If you are specifying an attribute SDK will take the first one automatically. This works well most of the time since attributes have typically just one label. But sometimes they have many so you need to me more specific. Coincidently our attribute has 2 labels.
+
+    a.display_forms.count
+    > 2
+
+The identifiers are "label.devs.email" and "label.devs.id". Let's try using those
+
+    GoodData::ReportDefinition.execute(:top => ["label.devs.id"], :left => [m])
+    GoodData::ReportDefinition.execute(:top => ["label.devs.email"], :left => [m])
+
+You can even do something that you cannot do on UI and that is using both of the labels at the same time (sic).
+
+    GoodData::ReportDefinition.execute(:top => ["label.devs.id", "label.devs.email"], :left => [m])
+
+In almost all above cases we had only one thing in left or top section. You can save your fingers and not use the array literal if there is only one item in the section. SDK will wrap them for you.
+
+    GoodData::ReportDefinition.execute(:top => "label.devs.id", :left => m)
+
+Sometimes it might be useful to refer to the objects in a different way. You can do it by title
+
+    GoodData::ReportDefinition.execute(
+      :top => [{:type => :attribute, :title => "Month/Year (committed_on)"}],
+      :left => m)
+
+Since underneath it uses MdObject.find_first_by_title it also accepts RegExp literal
+
+    GoodData::ReportDefinition.execute(
+      :top => [{:type => :attribute, :title => /Month\/Year/}],
+      :left => m)
+
+    GoodData::ReportDefinition.execute(
+      :top => [{:type => :attribute, :title => /month\/year/i}],
+      :left => m)
+
+In our model we have two attributes of name Id. Since the title does not have to be unique this is ok. Currently it will pick the first for you but this behavior will likely change in the favor of throwing an ambiguous error much like your SQL client probably does.
+
+    GoodData::ReportDefinition.execute(:top => [{:type => :attribute, :title => "Id"}], :left => m)
+
+Of course you can combine all things we learned together.
+
+    a = GoodData::Attribute.find_first_by_title(/month\/year/i)
+    GoodData::ReportDefinition.execute(:top => [{:type => :attribute, :title => "Id"}], :left => [m, a])
+
+### Reports
+Up until now we have been computing the reports just because. Maybe you wonder how you can actually create a report that would be saved. It is simple
+
+    GoodData::ReportDefinition.execute(
+      :top => [{:type => :attribute, :title => /Month\/Year/}],
+      :left => m)
+
+    report = GoodData::Report.create(
+      :title => "Fantastic report",
+      :top => [{:type => :attribute, :title => /Month\/Year/}],
+      :left => m)
+
+    report.save
+
+### Results
+
+Ok now we know how to create a report. Now let's see what we can do with the result. The point of this framework is not only you being able to create reports programmatically and save them for consumption over UI. Yes this is incredibly useful but when we have that why stop there. With Ruby SDK  you can actually consume the result programmatically as well so you can use GD as a basis for your application. Or as we show in this section we build a foundation for Test Driven BI development.
+
+Let's execute one of the previously defined reports and this time let's store the result
+
+    result = GoodData::ReportDefinition.execute(
+      :top => [{:type => :attribute, :title => /month\/year/i}],
+      :left => m)
+    >
+    [Jan 2014 | Feb 2014]
+    [1.0      | 8.0     ]
+
+The class is ReportDataResult. As you will see further it tries to conform to API of an array in key aspoects
+
+    result.class
+    > GoodData::ReportDataResult
+
+    result[0][0]
+    > "Jan 2014"
+
+    result[1][0]
+    > 1.0
+
+All the numbers are of BigDecimal class so you should be able to perform additional computations without losing precision
+
+    result[1][0].class
+    > BigDecimal
+
+Let's look on couple of methods that are useful for validating the results of reports
+
+    result.include_row? [1, 8]
+    > true
+
+    result.include_column? ["Feb 2014", 8]
+    > true
+
+Result is coming from server in a special format but after some processing it is just an 2 D array so it is no wonder that you can test on equality of a whole report.
+
+    result == [["Jan 2014", "Feb 2014"], [1, 8]]
+
+<div class="section-nav">
+  <div class="left align-right">
+      <a href="/docs/file/doc/pages/tutorial/CREATING_A_MODEL.md" class="prev">
+        Back
+      </a>
+  </div>
+
+  <div class="right align-left">
+      <a href="/docs/file/doc/pages/tutorial/TEST_DRIVEN_DEVELOPMENT.md" class="next">
+        Next
+      </a>
+  </div>
+  <div class="clear"></div
+</div>