gd_bam 0.0.2 → 0.0.3

data/README.md

@@ -1,73 +1,73 @@
 #BAsh Machinery = BAM
 
-This thing is fresh from the oven. It is 0.0.1 so there are lots of rough edges. On the other hand there is enough to make you dangerous. Play with it, break it, let me know.
+BAM is a tool that helps you be more productive while creating and maintaining projects. 
 
-###What are goals of BAM
-* Be able to spin up a predefined project in hours not days
-* Everything implemented using CloudConnect so we can still leverage the infrastructure and deploy to secure
-* Make some specific modifications easy (new fields from sources)
 
-###What are not goals of BAM
-* Supersede Clover/CC
-* Provide templates for development. The generated grapsh are meant not to be tampered with.
-* define a whole processing language. This might be the next extension but right now the goal is to be able to spin up predefined projects as fast and easily as possible. I am not defining primitives for joins, reformats etc.
+##Installation
 
-##Overview
+make sure you have ruby (1.9 and 1.8.7 is currently supported) and that you have gem installed. On mac this is already on your machine.
 
-BAM is consisting of two parts. The underlying layer that allows you to build ETLs from prebuild constructs. Second part should make possible to express different configurations in user comprehensible way and configure first layer for specific projects so you do not need to deal with low level stuff when you decide that you want to use Amount instead of Total Price in GoodSales project.
+`gem install gd_bam`
 
-####1st layer
+Done.
 
-There are 3 basic pieces that you will be playing around. Let's have a look at those
+<!-- ##Spin up a new project
 
-1) Tap
-This is a fancy name for source of data. It can be downloader from SF or CSV file. Tap configurations are source specific. Currently there is SF implemented.
+Here project refers to the GoodData project.
 
-2) Sink
-This is the target of your data. The only sink we have currently is GD.
+`bam project`
 
-3) Graph
-This is a clover graph. So it plays well with Ultra it needs to be created in a specific way. You can use graphs from the library or those that you provide locally (N/A yet).
+This spins up a completely new empty project. You can further specify to spin up several predefined templates that bam knows about. Currently it is just goodsales. You do not need to worry about versions etc. Bam does this for you and spins up the latest available.
 
+###Scaffolding
+Scaffolding helps you create the files and provides you initial structure. Project here reffers to BAM project.
 
-4) Flow
-This is something that describes how the data are flowing. The previous three pieces are the things that you can use in the flow.
+create a project `bam scaffold project test`
 
+this will create a project. You can also scaffold a project from a know template. Currently it is again just goodsales. The goal is that after spinning you gooddata and BAM project you are good to run. If you want the defaults there is nothing you need to do besides filling credentials.
 
-###2nd layer
-TBD
+ -->
+##Sample project
 
-##Installation
+We will spin a goodsales project and load it with data. Prerequisite for this is a functioning Salesforce poject that you can grab at force.com.
 
-make sure you have ruby (1.9 and 1.8 is supported) and that you have gem installed
+`bam scaffold project test --blueprint goodsales`
 
-`gem install gd_bam`
+now you can go inside `cd test`. You will notice several directories and files. We will get to `flows`, `taps` and `sinks` late. Currently focus just on `params.json`
+If you open it you will see several parameters which you will have to fill. Common one should be predefined and empty. BAM should warn you when it needs something it does not have.
 
-create a project `bam scaffold project test`
+One of the parameters is project_pid. To get that you need a project.
 
-this will create a project
+`bam project --blueprint goodsales`
+This should spin for a while and eventually should give you project ID. Fill it in your params.json.
 
-##Sample project
+Now go ahead and generate the downloaders. 
+`bam generate_downloaders`
+
+We will talk in detail why we split etl for project into downloaders and rest of the ETL. Now just trust us. By default it is generated into downloader_project folder. You can go ahead and run it on platform.
+
+`bam run project_downloaders --email joe@example.com`
+
+You can watch the progress in our CloudConnect console.
+
+Now generate the etl. 
 
-now you can go inside `cd test` and generate it
 `bam generate`
 
-This tap will download users from sf (you have to provide credentials in params.json). It then runs graph called "process user" (this is part of the distribution). This graph concatenates first name and last name together.
+This works the same as with downloaders but its default target is clover_project
 
-	GoodData::CloverGenerator::DSL::flow("user") do |f|
-	  tap(:id => "user")
-	
-	  graph("process_owner")
-	  metadata("user") do |m|
-	    m.remove("FirstName")
-	    m.remove("LastName")
-	    m.add(:name => "Name")
-	  end
-	
-	  sink(:id => "user")
-	end
+`bam run clover_project --email joe@example.com`
 
-Now you have to provide it the definition of tap which you can do like this.
+After it is finished log in to gooddata go into your project and celebrate. You just did project using BAM.
+
+##Taps
+Taps are sources of data. There are various types (:-)) but right now you can use just salesforce tap.
+
+###Common properties
+Every tap has the source and an id. Source tells it where to go to grab data. DB, CSV, SalesForce. Depending on the type the definition might be different. Id is a field that holds a name with which you can reference particular tap. Obviously it needs to be unique.
+
+###Salesforce
+This is a tap that connects to SalesForce using credentials mentioned in params.json. Object tells the downlooader which SF onject to grab.
 
 	{
 	   "source" : "salesforce"
@@ -92,6 +92,83 @@ Now you have to provide it the definition of tap which you can do like this.
 	  ]
 	}
 
+####Limit
+Sometimes it is useful to limit number of grabbed values for example for testing purposes.
+
+	{
+	   "source" : "salesforce"
+	  ,"object" : "User"
+	  ,"id"     : "user"
+	  ,"fields" : [
+	    {
+	      "name" : "Id"
+	    }
+	    .
+	    .
+	    .
+	  ]
+	  ,"limit": 100
+	}
+
+####Acts as
+Sometime it is needed to use one field several times in a source of data or you want to "call" certain field differently because the ETL relies on partiular name. Both cases are handled using `acts_as`
+
+	{
+	   "source" : "salesforce"
+	  ,"object" : "User"
+	  ,"id"     : "user"
+	  ,"fields" : [
+	    {
+	      "name" : "Id", "acts_as" : ["Id", "Name"]
+	    },
+	    {
+	      "name" : "Custom_Amount__c", "acts_as" : ["RenamedAmount"]
+	    }
+	  ]
+	}
+
+Id will be routed to both Id and Name. Custom_Amount__c will be called RenamedAmount.
+
+####Condition
+You can also specify a condition during download. I recommend using it only if it drastically lowers the amount of data that goes over wire. Otherwise implement it elsewhere.
+
+####Incremental
+It is wasteful to download everything on and on again. If you specify the incremental=true you are telling BAM to download only incrementally. This means several things even if BAM tries to hide away as much as it can it might be condusing. Incremental tap means
+
+* tap is treated differently. If you generate the ETL with BAM generate the taps will not reach into SF but into some intermediary store.
+* with generate_downloaders you can generete graphs that will handle the incremental nature of downloaders and all the bookkeeping and make sure that nothing gets lost. It stores the data in an intermediary store.
+
+####Why split graphs?
+The reason for this is simple. When you download only incrementally you do not stress the wires that much and that means you can run it pretty often. By runnning it often it means that even if something horrible happens once it will probably run succesfully next time. And as we mentioned this is cheap. On the other hand runnning the main ETL is often very expensive and recovering from failure is usually different so splitting them simplifies development of each. Since they are independent they can be even developed by different people which is sometimes useful.
+
+####Taps validation
+Fail early. There is nothing more frustrating than when the ETL fails during exwcution. When you develop the taps you can ask BAM to connect to SF and validate that the fields are present. This is not bulletproof since some fields can go away at any time but it gives you good idea if you did not misspelled any fields.
+
+####Mandatory fields
+Sometimes it is necessary to move fields around in SF. In such case the tap will. If you know this upfront you can tell BAM that this field is not mandatory and it will silently go along filling the missing field with ''
+
+###Sinks
+
+
+This tap will download users from sf (you have to provide credentials in params.json). It then runs graph called "process user" (this is part of the distribution). This graph concatenates first name and last name together.
+
+	GoodData::CloverGenerator::DSL::flow("user") do |f|
+	  tap(:id => "user")
+	
+	  graph("process_owner")
+	  metadata("user") do |m|
+	    m.remove("FirstName")
+	    m.remove("LastName")
+	    m.add(:name => "Name")
+	  end
+	
+	  sink(:id => "user")
+	end
+
+Now you have to provide it the definition of tap which you can do like this.
+
+
+
 Also you need to provide a definition for sink which can look somwhow like this.
 
 	{
@@ -113,7 +190,7 @@ For this example to work you need to provide SF and gd credentials. Provide them
 Now run `bam generate` and there will be a folder with the clover project generated. Open it in CC find main.grf and run it. After crunching for a while you should see data in the project.
 
 ### Runtime commands
-Part of the distribution is the bam executable which lets you do several neat things.
+Part of the distribution is the bam executable which lets you do several neat things on the commandline
 
 Run `bam` to get the list of commands
 Run `bam help command` to get help about the command
@@ -148,4 +225,8 @@ sinks
 Currently works only for SF. Validates that the target SF instance has all the fields in the objects that are specified in the taps definitions.
 
 ### validate_datasets
-Vallidates the sinks (currently only GD) with the definitions in the proeject. It looks for fields that are defined inside sinks and are not in the projects missing references etc. More description needed.
+Vallidates the sinks (currently only GD) with the definitions in the proeject. It looks for fields that are defined inside sinks and are not in the projects missing references etc. More description needed.
+
+##Roadmap
+* Allow different storage then ES (Vertica)
+* Contract checkers

data/bin/bam

@@ -86,19 +86,114 @@ command :sinks_validate do |c|
   end
 end
 
+desc 'Creates project'
+command :project do |c|
+ 
+  c.desc 'blueprint name. Currently support goodsales'
+  c.arg_name 'blueprint'
+  c.flag :blueprint
+  
+  c.desc 'token'
+  c.arg_name 'token'
+  c.flag :token
+
+  c.action do |global_options,options,args|
+
+    pid = case options[:blueprint]
+      when "goodsales"
+        "i49w4c73c2mh75iiehte3fv3fbos8h2k"
+      when nil
+        fail "Empty project not supported now"
+    end
+    GoodData::CloverGenerator.connect_to_gd
+    with_users  = options[:with_users]
+    
+    export = {
+      :exportProject => {
+        :exportUsers => with_users ? 1 : 0,
+        :exportData => 1
+      }
+    }
+    
+    result = GoodData.post("/gdc/md/#{pid}/maintenance/export", export)
+    token = result["exportArtifact"]["token"]
+    status_url = result["exportArtifact"]["status"]["uri"]
+    
+
+    state = GoodData.get(status_url)["taskState"]["status"]
+    while state == "RUNNING"
+      sleep 5
+      result = GoodData.get(status_url) 
+      state = result["taskState"]["status"]
+    end
+    
+    old_project = GoodData::Project[pid]
+
+
+    pr = {
+      :project => {
+        :content => {
+          :guidedNavigation => 1,
+          :driver => "Pg",
+          :authorizationToken => options[:token]
+        },
+        :meta => {
+          :title => "Test Project",
+          :summary => "Testing Project"
+        }
+      }
+    }
+    result = GoodData.post("/gdc/projects/", pr)
+    uri = result["uri"]
+    while(GoodData.get(uri)["project"]["content"]["state"] == "LOADING")
+      sleep(5)
+    end
+
+    new_project = GoodData::Project[uri]
+    
+    import = {
+      :importProject => {
+        :token => token
+      }
+    }
+    
+    result = GoodData.post("/gdc/md/#{new_project.obj_id}/maintenance/import", import)
+    status_url = result["uri"]
+    state = GoodData.get(status_url)["taskState"]["status"]
+    while state == "RUNNING"
+      sleep 5
+      result = GoodData.get(status_url) 
+      state = result["taskState"]["status"]
+    end    
+
+  end
+
+end
+
 
 desc 'Generates structures'
 arg_name 'what you want to generate project, tap, flow, dataset'
 command :scaffold do |c|
+
+  c.desc 'blueprint name. Currently support goodsales'
+  c.arg_name 'blueprint'
+  c.flag :blueprint
+
   c.action do |global_options,options,args|
     command = args.first
     fail "You did not provide what I should scaffold. I can generate project, tap, flow, sink nothing else" unless ["project", "tap", "flow", "sink"].include?(command) 
     case command
     when "project"
-      puts "project"
       directory = args[1]
       fail "Directory has to be provided as an argument. See help" if directory.nil?
-      GoodData::CloverGenerator.setup_bash_structure(directory)
+      if options[:blueprint].nil?
+        GoodData::CloverGenerator.setup_bash_structure(directory)  
+      else
+        case options[:blueprint]
+        when "goodsales"
+          system "git clone git@github.com:gooddata/goodsales_base.git #{directory}"
+        end
+      end
     when "flow"
       name = args[1]
       fail "Name of the flow has to be provided as an argument. See help" if name.nil?
@@ -164,10 +259,8 @@ command :run do |c|
     verbose = global_options[:v]
 
     GoodData::CloverGenerator.connect_to_gd
-    GoodData::CloverGenerator.create_email_channel
     options = global_options.merge({:name => "temporary"})
-    GoodData::CloverGenerator.deploy(args.first, options) do |deploy_response|
-      
+    GoodData::CloverGenerator.deploy(dir, options) do |deploy_response|
       puts HighLine::color("Executing", HighLine::BOLD) if verbose
       GoodData::CloverGenerator.create_email_channel do
         GoodData::CloverGenerator.execute_process(deploy_response["cloverTransformation"]["links"]["executions"], dir)

data/lib/bam/version.rb

@@ -1,3 +1,3 @@
 module Bam
-  VERSION = '0.0.2'
+  VERSION = '0.0.3'
 end

data/lib/graphs/process_stage.grf

@@ -1,11 +1,11 @@
 <?xml version="1.0" encoding="UTF-8"?>
-<Graph author="fluke" created="Tue Feb 05 15:38:24 PST 2013" guiVersion="3.3.1" id="1360179808937" licenseType="Commercial" modified="Fri Feb 22 13:36:18 PST 2013" modifiedBy="fluke" name="process_name" revision="1.12" showComponentDetails="true">
+<Graph author="fluke" created="Tue Feb 05 15:38:24 PST 2013" guiVersion="3.3.2" id="1360179808937" licenseType="Commercial" modified="Wed Apr 17 09:21:46 PDT 2013" modifiedBy="josefvitek" name="process_name" revision="1.18" showComponentDetails="true">
 <Global>
 <Metadata fileURL="${PROJECT}/metadata/${FLOW}/${NAME}/1_in.xml" id="Metadata0"/>
 <Metadata fileURL="${PROJECT}/metadata/${FLOW}/${NAME}/1_out.xml" id="Metadata1"/>
 <MetadataGroup id="ComponentGroup0" name="metadata"/>
 <Property fileURL="params.txt" id="GraphParameter0"/>
-<Property fileURL="workspace.prm" id="GraphParameter1"/>
+<Property fileURL="workspace.prm" id="GraphParameter2"/>
 <Dictionary/>
 </Global>
 <Phase number="0">
@@ -20,9 +20,9 @@ function integer transform() {
 	
 	$out.0.* = $in.0.*;
 	$out.0.status = status;
-	$out.0.SortOrder = "0";
-	$out.0.label = $in.0.Id;
-
+    $out.0.label = $in.0.Id;
+    $out.0.SortOrder = nvl($in.0.SortOrder, "9999");
+  
 	return ALL;
 }
 

data/lib/runtime.rb

@@ -60,7 +60,7 @@ module GoodData
         FileUtils::mkdir_p base + dir
       end
       FileUtils::touch(base + 'params.txt')
-      render_template("project.erb", PARAMS, :to_file => base + '.project')
+      render_template("project.erb", PARAMS.merge(:project_name => "downloader-#{PARAMS[:project_name]}"), :to_file => base + '.project')
       render_template("workspace.prm.erb", PARAMS, :to_file => base + 'workspace.prm')
     end
 
@@ -165,38 +165,48 @@ module GoodData
       dry_run = options[:dry]
       project = build_project
       datasets = project.get_datasets
-      datasets.each do |ds|
-        dataset_path = "cl_file_#{ds[:id]}"
-        File.open(dataset_path, "w") do |temp|
-          builder = Builder::XmlMarkup.new(:target=>temp, :indent=>2)
-          builder.schema do |builder|
-            builder.name(ds[:gd_name])
-            builder.title(ds[:gd_name])
-            builder.columns do |b|
-              ds[:fields].each do |f|
-                builder.column do |builder|
-                  builder.title(f[:name])
-                  builder.ldmType(f[:type].upcase)
-                  builder.reference(f[:for]) if f.has_key?(:for)
-                  builder.reference(f[:ref]) if f.has_key?(:ref)
-                  builder.schemaReference(f[:schema]) if f.has_key?(:schema)
-                  if f[:type] == "date"
-                    builder.schemaReference("#{f[:dd]}")
-                    builder.name("#{f[:name]}")
-                  else
-                    builder.name(f[:name] || f[:ref])
+      model_update_dir = Pathname('model_update')
+      cl_home = ENV['CL_HOME'] || PARAMS['CL_HOME'] || fail("Home of cl tool cannot be found. Either set up CL_HOME in your env with 'export CL_HOME=path/to/cl or set it up in your params.json. Point to the directory of CL not to the bin dir.'")
+      cl_home = Pathname(cl_home) + 'bin/gdi.sh'
+      
+      FileUtils::mkdir_p(model_update_dir)
+      File.open(model_update_dir + 'xy', 'w')
+      FileUtils::cd(model_update_dir) do
+        datasets.each do |ds|
+          dataset_path = Pathname("cl_file_#{ds[:id]}")
+          File.open(dataset_path, "w") do |temp|
+            builder = Builder::XmlMarkup.new(:target=>temp, :indent=>2)
+            builder.schema do |builder|
+              builder.name(ds[:gd_name])
+              builder.title(ds[:gd_name])
+              builder.columns do |b|
+                ds[:fields].each do |f|
+                  builder.column do |builder|
+                    builder.title(f[:name])
+                    builder.ldmType(f[:type].upcase)
+                    builder.reference(f[:for]) if f.has_key?(:for)
+                    builder.reference(f[:ref]) if f.has_key?(:ref)
+                    builder.schemaReference(f[:schema]) if f.has_key?(:schema)
+                    if f[:type] == "date"
+                      builder.schemaReference("#{f[:dd]}")
+                      builder.name("#{f[:name]}")
+                    else
+                      builder.name(f[:name] || f[:ref])
+                    end
                   end
                 end
               end
             end
           end
+          template_name = dry_run ? "update_dataset_dry.script.erb" : "update_dataset.script.erb"
+          render_template(template_name, PARAMS.merge({"config_file" => dataset_path.expand_path}), :to_file => 'update_dataset.script')
+          puts "Generate #{ds[:id]}"
+
+          system("#{cl_home} update_dataset.script --username #{PARAMS[:gd_login]} --password #{PARAMS[:gd_pass]}")
+          File.delete(dataset_path)        
         end
-        template_name = dry_run ? "update_dataset_dry.script.erb" : "update_dataset.script.erb"
-        render_template(template_name, PARAMS.merge({"config_file" => dataset_path}), :to_file => 'update_dataset.script')
-        puts "Generate #{ds[:id]}"
-        system("~/Downloads/gooddata-cli-1.2.65/bin/gdi.sh update_dataset.script --username #{PARAMS[:gd_login]} --password #{PARAMS[:gd_pass]}")
-        File.delete(dataset_path)
       end
+      FileUtils::rm_rf(model_update_dir)
     end
 
     def self.generate_downloaders(options={})
@@ -216,6 +226,7 @@ module GoodData
 
 
     def self.execute_process(link, dir)
+      binding.pry
       result = GoodData.post(link, {
         :graphExecution => {
          :graph => "./#{dir}/graphs/main.grf",
@@ -228,7 +239,7 @@ module GoodData
     def self.connect_to_gd
         GoodData.logger = Logger.new(STDOUT)
         GoodData.connect(PARAMS[:gd_login], PARAMS[:gd_pass])
-        GoodData.project = PARAMS[:project_pid]
+        GoodData.project = PARAMS[:project_pid] if !PARAMS[:project_pid].nil? && !PARAMS[:project_pid].empty?
     end
 
     def self.create_email_channel(&block)
@@ -262,6 +273,7 @@ module GoodData
       deploy_name = options[:name]
       verbose = options[:verbose] || false
       puts HighLine::color("Deploying #{dir}", HighLine::BOLD) if verbose
+      res = nil
 
       Tempfile.open("deploy-graph-archive") do |temp|
         Zip::ZipOutputStream.open(temp.path) do |zio|
@@ -274,7 +286,6 @@ module GoodData
           end
         end
 
-
         GoodData.connection.upload(temp.path)
         process_id = options[:process]
 
@@ -291,6 +302,7 @@ module GoodData
         end
       end
       puts HighLine::color("Deploy DONE #{dir}", HighLine::BOLD) if verbose
+      res
     end
 
     def self.deploy(dir, options={}, &block) 

data/templates/update_dataset.script.erb

@@ -1,4 +1,4 @@
 OpenProject(id="<%= project_pid %>");
-UseCsv(csvDataFile="xy", configFile="<%= config_file %>", hasHeader="true", separator = ",");
+UseCsv(csvDataFile="model_update/xy", configFile="<%= config_file %>", hasHeader="true", separator = ",");
 GenerateUpdateMaql(maqlFile="<%= config_file %>.maql");
 ExecuteMaql(maqlFile="<%= config_file %>.maql");

data/templates/update_dataset_dry.script.erb

@@ -1,3 +1,3 @@
 OpenProject(id="<%= project_pid %>");
-UseCsv(csvDataFile="xy", configFile="<%= config_file %>", hasHeader="true", separator = ",");
+UseCsv(csvDataFile="model_update/xy", configFile="<%= config_file %>", hasHeader="true", separator = ",");
 GenerateUpdateMaql(maqlFile="<%= config_file %>.maql");

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: gd_bam
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.3
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-04-12 00:00:00.000000000 Z
+date: 2013-04-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -256,17 +256,17 @@ dependencies:
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - '='
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.5.9
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - '='
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.5.9
 - !ruby/object:Gem::Dependency
   name: highline
   requirement: !ruby/object:Gem::Requirement
@@ -393,7 +393,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -632830288147751058
+      hash: -596246612218275016
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -402,7 +402,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -632830288147751058
+      hash: -596246612218275016
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.25