goodot 0.0.2

Sign up to get free protection for your applications and to get access to all the features.
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in goodot.gemspec
4
+ gemspec
data/LICENSE ADDED
@@ -0,0 +1,22 @@
1
+ (BSD License)
2
+
3
+ Copyright (c) 2010-2016 GoodData Corporation. All rights reserved.
4
+
5
+ Redistribution and use in source and binary forms, with or without modification, are permitted provided
6
+ that the following conditions are met:
7
+
8
+ * Redistributions of source code must retain the above copyright notice, this list of conditions and
9
+ the following disclaimer.
10
+ * Redistributions in binary form must reproduce the above copyright notice, this list of conditions
11
+ and the following disclaimer in the documentation and/or other materials provided with the distribution.
12
+ * Neither the name of the GoodData Corporation nor the names of its contributors may be used to endorse
13
+ or promote products derived from this software without specific prior written permission.
14
+
15
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS
16
+ OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
17
+ AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
18
+ CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
19
+ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
20
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
21
+ WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
22
+ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
@@ -0,0 +1,5 @@
1
+ # encoding: UTF-8
2
+ #
3
+ # Copyright (c) 2010-2016 GoodData Corporation. All rights reserved.
4
+ # This source code is licensed under the BSD-style license found in the
5
+ # LICENSE file in the root directory of this source tree.
@@ -0,0 +1,550 @@
1
+ # Goodot (LCM CLI)
2
+
3
+ ## Status
4
+
5
+ [![Gem Version](https://badge.fury.io/rb/goodot.png)](http://badge.fury.io/rb/goodot)
6
+ [![Downloads](http://img.shields.io/gem/dt/goodot.svg)](http://rubygems.org/gems/goodot)
7
+ [![Dependency Status](https://gemnasium.com/gooddata/goodot.png)](https://gemnasium.com/gooddata/goodot)
8
+
9
+ ## What is it?
10
+
11
+ Toolset for helping with lifecycle management of the project, specifically
12
+
13
+ * CLI for doing typical tasks
14
+ * Best practices for creating applications
15
+
16
+ ### Installation
17
+
18
+ ### CLI
19
+
20
+ Currently the biggest piece of functionality is the CLI interface. It allows you to perform several typical tasks in the application. If you would like to see more open an issue on github pls.
21
+
22
+ As part of the gem installation executable `goodot` is installed on your machine. By running `goodot` you will see a list of commands you can use. We will walk through them here briefly to give you an overview.
23
+
24
+ #### Login parameters
25
+ In the same vein as GoodData SDK you have to either specify the login parameters on command line
26
+
27
+ goodot -U mustang@gooddata.com -P SECRET -s https://customer-domain.intgdc.com -d organization command
28
+
29
+ or you can rely on your GoodData SDK stored authentication information. In the examples the login information will be omitted for brevity.
30
+
31
+ #### Commands
32
+
33
+ ##### clients
34
+
35
+ ###### list
36
+
37
+ Will provide you table of clients in your application.
38
+
39
+ goodot clients list
40
+
41
+ ###### remove
42
+
43
+ Allows you to remove a particular client
44
+
45
+ goodot clients remove --segment SEGMENT_ID
46
+
47
+ By default it just disassociates the client from the segment but does not delete the project that was associated with the client. If you want to remove both the association between segment and client and also the client's project you can use
48
+
49
+ goodot clients remove --client CLIENT_ID --delete-project
50
+
51
+ ###### reset
52
+
53
+ Allows you to disassociate particular client's project from the client's definition. This means that during next "provision clients" phase (which you can invoke through `goodot app spin-up-projects` command) a fresh project will be created for that client.
54
+
55
+ goodot clients reset --client SEGMENT_ID
56
+
57
+ By default the project is just removed from client's defintion. If you also want to delete the project you can use.
58
+
59
+ goodot clients reset --client CLIENT_ID --delete-project
60
+
61
+ ##### segments
62
+
63
+ ###### list
64
+
65
+ Will provide you table of clients in your application.
66
+
67
+ goodot segments list
68
+
69
+ The response might look like this
70
+
71
+ +-----------+----------------------------------+
72
+ | Id | Master_project_id |
73
+ +-----------+----------------------------------+
74
+ | segment_1 | j87dughbcn2ybfjqua1i373j33sxy2pn |
75
+ | segment_2 | d7b70gzooqhijd3ozvoe6m2clp1620fg |
76
+ | segment_3 | al5v4rdvt8bhuoo84gp2ooize5kj9ahh |
77
+ +-----------+----------------------------------+
78
+
79
+ Sometimes output for machine processing is useful. This command can be also output as a json.
80
+
81
+ The response might look like this
82
+
83
+ [
84
+ {
85
+ "name": "segment_1",
86
+ "master_project_id": "j87dughbcn2ybfjqua1i373j33sxy2pn"
87
+ },
88
+ {
89
+ "name": "segment_2",
90
+ "master_project_id": "d7b70gzooqhijd3ozvoe6m2clp1620fg"
91
+ },
92
+ {
93
+ "name": "segment_3",
94
+ "master_project_id": "al5v4rdvt8bhuoo84gp2ooize5kj9ahh"
95
+ }
96
+ ]
97
+
98
+ ###### remove
99
+
100
+ Allows you to remove a particular segment.
101
+
102
+ goodot segments remove --segment SEGMENT_ID
103
+
104
+ By default it expects the segment to be empty ie it does not have any clients associated. If it is not the case it will fail. You can opt in to cascade delete all the clients in the segment by using `cascade` switch.
105
+
106
+ goodot segments remove --segment SEGMENT_ID --cascade
107
+
108
+ This will remove the clients and delete their respective projects.
109
+
110
+ ##### masters
111
+
112
+ ###### list
113
+
114
+ Will provide you table of master projects in your application along with their respective segment ids.
115
+
116
+ goodot masters list
117
+
118
+ The response might look like this
119
+
120
+ +----------------------------------+-----------------+------------+
121
+ | Pid | Title | Segment_id |
122
+ +----------------------------------+-----------------+------------+
123
+ | t4ygciqls3nonmi25qtc2hj9t6joi3w7 | HR Demo Project | segment_1 |
124
+ | lys4vtz363h5f2eq1ioglkjtimu2ze3c | HR Demo Project | segment_2 |
125
+ | yebbqafxmsb9eg4axgu3prunfrdh66r6 | Test project | segment-3 |
126
+ +----------------------------------+-----------------+------------+
127
+
128
+ ##### app
129
+
130
+ ###### create-segment
131
+
132
+ Allows you to add a segment. Useful for creating segments by hand. You have to provide a project that will serve as a master for the segment.
133
+
134
+ goodot app create-segment -s SEGMENT_ID -p PROJECT_ID
135
+
136
+ ###### add-client
137
+
138
+ Allows you to add a client to particular segment. Useful for populating the segments by hand.
139
+
140
+ goodot app add-client -c CLIENT_ID -s SEGMENT_ID
141
+
142
+ ###### spin-up-projects
143
+
144
+ With command `goodot app add-client` you will create clients in a segment but they by default will not have a project assigned. This command will allow to spin a new project for you. The project is spun up off a release. The release is created by `goodot app synchronize`.
145
+
146
+ goodot app spin-up-projects
147
+
148
+ ###### synchronize
149
+
150
+ Synchronize will do two things.
151
+
152
+ - It will package up current master proejct for each segment and aave them. This package will be used as a snapshot for spinning up all new projects by `spin-up-projects`.
153
+ - It will try to synchronize the existing projects that are not up to date.
154
+
155
+ goodot app synchronize
156
+
157
+ You can also specify segment to perform release and sync in just that particular segment.
158
+
159
+ goodot app synchronize -s SEGMENT_ID
160
+
161
+ ###### export-association
162
+
163
+ This will export the association between segments and clients for backup or documentation purposes.
164
+
165
+ It will export
166
+ - segment id
167
+ - client id
168
+ - project id
169
+
170
+ It either prints it to the STDOUT
171
+
172
+ goodot app export-association
173
+
174
+ or into a file if specified.
175
+
176
+ goodot app export-association -f file.json
177
+
178
+ Sometimes it might be useful to omit the project IDs. You can omit them in the output by using
179
+
180
+ goodot app export-association --without-project
181
+
182
+ ###### import-association
183
+
184
+ This will import the association stored in a file. This accepts the format which is output by `goodot app export-association`.
185
+
186
+ goodot app import-association -f file.json
187
+
188
+ The `segment` and `id` keys are required. The project key is optional. The way it is treated on API is that if you specify project it is set to that value. If you do not specify project it is kept as it is. The file is taken declaratively the result will be as specified in the file. Superfluous clients will be deleted including their projects
189
+
190
+
191
+ ## Lifecycle Tutorial
192
+
193
+ Let's walk through a sample implementation. We will use Ruby SDK and LCM SDK for all the steps.
194
+
195
+ ### Definitions
196
+
197
+ Before we start let's define and explain some terms
198
+
199
+ #### Segment
200
+
201
+ This is a term that we use for a group of customers that share the same model, same reports and same ETL. This does not mean the share the exact same instance of each but the get each their respective copy. We typically say "Customer X is in 'Basic' segment".
202
+
203
+ #### Client
204
+ This is another word for a customer. At gooddata a client (in context of LCM) has meaning of creating a relationship between specific segment and specific project (maybe not yet created). Again we can tyically say something like "Client 'acme_production' is in 'Basic' segment"
205
+
206
+ #### Master project
207
+
208
+ Each segment has a master project associated. This is the place where each client gets its model/etl/reports from.
209
+
210
+ ![Master-Segment](./img/master_segment.png)
211
+
212
+ While master project is defined in LCM as just a project that is associated with a segment as master project I would like to extend the definition a bit. The extension is here to allow you to
213
+
214
+ - Ability to identify a Master project which is no longer associated with any segment (API does not keep history of that)
215
+ - Ability to revert to a particular version in case something goes wrong
216
+ - Ability to keep track of the versions
217
+
218
+ If you will use LCM tooling all the low level stuff will be handled for you almost transparently. If you however wish to do it yourself but still would like to use LCM CLI for inspecting things this is the interface that it expects you to fulfill.
219
+
220
+ Master project has to have several metadata properties defined. You can easily define those using SDK by following this example. The properties are
221
+
222
+ "GD_LCM_TYPE" = "master_project"
223
+ "GD_LCM_VERSION" = "x.y.z" # for example '1.0.0'
224
+ "GD_LCM_SEGMENT" = "id_of_the_segment"
225
+
226
+ ### Hello world
227
+ This is a proverbial hello world of LCM applications. Let's assume we have an application that has 2 segments - Basic and Premium. These 2 have different models and different reports (very symbolically represented in this example). We would like to demonstrate the following in the hello world example
228
+
229
+ - there are 2 different segments (basic, premium) each having unique model and different reports
230
+ - each project has users synced automatically from data
231
+ - each project has data permissions synced automatically from data
232
+ - we will show how to perform update of ETL
233
+ - we will show how to perform update of model
234
+ - we will show how to perform addition of new dashboards
235
+ - we will show how to perform a rollback to earlier version of the application
236
+
237
+ ![Hello world](./img/hello_world.png)
238
+
239
+
240
+ #### Prerequisites
241
+ We assume that we have
242
+
243
+ - Whitelabeled GoodData Organization (contact support if you do not have that)
244
+ - ETL processes ready for data acquisition and data manipulation. I will call these "downloader" and "transformation". I will show only how to deploy them or update them not how to create them since this is beyond the scope of this tutorial
245
+ - we will refer to organization name with name 'organization_name' this will be different in your particular case
246
+ - we will refer to token with name 'token'. Again in your particular case this will be different.
247
+
248
+ #### Code
249
+
250
+ If you want to follow along all the code can be found on github [https://github.com/fluke777/lcm_example](https://github.com/fluke777/lcm_example)
251
+
252
+ ### Definition of segment masters
253
+
254
+ First we need to set up the application structure. We have to define 2 projects that will serve as masters for our segments' clients and then set up the segments itself. The masters of segments serve several purposes.
255
+
256
+ - They are the blueprint of the model. Every client project that will be in that particular segment will receive the same model
257
+ - Similarly it serves as a master for the ETLs. Every client project will get the ETLs (with some slight parametrization) as the master project.
258
+ - The master contains the dashboards. The clients' projects will receive those dashboards and all the dependent reports and metrics.
259
+
260
+ #### Basic segment's master
261
+
262
+ You can find the code for definition of the [basic segment](https://github.com/fluke777/lcm_example/blob/master/basic_segment.rb). The flow is pretty simple. We define a model. Then we create the processes and their schedules. When everything is ok we define the actual segment.
263
+
264
+ #### Premium segment's master
265
+
266
+ Let's repeat for the [premium segment](https://github.com/fluke777/lcm_example/blob/master/premium_segment.rb). It is almost identical code with the only difference of defining a different model. Again this is represented here very symbolically just by having a model with two datasets instead of one. In real situations they might be very different.
267
+
268
+ #### ETL
269
+
270
+ The ETL is compliant with the blueprint and it typically consists of three pieces executed in this order.
271
+
272
+ - load of data into the project
273
+ - creation of Data permissions
274
+ - addition of users into the project
275
+
276
+ The flow is illustrated on this picture
277
+
278
+ ![ETL flow](./img/project_etl.png)
279
+
280
+
281
+ #### Creation of segments
282
+
283
+ You can create segments by running code
284
+
285
+ ruby initial_deployment.rb
286
+
287
+ This will create the
288
+
289
+ Again you can verify everything went fine by inspecting the application
290
+
291
+ goodot -U DOMAIN_OWNER_LOGIN -P PASSWORD -s ORGANIZATION_URL -d ORGANIZATION_NAME masters list
292
+
293
+ The result will look something like this
294
+
295
+ +----------------------------------+-----------------------+-----------------+---------+
296
+ | PID | Title | Segment ID | Version |
297
+ +----------------------------------+-----------------------+-----------------+---------+
298
+ | zgetgfidjtcz5h0e2e5fafjl4y1zve2k | Basic Segment 1.0.0 | basic_segment | 1.0.0 |
299
+ | eun7nh25r0vx1f3bd8omtlsvna3tqplv | Premium Segment 1.0.0 | premium_segment | 1.0.0 |
300
+ +----------------------------------+-----------------------+-----------------+---------+
301
+
302
+ Notice we still do not have any clients defined. You can verify this by running
303
+
304
+ goodot -U DOMAIN_OWNER_LOGIN -P PASSWORD -s ORGANIZATION_URL -d ORGANIZATION_NAME clients list
305
+
306
+ ### Service project
307
+
308
+ Service segment is one place per application where the following happens. This is usually not a rule but it is typical since many things need to happen only once per application and it is beneficial if they happen only once and from the same place.
309
+
310
+ - the data downloads happen here for all projects
311
+ - the data preparation/mangling happen here
312
+ - the organization's users get created and/or updated here
313
+
314
+ The way service project is created is that we will create a project that is special. The project is created in a very similar way to the master project we showed previously.
315
+
316
+ ruby inital_deploy_service_project.rb
317
+
318
+ This project is currently not in any segment. You have to remember its PID.
319
+
320
+ #### Processes
321
+
322
+ Let me describe what the processes here do and why they are in the order they are in. The high level picture is looking like this
323
+
324
+ ![ETL in service project](./img/service_etl.png)
325
+
326
+ ##### Downloader/Data acquisition
327
+
328
+ This makes sure we have the data in place where we need them. If the customer loads the data directly to ADS or project specific storage this step might be superfluous. The case might be you are downloading the data from some API so this is the step where it would happen.
329
+
330
+ ##### Data transformation
331
+ Data does not necessarily need to arrive in the shape suitable to be consumed by subseqnet steps. Here is the place to slice and dice and mangle the data into appropriate shapes and put them into the right places. Again might not be needed.
332
+
333
+ ##### Segment/client association
334
+
335
+ Part of the data load will be information about which clients are in each segment. Here we are talking about creating or removing new clients. Moving them between segments is currently not covered (TODO: VERIFY). The association is handled by [Association brick](https://github.com/gooddata/app_store/tree/master/apps/segments_workspace_association_brick). Briefly what it does is
336
+
337
+ - makes sure old clients are discarded
338
+ - makes sure new clients are provisioned. When provisioned
339
+ - the association is added
340
+ - when deprovisioned
341
+ - the association is removed
342
+ - the client project is discarded
343
+
344
+ Take note that the association is expected to be defined declaratively. What should be provisioned is determined by comapring to current state. You can find more details about this brick here. [Association brick docs](https://github.com/gooddata/app_store/tree/master/apps/segments_workspace_association_brick)
345
+
346
+ #### Project provisioning
347
+
348
+ The previous step creates only association between a segment and a client. It says "This client is in this segment". This still does not mean the project is created for him. The project is created in this step. This means
349
+
350
+ - the model, ETL and reports are cloned from the released master project
351
+ - the schedules are enabled
352
+
353
+ ##### Organization users synchronization
354
+
355
+ This makes sure the users are created and updated in your organization. This only adds the information about users into GD - creates accounts for them. The actual assignement happens in the respective projects. You can read more about this brick [here](https://github.com/gooddata/app_store/tree/master/apps/users_brick). It is the same brick as the one used to add users into projects. It is used in mode 'add_to_organization'.
356
+
357
+ #### Client ETL kick off
358
+
359
+ The ETL's in the client are usually not scheduled at all. The reason for this is that they run only when the data are prepared. There are couple of strategies how to achieve this but this is probably the least error prone.
360
+
361
+ Since the platform does not provide a way how to declaratively define schedule triggers based on succesfull finished of a schedule in a different project we use (this brick)[https://github.com/gooddata/app_store/tree/master/apps/schedule_execution_brick] to achieve something similar. Take note that when we deployed the ETLs to clients we marked them with a tag. This will search for all the schedules marked with that tag and execute them.
362
+
363
+
364
+ ### Making initial release
365
+
366
+ Part of the previous script was also a relase. This means several things.
367
+
368
+ - Release is something like a marker in time
369
+ - Every new project spun up will get the model + that was in the segment master at the time of a release
370
+ - All current projects will be updated to the currently released version
371
+
372
+ Also note that we prepared the whole master upfront from scratch and only then we created the actual segments and released them. We will use similar approach when we update the segments. Specifically we will always be following one simple rule
373
+
374
+ - We will never update a master that is live.
375
+
376
+ Segment master is treated like an immutable thing. This will help us make sure that we will always be able to go back to a certain version of a master if something goes wrong. We will get in more detail how to update master (for example with new reports later)
377
+
378
+ ### Recap
379
+
380
+ Now we have everything working. Namely
381
+
382
+ - we have segments defined and released
383
+ - we have automatic processes in place that will provision new users, data_permissions and new clients based on incoming data.
384
+ - we have service project that contains processes to update users, provision new clients and kick off the ETL in clients
385
+
386
+ #### Try it out
387
+
388
+ Let's try it out. We have prepared a file that will upload some test data into appropriate places. Run
389
+
390
+ ruby service_data_load.rb SERVICE_PROJECT_PID
391
+
392
+ This will upload 2 files. One that defines several clients to be spun off. Two test users that will be added to domain. Now go to the service project in browser (the URI should look like this https://DOMAIN/admin/disc/#/projects/SERVICE_PROJECT_PID) and execute the first schedule in the pipeline (that should be *downloader*). After all finishes you should be able to verify that it went fine like this.
393
+
394
+ There should be additional clients
395
+
396
+ goodot -U DOMAIN_OWNER_LOGIN -P PASSWORD -s ORGANIZATION_URL -d ORGANIZATION_NAME clients list
397
+
398
+ The result should look like this
399
+
400
+ +------------+----------------------------------------------+------------------------------------------------+
401
+ | Client Id | Segment URI | Project URI |
402
+ +------------+----------------------------------------------+------------------------------------------------+
403
+ | acme | /gdc/domains/mustangs/segments/basic_segment | /gdc/projects/sjpi817nns6o8sszbsb6ohshevbtrbfe |
404
+ | hearst | /gdc/domains/mustangs/segments/basic_segment | /gdc/projects/c50pp64hh5m08v78118hiy2ehmruur0v |
405
+ | level_up | /gdc/domains/mustangs/segments/basic_segment | /gdc/projects/jmfwto12m0bvmd8vi6x4f9gbxf5l17hc |
406
+ | mastercard | /gdc/domains/mustangs/segments/basic_segment | /gdc/projects/if5k4ohvxyha8zdxe74j4rt5ggw93lst |
407
+ +------------+----------------------------------------------+------------------------------------------------+
408
+
409
+
410
+ There should be users in your domain. You can verify it that you jack in to your application
411
+
412
+ goodot -U DOMAIN_OWNER_LOGIN -P PASSWORD -s ORGANIZATION_URL -d ORGANIZATION_NAME jack-in
413
+
414
+ and execute this
415
+
416
+ domain.users.map(&:login)
417
+
418
+ This should return something like
419
+
420
+ ["admin@gooddata.com", "john@mustangs.com", "jane@mustangs.com"]
421
+
422
+ Exit with executing
423
+
424
+ exit
425
+
426
+ Now you should have your client projects spun up with ETLs and models and everything. Let's try to run one by hand. First you need data for the processes. Run
427
+
428
+ ruby client_data_load.rb CLIENT_ID
429
+
430
+ For example
431
+
432
+ ruby client_data_load.rb hearst
433
+
434
+ This will upload files for users and filters. The data load does not do much in the example apart from outputting a messeage into log. After the script finishes go to the project in the browser and run the *load* schedule. This will execute all the processes under that project. Once done we expect 2 things to happen.
435
+
436
+ - have some users inside a project
437
+ - have data permissions defined for them
438
+
439
+ The first one you can verify like this. Jack in
440
+
441
+ goodot -U DOMAIN_OWNER_LOGIN -P PASSWORD -s ORGANIZATION_URL -d ORGANIZATION_NAME jack-in
442
+
443
+ and execute
444
+
445
+ domain.clients('hearst').project.users.map { |u| [u.login, u.role.identifier] }
446
+
447
+ The second one can be verified by running
448
+
449
+ domain.clients('hearst').project.data_permissions.pmap { |f| [f.login, f.pretty_expression]}
450
+
451
+ ### What's next?
452
+
453
+ Now when we have the basis up and running we would like to cover couple of additional typical scenarios that will occur.
454
+
455
+ - How to release an updated version of ETL
456
+ - How to release new dashboards
457
+ - How to introduce new model changes
458
+
459
+ #### Releasing new changes
460
+
461
+ The process is basically the same as the one we used during initial release. Since all the changes we described are codified in the master for particular segment the steps are very similar for all the cases.
462
+
463
+ - We will clone the current master project
464
+ - We will update the master project. This means
465
+ - Changing the model
466
+ - Add new dasshboards
467
+ - Change the model
468
+ - We bump the version of the segments
469
+ - We swap the master project for the new one
470
+ - We make a release
471
+ - We update all the existing projects
472
+
473
+ The following picture illustrates the process
474
+
475
+ ![ETL in service project](./img/release.png)
476
+
477
+ There are couple of very important things to notice.
478
+
479
+ - We never changed the old master so if there is a need we can always downgrade back.
480
+ - Since we are swapping the masters in an instance there is no risk that some project might have been created from the old version and some from the new version.
481
+
482
+ Unfortunately everything is not perfect. There are couple of things that have to be considered durng the design of the application or that particular release. Let's discuss them briefly here.
483
+
484
+ #### The problem of change
485
+
486
+ There are couple of innate problems with any change introduced in the project and that is "what is the source of truth" and "what happens if there is some clash of old data and the new changes". Let's walk through typical examples of change in a project and how it affects thecomplexity of an application.
487
+
488
+ #### Model changes
489
+
490
+ There are basically two scenarios here. In scenario one you as a application provider is completely responsible for all the reports and dashboards provided. Customer has no ability to create new content. In such a case everything should be fine and you are able to introduce arbitrary changes into the model. This case is prevalent in scenarios where customers are accesing GD through some other application and typically do not have access to ad hoc capabilities.
491
+
492
+ The second scenario is that end users do have access to adhoc content creation capabilities. Now when they create some new content they are depending on model or metric that you provided. If you change that in future version you are risking that you either change the meaning of the report (which is very dangerous) or that you break outright it. The solution is either do not change anything just add new stuff which is something that cannot be done indefinitely becuase it adds clutter to the project or you have to migrate the reports in some way. Which generally requires some manual work.
493
+
494
+ #### ETL/Model changes
495
+
496
+ Here the problem arises from the fact that different versions of the application might need different data. You need to make sure that changes to several parts of the application happens at the same time which might be difficult to do and this is something that should be thought of in the innitial design. The worst case scenario is to shut down the ETLs and enable them only after the release. This is difficult since it requires to "stop the world" which is inherently difficult.
497
+
498
+ #### Example of a migration
499
+
500
+ When migrating to a new version we need to ensure several things happen. To emphasize again we want to uphold our rule of "not touching a live master project" in any way.
501
+
502
+ - We need to create a new master project. The change we want to make is
503
+ - adding a report
504
+ - adding a field
505
+ - changing the ETL
506
+ - We need to tag it appropriately
507
+ - We need to make new release
508
+ - We need to update existing projects
509
+
510
+ There is a small helper method that helps you doing that. We will try illustrating two examples how to use it. Imagine, we want to do two things in a migration
511
+
512
+ - Change a project model. Let's add a field
513
+ - We need to change the upload process since we need to fill it with new data
514
+ - Let's add a new dashboard that will take leverage of that new field
515
+
516
+ ##### Code based migration
517
+
518
+ The migration is fleshed out in file. Let's pay attention to this particular piece of code
519
+
520
+ release(domain, '2.0.0', auth_token: TOKEN, service_project: service_project) do |release|
521
+ release.with_segment('basic_segment') do |segment, new_master_project|
522
+ blueprint = new_master_project.blueprint
523
+ blueprint.datasets('dataset.departments').change do |d|
524
+ d.add_fact('fact.departments.number', title: 'NUMBER')
525
+ end
526
+ new_master_project.update_from_blueprint(blueprint)
527
+ redeploy_or_create_process(new_master_project, './scripts/load/2.0.0', name: 'load')
528
+ end
529
+ end
530
+
531
+ The method release will go over all the segments. Take its masters, clone them and give them to you to adjust how you would like to. This is accomplished by `release.with_segment('basic_segment')` call. We are saying that we want to change that particular master. We will get a new master that we can change. If particular segment's master is not requested it is just cloned. The old version is kept around. This is for the reason that all the masters are on the same version so you can reason about migrations easily. Once all the udates finish succesfully. The new masters are taken, tagged, and swapped as new segments' masters. Then the release is performed. Since this point any new spun up clients will receive the data from these new masters. We also have to update the exeisting projects which happens automatically as part of the release method.
532
+
533
+ ##### Hand based migration
534
+
535
+ While the previous example is nice because it is all done inside code this is not a typical flow for many changes. Usually the new reports are created by people sometimes over several weeks. We wanted to support this scenario as well so there is another way how to use the migration. You can basically opt out of using the proposed clone. This is done by using different method.
536
+
537
+
538
+ release(domain, '2.0.0', auth_token: TOKEN, service_project: service_project) do |release|
539
+ release.provide_segment('basic_segment') do |segment|
540
+ client.projects('PID_OF_YOUR_PROJECT')
541
+ end
542
+ end
543
+
544
+ In such case the framework does not create a clone for you. The project you return is used (given it has all the appropriate tags as defined above). It is your responsibility to provide a master project that is compliant with the expectations.
545
+
546
+ In this flow it is typical that you want to start from a project that is actually a clone of the current version so you can just tweak it here and there. There is a method in goodot that can help you with this.
547
+
548
+ (TODO add goodot command)
549
+
550
+