screwdriver-api 4.1.158 → 4.1.162
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
Binary file
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
@startuml sd-zip-artifacts-overview
|
|
2
|
+
|
|
3
|
+
skinparam sequenceMessageAlign center
|
|
4
|
+
|
|
5
|
+
participant "sd build" as build
|
|
6
|
+
participant "store" as store
|
|
7
|
+
database "s3(Object Storage)" as s3
|
|
8
|
+
participant "api" as api
|
|
9
|
+
participant "queue-service" as qservice
|
|
10
|
+
queue "Resque(Redis)" as resque
|
|
11
|
+
participant "unzip worker" as worker
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
build -> store: store-cli
|
|
15
|
+
store -> s3: upload zip file
|
|
16
|
+
build -> api: POST /builds/{id}/artifacts/unzip
|
|
17
|
+
api -> qservice: POST /queue/message
|
|
18
|
+
qservice -> resque: enqueue
|
|
19
|
+
qservice --> api: response\n(POST /queue/message)
|
|
20
|
+
api --> build: response\n(POST /builds/{id}/artifacts/unzip)
|
|
21
|
+
worker -> resque : dequeue
|
|
22
|
+
worker -> store: GET zip artifact
|
|
23
|
+
store --> worker: response\nGET zip artifact
|
|
24
|
+
worker -> store: upload unzip artifacts
|
|
25
|
+
|
|
26
|
+
@enduml
|
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
# SD ZIP ARTIFACTS
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
Currently SD_ZIP_ARTIFACTS function depends on AWS Lambda which has some resource limitation. To be free from the limitation, Screwdriver will have all role to realize the SD_ZIP_ARTIFACTS function.
|
|
6
|
+
|
|
7
|
+
## Proposal
|
|
8
|
+
|
|
9
|
+
After uploading a zip file in `sd-teardown-screwdriver-artifact-bookend` step, send a message to the queue.
|
|
10
|
+
The message will be received by the consumer(unzip worker) that unzips the zip file and re-uploads it.
|
|
11
|
+
|
|
12
|
+
## Overview
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
Outline:
|
|
17
|
+
|
|
18
|
+
1. upload zip file to Object Storage(S3) (sd build -> store -> S3)
|
|
19
|
+
1. notify to queue-service after upload zip file (sd build -> api -> queue-service)
|
|
20
|
+
1. send a message to unzip file (queue-service -> Resque)
|
|
21
|
+
1. a consumer(unzip worker) receives the message, then the consumer downloads the zip file and unzips it, re-uploads unzipped files
|
|
22
|
+
|
|
23
|
+
## Flow(Details)
|
|
24
|
+
|
|
25
|
+
### sd-teardown-screwdriver-artifact-bookend (sd build)
|
|
26
|
+
|
|
27
|
+
1. When SD_ZIP_ARTIFACTS=true, zip artifacts to SD_ARTIFACT.zip and upload to S3
|
|
28
|
+
1. After uploading SD_ARTIFACT.zip, send request to API(/v4/builds/{id}/artifacts/unzip)
|
|
29
|
+
|
|
30
|
+
### SD API (api)
|
|
31
|
+
|
|
32
|
+
1. Send request to queue-service
|
|
33
|
+
|
|
34
|
+
#### API
|
|
35
|
+
|
|
36
|
+
|Method|URL|Description|
|
|
37
|
+
|:--|:--|:--|
|
|
38
|
+
|POST|/builds/{id}/artifacts/unzip|send a request to the queue-worker to add a message to unzip the file|
|
|
39
|
+
|
|
40
|
+
##### Authentication & Authorization
|
|
41
|
+
|
|
42
|
+
Use sd-token to allow only requests when the build ID in the token and the id in the path are the same.
|
|
43
|
+
|
|
44
|
+
##### POST DATA
|
|
45
|
+
|
|
46
|
+
|Name|Type|In|Description|Example|
|
|
47
|
+
|:--|:--|:--|:--|:--|
|
|
48
|
+
|id|integer|path|build ID|12|
|
|
49
|
+
|
|
50
|
+
##### Response Data
|
|
51
|
+
|
|
52
|
+
Status(Success): 202 Accepted (Successfully sent a message to queue-service)
|
|
53
|
+
|
|
54
|
+
|Name|Type|In|Description|Example|
|
|
55
|
+
|:--|:--|:--|:--|:--|
|
|
56
|
+
|
|
57
|
+
No data to response
|
|
58
|
+
|
|
59
|
+
Status(Error):
|
|
60
|
+
401 Unauthorized (Token is not valid)
|
|
61
|
+
403 Forbidden (Build ID has no authority)
|
|
62
|
+
404 Not Found (The specified build id does not exist)
|
|
63
|
+
|
|
64
|
+
|Name|Type|In|Description|Example|
|
|
65
|
+
|:--|:--|:--|:--|:--|
|
|
66
|
+
|statusCode|integer|body|status code|401|
|
|
67
|
+
|error|string|body|outline of error|Unauthorized|
|
|
68
|
+
|message|string|body|detail message of error|Missing authentication|
|
|
69
|
+
|
|
70
|
+
### queue-service (queue-service)
|
|
71
|
+
|
|
72
|
+
1. Create token that has an unzip worker scope.
|
|
73
|
+
1. Enqueue the message to queue(Resque) (Queue Name: unzip)
|
|
74
|
+
|
|
75
|
+
#### JWT Information
|
|
76
|
+
|
|
77
|
+
|Name|Value|Description|Example|
|
|
78
|
+
|:--|:--|:--|:--|
|
|
79
|
+
|username|build ID|add build ID as username|12|
|
|
80
|
+
|scope|new component name|scope that shows it's a unzip worker|unzip_worker|
|
|
81
|
+
|exp|time until unzip|expiration time for JWT. It needs to be valid from sending queue message to re-uploading. Default time should be current time + 2 hours|1634198203|
|
|
82
|
+
|
|
83
|
+
#### Enqueue information
|
|
84
|
+
|
|
85
|
+
|Name|Type|Description|Example|
|
|
86
|
+
|:--|:--|:--|:--|
|
|
87
|
+
|buildId|integer|build ID|12|
|
|
88
|
+
|token|string|unzip worker scope JWT|jwt.unzip.token|
|
|
89
|
+
|
|
90
|
+
#### API
|
|
91
|
+
|
|
92
|
+
|Method|URL|Description|
|
|
93
|
+
|:--|:--|:--|
|
|
94
|
+
|POST|/queue/message|enqueue message to unzip file|
|
|
95
|
+
|
|
96
|
+
##### Authentication & Authorization
|
|
97
|
+
|
|
98
|
+
Use sd-token to only allow requests from SD API
|
|
99
|
+
|
|
100
|
+
##### POST DATA
|
|
101
|
+
|
|
102
|
+
|Name|Type|In|Description|Example|
|
|
103
|
+
|:--|:--|:--|:--|:--|
|
|
104
|
+
|type|string|query|what kind of operation|unzip|
|
|
105
|
+
|buildId|integer|body|build ID|12|
|
|
106
|
+
|
|
107
|
+
##### Response Data
|
|
108
|
+
|
|
109
|
+
Status(Success): 200 OK (Successfully sent a message to queue)
|
|
110
|
+
|
|
111
|
+
|Name|Type|In|Description|Example|
|
|
112
|
+
|:--|:--|:--|:--|:--|
|
|
113
|
+
|
|
114
|
+
No data to response.
|
|
115
|
+
|
|
116
|
+
Status(Error):
|
|
117
|
+
401 Unauthorized (Token is not valid)
|
|
118
|
+
403 Forbidden (Build ID has no authority)
|
|
119
|
+
|
|
120
|
+
|Name|Type|In|Description|Example|
|
|
121
|
+
|:--|:--|:--|:--|:--|
|
|
122
|
+
|statusCode|integer|body|status code|401|
|
|
123
|
+
|error|string|body|outline of error|Unauthorized|
|
|
124
|
+
|message|string|body|detail message of error|Missing authentication|
|
|
125
|
+
|
|
126
|
+
### Unzip Worker (unzip worker)
|
|
127
|
+
|
|
128
|
+
This is a new component.
|
|
129
|
+
|
|
130
|
+
1. Receive message from queue(Resque) the name is unzip.
|
|
131
|
+
1. Get SD_ARTIFACT.zip from SD Store by using build ID and Token
|
|
132
|
+
1. Unzip the SD_ARTIFACT.zip file and re-upload the unzip files to Store
|
|
133
|
+
1. If above process fails, retry.
|
|
134
|
+
1. If retry fails
|
|
135
|
+
1. Add a statusMessage to the build to notify the user that the unzip has failed
|
|
136
|
+
1. Log the failure.
|
|
137
|
+
|
|
138
|
+
### SD Store (store)
|
|
139
|
+
|
|
140
|
+
1. Upload and Download artifact files
|
|
141
|
+
|
|
142
|
+
SD Store do not need to add new function, but need to add new Authentication.
|
|
143
|
+
|
|
144
|
+
- Able to Upload and Download artifact files by unzip worker scope token
|
|
145
|
+
- Return an error if the build id of the build artifacts to be operated is different from the build id contained in the token
|