calmlens 0.20.0 → 0.22.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +4 -1
- data/lib/calmlens/assets.rb +149 -110
- data/lib/calmlens/cdn.rb +197 -41
- data/lib/calmlens/error_codes.rb +100 -84
- data/lib/calmlens/errors.rb +11 -2
- data/lib/calmlens/events.rb +50 -1
- data/lib/calmlens/folders.rb +14 -9
- data/lib/calmlens/hash_lists.rb +29 -9
- data/lib/calmlens/internal.rb +9 -19
- data/lib/calmlens/lenses.rb +79 -6
- data/lib/calmlens/option_keys.rb +56 -26
- data/lib/calmlens/policies.rb +50 -18
- data/lib/calmlens/project_configuration.rb +113 -0
- data/lib/calmlens/resource.rb +24 -6
- data/lib/calmlens/server_client.rb +17 -6
- data/lib/calmlens/session.rb +29 -12
- data/lib/calmlens/testing/asset_filters.rb +39 -0
- data/lib/calmlens/testing/classification.rb +8 -2
- data/lib/calmlens/testing/fake_api.rb +950 -397
- data/lib/calmlens/testing/mock_decision.rb +6 -4
- data/lib/calmlens/testing/rsql.rb +245 -0
- data/lib/calmlens/testing/test_input_detections.rb +61 -0
- data/lib/calmlens/testing/test_input_scenario.rb +6 -2
- data/lib/calmlens/testing/test_inputs.rb +30 -10
- data/lib/calmlens/testing.rb +56 -15
- data/lib/calmlens/transform_rules.rb +1 -1
- data/lib/calmlens/upload_body.rb +85 -2
- data/lib/calmlens/upload_mime_types.rb +9 -1
- data/lib/calmlens/upload_tokens.rb +3 -3
- data/lib/calmlens/version.rb +1 -1
- data/lib/calmlens/webhooks.rb +58 -13
- data/lib/calmlens.rb +1 -0
- data/sig/calmlens.rbs +112 -13
- data/sig/error_codes.rbs +28 -20
- data/sig/qualifiers.rbs +2 -0
- data/sig/types.rbs +488 -190
- metadata +6 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5fd7cb8e811f68c6be7989deb980a259d4a5643bc6f4402324c64236d258661b
|
|
4
|
+
data.tar.gz: f5e03fc720335e9c2ee2116549ddad9407a627449aed230f31eacf9a31743a3d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ea06ee7b551c7e4a4336f3fb972e0e1181a55b5a148cb89c3b281dff789efe60289db280c18ffe700f1e883f763be60b10560b6a8321773c776aa40df2c57c13
|
|
7
|
+
data.tar.gz: b50571100307aaa792130cf4ca1d980232185613d9376a244b7857a7f052e0e43ffcbaa584fa661e869d349c3971e9d5199a07ac053844ee3ddbaa496db5aa81
|
data/README.md
CHANGED
|
@@ -16,6 +16,8 @@ gem "calmlens"
|
|
|
16
16
|
|
|
17
17
|
The gem has no runtime dependencies: it uses only the standard library (Net::HTTP, JSON, OpenSSL). It ships YARD documentation and RBS signatures in `sig/`, so Steep and other RBS tools can check your calls and the response keys you read.
|
|
18
18
|
|
|
19
|
+
Versions follow semantic versioning: see [Versioning and support](https://calmlens.com/docs/sdk#versioning-and-support).
|
|
20
|
+
|
|
19
21
|
## Quick start
|
|
20
22
|
|
|
21
23
|
Create an API key and copy your project ID in the CalmLens portal, and keep both on your server.
|
|
@@ -67,6 +69,7 @@ New statuses and fields may appear in later API versions. The SDK returns them u
|
|
|
67
69
|
- [Serve content from the CDN](https://calmlens.com/docs/sdk/ruby#cdn-methods), with [lenses, transforms and signed URLs](https://calmlens.com/docs/sdk/ruby/reference#cdn-urls) and [age-restricted delivery](https://calmlens.com/docs/guides/delivering-assets#age-restricted-assets).
|
|
68
70
|
- [Organise assets in folders](https://calmlens.com/docs/sdk/ruby/reference#folders), [search them](https://calmlens.com/docs/guides/search) and [page through lists](https://calmlens.com/docs/sdk/ruby#pagination).
|
|
69
71
|
- [Match images against hash lists](https://calmlens.com/docs/sdk/ruby/reference#hash-lists).
|
|
72
|
+
- Keep [upload presets](https://calmlens.com/docs/guides/upload-presets), project settings and metadata fields in your code: `upload_presets` (`list`, `each`, `ensure`, `delete`), `project_settings` (`get`, `update`) and `metadata_fields` (`list`, `ensure`, `delete`). `ensure` and `update` create or replace by key or ID, and the `revision` you read refuses a change made since.
|
|
70
73
|
- [Read the events feed](https://calmlens.com/docs/sdk/ruby#pagination) and [verify webhooks in Rails or Rack](https://calmlens.com/docs/sdk/ruby#webhooks).
|
|
71
74
|
- [Handle errors](https://calmlens.com/docs/sdk/ruby/reference#errors) with automatic [retries, timeouts and idempotency keys](https://calmlens.com/docs/sdk/ruby/reference#retries-and-timeouts).
|
|
72
75
|
- [Type-check your code with RBS signatures](https://calmlens.com/docs/sdk/ruby/reference#types).
|
|
@@ -77,7 +80,7 @@ Every method is listed in the [Ruby SDK documentation](https://calmlens.com/docs
|
|
|
77
80
|
|
|
78
81
|
`CalmLens::MockServerClient` from `calmlens/testing` runs the real client against an in-memory stand-in for the API and CDN, so you can test your integration offline. See [Ruby testing](https://calmlens.com/docs/sdk/ruby/testing).
|
|
79
82
|
|
|
80
|
-
`CalmLens::TestInputs` signs test inputs with your project's test input key from the portal: `generate_text_code` turns a scenario, preset or signal into a text code, `paint_image` paints it as a test image, and `generate_media_code` gives a phrase for audio and video, so the real API returns the scenario you declared.
|
|
83
|
+
`CalmLens::TestInputs` signs test inputs with your project's test input key from the portal: `generate_text_code` turns a scenario, preset or signal into a text code, `paint_image` paints it as a test image, and `generate_media_code` gives a phrase for audio and video, so the real API returns the scenario you declared. Submit a text code as the whole text and a test image as the whole upload.
|
|
81
84
|
|
|
82
85
|
## Links
|
|
83
86
|
|
data/lib/calmlens/assets.rb
CHANGED
|
@@ -15,30 +15,30 @@ module CalmLens
|
|
|
15
15
|
# The prefix of this resource's operations in Internal::OPTION_KEYS.
|
|
16
16
|
OPERATIONS = 'assets'
|
|
17
17
|
|
|
18
|
-
|
|
19
|
-
|
|
18
|
+
# The asset fields an upload carries in its body: the JSON body for text, the `options` part of
|
|
19
|
+
# an upload form for files. They never go in the URL, which proxies and servers log, and the
|
|
20
|
+
# API refuses them there.
|
|
21
|
+
UPLOAD_OPTION_KEYS = %w[name description processingOptions visibility metadata tags authorId context externalId
|
|
22
|
+
externalUrl folderId uploadPresetId].freeze
|
|
20
23
|
MAX_MULTIPART_IMAGE_BYTES = 20 * 1024 * 1024
|
|
21
24
|
# The API's limits for one-request uploads: images up to 10 MiB (larger ones are uploaded in
|
|
22
25
|
# parts), audio and video up to 50 MiB and other files up to 5,000 MiB.
|
|
23
26
|
MAX_DIRECT_IMAGE_BYTES = 10 * 1024 * 1024
|
|
24
27
|
MAX_MEDIA_BYTES = 50 * 1024 * 1024
|
|
25
28
|
MAX_FILE_BYTES = 5000 * 1024 * 1024
|
|
29
|
+
# Audio and video sent in parts can be up to 2 GiB; the API holds free uploads, made before
|
|
30
|
+
# billing is set up, to MAX_MEDIA_BYTES itself.
|
|
31
|
+
MAX_LONG_MEDIA_BYTES = 2 * 1024 * 1024 * 1024
|
|
26
32
|
# The longest `idempotency_key` of an upload sent in parts: each part's key adds `:part-<n>`.
|
|
27
33
|
MAX_SPLIT_IDEMPOTENCY_KEY_LENGTH = 240
|
|
28
|
-
# The
|
|
29
|
-
#
|
|
30
|
-
|
|
31
|
-
# Processing statuses after which no decision will arrive without a retry.
|
|
32
|
-
FINISHED_PROCESSING_STATUSES = %w[failed].freeze
|
|
34
|
+
# The classification statuses of a settled asset: `completed` (with its decision in
|
|
35
|
+
# `classification.moderation.status`) or `failed`. {#wait_for_decision} returns at the first.
|
|
36
|
+
SETTLED_STATUSES = %w[completed failed].freeze
|
|
33
37
|
DECISION_POLL_FIRST_DELAY = 0.5
|
|
34
38
|
# Each wait between decision checks is this many times the one before, up to DECISION_POLL_MAX_DELAY.
|
|
35
39
|
DECISION_POLL_FACTOR = 1.5
|
|
36
40
|
DECISION_POLL_MAX_DELAY = 2.0
|
|
37
41
|
DEFAULT_DECISION_TIMEOUT = 60
|
|
38
|
-
# The lifetime of a signed URL when none is given: 15 minutes.
|
|
39
|
-
DEFAULT_SIGNED_URL_SECONDS = 15 * 60
|
|
40
|
-
# The longest signed URL lifetime the API accepts: 24 hours.
|
|
41
|
-
MAX_SIGNED_URL_SECONDS = 24 * 60 * 60
|
|
42
42
|
# The operations {#batch} runs: `delete` deletes each asset with its child assets and stored
|
|
43
43
|
# content; `remove-content` removes its stored content and keeps the record.
|
|
44
44
|
BATCH_OPERATIONS = %w[delete remove-content].freeze
|
|
@@ -48,18 +48,20 @@ module CalmLens
|
|
|
48
48
|
# Searches assets by meaning or keyword.
|
|
49
49
|
# @option options [String] :query
|
|
50
50
|
# @option options [String] :mode `hybrid`, `semantic` or `keyword`
|
|
51
|
-
# @option options [Integer] :
|
|
51
|
+
# @option options [Integer] :page_size how many results, 1 to 20 (default 10)
|
|
52
52
|
# @return [Hash] `{'query', 'items' => [{'asset', 'score', 'matchedFields'}], 'semanticStatus'}`
|
|
53
53
|
def search(options = {})
|
|
54
54
|
call(options, __method__) { |api, request| request('GET', project('assets/search'), query: api, request_options: request) }
|
|
55
55
|
end
|
|
56
56
|
|
|
57
|
-
# Lists one page of assets.
|
|
57
|
+
# Lists one page of assets, newest first unless `sort` says otherwise.
|
|
58
58
|
# @option options [String] :cursor from the previous page
|
|
59
59
|
# @option options [Integer] :page_size 1 to 100, default 50
|
|
60
|
-
# @option options [String] :filter an RSQL filter such as `classification.moderation.status==allowed
|
|
61
|
-
#
|
|
62
|
-
# @
|
|
60
|
+
# @option options [String, nil] :filter an RSQL filter such as `classification.moderation.status==allowed`;
|
|
61
|
+
# `folderId==fldr_…` lists one folder's assets and `folderId==null` the project root's
|
|
62
|
+
# @option options [String] :sort `createdAt` and `id`, comma-separated, descending when prefixed
|
|
63
|
+
# with `-`, such as `-createdAt,id`; `-createdAt` by default
|
|
64
|
+
# @return [Hash] a cursor page: items, nextCursor and hasMore.
|
|
63
65
|
def list(options = {})
|
|
64
66
|
call(options, __method__) do |api, request|
|
|
65
67
|
request('GET', project('assets'), query: api, request_options: request)
|
|
@@ -73,14 +75,14 @@ module CalmLens
|
|
|
73
75
|
def each(options = {}, &block)
|
|
74
76
|
return enum_for(:each, options) unless block
|
|
75
77
|
|
|
76
|
-
|
|
77
|
-
listing = pages(api['cursor']) { |index| list(api.merge('cursor' => index, 'request_options' => request)) }
|
|
78
|
-
listing.each { |page| page['items'].each(&block) }
|
|
79
|
-
nil
|
|
78
|
+
each_item(options, __method__, method(:list), &block)
|
|
80
79
|
end
|
|
81
80
|
|
|
82
81
|
# Lists one page of an asset's children, such as frames extracted from a video.
|
|
83
82
|
# @option options [String] :asset_id
|
|
83
|
+
# @option options [String] :cursor
|
|
84
|
+
# @option options [Integer] :page_size
|
|
85
|
+
# @option options [String] :sort as for {#list}
|
|
84
86
|
# @return [Hash] a page like {#list}
|
|
85
87
|
def list_children(options)
|
|
86
88
|
call(options, __method__) do |api, request|
|
|
@@ -89,6 +91,30 @@ module CalmLens
|
|
|
89
91
|
end
|
|
90
92
|
end
|
|
91
93
|
|
|
94
|
+
# Lists one page of an audio or video asset's findings timeline, in time order.
|
|
95
|
+
# `classification.moderation.timeline.segmentCount` says how many segments there are. Pass each
|
|
96
|
+
# page's `nextCursor` back unchanged; cursors are opaque.
|
|
97
|
+
# @option options [String] :asset_id
|
|
98
|
+
# @option options [String] :cursor
|
|
99
|
+
# @option options [Integer] :page_size
|
|
100
|
+
# @return [Hash] a cursor page of timeline segments: items, nextCursor and hasMore.
|
|
101
|
+
def list_timeline(options)
|
|
102
|
+
call(options, __method__) do |api, request|
|
|
103
|
+
query = without(api, 'assetId')
|
|
104
|
+
request('GET', project("assets/#{Internal.segment(api['assetId'])}/timeline"), query: query, request_options: request)
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
# Iterates every segment of an audio or video asset's findings timeline, in time order.
|
|
109
|
+
# @param options [Hash] the options of {#list_timeline}
|
|
110
|
+
# @yieldparam segment [Hash]
|
|
111
|
+
# @return [Enumerator, nil] an Enumerator when no block is given.
|
|
112
|
+
def each_timeline_segment(options, &block)
|
|
113
|
+
return enum_for(:each_timeline_segment, options) unless block
|
|
114
|
+
|
|
115
|
+
each_item(options, __method__, method(:list_timeline), &block)
|
|
116
|
+
end
|
|
117
|
+
|
|
92
118
|
# Iterates every child of an asset, fetching pages as needed.
|
|
93
119
|
# @param options [Hash] the options of {#list_children}
|
|
94
120
|
# @yieldparam asset [Hash]
|
|
@@ -96,10 +122,7 @@ module CalmLens
|
|
|
96
122
|
def each_child(options, &block)
|
|
97
123
|
return enum_for(:each_child, options) unless block
|
|
98
124
|
|
|
99
|
-
|
|
100
|
-
listing = pages(api['cursor']) { |index| list_children(api.merge('cursor' => index, 'request_options' => request)) }
|
|
101
|
-
listing.each { |page| page['items'].each(&block) }
|
|
102
|
-
nil
|
|
125
|
+
each_item(options, __method__, method(:list_children), &block)
|
|
103
126
|
end
|
|
104
127
|
|
|
105
128
|
# @option options [String] :asset_id
|
|
@@ -110,18 +133,19 @@ module CalmLens
|
|
|
110
133
|
end
|
|
111
134
|
end
|
|
112
135
|
|
|
113
|
-
# Waits until an asset
|
|
114
|
-
# `
|
|
115
|
-
#
|
|
116
|
-
# longer each time after that, up to 2
|
|
136
|
+
# Waits until an asset is settled, its `classification.status` `completed` (with its decision in
|
|
137
|
+
# `classification.moderation.status`) or `failed`, and returns the asset. A `queued` or `processing`
|
|
138
|
+
# asset is still waiting, even when a reviewer has already decided it. It retrieves the asset,
|
|
139
|
+
# waiting 0.5 seconds before the second check and 1.5 times longer each time after that, up to 2
|
|
140
|
+
# seconds.
|
|
117
141
|
# Webhooks report decisions without polling.
|
|
118
142
|
#
|
|
119
143
|
# `timeout:` is the whole wait in seconds; `request_options: {timeout:}` still limits each request.
|
|
120
144
|
# To stop waiting early, run the call in a thread you can stop: it only sleeps and sends requests.
|
|
121
145
|
# @option options [String] :asset_id
|
|
122
146
|
# @option options [Numeric] :timeout seconds to wait for a decision (default 60)
|
|
123
|
-
# @return [Hash] the
|
|
124
|
-
# @raise [TimeoutError] `sdk/timeout` when
|
|
147
|
+
# @return [Hash] the settled asset
|
|
148
|
+
# @raise [TimeoutError] `sdk/timeout` when the asset isn't settled in time; `details` holds
|
|
125
149
|
# `'assetId'` and the last `'classificationStatus'` and `'moderationStatus'`.
|
|
126
150
|
def wait_for_decision(options)
|
|
127
151
|
call(options, __method__) do |api, request|
|
|
@@ -155,7 +179,7 @@ module CalmLens
|
|
|
155
179
|
|
|
156
180
|
# Records a reviewer's decision, replacing the policy's: `classification.moderation.status`
|
|
157
181
|
# becomes the decision, the decision and note join `classification.moderation.manualReviews` and
|
|
158
|
-
# the matching `asset.<status>` event is sent with `
|
|
182
|
+
# the matching `asset.<status>` event is sent with the API key as its `actor`. An asset still pending
|
|
159
183
|
# moderation can only be blocked or purged. Purging needs `confirm_purge: true` and the
|
|
160
184
|
# `assets:delete` permission, and permanently deletes the stored content. Not retried: it carries
|
|
161
185
|
# no `Idempotency-Key`.
|
|
@@ -173,8 +197,12 @@ module CalmLens
|
|
|
173
197
|
|
|
174
198
|
# Completes the policy evidence behind a decided asset's moderation decision: the report's
|
|
175
199
|
# `classification.moderation.explanationComplete` becomes `true` and every action stays as decided.
|
|
176
|
-
# With `rerun: true` the asset is moderated again with the project's current policy and
|
|
177
|
-
#
|
|
200
|
+
# With `rerun: true` the asset is moderated again with the project's current policy and an
|
|
201
|
+
# `asset.rerun-requested` event is recorded; use it when the policy changed since the decision
|
|
202
|
+
# (`asset/policy-changed`). The asset returns unchanged: it keeps its decision until the re-run's
|
|
203
|
+
# is stored, reported by `asset.classified` and `asset.<status>` events, and a failed re-run keeps
|
|
204
|
+
# it. A reviewed asset raises `asset/reviewed`, and one whose content is gone
|
|
205
|
+
# `asset/content-unavailable`.
|
|
178
206
|
# @option options [String] :asset_id
|
|
179
207
|
# @option options [Boolean] :rerun
|
|
180
208
|
# @return [Hash] the asset
|
|
@@ -189,13 +217,14 @@ module CalmLens
|
|
|
189
217
|
# `requested`); the record and classification results are kept.
|
|
190
218
|
# @option options [String] :asset_id
|
|
191
219
|
# @return [Hash] the asset
|
|
192
|
-
def
|
|
220
|
+
def remove_content(options)
|
|
193
221
|
call(options, __method__) do |api, request|
|
|
194
222
|
request('DELETE', "/assets/#{Internal.segment(api['assetId'])}/content", request_options: request)
|
|
195
223
|
end
|
|
196
224
|
end
|
|
197
225
|
|
|
198
|
-
# Deletes the asset record and its stored content.
|
|
226
|
+
# Deletes the asset record and its stored content. An asset extracted from another (one with a
|
|
227
|
+
# `parentId`) raises `asset/child-not-deletable`: delete its parent instead.
|
|
199
228
|
# @option options [String] :asset_id
|
|
200
229
|
# @return [nil]
|
|
201
230
|
def delete(options)
|
|
@@ -207,12 +236,14 @@ module CalmLens
|
|
|
207
236
|
|
|
208
237
|
# Deletes up to 500 assets (`operation: 'delete'`) or removes their stored content
|
|
209
238
|
# (`'remove-content'`) in one request. Every asset must exist in the project and the key needs
|
|
210
|
-
# `assets:delete` for its folder,
|
|
239
|
+
# `assets:delete` for its folder, and a deletion can't name an asset extracted from another
|
|
240
|
+
# (`asset/child-not-deletable`), or nothing is queued. Returns once the operation is queued; the
|
|
211
241
|
# work runs in the background and records `asset.deleted` or `asset.content-removed` for each
|
|
212
|
-
# asset
|
|
242
|
+
# asset, with a `batch-id` fact naming `batchId`. Assets under a safety hold are skipped. Not
|
|
243
|
+
# retried: it carries no `Idempotency-Key`.
|
|
213
244
|
# @option options [Array<String>] :asset_ids 1 to 500 asset IDs
|
|
214
245
|
# @option options [String] :operation one of {BATCH_OPERATIONS}
|
|
215
|
-
# @return [Hash] `{'
|
|
246
|
+
# @return [Hash] `{'batchId', 'status' => 'queued', 'assetCount'}`
|
|
216
247
|
def batch(options)
|
|
217
248
|
call(options, __method__) do |api, request|
|
|
218
249
|
ids = api['assetIds']
|
|
@@ -238,27 +269,8 @@ module CalmLens
|
|
|
238
269
|
end
|
|
239
270
|
end
|
|
240
271
|
|
|
241
|
-
#
|
|
242
|
-
#
|
|
243
|
-
# @option options [String] :resource `rendered` (the API's default) or `original`, which takes no lens or transform
|
|
244
|
-
# @option options [Integer] :expires_in_seconds default 900
|
|
245
|
-
# @option options [String] :lens a lens identifier
|
|
246
|
-
# @option options [String, Hash] :transform a selector String or structured options, as for
|
|
247
|
-
# {CDN#get_asset_url}; they replace the lens's options of the same name
|
|
248
|
-
# @return [Hash] `{'url', 'expiresAt'}`
|
|
249
|
-
def create_signed_url(options)
|
|
250
|
-
call(options, __method__) do |api, request|
|
|
251
|
-
path = "/assets/#{Internal.segment(api['assetId'])}/signed-url"
|
|
252
|
-
body = { 'expiresInSeconds' => DEFAULT_SIGNED_URL_SECONDS }.merge(without(api, 'assetId'))
|
|
253
|
-
check_signed_url(body)
|
|
254
|
-
body['transform'] = Transforms.selector(body['transform']) if body.key?('transform')
|
|
255
|
-
request('POST', path,
|
|
256
|
-
body: body,
|
|
257
|
-
request_options: request)
|
|
258
|
-
end
|
|
259
|
-
end
|
|
260
|
-
|
|
261
|
-
# Uploads content and queues moderation; the call returns before classification finishes.
|
|
272
|
+
# Uploads content and returns without waiting for its decision (`wait=false`), with
|
|
273
|
+
# `classification.status` `queued`.
|
|
262
274
|
# Watch for `asset.allowed`, `asset.limited`, `asset.age-restricted`, `asset.blocked` and
|
|
263
275
|
# `asset.purged` events or retrieve the asset later.
|
|
264
276
|
#
|
|
@@ -267,6 +279,11 @@ module CalmLens
|
|
|
267
279
|
# `{type:, data:}` where `data` is a binary String or an IO. An IO streams without being read into
|
|
268
280
|
# memory; pass `size_in_bytes:` when its length is unknown (a pipe or socket).
|
|
269
281
|
#
|
|
282
|
+
# Text is sent as a JSON body. Other content is sent as `multipart/form-data`: the asset's fields
|
|
283
|
+
# (name, description, metadata, author ID, context and so on) in its `options` part, then the
|
|
284
|
+
# file. The URL carries only transport values (`wait`, the content type and sizes), never the
|
|
285
|
+
# asset's fields.
|
|
286
|
+
#
|
|
270
287
|
# The request carries an `Idempotency-Key` (generated unless you pass
|
|
271
288
|
# `request_options: {idempotency_key:}`), so a retried upload never creates a second asset.
|
|
272
289
|
#
|
|
@@ -281,6 +298,7 @@ module CalmLens
|
|
|
281
298
|
# @option options [Hash] :part `{type:, data:, is_final:}` for the first multipart part
|
|
282
299
|
# @option options [Integer] :total_size_in_bytes the whole file's size, for multipart uploads
|
|
283
300
|
# @option options [String] :url
|
|
301
|
+
# @option options [Array<String>] :tags tags for the asset, such as `['catalogue', 'backpacks']`
|
|
284
302
|
# @option options [String] :author_id your identifier for who wrote the content; CalmLens only
|
|
285
303
|
# compares it with the author IDs in `context`, to tell which posts share a speaker.
|
|
286
304
|
# @option options [Hash] :context `{items: [{text:, author_id:}]}`, earlier posts in the
|
|
@@ -288,7 +306,7 @@ module CalmLens
|
|
|
288
306
|
# {ModerationContext::USED_ITEMS} that fit in {ModerationContext::USED_CHARACTERS} characters.
|
|
289
307
|
# @return [Hash] the asset
|
|
290
308
|
def create(options)
|
|
291
|
-
call(options, __method__) { |api, request| submit(api, request,
|
|
309
|
+
call(options, __method__) { |api, request| submit(api, request, false) }
|
|
292
310
|
end
|
|
293
311
|
|
|
294
312
|
# Uploads content and waits for the moderation decision where the content type supports it.
|
|
@@ -303,15 +321,17 @@ module CalmLens
|
|
|
303
321
|
raise Internal.invalid('Text to classify must be a non-empty string', ErrorCodes::INVALID_UPLOAD_PAYLOAD)
|
|
304
322
|
end
|
|
305
323
|
|
|
306
|
-
submit(api, request,
|
|
324
|
+
submit(api, request, true)
|
|
307
325
|
end
|
|
308
326
|
end
|
|
309
327
|
|
|
310
328
|
# Uploads a later part of a multipart upload started with {#create} or {#classify} and `part:`.
|
|
311
|
-
#
|
|
329
|
+
# Whether the final part waits for the decision was chosen when the upload started; pass `wait:`
|
|
330
|
+
# only to override it.
|
|
312
331
|
# @option options [String] :asset_id
|
|
313
332
|
# @option options [Hash] :part `{data:, part_number:, is_final:}`; part numbers start at 2.
|
|
314
|
-
# @option options [Boolean] :
|
|
333
|
+
# @option options [Boolean] :wait `true` to wait for the decision, `false` to return with the
|
|
334
|
+
# asset `queued`
|
|
315
335
|
# @return [Hash] the asset; `classification.status` stays `uploading` until the final part.
|
|
316
336
|
def upload_part(options)
|
|
317
337
|
call(options, __method__) do |api, request|
|
|
@@ -326,7 +346,7 @@ module CalmLens
|
|
|
326
346
|
end
|
|
327
347
|
|
|
328
348
|
body = UploadBody.from(part, multipart: true, part_size: !@session.defer_upload_checks)
|
|
329
|
-
send_part(api['assetId'], number, part.fetch('isFinal', false), body, api.fetch('
|
|
349
|
+
send_part(api['assetId'], number, part.fetch('isFinal', false), body, api.fetch('wait', nil), request)
|
|
330
350
|
end
|
|
331
351
|
end
|
|
332
352
|
|
|
@@ -342,12 +362,12 @@ module CalmLens
|
|
|
342
362
|
asset = retrieve('assetId' => asset_id, 'request_options' => request_options)
|
|
343
363
|
processing = asset.dig('classification', 'status')
|
|
344
364
|
moderation = asset.dig('classification', 'moderation', 'status')
|
|
345
|
-
return asset if
|
|
365
|
+
return asset if SETTLED_STATUSES.include?(processing)
|
|
346
366
|
|
|
347
367
|
remaining = deadline - @session.monotonic
|
|
348
368
|
unless remaining.positive?
|
|
349
369
|
raise TimeoutError.new(SdkErrorCodes::TIMEOUT,
|
|
350
|
-
"Asset #{asset['id']}
|
|
370
|
+
"Asset #{asset['id']} was not settled after #{timeout}s " \
|
|
351
371
|
"(processing #{processing}, moderation #{moderation})",
|
|
352
372
|
details: { 'assetId' => asset['id'], 'classificationStatus' => processing,
|
|
353
373
|
'moderationStatus' => moderation })
|
|
@@ -358,7 +378,7 @@ module CalmLens
|
|
|
358
378
|
end
|
|
359
379
|
end
|
|
360
380
|
|
|
361
|
-
def submit(options, request_options,
|
|
381
|
+
def submit(options, request_options, wait)
|
|
362
382
|
raise Internal.invalid('Asset name is required', ErrorCodes::INVALID_UPLOAD_PAYLOAD) if options['name'].to_s.empty?
|
|
363
383
|
unless %w[content part url].one? { |key| options.key?(key) } && !options.key?('isFinal')
|
|
364
384
|
raise Internal.invalid('Supply exactly one of content, part, or url; isFinal belongs in part',
|
|
@@ -366,7 +386,7 @@ module CalmLens
|
|
|
366
386
|
end
|
|
367
387
|
options = with_used_context(options)
|
|
368
388
|
if options.key?('url')
|
|
369
|
-
return expect_asset(request('POST', project('assets'), query: { '
|
|
389
|
+
return expect_asset(request('POST', project('assets'), query: { 'wait' => wait }, body: options,
|
|
370
390
|
idempotent: true, request_options: request_options))
|
|
371
391
|
end
|
|
372
392
|
|
|
@@ -380,16 +400,40 @@ module CalmLens
|
|
|
380
400
|
end
|
|
381
401
|
|
|
382
402
|
body = UploadBody.from(content, multipart: multipart, part_size: !@session.defer_upload_checks)
|
|
383
|
-
return upload_in_parts(options, request_options,
|
|
403
|
+
return upload_in_parts(options, request_options, wait, mime, body) if !multipart && splits?(content, body)
|
|
404
|
+
|
|
405
|
+
check_upload_size(mime, body.size) unless multipart || @session.defer_upload_checks
|
|
406
|
+
part_query = first_part_query(options, content, mime, body) if multipart
|
|
407
|
+
if content.key?('plainText')
|
|
408
|
+
return expect_asset(request('POST', project('assets'), query: { 'wait' => wait },
|
|
409
|
+
body: text_body(options, content, multipart),
|
|
410
|
+
idempotent: true, request_options: request_options))
|
|
411
|
+
end
|
|
412
|
+
|
|
413
|
+
query = { 'wait' => wait, 'contentType' => content['type'], 'sizeInBytes' => body.size }
|
|
414
|
+
send_form(query.merge(part_query || {}), upload_fields(options), content['type'], body, multipart, request_options)
|
|
415
|
+
end
|
|
416
|
+
|
|
417
|
+
# The asset fields an upload sends, without unset values.
|
|
418
|
+
def upload_fields(options) = options.slice(*UPLOAD_OPTION_KEYS).compact
|
|
384
419
|
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
420
|
+
# The JSON body of a text upload: its fields and `content`, or a first `part` with the sizes.
|
|
421
|
+
def text_body(options, content, multipart)
|
|
422
|
+
text = content.except('sizeInBytes')
|
|
423
|
+
return upload_fields(options).merge('content' => text) unless multipart
|
|
424
|
+
|
|
425
|
+
sizes = { 'sizeInBytes' => options['sizeInBytes'] || content['sizeInBytes'],
|
|
426
|
+
'totalSizeInBytes' => options['totalSizeInBytes'] }.compact
|
|
427
|
+
upload_fields(options).merge('part' => text, **sizes)
|
|
428
|
+
end
|
|
429
|
+
|
|
430
|
+
# Sends file content, or a first part, as an upload form: the fields in its `options` part and
|
|
431
|
+
# the bytes in its `file` part. The query holds only transport values.
|
|
432
|
+
def send_form(query, fields, type, body, multipart, request_options)
|
|
433
|
+
form, form_type = body.to_form(JSON.generate(fields), type)
|
|
389
434
|
expect_asset(request('POST', project('assets'),
|
|
390
|
-
query: query, raw:
|
|
391
|
-
|
|
392
|
-
headers: { 'Content-Type' => content['type'], 'X-CalmLens-Upload' => multipart ? 'part' : 'content' }))
|
|
435
|
+
query: query, raw: form, idempotent: true, request_options: request_options, upload: true,
|
|
436
|
+
headers: { 'Content-Type' => form_type, 'X-CalmLens-Upload' => multipart ? 'part' : 'content' }))
|
|
393
437
|
end
|
|
394
438
|
|
|
395
439
|
# Whether content goes in parts: not text, of known size, readable again and larger than a part.
|
|
@@ -400,14 +444,14 @@ module CalmLens
|
|
|
400
444
|
# Uploads `body` as a multipart upload of `part_size_bytes` parts and returns the final part's
|
|
401
445
|
# asset. Part `n` after the first carries the key `<key>:part-<n>`, so repeating the call with
|
|
402
446
|
# the same key replays the parts the API already has and sends the rest.
|
|
403
|
-
def upload_in_parts(options, request_options,
|
|
447
|
+
def upload_in_parts(options, request_options, wait, mime, body)
|
|
404
448
|
key = request_options[:idempotency_key] || SecureRandom.uuid
|
|
405
449
|
if key.length > MAX_SPLIT_IDEMPOTENCY_KEY_LENGTH
|
|
406
450
|
raise Internal.invalid("idempotency_key can have at most #{MAX_SPLIT_IDEMPOTENCY_KEY_LENGTH} characters " \
|
|
407
451
|
'for content uploaded in parts')
|
|
408
452
|
end
|
|
409
453
|
unless @session.defer_upload_checks
|
|
410
|
-
mime.start_with?('image/') ? check_image_size(body.size) :
|
|
454
|
+
mime.start_with?('image/') ? check_image_size(body.size) : check_upload_size(mime, body.size, in_parts: true)
|
|
411
455
|
end
|
|
412
456
|
|
|
413
457
|
total = body.size
|
|
@@ -416,12 +460,9 @@ module CalmLens
|
|
|
416
460
|
piece = ->(number) { body.slice((number - 1) * part_size, [part_size, total - ((number - 1) * part_size)].min) }
|
|
417
461
|
first = piece.call(1)
|
|
418
462
|
type = options.dig('content', 'type')
|
|
419
|
-
query =
|
|
420
|
-
|
|
421
|
-
asset =
|
|
422
|
-
query: query, raw: first, idempotent: true, upload: true,
|
|
423
|
-
request_options: request_options.merge(idempotency_key: key),
|
|
424
|
-
headers: { 'Content-Type' => type, 'X-CalmLens-Upload' => 'part' }))
|
|
463
|
+
query = { 'wait' => wait, 'contentType' => type, 'sizeInBytes' => first.size, 'partNumber' => 1, 'isFinal' => false,
|
|
464
|
+
'totalSizeInBytes' => total }
|
|
465
|
+
asset = send_form(query, upload_fields(options), type, first, true, request_options.merge(idempotency_key: key))
|
|
425
466
|
send_remaining_parts(asset, count, piece, key, request_options)
|
|
426
467
|
end
|
|
427
468
|
|
|
@@ -452,16 +493,16 @@ module CalmLens
|
|
|
452
493
|
asset
|
|
453
494
|
end
|
|
454
495
|
|
|
455
|
-
def send_part(asset_id, number, final, body,
|
|
496
|
+
def send_part(asset_id, number, final, body, wait, request_options)
|
|
456
497
|
query = { 'partNumber' => number, 'isFinal' => final, 'sizeInBytes' => body.size }
|
|
457
|
-
query['
|
|
498
|
+
query['wait'] = wait unless wait.nil?
|
|
458
499
|
headers = { 'Content-Type' => 'application/octet-stream', 'X-CalmLens-Upload' => 'part' }
|
|
459
500
|
expect_asset(request('POST', "/assets/#{Internal.segment(asset_id)}/part", query: query, raw: body, idempotent: true,
|
|
460
501
|
request_options: request_options, upload: true,
|
|
461
502
|
headers: headers))
|
|
462
503
|
end
|
|
463
504
|
|
|
464
|
-
# Keeps only the earlier posts CalmLens uses, which keeps
|
|
505
|
+
# Keeps only the earlier posts CalmLens uses, which keeps an upload's body short.
|
|
465
506
|
def with_used_context(options)
|
|
466
507
|
return options unless options.key?('context')
|
|
467
508
|
|
|
@@ -471,7 +512,8 @@ module CalmLens
|
|
|
471
512
|
options.merge('context' => { 'items' => ModerationContext.select_items(context['items']) })
|
|
472
513
|
end
|
|
473
514
|
|
|
474
|
-
# The
|
|
515
|
+
# The transport values of a first multipart part: its number, whether it is the last and the
|
|
516
|
+
# declared total.
|
|
475
517
|
def first_part_query(options, content, mime, body)
|
|
476
518
|
total = options['totalSizeInBytes']
|
|
477
519
|
unless total.nil? || (total.is_a?(Integer) && total.positive?)
|
|
@@ -479,7 +521,10 @@ module CalmLens
|
|
|
479
521
|
end
|
|
480
522
|
|
|
481
523
|
final = content.fetch('isFinal', false)
|
|
482
|
-
|
|
524
|
+
unless @session.defer_upload_checks
|
|
525
|
+
# A declared whole-file size over the limit for parts can never finish.
|
|
526
|
+
mime.start_with?('image/') ? check_image_size(total || body.size) : check_upload_size(mime, total, in_parts: true)
|
|
527
|
+
end
|
|
483
528
|
{ 'partNumber' => 1, 'isFinal' => final, 'totalSizeInBytes' => total }
|
|
484
529
|
end
|
|
485
530
|
|
|
@@ -495,45 +540,34 @@ module CalmLens
|
|
|
495
540
|
raise InvalidResponseError.new(SdkErrorCodes::INVALID_RESPONSE, 'Expected an asset response', status: 200)
|
|
496
541
|
end
|
|
497
542
|
|
|
498
|
-
def check_signed_url(body)
|
|
499
|
-
resource = body.fetch('resource', 'rendered')
|
|
500
|
-
raise Internal.invalid("A signed URL's resource must be original or rendered") unless %w[original rendered].include?(resource)
|
|
501
|
-
if resource == 'original' && (body.key?('lens') || body.key?('transform'))
|
|
502
|
-
raise Internal.invalid('Lenses and transforms apply only to the rendered resource')
|
|
503
|
-
end
|
|
504
|
-
|
|
505
|
-
expires = body['expiresInSeconds']
|
|
506
|
-
return if expires.is_a?(Integer) && expires.between?(1, MAX_SIGNED_URL_SECONDS)
|
|
507
|
-
|
|
508
|
-
raise Internal.invalid('expires_in_seconds must be a whole number from 1 to 86400')
|
|
509
|
-
end
|
|
510
|
-
|
|
511
543
|
def check_mime(type)
|
|
512
544
|
mime = type.to_s.split(';').first.to_s.strip.downcase
|
|
513
545
|
unless Internal.accepted_upload_mime_type?(mime)
|
|
514
|
-
raise InvalidArgumentError.new(ErrorCodes::
|
|
546
|
+
raise InvalidArgumentError.new(ErrorCodes::UNSUPPORTED_UPLOAD_TYPE,
|
|
515
547
|
"#{mime} is not a valid upload file type. Uploads must use a concrete file MIME " \
|
|
516
548
|
'type that CalmLens can store and process, such as image/png, image/jpeg, ' \
|
|
517
|
-
'application/pdf, or text/plain.', details: { 'mimeType' => type })
|
|
549
|
+
'application/pdf, or text/plain.', details: { 'mimeType' => type, 'source' => 'calmlens' })
|
|
518
550
|
end
|
|
519
551
|
raise Internal.invalid('Content type cannot contain line breaks') unless type.match?(Internal::HEADER_VALUE)
|
|
520
552
|
|
|
521
553
|
mime
|
|
522
554
|
end
|
|
523
555
|
|
|
524
|
-
# Refuses
|
|
525
|
-
#
|
|
526
|
-
|
|
556
|
+
# Refuses content whose known size is over the API's limit for its type, before any of it is
|
|
557
|
+
# sent: `in_parts` selects the limits for content sent in parts. Images in parts are checked by
|
|
558
|
+
# {#check_image_size}.
|
|
559
|
+
def check_upload_size(mime, size, in_parts: false)
|
|
527
560
|
return if size.nil?
|
|
528
561
|
|
|
529
562
|
limit, message = if mime.start_with?('image/')
|
|
530
563
|
[MAX_DIRECT_IMAGE_BYTES, 'Images uploaded in one request can be at most 10 MiB; upload larger images in parts']
|
|
531
564
|
elsif mime.start_with?('audio/', 'video/')
|
|
532
|
-
[
|
|
565
|
+
[in_parts ? MAX_LONG_MEDIA_BYTES : MAX_MEDIA_BYTES,
|
|
566
|
+
'Audio and video files can be at most 50 MiB in one request, or 2 GiB in parts']
|
|
533
567
|
else
|
|
534
568
|
[MAX_FILE_BYTES, 'Files can be at most 5,000 MiB']
|
|
535
569
|
end
|
|
536
|
-
raise
|
|
570
|
+
raise too_large(message, limit) if size > limit
|
|
537
571
|
end
|
|
538
572
|
|
|
539
573
|
# Images in parts total at most 20 MiB, at any size below that; a declared total is checked
|
|
@@ -541,7 +575,12 @@ module CalmLens
|
|
|
541
575
|
def check_image_size(size)
|
|
542
576
|
return unless size > MAX_MULTIPART_IMAGE_BYTES
|
|
543
577
|
|
|
544
|
-
raise
|
|
578
|
+
raise too_large('Multipart image uploads cannot exceed 20 MiB.', MAX_MULTIPART_IMAGE_BYTES)
|
|
579
|
+
end
|
|
580
|
+
|
|
581
|
+
# `upload/too-large` for content over the plan's `limit` bytes, as the API answers it.
|
|
582
|
+
def too_large(message, limit)
|
|
583
|
+
InvalidArgumentError.new(ErrorCodes::UPLOAD_TOO_LARGE, message, details: { 'limitBytes' => limit, 'source' => 'plan' })
|
|
545
584
|
end
|
|
546
585
|
end
|
|
547
586
|
end
|