dex-connectors-sdk 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dex_connectors_sdk-0.1.0/.gitignore +7 -0
- dex_connectors_sdk-0.1.0/PKG-INFO +310 -0
- dex_connectors_sdk-0.1.0/README.md +298 -0
- dex_connectors_sdk-0.1.0/examples/core.py +91 -0
- dex_connectors_sdk-0.1.0/examples/credentials_and_project.py +214 -0
- dex_connectors_sdk-0.1.0/examples/operations_and_oauth.py +161 -0
- dex_connectors_sdk-0.1.0/examples/single_dispatch.py +60 -0
- dex_connectors_sdk-0.1.0/integration/test_dex_flows.py +553 -0
- dex_connectors_sdk-0.1.0/pyproject.toml +36 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/__init__.py +98 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/configuration_ui.py +111 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/connectormock/__init__.py +545 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/credentials.py +157 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/errors.py +82 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/oauthtoken/__init__.py +298 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/operations.py +182 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/projectconfig/__init__.py +329 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/projectconfig/broker.py +262 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/projectconfig/connections.py +331 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/projectconfig/dexweb_store.py +175 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/projectconfig/models.py +480 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/projectconfig/provider.py +379 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/projectconfig/storage.py +387 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/providerhttp/__init__.py +184 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/py.typed +0 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/retry_policy.py +78 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/singledispatch/__init__.py +18 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/singledispatch/fence.py +173 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/singledispatch/markers.py +126 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/steps.py +363 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/timestamps.py +48 -0
- dex_connectors_sdk-0.1.0/src/dex_connectors/sdk/types.py +610 -0
- dex_connectors_sdk-0.1.0/tests/conftest.py +16 -0
- dex_connectors_sdk-0.1.0/tests/test_connectormock.py +167 -0
- dex_connectors_sdk-0.1.0/tests/test_core.py +217 -0
- dex_connectors_sdk-0.1.0/tests/test_oauthtoken.py +125 -0
- dex_connectors_sdk-0.1.0/tests/test_operations.py +216 -0
- dex_connectors_sdk-0.1.0/tests/test_project_credential_provider.py +417 -0
- dex_connectors_sdk-0.1.0/tests/test_projectconfig.py +594 -0
- dex_connectors_sdk-0.1.0/tests/test_providerhttp.py +123 -0
- dex_connectors_sdk-0.1.0/tests/test_singledispatch.py +207 -0
- dex_connectors_sdk-0.1.0/tests/test_steps.py +167 -0
- dex_connectors_sdk-0.1.0/uv.lock +902 -0
|
@@ -0,0 +1,310 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: dex-connectors-sdk
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Python core for Dex-native provider connectors
|
|
5
|
+
License-Expression: MIT
|
|
6
|
+
Requires-Python: >=3.11
|
|
7
|
+
Requires-Dist: boto3>=1.35
|
|
8
|
+
Requires-Dist: cryptography>=43
|
|
9
|
+
Requires-Dist: dex-python-sdk>=1.28.0
|
|
10
|
+
Requires-Dist: httpx>=0.27
|
|
11
|
+
Description-Content-Type: text/markdown
|
|
12
|
+
|
|
13
|
+
# Python Dex Connector SDK core
|
|
14
|
+
|
|
15
|
+
`dex-connectors-sdk` is the Python core for Dex-native provider connectors. It
|
|
16
|
+
uses the PEP 420 `dex_connectors` namespace and imports as `dex_connectors.sdk`.
|
|
17
|
+
Version `0.1.0` supplies the models, provider
|
|
18
|
+
operation contracts, credential helpers, OAuth token exchanges, and
|
|
19
|
+
single-dispatch checkpoint values below. Query and Mutation Step factories
|
|
20
|
+
use the released Dex SDK's instance-addressed async Steps.
|
|
21
|
+
Dex Web FDG 2.0 connector metadata for Python requires upstream R4.
|
|
22
|
+
|
|
23
|
+
Python 3.11 or newer and `dex-python-sdk>=1.28.0` are required. APIs are
|
|
24
|
+
async-first where they perform I/O. Connector Steps run in `AsyncWorker` and
|
|
25
|
+
never place credentials in Flow state.
|
|
26
|
+
|
|
27
|
+
Runnable examples are [`examples/core.py`](examples/core.py),
|
|
28
|
+
[`examples/credentials_and_project.py`](examples/credentials_and_project.py),
|
|
29
|
+
[`examples/operations_and_oauth.py`](examples/operations_and_oauth.py),
|
|
30
|
+
and [`examples/single_dispatch.py`](examples/single_dispatch.py).
|
|
31
|
+
Run each with `uv run python examples/<name>.py`.
|
|
32
|
+
|
|
33
|
+
## Core values, failures, and retry policy
|
|
34
|
+
|
|
35
|
+
`CallID` is a UUID string stable across retries. `Call` binds that identity to
|
|
36
|
+
the logical `ConnectionRef` and `OperationRef`. Its `AsyncContext` field is live
|
|
37
|
+
only during a Step handler and never enters JSON or Flow state. `QueryResult`,
|
|
38
|
+
`MutationResult`, `QueryDefinition`, and `MutationDefinition` encode the same
|
|
39
|
+
camel-case JSON as `sdkgo`, including optional
|
|
40
|
+
failure omission and RFC 3339 receipt timestamps. `FailureKind` describes a
|
|
41
|
+
fact; an operation decides whether to choose a branch or retry. The default
|
|
42
|
+
Execute retry policy starts at one second, doubles to five minutes, has no
|
|
43
|
+
attempt count limit, and ends after one hour. Exhaustion fails the Flow.
|
|
44
|
+
|
|
45
|
+
The core example constructs and round-trips a `QueryResult`, models a retry
|
|
46
|
+
attempt, and reads the default policy. `MutationAttempt.uncertain` is reserved
|
|
47
|
+
for a possibly accepted write that cannot be repeated or confirmed by read-back.
|
|
48
|
+
|
|
49
|
+
## Configuration UI
|
|
50
|
+
|
|
51
|
+
`ConnectorConfigurationUI` contains ordered `ConnectorUIUnit` records. Each
|
|
52
|
+
unit requires an instructional description and one or more
|
|
53
|
+
`ConnectorUIBinding` JSON Pointers. The core example constructs and validates a
|
|
54
|
+
complete unit. It carries presentation metadata only; configuration values
|
|
55
|
+
are loaded by `projectconfig` at application startup.
|
|
56
|
+
|
|
57
|
+
## Credentials
|
|
58
|
+
|
|
59
|
+
`CredentialProvider` is an async Protocol that resolves one `Call` at provider
|
|
60
|
+
call time.
|
|
61
|
+
`CredentialRefreshDriver` declares a deterministic refresh decision and async
|
|
62
|
+
exchange but does not store results. `SecretString` redacts string and repr
|
|
63
|
+
rendering and rejects serialization; call `reveal()` only when building a
|
|
64
|
+
provider request. The examples implement both credential Protocols and show
|
|
65
|
+
safe rendering. A connector
|
|
66
|
+
should return `ReauthorizationRequiredError` only when interactive provider
|
|
67
|
+
authorization is required. `StaticCredentialProvider` resolves an in-memory
|
|
68
|
+
connection map for examples and tests. `resolve_credential` delegates to
|
|
69
|
+
coordinated refresh when a provider supports it;
|
|
70
|
+
`resolve_credential_after_rejection` requires that support and should be called
|
|
71
|
+
at most once after a provider authentication rejection. Refreshing providers
|
|
72
|
+
serialize per connection, reload after taking that lock, and persist complete
|
|
73
|
+
replacement credentials atomically.
|
|
74
|
+
|
|
75
|
+
From the runnable [operation example](examples/operations_and_oauth.py):
|
|
76
|
+
|
|
77
|
+
```python
|
|
78
|
+
credentials = StaticCredentialProvider({connection: SecretString("INERT_SECRET")})
|
|
79
|
+
credential = await resolve_credential(credentials, call, ExampleRefreshDriver())
|
|
80
|
+
assert credential.reveal() == "INERT_SECRET"
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
## Provider operations
|
|
84
|
+
|
|
85
|
+
`Query` and `Mutation` are async provider contracts independent of Dex.
|
|
86
|
+
Each has `definition()` and `invoke(call, input)`; a Mutation also derives a
|
|
87
|
+
stable `idempotency_key(call_id, input)`. `run_query` and `run_mutation` validate
|
|
88
|
+
the attempt against declared branches, complete the safe receipt, and raise
|
|
89
|
+
`RetryError` for a safe-to-repeat failure. A returned unknown branch, foreign
|
|
90
|
+
receipt, or invalid attempt selects `defect`. An `uncertain` Mutation requires a declared route and
|
|
91
|
+
the narrow unconfirmable-write conditions in the repository rules. Retry
|
|
92
|
+
exhaustion is handled by Dex and fails the Flow.
|
|
93
|
+
`RetryError.retry_after_whole_seconds` truncates positive fractional delays to
|
|
94
|
+
the whole seconds accepted by Dex; zero means no explicit retry hint.
|
|
95
|
+
|
|
96
|
+
From the same runnable example:
|
|
97
|
+
|
|
98
|
+
```python
|
|
99
|
+
result = await run_mutation(CreateExampleItem(), call, "item-1")
|
|
100
|
+
assert result.branch == "created" and result.receipt.idempotency_key == str(call_id)
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## Connector Steps
|
|
104
|
+
|
|
105
|
+
`QueryStep` and `MutationStep` are async Dex `Step` instances configured with
|
|
106
|
+
their corresponding `QueryStepConfig` or `MutationStepConfig`. The instance's
|
|
107
|
+
stable `step_type` and concrete `step_input_type` let Dex register multiple
|
|
108
|
+
instances of one generic class. A required branch needs a target; an unwired
|
|
109
|
+
optional branch fails the Flow with a bounded, secret-safe reason. `StepOptions`
|
|
110
|
+
overrides may change Execute settings but cannot add WaitFor behavior. The
|
|
111
|
+
factory maps a connector `RetryError` to Dex retry, truncating RetryAfter to
|
|
112
|
+
whole seconds; retry exhaustion remains a failed Flow. A configured result
|
|
113
|
+
Attribute is written before routing and belongs in the Flow persistence schema.
|
|
114
|
+
`ProgressUpdate` Stream messages include call ID, Dex attempt, and monotonic
|
|
115
|
+
connector sequence. `Call.report_progress(Progress(...))` emits them when a
|
|
116
|
+
progress Stream is configured; `Call.write_text` writes ordered buffered text
|
|
117
|
+
when a text Stream is configured. These outputs are best effort and belong in
|
|
118
|
+
the consuming Flow persistence schema. The default text flush interval is one
|
|
119
|
+
second with a 16 KiB buffer; Step configs can override both settings.
|
|
120
|
+
|
|
121
|
+
From [the runnable operation example](examples/operations_and_oauth.py):
|
|
122
|
+
|
|
123
|
+
```python
|
|
124
|
+
created = CreatedItem()
|
|
125
|
+
result_attribute = dex.Attribute("createdItem", MutationResult[str])
|
|
126
|
+
progress_stream = dex.Stream("createProgress", ProgressUpdate, 64 * 1024)
|
|
127
|
+
text_stream = dex.Stream("createText", str, 64 * 1024)
|
|
128
|
+
create = MutationStep(MutationStepConfig(
|
|
129
|
+
"CreateItem", str, StepAnnotations("sample", "Sample", "Create one sample item."),
|
|
130
|
+
CreateExampleItem(), connection, lambda value: value,
|
|
131
|
+
{"created": created},
|
|
132
|
+
result_attribute=result_attribute,
|
|
133
|
+
progress_stream=progress_stream, text_stream=text_stream,
|
|
134
|
+
))
|
|
135
|
+
dex.Registry([
|
|
136
|
+
CreateItemFlow(create, created, result_attribute, progress_stream, text_stream)
|
|
137
|
+
], allow_async_handlers=True)
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
## OAuth token refresh
|
|
141
|
+
|
|
142
|
+
`oauthtoken.TokenEndpoint` performs bounded, redirect-disabled refresh-token,
|
|
143
|
+
client-credentials, and JWT-bearer exchanges. The connector owns its endpoint,
|
|
144
|
+
terminal error codes, scopes, and credential type. A declared terminal grant
|
|
145
|
+
error below HTTP 500 raises `ReauthorizationRequiredError`; server outages and
|
|
146
|
+
other failures stay retryable. `TokenResponse.next_refresh_token` retains the
|
|
147
|
+
prior refresh token when a provider does not rotate it. The package also has
|
|
148
|
+
RS256 JWT assertion signing, scope checks, and proportional expiry skew.
|
|
149
|
+
|
|
150
|
+
From the runnable example:
|
|
151
|
+
|
|
152
|
+
```python
|
|
153
|
+
endpoint = TokenEndpoint("Example", "https://oauth.example/token", http_client)
|
|
154
|
+
oauth_client = ClientCredentials(
|
|
155
|
+
"example-client", SecretString("INERT_CLIENT_SECRET"),
|
|
156
|
+
ClientAuthenticationMethod.CLIENT_SECRET_POST,
|
|
157
|
+
)
|
|
158
|
+
token = await endpoint.exchange_refresh_token(oauth_client, SecretString("INERT_REFRESH"))
|
|
159
|
+
assert token.access_token.reveal() == "INERT_ACCESS"
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
## Single dispatch checkpoints
|
|
163
|
+
|
|
164
|
+
`singledispatch.Marker` uses the Go heartbeat JSON shape. `decide_dispatch`
|
|
165
|
+
allows an initial send or a resend only after the immediately prior attempt
|
|
166
|
+
recorded `notSent` or `canResend` without a provider ID. A claimed, missing,
|
|
167
|
+
stale, or accepted checkpoint requires read-back; malformed or foreign data
|
|
168
|
+
fails closed. `singledispatch.Fence` reads `Call.context.deadline` and the
|
|
169
|
+
previous Dex heartbeat before credential lookup. `dispatch(send)` checkpoints
|
|
170
|
+
`claimed` before one provider send and repeats that claim every five seconds.
|
|
171
|
+
The callback receives the write deadline and must avoid transport replay.
|
|
172
|
+
Call `record_not_sent` before retrying a credential or pre-send failure;
|
|
173
|
+
`record_can_resend` requires positive proof that the provider did not process
|
|
174
|
+
the request. `record_provider_id` preserves an accepted resource identity for
|
|
175
|
+
read-back. Use sync Execute durability because local async activities ignore
|
|
176
|
+
heartbeats.
|
|
177
|
+
|
|
178
|
+
From [the runnable single-dispatch example](examples/single_dispatch.py):
|
|
179
|
+
|
|
180
|
+
```python
|
|
181
|
+
fence = Fence(call, timedelta(seconds=10))
|
|
182
|
+
assert (await fence.decide()).kind == DecisionKind.SEND_ALLOWED
|
|
183
|
+
|
|
184
|
+
async def send(write_deadline: datetime) -> str:
|
|
185
|
+
"""Use the deadline when constructing a real non-replaying provider request."""
|
|
186
|
+
assert write_deadline > datetime.now(UTC)
|
|
187
|
+
return "INERT_PROVIDER_OBJECT"
|
|
188
|
+
|
|
189
|
+
provider_id = await fence.dispatch(send)
|
|
190
|
+
await fence.record_provider_id(provider_id)
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
## Provider HTTP
|
|
194
|
+
|
|
195
|
+
`new_provider_http_client` returns an `httpx.AsyncClient` with redirects
|
|
196
|
+
disabled so authorization headers are never replayed to another host.
|
|
197
|
+
`read_bounded_body` rejects an oversized response. `read_error_tokens` extracts
|
|
198
|
+
only safe machine tokens from bounded JSON error bodies, and
|
|
199
|
+
`parse_retry_after` accepts delta seconds or HTTP dates capped at one hour.
|
|
200
|
+
`send_traced` reports `ProviderTransportError.provably_not_sent` only after a
|
|
201
|
+
traced pre-send connection failure; all other transport failures fail closed.
|
|
202
|
+
`BodyTooLargeError` retains the response status and headers for classification
|
|
203
|
+
without putting header values in its message.
|
|
204
|
+
The caller closes the streaming response returned by `send_traced`; the bounded
|
|
205
|
+
reader closes it on success and failure.
|
|
206
|
+
The core example sends through an in-process HTTP transport.
|
|
207
|
+
|
|
208
|
+
## Project configuration
|
|
209
|
+
|
|
210
|
+
`load_from_environment()` reads `DEX_CONNECTOR_CONFIG_FILE` for Dex Web local
|
|
211
|
+
development or the trusted `DEX_PROJECT_*` hosted S3 contract. Local
|
|
212
|
+
`connections.json` and `use-configurations.json` must be regular owner-only
|
|
213
|
+
`0600` files. Hosted reads pin an exact version and SHA-256 digest, and
|
|
214
|
+
`ConfigurationStore` uses revision CAS. `S3ObjectStore.create` checks bucket
|
|
215
|
+
versioning and its KMS boundary. Versioned documents use Go-compatible compact
|
|
216
|
+
JSON bytes so Unicode values produce the same snapshot digests.
|
|
217
|
+
`RFC3339NanoTimestamp` preserves provider offsets and nanoseconds when a
|
|
218
|
+
credential or receipt is decoded. `LoadedProject.resolve_credential` reads
|
|
219
|
+
credentials at call time; shared connections use the credential broker
|
|
220
|
+
protocol, with a forced revision read after provider rejection. Never log or
|
|
221
|
+
persist the returned credential object. The examples freeze an in-memory
|
|
222
|
+
snapshot, resolve a private application secret and broker credential, load an
|
|
223
|
+
owner-only Dex Web file, and exercise S3's conditional-write adapter.
|
|
224
|
+
|
|
225
|
+
`projectconfig.provider.CredentialProvider` binds a non-refreshing connector
|
|
226
|
+
credential decoder to a stored connection. `RefreshingCredentialProvider` also
|
|
227
|
+
takes an encoder for complete replacement material. It conditionally admits one
|
|
228
|
+
refresh request per credential generation across replicas, stores an immutable
|
|
229
|
+
result, then publishes it through a revision CAS. Concurrent callers recover
|
|
230
|
+
the result without another provider request. A retryable refresh failure
|
|
231
|
+
restores the prior credential; a terminal grant failure fences it for
|
|
232
|
+
reauthorization. A generic provider 401 can force refresh only after the
|
|
233
|
+
authoritative stored expiry has elapsed. Under `dexcli dev`, the same provider
|
|
234
|
+
uses process-local connection heads and writes refreshed material back to the
|
|
235
|
+
owner-only `connections.json` file only while it still holds the prior
|
|
236
|
+
credential. A newer Dex Web authorization wins the race.
|
|
237
|
+
Providers built with `project.connections` route a `sharedConnectionId` to
|
|
238
|
+
the credential broker. They never read a stored connection head or call the
|
|
239
|
+
refresh driver for that entry. Pass `decode_shared` when the broker omits
|
|
240
|
+
host-kept renewal secrets required by the ordinary decoder. Broker
|
|
241
|
+
`REAUTHORIZATION_REQUIRED` and `CONNECTION_NOT_SELECTED` propagate to callers.
|
|
242
|
+
|
|
243
|
+
From the same runnable example:
|
|
244
|
+
|
|
245
|
+
```python
|
|
246
|
+
shared_provider = ProjectCredentialProvider(
|
|
247
|
+
project.connections,
|
|
248
|
+
ConnectionKey("sample", "main"),
|
|
249
|
+
lambda value: {"apiKey": str(value["apiKey"]), "privateKey": str(value["privateKey"])},
|
|
250
|
+
decode_shared=lambda value: {"apiKey": str(value["apiKey"])},
|
|
251
|
+
)
|
|
252
|
+
assert (await shared_provider.resolve(call))["apiKey"] == "REDACTED_SHARED_KEY"
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
From [the runnable project example](examples/credentials_and_project.py):
|
|
256
|
+
|
|
257
|
+
```python
|
|
258
|
+
refreshing = RefreshingCredentialProvider(
|
|
259
|
+
project_connections, key, lambda value: value,
|
|
260
|
+
lambda value: value,
|
|
261
|
+
)
|
|
262
|
+
refreshed = await resolve_credential(refreshing, call, ExampleProjectRefreshDriver())
|
|
263
|
+
assert refreshed["access"] == "INERT_NEW_ACCESS"
|
|
264
|
+
assert (await project_connections.read_connection(key)).status == "READY"
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
## Connector mocks
|
|
268
|
+
|
|
269
|
+
`ScriptedOperation` holds a provider Query or Mutation script without Dex Step
|
|
270
|
+
registration. `respond`, `when`, `default`, `strict`, and `for_flow` model the Go
|
|
271
|
+
mock's precedence. Keyed terminal Mutation outcomes replay; retries consume
|
|
272
|
+
the next case. `load_fixture` strictly reads JSON fixture objects and rejects
|
|
273
|
+
duplicate fields. `assert_no_failures()` fails a test after an unscripted call
|
|
274
|
+
or invalid page; `DEX_CONNECTOR_MOCK_REPORT_DIR` enables safe call rows in
|
|
275
|
+
`connector-calls.json`. In pytest, pass `report_failure=pytest.fail` to
|
|
276
|
+
`ScriptedOperation` so a missing script fails immediately. The core example
|
|
277
|
+
answers a Query with a declared branch.
|
|
278
|
+
`connectormock.Query` and `connectormock.Mutation` wrap that script as real
|
|
279
|
+
async operation implementations using the connector's actual definition.
|
|
280
|
+
Their `invoke` requires the live `Call.context` so Flow-scoped scripts use
|
|
281
|
+
Dex's Flow ID. Real Dex Flow tests exercise keyed terminal replay after a
|
|
282
|
+
lost mock response.
|
|
283
|
+
|
|
284
|
+
## Verification
|
|
285
|
+
|
|
286
|
+
Run `uv sync`, `uv run pytest`, `uv run mypy --strict src tests examples
|
|
287
|
+
integration`, and `uv run ruff check src tests examples integration`. For real
|
|
288
|
+
Dex Flow tests, install the latest stable CLI with
|
|
289
|
+
`python3 ../script/dex_compatibility.py install-dexcli --output ../.bin/dexcli`,
|
|
290
|
+
then from `sdkpython/` run `mkdir -p .dexcli-data/logs` and start
|
|
291
|
+
`../.bin/dexcli dev -dex-port 19881 -web-port 19882 -open=false
|
|
292
|
+
-zd-data-dir .dexcli-data/zd -server-log-folder .dexcli-data/logs`. Run
|
|
293
|
+
`make test-sdkpython-integration` from the repository root. CI runs this suite
|
|
294
|
+
for Python SDK changes against the Dex CLI selected once for the workflow run.
|
|
295
|
+
The integration suite checks
|
|
296
|
+
branch routing, retry and exhaustion, uncertainty, the lost-response fence,
|
|
297
|
+
safe resend from a persisted checkpoint, heartbeat cadence, mock replay,
|
|
298
|
+
result Attribute, and progress/text Streams.
|
|
299
|
+
|
|
300
|
+
The provider address-policy client and SSE reader are outside this release.
|
|
301
|
+
|
|
302
|
+
## 0.1.0 release notes
|
|
303
|
+
|
|
304
|
+
First Python release of the provider-neutral Connector SDK: Go-compatible
|
|
305
|
+
identities, failures, results, retry defaults, project configuration,
|
|
306
|
+
credential resolution, provider HTTP, connector mocks, async operation
|
|
307
|
+
contracts, OAuth token exchange, Query and Mutation Step registration,
|
|
308
|
+
result Attribute and progress Streams, and the full single-dispatch fence.
|
|
309
|
+
Real `dexcli dev` Flows verify the async Step paths. Python FDG metadata
|
|
310
|
+
awaits the upstream R4 analyzer; it is not generated by this release.
|
|
@@ -0,0 +1,298 @@
|
|
|
1
|
+
# Python Dex Connector SDK core
|
|
2
|
+
|
|
3
|
+
`dex-connectors-sdk` is the Python core for Dex-native provider connectors. It
|
|
4
|
+
uses the PEP 420 `dex_connectors` namespace and imports as `dex_connectors.sdk`.
|
|
5
|
+
Version `0.1.0` supplies the models, provider
|
|
6
|
+
operation contracts, credential helpers, OAuth token exchanges, and
|
|
7
|
+
single-dispatch checkpoint values below. Query and Mutation Step factories
|
|
8
|
+
use the released Dex SDK's instance-addressed async Steps.
|
|
9
|
+
Dex Web FDG 2.0 connector metadata for Python requires upstream R4.
|
|
10
|
+
|
|
11
|
+
Python 3.11 or newer and `dex-python-sdk>=1.28.0` are required. APIs are
|
|
12
|
+
async-first where they perform I/O. Connector Steps run in `AsyncWorker` and
|
|
13
|
+
never place credentials in Flow state.
|
|
14
|
+
|
|
15
|
+
Runnable examples are [`examples/core.py`](examples/core.py),
|
|
16
|
+
[`examples/credentials_and_project.py`](examples/credentials_and_project.py),
|
|
17
|
+
[`examples/operations_and_oauth.py`](examples/operations_and_oauth.py),
|
|
18
|
+
and [`examples/single_dispatch.py`](examples/single_dispatch.py).
|
|
19
|
+
Run each with `uv run python examples/<name>.py`.
|
|
20
|
+
|
|
21
|
+
## Core values, failures, and retry policy
|
|
22
|
+
|
|
23
|
+
`CallID` is a UUID string stable across retries. `Call` binds that identity to
|
|
24
|
+
the logical `ConnectionRef` and `OperationRef`. Its `AsyncContext` field is live
|
|
25
|
+
only during a Step handler and never enters JSON or Flow state. `QueryResult`,
|
|
26
|
+
`MutationResult`, `QueryDefinition`, and `MutationDefinition` encode the same
|
|
27
|
+
camel-case JSON as `sdkgo`, including optional
|
|
28
|
+
failure omission and RFC 3339 receipt timestamps. `FailureKind` describes a
|
|
29
|
+
fact; an operation decides whether to choose a branch or retry. The default
|
|
30
|
+
Execute retry policy starts at one second, doubles to five minutes, has no
|
|
31
|
+
attempt count limit, and ends after one hour. Exhaustion fails the Flow.
|
|
32
|
+
|
|
33
|
+
The core example constructs and round-trips a `QueryResult`, models a retry
|
|
34
|
+
attempt, and reads the default policy. `MutationAttempt.uncertain` is reserved
|
|
35
|
+
for a possibly accepted write that cannot be repeated or confirmed by read-back.
|
|
36
|
+
|
|
37
|
+
## Configuration UI
|
|
38
|
+
|
|
39
|
+
`ConnectorConfigurationUI` contains ordered `ConnectorUIUnit` records. Each
|
|
40
|
+
unit requires an instructional description and one or more
|
|
41
|
+
`ConnectorUIBinding` JSON Pointers. The core example constructs and validates a
|
|
42
|
+
complete unit. It carries presentation metadata only; configuration values
|
|
43
|
+
are loaded by `projectconfig` at application startup.
|
|
44
|
+
|
|
45
|
+
## Credentials
|
|
46
|
+
|
|
47
|
+
`CredentialProvider` is an async Protocol that resolves one `Call` at provider
|
|
48
|
+
call time.
|
|
49
|
+
`CredentialRefreshDriver` declares a deterministic refresh decision and async
|
|
50
|
+
exchange but does not store results. `SecretString` redacts string and repr
|
|
51
|
+
rendering and rejects serialization; call `reveal()` only when building a
|
|
52
|
+
provider request. The examples implement both credential Protocols and show
|
|
53
|
+
safe rendering. A connector
|
|
54
|
+
should return `ReauthorizationRequiredError` only when interactive provider
|
|
55
|
+
authorization is required. `StaticCredentialProvider` resolves an in-memory
|
|
56
|
+
connection map for examples and tests. `resolve_credential` delegates to
|
|
57
|
+
coordinated refresh when a provider supports it;
|
|
58
|
+
`resolve_credential_after_rejection` requires that support and should be called
|
|
59
|
+
at most once after a provider authentication rejection. Refreshing providers
|
|
60
|
+
serialize per connection, reload after taking that lock, and persist complete
|
|
61
|
+
replacement credentials atomically.
|
|
62
|
+
|
|
63
|
+
From the runnable [operation example](examples/operations_and_oauth.py):
|
|
64
|
+
|
|
65
|
+
```python
|
|
66
|
+
credentials = StaticCredentialProvider({connection: SecretString("INERT_SECRET")})
|
|
67
|
+
credential = await resolve_credential(credentials, call, ExampleRefreshDriver())
|
|
68
|
+
assert credential.reveal() == "INERT_SECRET"
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## Provider operations
|
|
72
|
+
|
|
73
|
+
`Query` and `Mutation` are async provider contracts independent of Dex.
|
|
74
|
+
Each has `definition()` and `invoke(call, input)`; a Mutation also derives a
|
|
75
|
+
stable `idempotency_key(call_id, input)`. `run_query` and `run_mutation` validate
|
|
76
|
+
the attempt against declared branches, complete the safe receipt, and raise
|
|
77
|
+
`RetryError` for a safe-to-repeat failure. A returned unknown branch, foreign
|
|
78
|
+
receipt, or invalid attempt selects `defect`. An `uncertain` Mutation requires a declared route and
|
|
79
|
+
the narrow unconfirmable-write conditions in the repository rules. Retry
|
|
80
|
+
exhaustion is handled by Dex and fails the Flow.
|
|
81
|
+
`RetryError.retry_after_whole_seconds` truncates positive fractional delays to
|
|
82
|
+
the whole seconds accepted by Dex; zero means no explicit retry hint.
|
|
83
|
+
|
|
84
|
+
From the same runnable example:
|
|
85
|
+
|
|
86
|
+
```python
|
|
87
|
+
result = await run_mutation(CreateExampleItem(), call, "item-1")
|
|
88
|
+
assert result.branch == "created" and result.receipt.idempotency_key == str(call_id)
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
## Connector Steps
|
|
92
|
+
|
|
93
|
+
`QueryStep` and `MutationStep` are async Dex `Step` instances configured with
|
|
94
|
+
their corresponding `QueryStepConfig` or `MutationStepConfig`. The instance's
|
|
95
|
+
stable `step_type` and concrete `step_input_type` let Dex register multiple
|
|
96
|
+
instances of one generic class. A required branch needs a target; an unwired
|
|
97
|
+
optional branch fails the Flow with a bounded, secret-safe reason. `StepOptions`
|
|
98
|
+
overrides may change Execute settings but cannot add WaitFor behavior. The
|
|
99
|
+
factory maps a connector `RetryError` to Dex retry, truncating RetryAfter to
|
|
100
|
+
whole seconds; retry exhaustion remains a failed Flow. A configured result
|
|
101
|
+
Attribute is written before routing and belongs in the Flow persistence schema.
|
|
102
|
+
`ProgressUpdate` Stream messages include call ID, Dex attempt, and monotonic
|
|
103
|
+
connector sequence. `Call.report_progress(Progress(...))` emits them when a
|
|
104
|
+
progress Stream is configured; `Call.write_text` writes ordered buffered text
|
|
105
|
+
when a text Stream is configured. These outputs are best effort and belong in
|
|
106
|
+
the consuming Flow persistence schema. The default text flush interval is one
|
|
107
|
+
second with a 16 KiB buffer; Step configs can override both settings.
|
|
108
|
+
|
|
109
|
+
From [the runnable operation example](examples/operations_and_oauth.py):
|
|
110
|
+
|
|
111
|
+
```python
|
|
112
|
+
created = CreatedItem()
|
|
113
|
+
result_attribute = dex.Attribute("createdItem", MutationResult[str])
|
|
114
|
+
progress_stream = dex.Stream("createProgress", ProgressUpdate, 64 * 1024)
|
|
115
|
+
text_stream = dex.Stream("createText", str, 64 * 1024)
|
|
116
|
+
create = MutationStep(MutationStepConfig(
|
|
117
|
+
"CreateItem", str, StepAnnotations("sample", "Sample", "Create one sample item."),
|
|
118
|
+
CreateExampleItem(), connection, lambda value: value,
|
|
119
|
+
{"created": created},
|
|
120
|
+
result_attribute=result_attribute,
|
|
121
|
+
progress_stream=progress_stream, text_stream=text_stream,
|
|
122
|
+
))
|
|
123
|
+
dex.Registry([
|
|
124
|
+
CreateItemFlow(create, created, result_attribute, progress_stream, text_stream)
|
|
125
|
+
], allow_async_handlers=True)
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
## OAuth token refresh
|
|
129
|
+
|
|
130
|
+
`oauthtoken.TokenEndpoint` performs bounded, redirect-disabled refresh-token,
|
|
131
|
+
client-credentials, and JWT-bearer exchanges. The connector owns its endpoint,
|
|
132
|
+
terminal error codes, scopes, and credential type. A declared terminal grant
|
|
133
|
+
error below HTTP 500 raises `ReauthorizationRequiredError`; server outages and
|
|
134
|
+
other failures stay retryable. `TokenResponse.next_refresh_token` retains the
|
|
135
|
+
prior refresh token when a provider does not rotate it. The package also has
|
|
136
|
+
RS256 JWT assertion signing, scope checks, and proportional expiry skew.
|
|
137
|
+
|
|
138
|
+
From the runnable example:
|
|
139
|
+
|
|
140
|
+
```python
|
|
141
|
+
endpoint = TokenEndpoint("Example", "https://oauth.example/token", http_client)
|
|
142
|
+
oauth_client = ClientCredentials(
|
|
143
|
+
"example-client", SecretString("INERT_CLIENT_SECRET"),
|
|
144
|
+
ClientAuthenticationMethod.CLIENT_SECRET_POST,
|
|
145
|
+
)
|
|
146
|
+
token = await endpoint.exchange_refresh_token(oauth_client, SecretString("INERT_REFRESH"))
|
|
147
|
+
assert token.access_token.reveal() == "INERT_ACCESS"
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
## Single dispatch checkpoints
|
|
151
|
+
|
|
152
|
+
`singledispatch.Marker` uses the Go heartbeat JSON shape. `decide_dispatch`
|
|
153
|
+
allows an initial send or a resend only after the immediately prior attempt
|
|
154
|
+
recorded `notSent` or `canResend` without a provider ID. A claimed, missing,
|
|
155
|
+
stale, or accepted checkpoint requires read-back; malformed or foreign data
|
|
156
|
+
fails closed. `singledispatch.Fence` reads `Call.context.deadline` and the
|
|
157
|
+
previous Dex heartbeat before credential lookup. `dispatch(send)` checkpoints
|
|
158
|
+
`claimed` before one provider send and repeats that claim every five seconds.
|
|
159
|
+
The callback receives the write deadline and must avoid transport replay.
|
|
160
|
+
Call `record_not_sent` before retrying a credential or pre-send failure;
|
|
161
|
+
`record_can_resend` requires positive proof that the provider did not process
|
|
162
|
+
the request. `record_provider_id` preserves an accepted resource identity for
|
|
163
|
+
read-back. Use sync Execute durability because local async activities ignore
|
|
164
|
+
heartbeats.
|
|
165
|
+
|
|
166
|
+
From [the runnable single-dispatch example](examples/single_dispatch.py):
|
|
167
|
+
|
|
168
|
+
```python
|
|
169
|
+
fence = Fence(call, timedelta(seconds=10))
|
|
170
|
+
assert (await fence.decide()).kind == DecisionKind.SEND_ALLOWED
|
|
171
|
+
|
|
172
|
+
async def send(write_deadline: datetime) -> str:
|
|
173
|
+
"""Use the deadline when constructing a real non-replaying provider request."""
|
|
174
|
+
assert write_deadline > datetime.now(UTC)
|
|
175
|
+
return "INERT_PROVIDER_OBJECT"
|
|
176
|
+
|
|
177
|
+
provider_id = await fence.dispatch(send)
|
|
178
|
+
await fence.record_provider_id(provider_id)
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
## Provider HTTP
|
|
182
|
+
|
|
183
|
+
`new_provider_http_client` returns an `httpx.AsyncClient` with redirects
|
|
184
|
+
disabled so authorization headers are never replayed to another host.
|
|
185
|
+
`read_bounded_body` rejects an oversized response. `read_error_tokens` extracts
|
|
186
|
+
only safe machine tokens from bounded JSON error bodies, and
|
|
187
|
+
`parse_retry_after` accepts delta seconds or HTTP dates capped at one hour.
|
|
188
|
+
`send_traced` reports `ProviderTransportError.provably_not_sent` only after a
|
|
189
|
+
traced pre-send connection failure; all other transport failures fail closed.
|
|
190
|
+
`BodyTooLargeError` retains the response status and headers for classification
|
|
191
|
+
without putting header values in its message.
|
|
192
|
+
The caller closes the streaming response returned by `send_traced`; the bounded
|
|
193
|
+
reader closes it on success and failure.
|
|
194
|
+
The core example sends through an in-process HTTP transport.
|
|
195
|
+
|
|
196
|
+
## Project configuration
|
|
197
|
+
|
|
198
|
+
`load_from_environment()` reads `DEX_CONNECTOR_CONFIG_FILE` for Dex Web local
|
|
199
|
+
development or the trusted `DEX_PROJECT_*` hosted S3 contract. Local
|
|
200
|
+
`connections.json` and `use-configurations.json` must be regular owner-only
|
|
201
|
+
`0600` files. Hosted reads pin an exact version and SHA-256 digest, and
|
|
202
|
+
`ConfigurationStore` uses revision CAS. `S3ObjectStore.create` checks bucket
|
|
203
|
+
versioning and its KMS boundary. Versioned documents use Go-compatible compact
|
|
204
|
+
JSON bytes so Unicode values produce the same snapshot digests.
|
|
205
|
+
`RFC3339NanoTimestamp` preserves provider offsets and nanoseconds when a
|
|
206
|
+
credential or receipt is decoded. `LoadedProject.resolve_credential` reads
|
|
207
|
+
credentials at call time; shared connections use the credential broker
|
|
208
|
+
protocol, with a forced revision read after provider rejection. Never log or
|
|
209
|
+
persist the returned credential object. The examples freeze an in-memory
|
|
210
|
+
snapshot, resolve a private application secret and broker credential, load an
|
|
211
|
+
owner-only Dex Web file, and exercise S3's conditional-write adapter.
|
|
212
|
+
|
|
213
|
+
`projectconfig.provider.CredentialProvider` binds a non-refreshing connector
|
|
214
|
+
credential decoder to a stored connection. `RefreshingCredentialProvider` also
|
|
215
|
+
takes an encoder for complete replacement material. It conditionally admits one
|
|
216
|
+
refresh request per credential generation across replicas, stores an immutable
|
|
217
|
+
result, then publishes it through a revision CAS. Concurrent callers recover
|
|
218
|
+
the result without another provider request. A retryable refresh failure
|
|
219
|
+
restores the prior credential; a terminal grant failure fences it for
|
|
220
|
+
reauthorization. A generic provider 401 can force refresh only after the
|
|
221
|
+
authoritative stored expiry has elapsed. Under `dexcli dev`, the same provider
|
|
222
|
+
uses process-local connection heads and writes refreshed material back to the
|
|
223
|
+
owner-only `connections.json` file only while it still holds the prior
|
|
224
|
+
credential. A newer Dex Web authorization wins the race.
|
|
225
|
+
Providers built with `project.connections` route a `sharedConnectionId` to
|
|
226
|
+
the credential broker. They never read a stored connection head or call the
|
|
227
|
+
refresh driver for that entry. Pass `decode_shared` when the broker omits
|
|
228
|
+
host-kept renewal secrets required by the ordinary decoder. Broker
|
|
229
|
+
`REAUTHORIZATION_REQUIRED` and `CONNECTION_NOT_SELECTED` propagate to callers.
|
|
230
|
+
|
|
231
|
+
From the same runnable example:
|
|
232
|
+
|
|
233
|
+
```python
|
|
234
|
+
shared_provider = ProjectCredentialProvider(
|
|
235
|
+
project.connections,
|
|
236
|
+
ConnectionKey("sample", "main"),
|
|
237
|
+
lambda value: {"apiKey": str(value["apiKey"]), "privateKey": str(value["privateKey"])},
|
|
238
|
+
decode_shared=lambda value: {"apiKey": str(value["apiKey"])},
|
|
239
|
+
)
|
|
240
|
+
assert (await shared_provider.resolve(call))["apiKey"] == "REDACTED_SHARED_KEY"
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
From [the runnable project example](examples/credentials_and_project.py):
|
|
244
|
+
|
|
245
|
+
```python
|
|
246
|
+
refreshing = RefreshingCredentialProvider(
|
|
247
|
+
project_connections, key, lambda value: value,
|
|
248
|
+
lambda value: value,
|
|
249
|
+
)
|
|
250
|
+
refreshed = await resolve_credential(refreshing, call, ExampleProjectRefreshDriver())
|
|
251
|
+
assert refreshed["access"] == "INERT_NEW_ACCESS"
|
|
252
|
+
assert (await project_connections.read_connection(key)).status == "READY"
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
## Connector mocks
|
|
256
|
+
|
|
257
|
+
`ScriptedOperation` holds a provider Query or Mutation script without Dex Step
|
|
258
|
+
registration. `respond`, `when`, `default`, `strict`, and `for_flow` model the Go
|
|
259
|
+
mock's precedence. Keyed terminal Mutation outcomes replay; retries consume
|
|
260
|
+
the next case. `load_fixture` strictly reads JSON fixture objects and rejects
|
|
261
|
+
duplicate fields. `assert_no_failures()` fails a test after an unscripted call
|
|
262
|
+
or invalid page; `DEX_CONNECTOR_MOCK_REPORT_DIR` enables safe call rows in
|
|
263
|
+
`connector-calls.json`. In pytest, pass `report_failure=pytest.fail` to
|
|
264
|
+
`ScriptedOperation` so a missing script fails immediately. The core example
|
|
265
|
+
answers a Query with a declared branch.
|
|
266
|
+
`connectormock.Query` and `connectormock.Mutation` wrap that script as real
|
|
267
|
+
async operation implementations using the connector's actual definition.
|
|
268
|
+
Their `invoke` requires the live `Call.context` so Flow-scoped scripts use
|
|
269
|
+
Dex's Flow ID. Real Dex Flow tests exercise keyed terminal replay after a
|
|
270
|
+
lost mock response.
|
|
271
|
+
|
|
272
|
+
## Verification
|
|
273
|
+
|
|
274
|
+
Run `uv sync`, `uv run pytest`, `uv run mypy --strict src tests examples
|
|
275
|
+
integration`, and `uv run ruff check src tests examples integration`. For real
|
|
276
|
+
Dex Flow tests, install the latest stable CLI with
|
|
277
|
+
`python3 ../script/dex_compatibility.py install-dexcli --output ../.bin/dexcli`,
|
|
278
|
+
then from `sdkpython/` run `mkdir -p .dexcli-data/logs` and start
|
|
279
|
+
`../.bin/dexcli dev -dex-port 19881 -web-port 19882 -open=false
|
|
280
|
+
-zd-data-dir .dexcli-data/zd -server-log-folder .dexcli-data/logs`. Run
|
|
281
|
+
`make test-sdkpython-integration` from the repository root. CI runs this suite
|
|
282
|
+
for Python SDK changes against the Dex CLI selected once for the workflow run.
|
|
283
|
+
The integration suite checks
|
|
284
|
+
branch routing, retry and exhaustion, uncertainty, the lost-response fence,
|
|
285
|
+
safe resend from a persisted checkpoint, heartbeat cadence, mock replay,
|
|
286
|
+
result Attribute, and progress/text Streams.
|
|
287
|
+
|
|
288
|
+
The provider address-policy client and SSE reader are outside this release.
|
|
289
|
+
|
|
290
|
+
## 0.1.0 release notes
|
|
291
|
+
|
|
292
|
+
First Python release of the provider-neutral Connector SDK: Go-compatible
|
|
293
|
+
identities, failures, results, retry defaults, project configuration,
|
|
294
|
+
credential resolution, provider HTTP, connector mocks, async operation
|
|
295
|
+
contracts, OAuth token exchange, Query and Mutation Step registration,
|
|
296
|
+
result Attribute and progress Streams, and the full single-dispatch fence.
|
|
297
|
+
Real `dexcli dev` Flows verify the async Step paths. Python FDG metadata
|
|
298
|
+
awaits the upstream R4 analyzer; it is not generated by this release.
|