@adcp/sdk 16.0.0-beta.2 → 16.0.0-beta.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +54 -0
- package/dist/lib/auth/oauth/diagnose.js +3 -11
- package/dist/lib/auth/oauth/diagnose.mjs +4 -12
- package/dist/lib/auth/oauth/diagnostics.d.mts +2 -1
- package/dist/lib/auth/oauth/diagnostics.d.ts +2 -1
- package/dist/lib/auth/oauth/diagnostics.js +8 -10
- package/dist/lib/auth/oauth/diagnostics.mjs +7 -10
- package/dist/lib/core/ConversationTypes.d.mts +8 -3
- package/dist/lib/core/ConversationTypes.d.ts +8 -3
- package/dist/lib/core/SingleAgentClient.d.mts +5 -3
- package/dist/lib/core/SingleAgentClient.d.ts +5 -3
- package/dist/lib/core/TaskExecutor.js +5 -1
- package/dist/lib/core/TaskExecutor.mjs +5 -1
- package/dist/lib/errors/index.d.mts +9 -4
- package/dist/lib/errors/index.d.ts +9 -4
- package/dist/lib/index.d.mts +3 -3
- package/dist/lib/index.d.ts +3 -3
- package/dist/lib/index.js +8 -0
- package/dist/lib/index.mjs +14 -2
- package/dist/lib/negotiation/types.d.mts +20 -39
- package/dist/lib/negotiation/types.d.ts +20 -39
- package/dist/lib/protocols/index.d.mts +60 -24
- package/dist/lib/protocols/index.d.ts +60 -24
- package/dist/lib/protocols/index.js +9 -4
- package/dist/lib/protocols/index.mjs +3 -0
- package/dist/lib/protocols/mcp-modern.js +1 -0
- package/dist/lib/protocols/mcp-modern.mjs +2 -1
- package/dist/lib/protocols/mcp.js +1 -0
- package/dist/lib/protocols/mcp.mjs +2 -1
- package/dist/lib/protocols/responseSizeLimit.d.mts +91 -5
- package/dist/lib/protocols/responseSizeLimit.d.ts +91 -5
- package/dist/lib/protocols/responseSizeLimit.js +195 -31
- package/dist/lib/protocols/responseSizeLimit.mjs +192 -31
- package/dist/lib/reporting/consumer-runtime.js +2 -1
- package/dist/lib/reporting/consumer-runtime.mjs +2 -1
- package/dist/lib/reporting/ledger/handler.js +50 -11
- package/dist/lib/reporting/ledger/handler.mjs +50 -12
- package/dist/lib/reporting/ledger/index.d.mts +17 -14
- package/dist/lib/reporting/ledger/index.d.ts +17 -14
- package/dist/lib/reporting/ledger/index.js +32 -20
- package/dist/lib/reporting/ledger/index.mjs +21 -10
- package/dist/lib/reporting/ledger/offering-validation.d.mts +9 -0
- package/dist/lib/reporting/ledger/offering-validation.d.ts +9 -0
- package/dist/lib/reporting/{gcs/bucket-policy.js → ledger/offering-validation.js} +9 -8
- package/dist/lib/reporting/ledger/offering-validation.mjs +8 -0
- package/dist/lib/reporting/ledger/receipts.d.mts +48 -0
- package/dist/lib/reporting/ledger/receipts.d.ts +48 -0
- package/dist/lib/reporting/ledger/receipts.js +186 -0
- package/dist/lib/reporting/ledger/receipts.mjs +161 -0
- package/dist/lib/reporting/ledger/status-ingest.js +3 -3
- package/dist/lib/reporting/ledger/status-ingest.mjs +4 -5
- package/dist/lib/reporting/ledger/types.d.mts +43 -0
- package/dist/lib/reporting/ledger/types.d.ts +43 -0
- package/dist/lib/reporting/ledger/types.js +31 -2
- package/dist/lib/reporting/ledger/types.mjs +26 -1
- package/dist/lib/reporting/pagination.d.mts +6 -0
- package/dist/lib/reporting/pagination.d.ts +6 -0
- package/dist/lib/reporting/{gcs/errors.js → pagination.js} +12 -17
- package/dist/lib/reporting/pagination.mjs +12 -0
- package/dist/lib/reporting/reconciliation.d.mts +1 -0
- package/dist/lib/reporting/reconciliation.d.ts +1 -0
- package/dist/lib/reporting/reconciliation.js +3 -2
- package/dist/lib/reporting/reconciliation.mjs +3 -2
- package/dist/lib/reporting/service/index.d.mts +79 -153
- package/dist/lib/reporting/service/index.d.ts +79 -153
- package/dist/lib/reporting/service/index.js +34 -491
- package/dist/lib/reporting/service/index.mjs +36 -510
- package/dist/lib/reporting/service/scheduler-support.d.mts +43 -0
- package/dist/lib/reporting/service/scheduler-support.d.ts +43 -0
- package/dist/lib/reporting/service/scheduler-support.js +125 -0
- package/dist/lib/reporting/service/scheduler-support.mjs +96 -0
- package/dist/lib/schemas-data/v2.5/_provenance.json +1 -1
- package/dist/lib/signing/canonicalize.d.mts +4 -0
- package/dist/lib/signing/canonicalize.d.ts +4 -0
- package/dist/lib/signing/canonicalize.js +14 -8
- package/dist/lib/signing/canonicalize.mjs +14 -8
- package/dist/lib/signing/content-digest.d.mts +8 -0
- package/dist/lib/signing/content-digest.d.ts +8 -0
- package/dist/lib/signing/content-digest.js +11 -5
- package/dist/lib/signing/content-digest.mjs +10 -5
- package/dist/lib/signing/crypto.d.mts +15 -0
- package/dist/lib/signing/crypto.d.ts +15 -0
- package/dist/lib/signing/crypto.js +75 -3
- package/dist/lib/signing/crypto.mjs +78 -3
- package/dist/lib/signing/server.d.mts +1 -1
- package/dist/lib/signing/server.d.ts +1 -1
- package/dist/lib/signing/server.js +2 -0
- package/dist/lib/signing/server.mjs +2 -1
- package/dist/lib/signing/signer.js +11 -14
- package/dist/lib/signing/signer.mjs +13 -16
- package/dist/lib/signing/testing.js +2 -6
- package/dist/lib/signing/testing.mjs +3 -7
- package/dist/lib/signing/verifier.js +2 -2
- package/dist/lib/signing/verifier.mjs +3 -3
- package/dist/lib/signing/web-bot-auth.js +122 -43
- package/dist/lib/signing/web-bot-auth.mjs +128 -44
- package/dist/lib/signing/webhook-verifier.js +2 -2
- package/dist/lib/signing/webhook-verifier.mjs +3 -3
- package/dist/lib/substitution/compile.js +2 -1
- package/dist/lib/substitution/compile.mjs +2 -1
- package/dist/lib/substitution/consent-macros.d.mts +4 -0
- package/dist/lib/substitution/consent-macros.d.ts +4 -0
- package/dist/lib/substitution/consent-macros.js +45 -0
- package/dist/lib/substitution/consent-macros.mjs +20 -0
- package/dist/lib/substitution/index.d.mts +1 -1
- package/dist/lib/substitution/index.d.ts +1 -1
- package/dist/lib/substitution/index.js +2 -0
- package/dist/lib/substitution/index.mjs +2 -1
- package/dist/lib/substitution/translate.d.mts +15 -2
- package/dist/lib/substitution/translate.d.ts +15 -2
- package/dist/lib/substitution/translate.js +148 -26
- package/dist/lib/substitution/translate.mjs +147 -26
- package/dist/lib/testing/client.js +2 -1
- package/dist/lib/testing/client.mjs +2 -1
- package/dist/lib/testing/scenarios/capabilities.js +1 -1
- package/dist/lib/testing/scenarios/capabilities.mjs +1 -1
- package/dist/lib/testing/storyboard/junit.js +16 -3
- package/dist/lib/testing/storyboard/junit.mjs +16 -3
- package/dist/lib/testing/storyboard/request-signing/builder.js +12 -14
- package/dist/lib/testing/storyboard/request-signing/builder.mjs +13 -15
- package/dist/lib/testing/storyboard/runner.js +94 -25
- package/dist/lib/testing/storyboard/runner.mjs +94 -25
- package/dist/lib/testing/storyboard/signer-grader/grader.js +2 -7
- package/dist/lib/testing/storyboard/signer-grader/grader.mjs +2 -7
- package/dist/lib/testing/storyboard/types.d.mts +6 -0
- package/dist/lib/testing/storyboard/types.d.ts +6 -0
- package/dist/lib/testing/storyboard/validations.d.mts +1 -1
- package/dist/lib/testing/storyboard/validations.d.ts +1 -1
- package/dist/lib/types/enums.generated.d.mts +3 -0
- package/dist/lib/types/enums.generated.d.ts +3 -0
- package/dist/lib/types/enums.generated.js +9 -0
- package/dist/lib/types/enums.generated.mjs +6 -0
- package/dist/lib/validation/schema-loader.js +25 -0
- package/dist/lib/validation/schema-loader.mjs +24 -0
- package/dist/lib/version.d.mts +3 -3
- package/dist/lib/version.d.ts +3 -3
- package/dist/lib/version.js +3 -3
- package/dist/lib/version.mjs +3 -3
- package/docs/TYPE-SUMMARY.md +28 -5
- package/docs/guides/DISCOVERY-CACHE-AND-SESSION-REUSE.md +3 -2
- package/docs/guides/REPORTING-GCS-FENCE.md +3 -3
- package/docs/guides/REPORTING-GCS-MANAGED.md +4 -4
- package/docs/guides/REPORTING-LEDGER-CUTOVER.md +7 -7
- package/docs/guides/REPORTING-LEDGER.md +107 -51
- package/docs/guides/REPORTING-OPERATIONS.md +66 -5
- package/docs/guides/REPORTING-STABILITY.md +36 -32
- package/docs/guides/UNIVERSAL-MACRO-COMPILATION.md +38 -0
- package/docs/guides/VALIDATE-YOUR-AGENT.md +12 -0
- package/docs/llms.txt +4 -3
- package/docs/migration-14.x-rc-worksheet.md +4 -7
- package/docs/migration-reporting-storage-package.md +186 -0
- package/examples/README.md +1 -1
- package/package.json +14 -78
- package/dist/lib/reporting/azure/index.d.mts +0 -31
- package/dist/lib/reporting/azure/index.d.ts +0 -31
- package/dist/lib/reporting/azure/index.js +0 -266
- package/dist/lib/reporting/azure/index.mjs +0 -249
- package/dist/lib/reporting/bigquery/index.d.mts +0 -246
- package/dist/lib/reporting/bigquery/index.d.ts +0 -246
- package/dist/lib/reporting/bigquery/index.js +0 -633
- package/dist/lib/reporting/bigquery/index.mjs +0 -594
- package/dist/lib/reporting/gcs/adapter.d.mts +0 -38
- package/dist/lib/reporting/gcs/adapter.d.ts +0 -38
- package/dist/lib/reporting/gcs/adapter.js +0 -342
- package/dist/lib/reporting/gcs/adapter.mjs +0 -325
- package/dist/lib/reporting/gcs/bucket-policy.d.mts +0 -3
- package/dist/lib/reporting/gcs/bucket-policy.d.ts +0 -3
- package/dist/lib/reporting/gcs/bucket-policy.mjs +0 -7
- package/dist/lib/reporting/gcs/errors.d.mts +0 -11
- package/dist/lib/reporting/gcs/errors.d.ts +0 -11
- package/dist/lib/reporting/gcs/errors.mjs +0 -16
- package/dist/lib/reporting/gcs/fence.d.mts +0 -41
- package/dist/lib/reporting/gcs/fence.d.ts +0 -41
- package/dist/lib/reporting/gcs/fence.js +0 -250
- package/dist/lib/reporting/gcs/fence.mjs +0 -230
- package/dist/lib/reporting/gcs/index.d.mts +0 -18
- package/dist/lib/reporting/gcs/index.d.ts +0 -18
- package/dist/lib/reporting/gcs/index.js +0 -44
- package/dist/lib/reporting/gcs/index.mjs +0 -14
- package/dist/lib/reporting/gcs/io.d.mts +0 -7
- package/dist/lib/reporting/gcs/io.d.ts +0 -7
- package/dist/lib/reporting/gcs/io.js +0 -95
- package/dist/lib/reporting/gcs/io.mjs +0 -69
- package/dist/lib/reporting/gcs/reader.d.mts +0 -67
- package/dist/lib/reporting/gcs/reader.d.ts +0 -67
- package/dist/lib/reporting/gcs/reader.js +0 -193
- package/dist/lib/reporting/gcs/reader.mjs +0 -169
- package/dist/lib/reporting/gcs/row-provider.d.mts +0 -39
- package/dist/lib/reporting/gcs/row-provider.d.ts +0 -39
- package/dist/lib/reporting/gcs/row-provider.js +0 -225
- package/dist/lib/reporting/gcs/row-provider.mjs +0 -208
- package/dist/lib/reporting/gcs/validation.d.mts +0 -5
- package/dist/lib/reporting/gcs/validation.d.ts +0 -5
- package/dist/lib/reporting/gcs/validation.js +0 -49
- package/dist/lib/reporting/gcs/validation.mjs +0 -25
- package/dist/lib/reporting/ledger/change-feed.d.mts +0 -69
- package/dist/lib/reporting/ledger/change-feed.d.ts +0 -69
- package/dist/lib/reporting/ledger/change-feed.js +0 -107
- package/dist/lib/reporting/ledger/change-feed.mjs +0 -80
- package/dist/lib/reporting/ledger/managed-postgres.d.mts +0 -293
- package/dist/lib/reporting/ledger/managed-postgres.d.ts +0 -293
- package/dist/lib/reporting/ledger/managed-postgres.js +0 -2346
- package/dist/lib/reporting/ledger/managed-postgres.mjs +0 -2332
- package/dist/lib/reporting/ledger/managed-router.d.mts +0 -148
- package/dist/lib/reporting/ledger/managed-router.d.ts +0 -148
- package/dist/lib/reporting/ledger/managed-router.js +0 -280
- package/dist/lib/reporting/ledger/managed-router.mjs +0 -258
- package/dist/lib/reporting/ledger/managed.d.mts +0 -535
- package/dist/lib/reporting/ledger/managed.d.ts +0 -535
- package/dist/lib/reporting/ledger/managed.js +0 -761
- package/dist/lib/reporting/ledger/managed.mjs +0 -729
- package/dist/lib/reporting/ledger/notification-activity.d.mts +0 -332
- package/dist/lib/reporting/ledger/notification-activity.d.ts +0 -332
- package/dist/lib/reporting/ledger/notification-activity.js +0 -1366
- package/dist/lib/reporting/ledger/notification-activity.mjs +0 -1337
- package/dist/lib/reporting/ledger/object-writes.d.mts +0 -128
- package/dist/lib/reporting/ledger/object-writes.d.ts +0 -128
- package/dist/lib/reporting/ledger/object-writes.js +0 -147
- package/dist/lib/reporting/ledger/object-writes.mjs +0 -114
- package/dist/lib/reporting/ledger/postgres.d.mts +0 -689
- package/dist/lib/reporting/ledger/postgres.d.ts +0 -689
- package/dist/lib/reporting/ledger/postgres.js +0 -4462
- package/dist/lib/reporting/ledger/postgres.mjs +0 -4462
- package/dist/lib/reporting/ledger/row-storage-migration.d.mts +0 -59
- package/dist/lib/reporting/ledger/row-storage-migration.d.ts +0 -59
- package/dist/lib/reporting/ledger/row-storage-migration.js +0 -365
- package/dist/lib/reporting/ledger/row-storage-migration.mjs +0 -339
- package/dist/lib/reporting/ledger/row-storage-object.d.mts +0 -203
- package/dist/lib/reporting/ledger/row-storage-object.d.ts +0 -203
- package/dist/lib/reporting/ledger/row-storage-object.js +0 -379
- package/dist/lib/reporting/ledger/row-storage-object.mjs +0 -335
- package/dist/lib/reporting/ledger/row-storage-provider-io.d.mts +0 -44
- package/dist/lib/reporting/ledger/row-storage-provider-io.d.ts +0 -44
- package/dist/lib/reporting/ledger/row-storage-provider-io.js +0 -142
- package/dist/lib/reporting/ledger/row-storage-provider-io.mjs +0 -111
- package/dist/lib/reporting/ledger/row-storage.d.mts +0 -316
- package/dist/lib/reporting/ledger/row-storage.d.ts +0 -316
- package/dist/lib/reporting/ledger/row-storage.js +0 -812
- package/dist/lib/reporting/ledger/row-storage.mjs +0 -802
- package/dist/lib/reporting/s3/index.d.mts +0 -34
- package/dist/lib/reporting/s3/index.d.ts +0 -34
- package/dist/lib/reporting/s3/index.js +0 -347
- package/dist/lib/reporting/s3/index.mjs +0 -340
- package/dist/lib/reporting/service/maintenance.d.mts +0 -148
- package/dist/lib/reporting/service/maintenance.d.ts +0 -148
- package/dist/lib/reporting/service/maintenance.js +0 -185
- package/dist/lib/reporting/service/maintenance.mjs +0 -159
- package/examples/reliable-reporting-service/README.md +0 -169
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
# Seller Reporting Ledger
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Reliable Reporting Core turns a conforming reporting source into durable seller-side reporting. It is separate from the buyer-side `reconcileReporting` API.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Two packages make it up. `@adcp/sdk` holds the protocol layer: the `ReportingLedgerStore` interface, the protocol handlers, the producer, `createReliableReportingService` and row encoding (`@adcp/sdk/reporting/ledger`, `@adcp/sdk/reporting/service`). [`@adcp/reporting-storage`](https://github.com/adcontextprotocol/adcp-client/blob/main/packages/reporting-storage/README.md) holds the storage behind it: the PostgreSQL ledger store and migrations, row storage and its GCS, S3 and Azure providers, managed delivery, notification activity, the BigQuery sink and the production composition. Install both (`npm install @adcp/sdk@beta @adcp/reporting-storage@beta pg` while SDK 16 is in beta). Moving from the pre-split `@adcp/sdk/reporting/*` imports? See [Reliable Reporting storage moved to `@adcp/reporting-storage`](../migration-reporting-storage-package.md).
|
|
6
|
+
|
|
7
|
+
> **Stability:** `@adcp/reporting-storage` is pre-1.0 and every export of it is experimental: breaking changes ship in its minor releases. Pin it with `~`. The store, transaction, lease and row-read interfaces in `@adcp/sdk/reporting/ledger` that it implements are `@experimental` too. See the [Reliable Reporting stability policy](REPORTING-STABILITY.md).
|
|
6
8
|
|
|
7
9
|
## Recommended: install the lifecycle service
|
|
8
10
|
|
|
@@ -13,7 +15,7 @@ producer, handlers, and decisioning-platform account resolver.
|
|
|
13
15
|
|
|
14
16
|
```ts
|
|
15
17
|
import { Pool } from 'pg';
|
|
16
|
-
import { PostgresReportingLedgerStore } from '@adcp/
|
|
18
|
+
import { PostgresReportingLedgerStore } from '@adcp/reporting-storage';
|
|
17
19
|
import { createReliableReportingService } from '@adcp/sdk/reporting/service';
|
|
18
20
|
import { createAdcpServerFromPlatform } from '@adcp/sdk/server';
|
|
19
21
|
|
|
@@ -53,7 +55,7 @@ const reporting = createReliableReportingService({
|
|
|
53
55
|
},
|
|
54
56
|
});
|
|
55
57
|
|
|
56
|
-
|
|
58
|
+
for (const sql of reporting.setup.migrations) await pool.query(sql);
|
|
57
59
|
const installedPlatform = reporting.install(platform);
|
|
58
60
|
const server = createAdcpServerFromPlatform(installedPlatform, serverOptions);
|
|
59
61
|
|
|
@@ -86,7 +88,7 @@ still use an `accountIds` roster. Production setup requires
|
|
|
86
88
|
`activity.tenantScopeForAccount`; when an offering uses `consumer_receipt`, it
|
|
87
89
|
also requires a trusted `obligatedConsumers` callback so lifecycle health can
|
|
88
90
|
become reconciled only after every obligated consumer has accepted.
|
|
89
|
-
See the [integrated seller example](
|
|
91
|
+
See the [integrated seller example](https://github.com/adcontextprotocol/adcp-client/blob/main/packages/reporting-storage/examples/reliable-reporting-service/README.md#integrated-seller-production-service)
|
|
90
92
|
for the full option shape.
|
|
91
93
|
|
|
92
94
|
Install a buyer declaration after the account and its media-buy scope have
|
|
@@ -135,6 +137,8 @@ import {
|
|
|
135
137
|
PostgresReportingLedgerStore,
|
|
136
138
|
REPORTING_LEDGER_MIGRATION,
|
|
137
139
|
sweepExpiredReportingLedgerState,
|
|
140
|
+
} from '@adcp/reporting-storage';
|
|
141
|
+
import {
|
|
138
142
|
createReportingDeliveryHandler,
|
|
139
143
|
createReportingProducer,
|
|
140
144
|
createReportingStatusHandler,
|
|
@@ -162,7 +166,7 @@ const getMediaBuyDelivery = createReportingDeliveryHandler(store);
|
|
|
162
166
|
await sweepExpiredReportingLedgerState(pool);
|
|
163
167
|
```
|
|
164
168
|
|
|
165
|
-
Pass `getReportingStatus` and `getMediaBuyDelivery` directly to the matching `createAdcpServer` slots. The delivery helper serves only exact `reporting_revision_id` reads and returns the row payload bound by the ledger revision. Advertise `media_buy.reporting_delivery` in `experimental_features` together with a `media_buy.reporting_delivery` capability whose Reliable Reporting version is `1.0` only after both handlers are wired. Configure account resolution on the server: both handlers require the framework-resolved, caller-scoped account identity and never trust a request-body identity as an authorization boundary. If two callers can name the same upstream account, the resolver must issue distinct internal account IDs for their ledger namespaces. Install immutable delivery-configuration generations through `producer.installConfiguration`, call `planObligations()` after period close, and run `runWorker()` from a durable scheduler. Multiple workers are safe: PostgreSQL claims use `SKIP LOCKED`, expiring leases, and fencing generations.
|
|
169
|
+
Pass `getReportingStatus` and `getMediaBuyDelivery` directly to the matching `createAdcpServer` slots. The delivery helper serves only exact `reporting_revision_id` reads and returns the row payload bound by the ledger revision. Both handlers take `pagination.max_results` as AdCP defines it, from `core/pagination-request.json`: an integer from 1 to 100, 50 when omitted. Any other value (and a `pagination` that is not an object or a `cursor` that is not a string) is refused with `VALIDATION_ERROR` (`field: 'pagination.max_results'`, `'pagination'` or `'pagination.cursor'`) before the ledger is read, so the same request behaves the same on every seller; page with `pagination.cursor`. The buyer-side walkers ask for the schema maximum, 100. Advertise `media_buy.reporting_delivery` in `experimental_features` together with a `media_buy.reporting_delivery` capability whose Reliable Reporting version is `1.0` only after both handlers are wired. Configure account resolution on the server: both handlers require the framework-resolved, caller-scoped account identity and never trust a request-body identity as an authorization boundary. If two callers can name the same upstream account, the resolver must issue distinct internal account IDs for their ledger namespaces. Install immutable delivery-configuration generations through `producer.installConfiguration`, call `planObligations()` after period close, and run `runWorker()` from a durable scheduler. Multiple workers are safe: PostgreSQL claims use `SKIP LOCKED`, expiring leases, and fencing generations.
|
|
166
170
|
|
|
167
171
|
The planner supports fixed millisecond schedules and an explicit source-calendar
|
|
168
172
|
day schedule. For the latter, set the existing schedule identity fields to
|
|
@@ -291,7 +295,7 @@ rows as verified canonical JSONL chunks (the shared SDK persistence layout,
|
|
|
291
295
|
adcontextprotocol/adcp#7996):
|
|
292
296
|
|
|
293
297
|
```ts
|
|
294
|
-
import { REPORTING_ROW_STORAGE_MIGRATION } from '@adcp/
|
|
298
|
+
import { REPORTING_ROW_STORAGE_MIGRATION } from '@adcp/reporting-storage';
|
|
295
299
|
|
|
296
300
|
await pool.query(REPORTING_ROW_STORAGE_MIGRATION); // after REPORTING_LEDGER_MIGRATION; PostgreSQL 13+
|
|
297
301
|
const store = new PostgresReportingLedgerStore(pool, {
|
|
@@ -364,7 +368,7 @@ point reads or writes at a destination the host did not configure.
|
|
|
364
368
|
import {
|
|
365
369
|
PostgresReportingLedgerStore,
|
|
366
370
|
createFilesystemReportingRowObjectProviderV1,
|
|
367
|
-
} from '@adcp/
|
|
371
|
+
} from '@adcp/reporting-storage';
|
|
368
372
|
|
|
369
373
|
const store = new PostgresReportingLedgerStore(pool, {
|
|
370
374
|
acknowledgeIsolatedDatabase: true,
|
|
@@ -462,6 +466,14 @@ await store.readyRowStorage();
|
|
|
462
466
|
version at the recorded size is the same bytes) instead of downloading it
|
|
463
467
|
while holding the account and object locks. A provider without `stat` falls
|
|
464
468
|
back to reading the whole object there. All bundled providers implement it.
|
|
469
|
+
Also implement the optional `list({ prefix, pageToken, pageSize })`, which
|
|
470
|
+
returns `{ objects: [{ objectRef, nativeVersionRef, sizeBytes, createdAt }],
|
|
471
|
+
nextPageToken? }`: every stored version under the prefix once, in a stable
|
|
472
|
+
order (on a versioned store, noncurrent versions too, never delete markers),
|
|
473
|
+
with the version exactly as `putIfAbsent` recorded it and the time it was
|
|
474
|
+
written. The orphan collector needs it; a binding whose provider has no `list`
|
|
475
|
+
is reported as unsupported there. The conformance suite checks it (`list`) when
|
|
476
|
+
present. All bundled providers implement it.
|
|
465
477
|
|
|
466
478
|
#### Cloud providers
|
|
467
479
|
|
|
@@ -472,12 +484,12 @@ reference fails `UNSAFE_BINDING`. Provider errors surface only as stable
|
|
|
472
484
|
`ReportingRowStoreError` codes. Row buckets are separate from Managed Delivery
|
|
473
485
|
buckets.
|
|
474
486
|
|
|
475
|
-
**Google Cloud Storage** (`@adcp/
|
|
487
|
+
**Google Cloud Storage** (`@adcp/reporting-storage/gcs`, optional peer
|
|
476
488
|
`@google-cloud/storage`):
|
|
477
489
|
|
|
478
490
|
```ts
|
|
479
491
|
import { Storage } from '@google-cloud/storage';
|
|
480
|
-
import { createGcsReportingRowObjectProviderV1 } from '@adcp/
|
|
492
|
+
import { createGcsReportingRowObjectProviderV1 } from '@adcp/reporting-storage/gcs';
|
|
481
493
|
|
|
482
494
|
const gcs = createGcsReportingRowObjectProviderV1({
|
|
483
495
|
storage: new Storage(), // keyless credentials (workload identity / ADC)
|
|
@@ -500,12 +512,12 @@ const gcs = createGcsReportingRowObjectProviderV1({
|
|
|
500
512
|
- The client needs `storage.buckets.get` for the probe plus object create, get
|
|
501
513
|
and delete on the bucket.
|
|
502
514
|
|
|
503
|
-
**Amazon S3 and S3-compatible stores** (`@adcp/
|
|
515
|
+
**Amazon S3 and S3-compatible stores** (`@adcp/reporting-storage/s3`, optional peer
|
|
504
516
|
`@aws-sdk/client-s3`):
|
|
505
517
|
|
|
506
518
|
```ts
|
|
507
519
|
import { S3Client } from '@aws-sdk/client-s3';
|
|
508
|
-
import { createS3ReportingRowObjectProviderV1 } from '@adcp/
|
|
520
|
+
import { createS3ReportingRowObjectProviderV1 } from '@adcp/reporting-storage/s3';
|
|
509
521
|
|
|
510
522
|
const s3 = createS3ReportingRowObjectProviderV1({
|
|
511
523
|
client: new S3Client({ region: 'us-east-1' }), // default credential chain
|
|
@@ -542,13 +554,13 @@ const s3 = createS3ReportingRowObjectProviderV1({
|
|
|
542
554
|
`s3:GetBucketPolicyStatus` is optional. For stores without flexible
|
|
543
555
|
checksums, construct the client with `requestChecksumCalculation: 'WHEN_REQUIRED'`.
|
|
544
556
|
|
|
545
|
-
**Azure Blob Storage** (`@adcp/
|
|
557
|
+
**Azure Blob Storage** (`@adcp/reporting-storage/azure`, optional peer
|
|
546
558
|
`@azure/storage-blob`):
|
|
547
559
|
|
|
548
560
|
```ts
|
|
549
561
|
import { DefaultAzureCredential } from '@azure/identity';
|
|
550
562
|
import { BlobServiceClient } from '@azure/storage-blob';
|
|
551
|
-
import { createAzureBlobReportingRowObjectProviderV1 } from '@adcp/
|
|
563
|
+
import { createAzureBlobReportingRowObjectProviderV1 } from '@adcp/reporting-storage/azure';
|
|
552
564
|
|
|
553
565
|
const azure = createAzureBlobReportingRowObjectProviderV1({
|
|
554
566
|
client: new BlobServiceClient('https://acmerows.blob.core.windows.net', new DefaultAzureCredential()),
|
|
@@ -677,7 +689,7 @@ an unknown ID. A revision in (a) to (d) whose content cannot be served is
|
|
|
677
689
|
// Core-only service
|
|
678
690
|
createReliableReportingService({
|
|
679
691
|
// ...
|
|
680
|
-
maintenance: { compaction: { enabled: true, limit: 100 } },
|
|
692
|
+
maintenance: reportingLedgerMaintenance({ compaction: { enabled: true, limit: 100 } }),
|
|
681
693
|
});
|
|
682
694
|
|
|
683
695
|
// Production service
|
|
@@ -751,7 +763,8 @@ and `compactSupersededSnapshots` re-check before compacting.
|
|
|
751
763
|
|
|
752
764
|
You do not need the production service (managed delivery, notifications and
|
|
753
765
|
activity) to keep a ledger tidy. Build the store with the options you want and
|
|
754
|
-
give `createReliableReportingService` a `maintenance` option
|
|
766
|
+
give `createReliableReportingService` a `maintenance` option built by
|
|
767
|
+
`reportingLedgerMaintenance` from `@adcp/reporting-storage`. The production
|
|
755
768
|
service does the same work automatically; the two share one implementation, so
|
|
756
769
|
partial-failure reporting, the `maxFeedHoldDays` that retention honours, per-task
|
|
757
770
|
limits and abort handling are identical.
|
|
@@ -762,7 +775,8 @@ import {
|
|
|
762
775
|
REPORTING_LEDGER_MIGRATION,
|
|
763
776
|
REPORTING_ROW_STORAGE_MIGRATION,
|
|
764
777
|
REPORTING_LEDGER_CHANGES_MIGRATION,
|
|
765
|
-
|
|
778
|
+
reportingLedgerMaintenance,
|
|
779
|
+
} from '@adcp/reporting-storage';
|
|
766
780
|
import { createReliableReportingService } from '@adcp/sdk/reporting/service';
|
|
767
781
|
|
|
768
782
|
// Apply the three migrations, then:
|
|
@@ -777,11 +791,11 @@ const reporting = createReliableReportingService({
|
|
|
777
791
|
store,
|
|
778
792
|
statusRetentionDays: 90, // the commitment advertised to buyers; retention uses it
|
|
779
793
|
// ...adapters, contact, resolveSource, resolveCurrency, resolveCoverage
|
|
780
|
-
maintenance: {
|
|
794
|
+
maintenance: reportingLedgerMaintenance({
|
|
781
795
|
intervalMilliseconds: 300_000, // default: at most one pass per 5 minutes
|
|
782
796
|
retention: { enabled: true, recordRetentionDays: 120, limit: 100 },
|
|
783
797
|
changeFeed: { changeRetentionDays: 30, maxFeedHoldDays: 7 },
|
|
784
|
-
},
|
|
798
|
+
}),
|
|
785
799
|
});
|
|
786
800
|
|
|
787
801
|
reporting.start({ intervalMilliseconds: 60_000, deploymentWide: true, onError });
|
|
@@ -813,12 +827,18 @@ Tasks, in order:
|
|
|
813
827
|
| `change-feed pruning` | when `maintenance.changeFeed` is set | `pruneChanges` with `changeRetentionDays` and `maxFeedHoldDays`. |
|
|
814
828
|
| `retention` | when `maintenance.retention.enabled` is `true` | `retireExpiredPeriods` with the service's `statusRetentionDays` and the feed's `maxFeedHoldDays`. |
|
|
815
829
|
| `snapshot compaction` | when `maintenance.compaction.enabled` is `true` | `compactSupersededSnapshots` with the service's `statusRetentionDays` and `automatedRecoveryWindowSeconds` (see [Compacting superseded snapshots](#compacting-superseded-snapshots-opt-in)). |
|
|
830
|
+
| `orphan object collection` | when `maintenance.orphanCollection.enabled` is `true` | `collectOrphanedRowObjects`: lists each object binding under `{prefix}/{namespace_key}/` and deletes old objects nothing names. Opt-in, rate-limited and resumable; see [Orphaned object collection](REPORTING-OPERATIONS.md#adopting-row-storage-retention-and-the-change-feed). |
|
|
816
831
|
|
|
817
832
|
Notes:
|
|
818
833
|
|
|
819
|
-
- `maintenance`
|
|
834
|
+
- The service's `maintenance` option takes a factory, not an options object:
|
|
835
|
+
the SDK does not know what maintenance a store needs, so it calls the factory
|
|
836
|
+
once at construction with the store and the retention and recovery-window
|
|
837
|
+
commitments it advertises. `reportingLedgerMaintenance(options)` is the
|
|
838
|
+
PostgreSQL one. It requires a `PostgresReportingLedgerStore` (or a store with the
|
|
820
839
|
same `retireExpiredPeriods`, `pruneChanges` and `sweepRowWriteIntents`
|
|
821
|
-
methods
|
|
840
|
+
methods, plus `collectOrphanedRowObjects` when `orphanCollection` is set) and
|
|
841
|
+
fails at construction otherwise. `maintenance.changeFeed` also
|
|
822
842
|
requires `changeFeed: true` on the store, since pruning is otherwise a silent
|
|
823
843
|
no-op.
|
|
824
844
|
- Retention takes `statusRetentionDays` from the service, so it can never retire
|
|
@@ -838,7 +858,7 @@ In-process consumers such as pacing, alerts and warehouse loaders read from the
|
|
|
838
858
|
ledger, never from a warehouse copy.
|
|
839
859
|
|
|
840
860
|
```ts
|
|
841
|
-
import { REPORTING_LEDGER_CHANGES_MIGRATION } from '@adcp/
|
|
861
|
+
import { REPORTING_LEDGER_CHANGES_MIGRATION } from '@adcp/reporting-storage';
|
|
842
862
|
|
|
843
863
|
await pool.query(REPORTING_LEDGER_CHANGES_MIGRATION); // PostgreSQL 13+
|
|
844
864
|
const store = new PostgresReportingLedgerStore(pool, { acknowledgeIsolatedDatabase: true, changeFeed: true });
|
|
@@ -860,30 +880,60 @@ await store.saveFeedConsumerCursor('pacing', page.cursor); // holds back pruning
|
|
|
860
880
|
highest-numbered snapshot. An official revision is terminal, so this is always
|
|
861
881
|
one index lookup.
|
|
862
882
|
- `changesAfter` lists committed obligation, revision, adjustment and retirement
|
|
863
|
-
changes.
|
|
864
|
-
|
|
865
|
-
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
869
|
-
|
|
870
|
-
|
|
871
|
-
|
|
872
|
-
|
|
873
|
-
it
|
|
874
|
-
|
|
883
|
+
changes. Every feed, account-scoped or deployment-wide (omit `account_id`),
|
|
884
|
+
orders by `(xid, seq)` and returns only changes whose transaction precedes
|
|
885
|
+
every transaction still in flight (`xid` below the reading snapshot's
|
|
886
|
+
`xmin`), so a slow commit is never skipped. The order does not depend on the
|
|
887
|
+
account lock, on the order a writer drew `seq` in, or on which lock key a
|
|
888
|
+
writer took: a change with a lower `seq` and a higher `xid` that commits late
|
|
889
|
+
is simply returned on a later page, after the cursor. The cost is that any
|
|
890
|
+
older open transaction on the cluster, including an idle one in another
|
|
891
|
+
database, holds every feed back until it ends; keep transactions short. An
|
|
892
|
+
empty page therefore means either caught up or held back, so poll again rather
|
|
893
|
+
than treating it as the end of the feed. The account scan is served by the
|
|
894
|
+
`(account_id, xid, seq)` index that `REPORTING_LEDGER_CHANGES_MIGRATION`
|
|
895
|
+
creates.
|
|
875
896
|
- **A ledger database has exactly one writer SDK.** The store writes each change
|
|
876
|
-
row under the account advisory lock
|
|
877
|
-
|
|
878
|
-
|
|
879
|
-
|
|
880
|
-
|
|
881
|
-
|
|
882
|
-
|
|
883
|
-
|
|
884
|
-
`
|
|
885
|
-
|
|
886
|
-
|
|
897
|
+
row under the account advisory lock. The feed no longer needs that lock for
|
|
898
|
+
its order, but a second SDK is still unsupported: it can write rows the first
|
|
899
|
+
SDK's lifecycle, retention and idempotency logic does not know about. Give
|
|
900
|
+
another SDK (for example a Python deployment) its own database. `pruneChanges`
|
|
901
|
+
only deletes changes older than `changeRetentionDays`, far longer than any
|
|
902
|
+
transaction should stay open, and consumer holds compare against cursors the
|
|
903
|
+
feed itself returned, so neither adds an ordering assumption.
|
|
904
|
+
- **Upgrading from account cursors that carry only `seq`.** Account feeds used
|
|
905
|
+
to order by `seq` and cut each page before the first in-flight change. An old
|
|
906
|
+
account cursor is `{ seq }` with no `xid`. `changesAfter` accepts it and maps
|
|
907
|
+
it to the position `(xid, seq)` where `xid` is the smaller of the oldest `xid`
|
|
908
|
+
among the account's changes after that `seq` and the reading snapshot's `xmin`.
|
|
909
|
+
Every change after the old `seq` has an `xid` at or above both (a change that
|
|
910
|
+
has not committed yet belongs to a transaction at or above `xmin`), so nothing
|
|
911
|
+
is skipped. The cost is that changes the consumer already read can be returned
|
|
912
|
+
again: those with a lower `seq` and a higher `xid`, and, when a long
|
|
913
|
+
transaction was open at that moment, every change committed since it began.
|
|
914
|
+
Treat redelivery as normal, as for any at-least-once consumer. The returned cursor is in the new
|
|
915
|
+
format, even on an empty page. A legacy cursor fails with `CURSOR_EXPIRED` if
|
|
916
|
+
any change with a higher `seq` was pruned (`adcp_reporting_change_account_horizon.pruned_through_seq`
|
|
917
|
+
still records the highest pruned `seq` for this check). A consumer row saved
|
|
918
|
+
before the upgrade has a NULL `cursor_xid` and holds retention by `seq` until
|
|
919
|
+
the consumer saves its next cursor; `saveFeedConsumerCursor` given a legacy
|
|
920
|
+
cursor maps it the same way. Re-apply `REPORTING_LEDGER_CHANGES_MIGRATION`
|
|
921
|
+
before starting the new binary: it adds the account horizon's `xid` columns,
|
|
922
|
+
replaces the consumer table's check constraint, and creates the new index. On
|
|
923
|
+
a large changes table, create the index first with `CREATE INDEX CONCURRENTLY
|
|
924
|
+
IF NOT EXISTS adcp_reporting_changes_account_xid ON adcp_reporting_changes
|
|
925
|
+
(account_id, xid, seq)`, because a plain `CREATE INDEX` blocks appends while it
|
|
926
|
+
builds; the migration then skips it. The older
|
|
927
|
+
`adcp_reporting_changes_account (account_id, seq)` index is left in place and
|
|
928
|
+
serves the legacy-cursor mapping; drop it once no legacy cursor remains.
|
|
929
|
+
- **Do not run the old and new SDK's change-feed maintenance together.** An older
|
|
930
|
+
host's `pruneChanges`, `retireExpiredPeriods` and compaction hold an account
|
|
931
|
+
consumer's changes by `seq` alone and record only `pruned_through_seq`, so
|
|
932
|
+
against a consumer that holds new-format cursors they can prune a change the
|
|
933
|
+
consumer has not read and never advance the horizon the new reader checks.
|
|
934
|
+
Stop the older hosts' maintenance (or the hosts) before the new binary reads
|
|
935
|
+
the feed. Rolling back is not supported once a consumer has saved a new-format
|
|
936
|
+
account cursor: the older SDK rejects it, so reset those consumers.
|
|
887
937
|
- `pruneChanges` deletes changes older than `changeRetentionDays` (default 30)
|
|
888
938
|
that every consumer updated within `maxFeedHoldDays` (default 7) has passed.
|
|
889
939
|
An older cursor throws `ReportingChangeCursorExpiredError`; resynchronize from
|
|
@@ -918,7 +968,7 @@ await store.saveFeedConsumerCursor('pacing', page.cursor); // holds back pruning
|
|
|
918
968
|
|
|
919
969
|
### BigQuery warehouse sink
|
|
920
970
|
|
|
921
|
-
`@adcp/
|
|
971
|
+
`@adcp/reporting-storage/bigquery` loads committed revisions into your own BigQuery
|
|
922
972
|
tables for analysts. The warehouse copy is derived and non-authoritative. It
|
|
923
973
|
is never read for serving or verification, so its table names, columns,
|
|
924
974
|
partitioning and retention are yours.
|
|
@@ -928,7 +978,7 @@ import { BigQuery } from '@google-cloud/bigquery';
|
|
|
928
978
|
import {
|
|
929
979
|
REPORTING_WAREHOUSE_SINK_MIGRATION,
|
|
930
980
|
createBigQueryReportingWarehouseSinkV1,
|
|
931
|
-
} from '@adcp/
|
|
981
|
+
} from '@adcp/reporting-storage/bigquery';
|
|
932
982
|
|
|
933
983
|
await pool.query(REPORTING_WAREHOUSE_SINK_MIGRATION); // after REPORTING_LEDGER_CHANGES_MIGRATION
|
|
934
984
|
const bigquery = new BigQuery({ projectId: 'analytics-prod' }); // ADC / workload identity
|
|
@@ -1109,9 +1159,13 @@ Core remains the default and has no destination, external-resource, or receipt d
|
|
|
1109
1159
|
```ts
|
|
1110
1160
|
import {
|
|
1111
1161
|
PostgresReportingLedgerStore,
|
|
1162
|
+
} from '@adcp/reporting-storage';
|
|
1163
|
+
import {
|
|
1112
1164
|
PostgresReportingManagedDeliveryStore,
|
|
1113
1165
|
REPORTING_MANAGED_DELIVERY_MIGRATION,
|
|
1114
1166
|
createReportingManagedDeliveryRuntime,
|
|
1167
|
+
} from '@adcp/reporting-storage/managed';
|
|
1168
|
+
import {
|
|
1115
1169
|
reportingManagedDeliveryBindingV1,
|
|
1116
1170
|
} from '@adcp/sdk/reporting/ledger';
|
|
1117
1171
|
|
|
@@ -1182,7 +1236,7 @@ A seller often delivers one feed through several transports: one buyer takes fil
|
|
|
1182
1236
|
import {
|
|
1183
1237
|
createReportingManagedDeliveryAdapterRouterV1,
|
|
1184
1238
|
createReportingManagedDeliveryRuntime,
|
|
1185
|
-
} from '@adcp/
|
|
1239
|
+
} from '@adcp/reporting-storage/managed';
|
|
1186
1240
|
|
|
1187
1241
|
const adapter = createReportingManagedDeliveryAdapterRouterV1({
|
|
1188
1242
|
routes: [
|
|
@@ -1262,12 +1316,14 @@ only for health changes:
|
|
|
1262
1316
|
```ts
|
|
1263
1317
|
import { createPostgresPersistentNotificationRuntime } from '@adcp/sdk/server';
|
|
1264
1318
|
import {
|
|
1265
|
-
createPostgresReportingNotificationActivityRuntime,
|
|
1266
|
-
createPostgresReportingNotificationAttemptCheckpoint,
|
|
1267
1319
|
PostgresReportingLedgerStore,
|
|
1268
1320
|
REPORTING_LEDGER_FINALITY_WRITER_FENCE_MIGRATION,
|
|
1269
1321
|
REPORTING_LEDGER_MIGRATION,
|
|
1270
|
-
} from '@adcp/
|
|
1322
|
+
} from '@adcp/reporting-storage';
|
|
1323
|
+
import {
|
|
1324
|
+
createPostgresReportingNotificationActivityRuntime,
|
|
1325
|
+
createPostgresReportingNotificationAttemptCheckpoint,
|
|
1326
|
+
} from '@adcp/reporting-storage/notifications';
|
|
1271
1327
|
|
|
1272
1328
|
// Build the durable pre-POST checkpoint first. The notification runtime needs
|
|
1273
1329
|
// it, and the activity runtime verifies it targets the same durable store —
|
|
@@ -5,7 +5,7 @@ consumer runtime. Reliable Reporting controls evidence used for billing and
|
|
|
5
5
|
large media-spend decisions. Treat its PostgreSQL state, destination objects,
|
|
6
6
|
signing keys, and authentication registry as financial infrastructure.
|
|
7
7
|
|
|
8
|
-
> **Stability:** the storage
|
|
8
|
+
> **Stability:** the storage this runbook operates lives in `@adcp/reporting-storage`, which is pre-1.0 and experimental. Pin it with `~` (for example `~0.1.0`) and read each minor's changeset before upgrading. See the [Reliable Reporting stability policy](REPORTING-STABILITY.md).
|
|
9
9
|
|
|
10
10
|
## Production topology
|
|
11
11
|
|
|
@@ -80,7 +80,8 @@ Each of these features is opt-in and additive. Enable them one at a time on a
|
|
|
80
80
|
running deployment. See the [ledger guide](REPORTING-LEDGER.md#row-storage)
|
|
81
81
|
for configuration details.
|
|
82
82
|
|
|
83
|
-
Running Core only, without the production service? Pass `maintenance
|
|
83
|
+
Running Core only, without the production service? Pass `maintenance:
|
|
84
|
+
reportingLedgerMaintenance({ ... })` (from `@adcp/reporting-storage`) to
|
|
84
85
|
`createReliableReportingService` so its scheduler (or `runMaintenance()`) does
|
|
85
86
|
the retention, change-feed pruning and abandoned-upload sweeps the production
|
|
86
87
|
service runs for you. See
|
|
@@ -143,6 +144,58 @@ service runs for you. See
|
|
|
143
144
|
for the rule and what it keeps. Compaction cannot be undone; backups hold
|
|
144
145
|
any longer evidence.
|
|
145
146
|
|
|
147
|
+
8. **Orphaned object collection (optional).** The intent sweep deletes only
|
|
148
|
+
what an intent lists. It cannot see an upload whose `PUT` succeeded before the
|
|
149
|
+
intent recorded it, an intent for a binding no running host configures, or an
|
|
150
|
+
intent lost to a database restore, and a bucket lifecycle rule cannot tell
|
|
151
|
+
those objects from committed ones. Enable `orphanCollection`
|
|
152
|
+
(`maintenance.orphanCollection` on the Core-only service) to list each
|
|
153
|
+
configured object binding under `{prefix}/{namespace_key}/` and delete what
|
|
154
|
+
nothing names. Roll it out in this order:
|
|
155
|
+
1. Run it with `dryRun: true` and read `candidates` in the
|
|
156
|
+
`orphan object collection` task result. Confirm they are objects you
|
|
157
|
+
expect to lose.
|
|
158
|
+
2. Enable it for real, with `acknowledgeExclusiveNamespace: true` (see
|
|
159
|
+
below), a modest `limit` (objects examined per binding per pass, default
|
|
160
|
+
1000) and one host.
|
|
161
|
+
|
|
162
|
+
It is built for a long period: a listing that does not finish within `limit`
|
|
163
|
+
continues on the next pass, and a finished one rests for `intervalSeconds`
|
|
164
|
+
(default one day) before the next starts. That position lives in the process,
|
|
165
|
+
so a restart begins a new listing; the cost is bounded by `limit` per pass
|
|
166
|
+
either way. Reduce `limit`, or lengthen `intervalSeconds`, to spend fewer list
|
|
167
|
+
requests.
|
|
168
|
+
|
|
169
|
+
An object is deleted only when all of these hold: its key is a `.jsonl` or
|
|
170
|
+
`.jsonl.gz` object under the binding's owned prefix; it was written more than
|
|
171
|
+
`minAgeSeconds` ago (default 7 days, never less than 1 day, which must stay
|
|
172
|
+
well above the write deadline plus intent expiry); no row set that is not yet
|
|
173
|
+
`pruned` has a chunk at that key and no write intent lists it, at any version;
|
|
174
|
+
and, under the per-object advisory lock every other delete takes, no `live` or
|
|
175
|
+
`unavailable` row set references that exact version and no open intent
|
|
176
|
+
created it, checked under every binding because two bindings can share a
|
|
177
|
+
bucket and prefix (a retired binding kept `readOnly` next to its successor).
|
|
178
|
+
`readOnly` bindings are skipped unless `includeReadOnly: true`. A binding
|
|
179
|
+
whose provider has no `list` is reported as `unsupported` (and as a
|
|
180
|
+
maintenance warning), never silently skipped. Everything under the owned prefix
|
|
181
|
+
with a row-object key counts as written by this SDK, so do not store other
|
|
182
|
+
`.jsonl` files there.
|
|
183
|
+
|
|
184
|
+
**Run it only against the ledger database that owns the namespace.** The
|
|
185
|
+
namespace key comes from the database's installation identity. A clone or
|
|
186
|
+
point-in-time copy of that database that still points at the same bucket
|
|
187
|
+
shares the key, and its collector would delete the objects of every row set
|
|
188
|
+
the original wrote after the copy was taken, because the copy has never heard
|
|
189
|
+
of them. The age threshold does not protect against that, which is why a real
|
|
190
|
+
(not dry) pass needs `acknowledgeExclusiveNamespace: true`: it is your statement
|
|
191
|
+
that this database is the only owner. A clone made by copying the deployment
|
|
192
|
+
copies the flag too, so the flag does not protect it: remove it, or the whole
|
|
193
|
+
option, from every copy. Disable the option on clones, and after
|
|
194
|
+
restoring a database in place expect the collector to delete the objects of the
|
|
195
|
+
row sets the restore lost, which is its purpose.
|
|
196
|
+
The cost grows with bucket size: one listing request per page, plus one
|
|
197
|
+
database query per page and one transaction per deleted object.
|
|
198
|
+
|
|
146
199
|
Alert on these row-storage signals:
|
|
147
200
|
|
|
148
201
|
- `ROWS_INTEGRITY_FAILED` or `ROWS_UNAVAILABLE` from reads. These are never
|
|
@@ -154,6 +207,14 @@ Alert on these row-storage signals:
|
|
|
154
207
|
host does not configure, and `overdue` (expired more than 24 hours ago, still
|
|
155
208
|
unswept) means no host is handling them and their objects are leaking. The
|
|
156
209
|
maintenance helper reports both through `onError`.
|
|
210
|
+
- The `orphan object collection` task reporting `failed` or `unsupported`
|
|
211
|
+
bindings (their provider cannot list, so nothing there is collected). A listing
|
|
212
|
+
that fails is retried from the page that failed (or from the start, if it was
|
|
213
|
+
the first page of a resumed listing). A delete that fails is counted and the
|
|
214
|
+
pass moves on, so one stuck object cannot hold up the rest; it is tried again on
|
|
215
|
+
the next full listing. `referenced` and
|
|
216
|
+
`tooYoung` are normal; a `foreign` count means the prefix holds objects this
|
|
217
|
+
SDK did not write as row objects, which it leaves alone.
|
|
157
218
|
- The `snapshot compaction` task reporting `failed` revisions. Provider deletes
|
|
158
219
|
are failing; the revisions stay `pruning` (reads already answer not-found) and
|
|
159
220
|
the next pass resumes them.
|
|
@@ -328,7 +389,7 @@ For complete GCS file-transfer delivery, private contracts and scoped buyer read
|
|
|
328
389
|
|
|
329
390
|
## Live GCP validation
|
|
330
391
|
|
|
331
|
-
`scripts/validate-reporting-gcp.mjs` runs the built
|
|
392
|
+
`packages/reporting-storage/scripts/validate-reporting-gcp.mjs` (`npm run validate:reporting-gcp`) runs the built `@adcp/reporting-storage` against a real BigQuery
|
|
332
393
|
project and a real GCS bucket. The in-memory doubles in the test suite cannot
|
|
333
394
|
prove how BigQuery names jobs, where it stores load options, or what a real
|
|
334
395
|
bucket policy looks like; this script does. It is not published with the
|
|
@@ -336,7 +397,7 @@ package, so run it from a repository checkout.
|
|
|
336
397
|
|
|
337
398
|
What it checks:
|
|
338
399
|
|
|
339
|
-
- **BigQuery warehouse sink** (`@adcp/
|
|
400
|
+
- **BigQuery warehouse sink** (`@adcp/reporting-storage/bigquery`). It applies the
|
|
340
401
|
ledger migrations into fresh PostgreSQL schemas, commits revisions, creates a
|
|
341
402
|
uniquely named dataset in the destination project from `defaultTablesSql` and
|
|
342
403
|
`currentRowsViewSql`, and runs the sink. It verifies row and revision counts
|
|
@@ -372,7 +433,7 @@ What it checks:
|
|
|
372
433
|
`GCP_BQ_PROJECT`. Load jobs are then created in the destination project, and
|
|
373
434
|
the crash replay fails if the "job already exists" status check looks in the
|
|
374
435
|
client's default project instead.
|
|
375
|
-
- **GCS row provider** (`@adcp/
|
|
436
|
+
- **GCS row provider** (`@adcp/reporting-storage/gcs`). It prints the bucket policy,
|
|
376
437
|
runs the provider probe (uniform bucket-level access, public access
|
|
377
438
|
prevention, lifecycle rules) and `runReportingRowObjectProviderConformanceV1`,
|
|
378
439
|
then commits a revision through an object binding on the bucket and reads it
|
|
@@ -1,76 +1,80 @@
|
|
|
1
1
|
# Reliable Reporting stability policy
|
|
2
2
|
|
|
3
|
-
The Reliable Reporting storage surface is **experimental**. It is new, it has not yet run in production at an adopter, and its shape is still being settled by real use.
|
|
3
|
+
The Reliable Reporting storage surface is **experimental**. It is new, it has not yet run in production at an adopter, and its shape is still being settled by real use. It ships as its own package, `@adcp/reporting-storage`, on its own version line, so a breaking change there is a minor release of that package and never forces a major release of `@adcp/sdk`.
|
|
4
4
|
|
|
5
|
-
This follows how the AdCP protocol treats experimental surfaces (`x-status: experimental` in the schemas, with advance notice of breaking changes).
|
|
5
|
+
This follows how the AdCP protocol treats experimental surfaces (`x-status: experimental` in the schemas, with advance notice of breaking changes), and how other SDKs ship storage (Auth.js `@auth/pg-adapter`, LangGraph's checkpoint packages, OpenTelemetry exporters). The Python SDK is making the same split (adcp-client-python#1510).
|
|
6
6
|
|
|
7
7
|
## What is experimental
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
**Everything exported by `@adcp/reporting-storage`.** The package is pre-1.0 (`0.x`), and each export carries an `@experimental` TSDoc tag that links to this page. Editors show the tag on hover, and some mark the symbol as unstable.
|
|
10
10
|
|
|
11
|
-
| Subpath
|
|
12
|
-
|
|
|
13
|
-
| `@adcp/
|
|
14
|
-
| `@adcp/
|
|
15
|
-
| `@adcp/
|
|
16
|
-
| `@adcp/
|
|
17
|
-
| `@adcp/
|
|
18
|
-
| `@adcp/
|
|
11
|
+
| Subpath | Holds |
|
|
12
|
+
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
|
|
13
|
+
| `@adcp/reporting-storage` | the PostgreSQL ledger store (`PostgresReportingLedgerStore`), its migrations, `sweepExpiredReportingLedgerState`, row storage (`ReportingRowStorageV1`, row bindings, the provider contract and conformance suite, the filesystem provider, the row-storage migration and probe), object writes, the host change feed (`REPORTING_LEDGER_CHANGES_MIGRATION`, change records and cursors), and ledger maintenance (retention, compaction, orphan collection, change-feed pruning, `reportingLedgerMaintenance`) |
|
|
14
|
+
| `@adcp/reporting-storage/managed` | the managed-delivery runtime, `PostgresReportingManagedDeliveryStore`, the adapter contract and the adapter router |
|
|
15
|
+
| `@adcp/reporting-storage/notifications` | notification activity stores, checkpoints and runtime |
|
|
16
|
+
| `@adcp/reporting-storage/service` | `createPostgresReliableReportingProductionService` |
|
|
17
|
+
| `@adcp/reporting-storage/gcs` | the GCS write fence, reader, managed-delivery adapter and row-object provider |
|
|
18
|
+
| `@adcp/reporting-storage/s3` | the S3 row-object provider |
|
|
19
|
+
| `@adcp/reporting-storage/azure` | the Azure Blob row-object provider |
|
|
20
|
+
| `@adcp/reporting-storage/bigquery` | the BigQuery warehouse sink |
|
|
19
21
|
|
|
20
|
-
|
|
22
|
+
**The storage interfaces in `@adcp/sdk`.** The contract the package implements stays in the SDK, and it changes together with the package, so it is experimental too: the store, transaction, lease, row-read and revision-metadata interfaces in `@adcp/sdk/reporting/ledger` (`ReportingLedgerStore` and the types it uses), `ReportingReceiptSyncStoreV1`, the store-contract error classes (`ReportingLedgerLeaseLostError`, `ReportingLedgerPeriodRetiredError`, `ReportingLedgerContinuityError`) and the row-storage error types. Their TSDoc carries the `@experimental` tag. A breaking change to one ships as a minor of `@adcp/sdk` together with a matching minor of `@adcp/reporting-storage`, and the changesets say which pair to install.
|
|
21
23
|
|
|
22
|
-
- the
|
|
23
|
-
- row storage (`ReportingRowStorageV1`, row bindings, row encoding, the row-storage migration and probe, and row-store errors) and object writes;
|
|
24
|
-
- the host change feed (`REPORTING_LEDGER_CHANGES_MIGRATION`, change records and cursors);
|
|
25
|
-
- managed delivery (the runtime, the store, the adapter contract, the adapter router and the PostgreSQL store);
|
|
26
|
-
- notification activity stores and checkpoints;
|
|
27
|
-
- the store, transaction, lease and row-read interfaces in `ledger/types`.
|
|
28
|
-
|
|
29
|
-
Within `@adcp/sdk/reporting/service`, the experimental exports are `ReportingRetentionOptionsV1`, `ReportingCompactionOptionsV1`, `ReportingChangeFeedMaintenanceOptionsV1`, `runReportingLedgerMaintenance` and the related maintenance types and error.
|
|
24
|
+
**Storage-facing building blocks in `@adcp/sdk`.** A few exports exist so a store can be written against public API only, and they are covered by the same policy even where their TSDoc is silent: the schedule, instant and evidence helpers on `@adcp/sdk/reporting/ledger` (`compareReportingInstants`, `reportingPeriodDate`, `reportingPeriodSchedule`, `isFrozenCalendarRulesMismatch`, the receipt and verification evidence guards, `isWellFormedUnicodeString`, `getReportingDeliveryOfferingValidatorV1`), and the scheduler primitives and `REPORTING_MANAGED_DELIVERY_COMPOSITION_V1` on `@adcp/sdk/reporting/service`. Application code should not depend on them.
|
|
30
25
|
|
|
31
26
|
### Not experimental
|
|
32
27
|
|
|
33
|
-
These follow normal semver. A breaking change to them ships in a major release
|
|
28
|
+
These follow normal semver. A breaking change to them ships in a major release of `@adcp/sdk`.
|
|
34
29
|
|
|
35
30
|
- The protocol-facing handlers: `createReportingDeliveryHandler`, `createReportingStatusHandler`, `createSyncReportingStatusHandler`, `createSyncReportingReceiptsHandler`, and the status-ingest request and response schemas.
|
|
36
31
|
- `createReportingProducer`, `assertSupportedReportingAuthoritativeParty` and `UnsupportedReportingFeatureError`.
|
|
37
|
-
- The health, coverage, lifecycle, schedule and consumer-status-identity helpers in `@adcp/sdk/reporting/ledger
|
|
38
|
-
- `createReliableReportingService`, `
|
|
32
|
+
- The health, coverage, lifecycle, schedule and consumer-status-identity helpers in `@adcp/sdk/reporting/ledger`, and the canonical row encoding and digests (`row-encoding`) with their golden fixtures.
|
|
33
|
+
- `createReliableReportingService`, its `maintenance` hook and the service conformance runner in `@adcp/sdk/reporting/service`.
|
|
39
34
|
- `@adcp/sdk/reporting/consumer` (buyer reconciliation and worker) and `@adcp/sdk/reporting/source` (source contract and executor). They are separate subpaths with their own `V1` contract suffixes and are not covered by this policy.
|
|
40
35
|
|
|
41
36
|
If you are unsure about a symbol, read its TSDoc. An `@experimental` tag means it is covered. No tag means it is not.
|
|
42
37
|
|
|
43
38
|
## What "experimental" means
|
|
44
39
|
|
|
45
|
-
- **Breaking changes may ship in minor releases
|
|
40
|
+
- **Breaking changes may ship in minor releases of `@adcp/reporting-storage`.** That covers signatures, option names, stored field and column names, migrations, and defaults. While the package is `0.x`, a minor bump (`0.1` to `0.2`) is the one that may break.
|
|
46
41
|
- **Every such change is announced.** The changeset describes the break and includes a migration note, and the release notes carry it. Larger changes also get a `docs/migration-*.md` walkthrough.
|
|
47
|
-
- **The wire protocol is not affected.** Buyer-visible behavior
|
|
42
|
+
- **The wire protocol is not affected.** Buyer-visible behavior the package implements (reporting delivery, `revision_content_sha256`, status, receipts and so on) follows AdCP's own versioning, not this policy. A change to the storage layout does not change what a buyer sees.
|
|
48
43
|
- **Security and correctness fixes are not held back.** If a fix needs a breaking change, it ships in a minor.
|
|
49
44
|
|
|
45
|
+
## How the two packages line up
|
|
46
|
+
|
|
47
|
+
`@adcp/reporting-storage` declares `@adcp/sdk` as a **peer dependency** (`^16.0.0-beta.0` while SDK 16 is in beta), not a dependency. The store, its errors and the handlers share types and symbol-keyed guards, so the host must hold one copy of `@adcp/sdk`. A change to a storage interface in the SDK ships with a matching storage release; the changeset for each says which pair to install. The cloud SDKs (`@aws-sdk/client-s3`, `@azure/storage-blob`, `@google-cloud/storage`) are optional peers of the storage package. `@adcp/sdk` does not import `@adcp/reporting-storage`, and a test and a lint rule keep it that way.
|
|
48
|
+
|
|
50
49
|
## How to pin
|
|
51
50
|
|
|
52
|
-
|
|
51
|
+
Pin the storage package with a tilde range. A caret range on a `0.x` version already stops at the next minor, but `~` states the intent and survives the move to `1.0`.
|
|
53
52
|
|
|
54
53
|
```json
|
|
55
54
|
{
|
|
56
55
|
"dependencies": {
|
|
57
|
-
"@adcp/sdk": "
|
|
56
|
+
"@adcp/sdk": "^16.0.0-beta.0",
|
|
57
|
+
"@adcp/reporting-storage": "~0.1.0-beta.0"
|
|
58
58
|
}
|
|
59
59
|
}
|
|
60
60
|
```
|
|
61
61
|
|
|
62
|
-
|
|
63
|
-
|
|
62
|
+
Those are the beta pins. A prerelease range such as `~0.1.0` does not match `0.1.0-beta.0`, so use the `-beta.0` form until stable releases exist, then move to `^16.0.0` and `~0.1.0`.
|
|
63
|
+
|
|
64
|
+
- `~0.1.0` (after the first stable release) takes patches only (recommended).
|
|
65
|
+
- An exact version (`"0.1.0-beta.0"`) is stricter still.
|
|
64
66
|
- Read the changelog before widening to a new minor, and apply any migration it names.
|
|
65
67
|
|
|
66
|
-
|
|
68
|
+
`@adcp/sdk` itself can use a caret range as usual, because the SDK's own surface follows semver. While SDK 16 is in beta, install both packages with the `beta` dist-tag (`npm install @adcp/sdk@beta @adcp/reporting-storage@beta`) until stable releases exist.
|
|
67
69
|
|
|
68
70
|
## Migrations
|
|
69
71
|
|
|
70
|
-
Database migrations exported
|
|
72
|
+
Database migrations exported by `@adcp/reporting-storage` (`REPORTING_LEDGER_MIGRATION`, `REPORTING_ROW_STORAGE_MIGRATION` and the others) stay idempotent where practical, and a changeset says when you must re-apply one. See [Reliable Reporting production operations](REPORTING-OPERATIONS.md) and the [ledger guide](REPORTING-LEDGER.md).
|
|
73
|
+
|
|
74
|
+
Coming from the single-package layout (`@adcp/sdk/reporting/ledger`, `/gcs`, `/s3`, `/azure`, `/bigquery`)? See [Reliable Reporting storage moved to `@adcp/reporting-storage`](../migration-reporting-storage-package.md).
|
|
71
75
|
|
|
72
76
|
## When this will change
|
|
73
77
|
|
|
74
|
-
The surface is expected to stabilize after the first production adopter runs on SDK 16 stable. At that point the `@experimental` tags are removed in
|
|
78
|
+
The surface is expected to stabilize after the first production adopter runs on SDK 16 stable. At that point `@adcp/reporting-storage` moves to `1.0.0` and the `@experimental` tags are removed in that release with a changelog note. Breaking changes need a major release from then on.
|
|
75
79
|
|
|
76
80
|
Related: [Seller reporting ledger](REPORTING-LEDGER.md), [Reliable Reporting production operations](REPORTING-OPERATIONS.md).
|