@codelitdev/platform-conformance 0.1.0-alpha.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md ADDED
@@ -0,0 +1,93 @@
1
+ # @codelitdev/platform-conformance
2
+
3
+ Shared integration checks for products built with CodeLit [Platform](https://github.com/codelitdev/platform) packages.
4
+ Conformance tests the behavior of a product after it composes the packages with
5
+ its own routes, credentials, tenants, policies, database, and workers. Package
6
+ unit tests and product domain tests still have their own jobs.
7
+
8
+ The product supplies fixtures and adapters, calls the suites it uses, and
9
+ asserts that every returned `failures` array is empty. Installing this package
10
+ does not run a test or create a CI gate.
11
+
12
+ ## Scope
13
+
14
+ The [Platform architecture](../../docs/architecture.md) calls for these
15
+ product-level checks. The last column distinguishes working checks from gaps;
16
+ the reference runner is not yet a general adapter for every product.
17
+
18
+ | Area | Product-level invariant | Coverage today |
19
+ | --- | --- | --- |
20
+ | Credentials | Sessions, OAuth tokens, and API keys resolve to the right principal; ambiguous or invalid credentials fail without fallback; transport clients cannot present a trusted system identity. | Some cases in the reference runner. |
21
+ | OAuth integration | Advertised issuer, authorization and token endpoints, resource metadata, and declared DCR/CIMD support match the running product. | MCP protected-resource discovery has an independent suite; authorization-server and client-registration checks still need one. |
22
+ | Tenancy and identity lifecycle | Membership is rechecked, API keys stay fixed to one tenant, cross-tenant access is denied, and key revocation, invitations, and last-owner safeguards work. | Basic isolation in the reference runner; lifecycle cases remain product tests. |
23
+ | REST and OpenAPI | Runtime validation, stable safe errors, and generated OpenAPI agree with the product contract. | Reference-specific checks for `/v1/notes` and its operations. |
24
+ | MCP transport and parity | Anonymous requests receive an OAuth challenge; authenticated tools follow product policy and match equivalent REST capabilities; destructive actions require explicit proof. | Independent discovery suite and reference-specific tool checks. Static parity validation lives in `@codelitdev/mcp-server-kit`. |
25
+ | Observability | Disabled telemetry is safe, secrets are redacted, and a failed optional sink does not fail requests or jobs. | Reference runner hooks; independent composition suite still needed. |
26
+ | Billing composition | Product schema, action grants, audit hooks, fake-provider flow, and maintenance are wired correctly. | Reference runner hooks. Provider and workflow contracts live in `@codelitdev/billing/testing`. |
27
+ | Lifecycle | Health and readiness reflect state; shutdown stops new work and closes resources. | Reference runner checks fixed endpoints; independent suite still needed. |
28
+ | Version compatibility | Resolved packages satisfy supported and minimum-secure versions. | Owned by `platform-cli doctor` and the preset it ships; product CI should run that check. |
29
+
30
+ Conformance belongs at the product boundary. A fabricated response from a mock
31
+ adapter cannot establish that the real auth middleware, route, or worker works.
32
+ New suites should accept product-defined paths, operations, credentials, and
33
+ fixture setup rather than assume the reference product's notes resource. They
34
+ should reuse package-owned protocol or provider contracts instead of copying
35
+ them.
36
+
37
+ ## Available runners
38
+
39
+ `runPlatformConformance(adapter, capabilities)` exercises the reference
40
+ product's owner/member/outsider and two-tenant scenario. It covers parts of
41
+ auth, tenancy, REST, MCP, OpenAPI, observability, billing, readiness, and
42
+ shutdown. Its current requests assume `/v1/notes`, `notes.*` MCP tools, and
43
+ reference billing routes. Existing adopters can keep using it; products with
44
+ other domain models should not manufacture notes responses to make it pass.
45
+ See the [reference adapter](../../examples/reference-product/apps/api/src/conformance-adapter.ts)
46
+ for its fixture and HTTP shape.
47
+
48
+ `runMcpDiscoveryConformance(adapter, { resourceUrl })` is an independent HTTP
49
+ suite. It accepts the product's public MCP resource URL and a `request` adapter
50
+ that returns status, headers, and parsed body. It checks the anonymous `401`,
51
+ Bearer resource-metadata challenge, protected-resource metadata, and
52
+ advertised authorization servers. It does not assume a tool name, tenant
53
+ model, framework, or billing setup. See its [HTTP test](src/mcp-discovery.test.ts)
54
+ for an adapter using an ephemeral listener.
55
+
56
+ ```ts
57
+ import {
58
+ runMcpDiscoveryConformance,
59
+ runPlatformConformance,
60
+ } from "@codelitdev/platform-conformance";
61
+
62
+ // Use the composition runner only if your adapter implements its reference scenario.
63
+ const composition = await runPlatformConformance(referenceAdapter, {
64
+ mcp: true,
65
+ billing: true,
66
+ observability: true,
67
+ openapi: true,
68
+ shutdown: true,
69
+ });
70
+ expect(composition.failures).toEqual([]);
71
+
72
+ // Any MCP product can use its real HTTP route for this independent suite.
73
+ const discovery = await runMcpDiscoveryConformance(
74
+ { request: productHttpRequest },
75
+ { resourceUrl: "http://127.0.0.1:8000/mcp" },
76
+ );
77
+ expect(discovery.failures).toEqual([]);
78
+ ```
79
+
80
+ Run only the suites applicable to the product, but do not treat an omitted
81
+ suite as a pass. Products should declare their supported capabilities in their
82
+ test setup and keep the selected suites visible in review. A suite that needs
83
+ HTTP must exercise the real route, including auth and discovery middleware.
84
+
85
+ ## Product CI
86
+
87
+ Create disposable fixtures, start a local API listener or equivalent test
88
+ instance, run the selected suites, and close the listener, database, and
89
+ workers afterward. The test should be part of the API package's normal test
90
+ command. Run it after a frozen install on pull requests, and make that CI job a
91
+ required branch-protection check. A push-only workflow cannot guard a pull
92
+ request. Run `platform-cli doctor` in the same gate for installed versions and
93
+ managed configuration; it does not prove behavioral conformance.
package/dist/adapter.d.ts CHANGED
@@ -22,6 +22,7 @@ export type PlatformConformanceAdapter<Principal, Tenant, SessionCredential, OAu
22
22
  body: unknown;
23
23
  headers?: Record<string, string>;
24
24
  }>;
25
+ /** Return transport response headers so conformance can verify OAuth discovery challenges. */
25
26
  mcp?(input: McpInput): Promise<McpOutput>;
26
27
  readAuditEvents(): Promise<readonly AuditEvent[]>;
27
28
  openapiDocument?(): Promise<{
package/dist/index.d.ts CHANGED
@@ -1,2 +1,3 @@
1
1
  export type { ConformanceCapabilities, ConformanceFailure, PlatformConformanceAdapter, } from "./adapter.js";
2
+ export { type McpDiscoveryAdapter, type McpDiscoveryOptions, type McpDiscoveryRequest, type McpDiscoveryResponse, runMcpDiscoveryConformance, } from "./mcp-discovery.js";
2
3
  export { type ReferenceHttpInput, type ReferenceMcpInput, runPlatformConformance, } from "./run.js";
package/dist/index.js CHANGED
@@ -1 +1,2 @@
1
+ export { runMcpDiscoveryConformance, } from "./mcp-discovery.js";
1
2
  export { runPlatformConformance, } from "./run.js";
@@ -0,0 +1,28 @@
1
+ import type { ConformanceFailure } from "./adapter.js";
2
+ export type McpDiscoveryRequest = {
3
+ method: "GET" | "POST";
4
+ path: string;
5
+ headers?: Record<string, string>;
6
+ body?: unknown;
7
+ };
8
+ export type McpDiscoveryResponse = {
9
+ status: number;
10
+ headers?: Record<string, string>;
11
+ body?: unknown;
12
+ };
13
+ /** Send requests through the product's real HTTP stack, including its auth middleware. */
14
+ export type McpDiscoveryAdapter = {
15
+ request(input: McpDiscoveryRequest): Promise<McpDiscoveryResponse>;
16
+ };
17
+ export type McpDiscoveryOptions = {
18
+ /** The public OAuth resource identifier advertised by this MCP server. */
19
+ resourceUrl: string;
20
+ };
21
+ export declare function mcpResourceMetadataChallenge(headers?: Record<string, string>): string | undefined;
22
+ /**
23
+ * Check OAuth discovery at the HTTP boundary without assuming product routes,
24
+ * tool names, credentials, tenants, or billing behavior.
25
+ */
26
+ export declare function runMcpDiscoveryConformance(adapter: McpDiscoveryAdapter, options: McpDiscoveryOptions): Promise<{
27
+ failures: ConformanceFailure[];
28
+ }>;
@@ -0,0 +1,84 @@
1
+ function absoluteHttpUrl(value) {
2
+ try {
3
+ const url = new URL(value);
4
+ if ((url.protocol !== "http:" && url.protocol !== "https:") ||
5
+ url.username ||
6
+ url.password ||
7
+ url.search ||
8
+ url.hash) {
9
+ return undefined;
10
+ }
11
+ return url;
12
+ }
13
+ catch {
14
+ return undefined;
15
+ }
16
+ }
17
+ export function mcpResourceMetadataChallenge(headers) {
18
+ const challenge = Object.entries(headers ?? {}).find(([name]) => name.toLowerCase() === "www-authenticate")?.[1];
19
+ if (!challenge)
20
+ return undefined;
21
+ const parts = challenge.split(/,(?=(?:[^"]*"[^"]*")*[^"]*$)/);
22
+ let bearer = false;
23
+ for (const part of parts) {
24
+ const trimmed = part.trim();
25
+ const scheme = /^([A-Za-z][A-Za-z0-9_-]*)\s+(?=[A-Za-z])/.exec(trimmed);
26
+ if (scheme)
27
+ bearer = scheme[1]?.toLowerCase() === "bearer";
28
+ if (!bearer)
29
+ continue;
30
+ const metadataUrl = /(?:^|\s)resource_metadata\s*=\s*"([^"]+)"/i.exec(scheme ? trimmed.slice(scheme[0].length) : trimmed)?.[1];
31
+ if (metadataUrl)
32
+ return metadataUrl;
33
+ }
34
+ return undefined;
35
+ }
36
+ /**
37
+ * Check OAuth discovery at the HTTP boundary without assuming product routes,
38
+ * tool names, credentials, tenants, or billing behavior.
39
+ */
40
+ export async function runMcpDiscoveryConformance(adapter, options) {
41
+ const resourceUrl = absoluteHttpUrl(options.resourceUrl);
42
+ if (!resourceUrl) {
43
+ throw new Error("resourceUrl must be an absolute HTTP(S) URL without credentials, query, or fragment");
44
+ }
45
+ const resourcePath = resourceUrl.pathname.replace(/\/$/, "") || "/";
46
+ const metadataPath = `/.well-known/oauth-protected-resource${resourcePath === "/" ? "" : resourcePath}`;
47
+ const expectedMetadataUrl = `${resourceUrl.origin}${metadataPath}`;
48
+ const failures = [];
49
+ const fail = (message) => failures.push({ suite: "mcp-discovery", message });
50
+ const unauthorized = await adapter.request({
51
+ method: "POST",
52
+ path: resourceUrl.pathname,
53
+ headers: { "content-type": "application/json" },
54
+ body: { jsonrpc: "2.0", id: 1, method: "tools/list" },
55
+ });
56
+ if (unauthorized.status !== 401) {
57
+ fail(`anonymous MCP request returned ${unauthorized.status} instead of 401`);
58
+ }
59
+ const challengeUrl = mcpResourceMetadataChallenge(unauthorized.headers);
60
+ if (challengeUrl !== expectedMetadataUrl) {
61
+ fail("WWW-Authenticate must advertise the MCP protected-resource metadata URL");
62
+ }
63
+ const metadata = await adapter.request({ method: "GET", path: metadataPath });
64
+ if (metadata.status !== 200) {
65
+ fail(`protected-resource metadata returned ${metadata.status} instead of 200`);
66
+ return { failures };
67
+ }
68
+ if (!metadata.body ||
69
+ typeof metadata.body !== "object" ||
70
+ Array.isArray(metadata.body)) {
71
+ fail("protected-resource metadata is not a JSON object");
72
+ return { failures };
73
+ }
74
+ const document = metadata.body;
75
+ if (document.resource !== options.resourceUrl) {
76
+ fail("protected-resource metadata does not identify the MCP resource URL");
77
+ }
78
+ if (!Array.isArray(document.authorization_servers) ||
79
+ document.authorization_servers.length === 0 ||
80
+ !document.authorization_servers.every((issuer) => typeof issuer === "string" && Boolean(absoluteHttpUrl(issuer)))) {
81
+ fail("protected-resource metadata must list absolute HTTP(S) authorization server URLs");
82
+ }
83
+ return { failures };
84
+ }
package/dist/run.d.ts CHANGED
@@ -26,6 +26,7 @@ type HttpAdapter = PlatformConformanceAdapter<{
26
26
  }, ReferenceHttpInput, ReferenceMcpInput, {
27
27
  status: number;
28
28
  body: unknown;
29
+ headers?: Record<string, string>;
29
30
  }, {
30
31
  action: string;
31
32
  resourceId: string;
package/dist/run.js CHANGED
@@ -1,3 +1,4 @@
1
+ import { mcpResourceMetadataChallenge } from "./mcp-discovery.js";
1
2
  function errorCode(body) {
2
3
  if (!body || typeof body !== "object")
3
4
  return undefined;
@@ -12,6 +13,23 @@ function errorCode(body) {
12
13
  }
13
14
  return undefined;
14
15
  }
16
+ function hasMcpBearerChallenge(headers) {
17
+ const metadataUrl = mcpResourceMetadataChallenge(headers);
18
+ if (!metadataUrl)
19
+ return false;
20
+ try {
21
+ const url = new URL(metadataUrl);
22
+ return (["http:", "https:"].includes(url.protocol) &&
23
+ !url.username &&
24
+ !url.password &&
25
+ !url.search &&
26
+ !url.hash &&
27
+ url.pathname === "/.well-known/oauth-protected-resource/mcp");
28
+ }
29
+ catch {
30
+ return false;
31
+ }
32
+ }
15
33
  export async function runPlatformConformance(adapter, capabilities = {}) {
16
34
  const failures = [];
17
35
  const fail = (suite, message) => {
@@ -136,6 +154,14 @@ export async function runPlatformConformance(adapter, capabilities = {}) {
136
154
  }
137
155
  }
138
156
  if (capabilities.mcp) {
157
+ const anonymousMcp = await adapter.mcp({
158
+ headers: {},
159
+ tool: "notes.list",
160
+ args: {},
161
+ });
162
+ if (anonymousMcp.status !== 401 || !hasMcpBearerChallenge(anonymousMcp.headers)) {
163
+ fail("mcp", "unauthenticated MCP did not return a protected-resource bearer challenge");
164
+ }
139
165
  const mcpList = await adapter.mcp({
140
166
  headers: {
141
167
  authorization: oauth.authorization,
package/package.json CHANGED
@@ -1,10 +1,19 @@
1
1
  {
2
2
  "name": "@codelitdev/platform-conformance",
3
- "version": "0.1.0-alpha.0",
4
- "description": "Shared Platform composition suites for product adapters",
3
+ "version": "0.2.0",
4
+ "description": "Shared HTTP conformance suites for CodeLit Platform products",
5
5
  "author": "CodeLit",
6
6
  "type": "module",
7
7
  "license": "MIT",
8
+ "repository": {
9
+ "type": "git",
10
+ "url": "git+https://github.com/codelitdev/platform.git",
11
+ "directory": "packages/platform-conformance"
12
+ },
13
+ "homepage": "https://github.com/codelitdev/platform/tree/main/packages/platform-conformance#readme",
14
+ "bugs": {
15
+ "url": "https://github.com/codelitdev/platform/issues"
16
+ },
8
17
  "exports": {
9
18
  ".": {
10
19
  "types": "./dist/index.d.ts",
@@ -13,7 +22,8 @@
13
22
  }
14
23
  },
15
24
  "files": [
16
- "dist"
25
+ "dist",
26
+ "README.md"
17
27
  ],
18
28
  "scripts": {
19
29
  "build": "tsc -p tsconfig.json",
@@ -27,10 +37,10 @@
27
37
  "tag": "latest"
28
38
  },
29
39
  "peerDependencies": {
30
- "@codelitdev/platform": ">=0.1.0-alpha.0"
40
+ "@codelitdev/platform": ">=0.2.0"
31
41
  },
32
42
  "devDependencies": {
33
- "@codelitdev/platform": "0.1.0-alpha.0",
43
+ "@codelitdev/platform": "workspace:*",
34
44
  "@types/bun": "^1.4.1",
35
45
  "@types/node": "^22.14.1",
36
46
  "typescript": "^5.9.3"