@kortyx/usage-telemetry 0.3.0 → 0.5.0

package/CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## [0.5.0](https://github.com/kortyx-io/kortyx/compare/usage-telemetry-v0.4.0...usage-telemetry-v0.5.0) (2026-10-18)
+
+
+### Features
+
+* **cli:** validate topology and update compatible dependencies ([6c16ef3](https://github.com/kortyx-io/kortyx/commit/6c16ef3d1c9e8329d31f1a28e87aa26ae6dbe651))
+* **studio:** add contextual help and organize product documentation ([#366](https://github.com/kortyx-io/kortyx/issues/366)) ([aa4a624](https://github.com/kortyx-io/kortyx/commit/aa4a6242f6294e0f96968a8021440cdc55467121))
+* **studio:** add sample project and guided onboarding ([#363](https://github.com/kortyx-io/kortyx/issues/363)) ([25da0b9](https://github.com/kortyx-io/kortyx/commit/25da0b9c92072446a7503b23ba4f157aebec8185))
+
+## [0.4.0](https://github.com/kortyx-io/kortyx/compare/usage-telemetry-v0.3.0...usage-telemetry-v0.4.0) (2026-10-16)
+
+
+### Features
+
+* **studio:** standardize tables, filters and column behavior ([#353](https://github.com/kortyx-io/kortyx/issues/353)) ([c662e59](https://github.com/kortyx-io/kortyx/commit/c662e59d814a6a9f12c709595968ee7b1ece3191))
+* **telemetry:** improve failure classification and add CLI usage ([#355](https://github.com/kortyx-io/kortyx/issues/355)) ([c5fbbed](https://github.com/kortyx-io/kortyx/commit/c5fbbed3064ea7b2b3020617db7ce09d1b175db3))
+
 ## [0.3.0](https://github.com/kortyx-io/kortyx/compare/usage-telemetry-v0.2.0...usage-telemetry-v0.3.0) (2026-10-13)
 
 

package/README.md

@@ -1,6 +1,6 @@
 # Kortyx usage telemetry
 
-Maintainer usage telemetry measures adoption and reliability of the SDK and OSS Studio. It is separate from customer observability in `@kortyx/telemetry` and `@kortyx/otel`. The default destination is Kortyx's EU PostHog Production project, using its public capture key. Consumers need no personal API token.
+Maintainer usage telemetry measures adoption and reliability of the SDK, CLI and OSS Studio. It is separate from customer observability in `@kortyx/telemetry` and `@kortyx/otel`. The default destination is Kortyx's EU PostHog Production project, using its public capture key. Consumers need no personal API token.
 
 ## Controls
 
@@ -14,6 +14,7 @@ Set these in the SDK process or Studio deployment environment. CLI-generated Com
 | `KORTYX_USAGE_TELEMETRY_POSTHOG_HOST` | Overrides the ingestion host; defaults to `https://eu.i.posthog.com` |
 | `KORTYX_USAGE_TELEMETRY_ENDPOINT` | Replaces PostHog with a collector accepting `{ identity, records }` |
 | `KORTYX_USAGE_TELEMETRY_ENVIRONMENT=staging` | Labels maintainer staging; pair it with that project's capture key |
+| `KORTYX_USAGE_TELEMETRY_TRAFFIC_SOURCE` | Explicit `internal`, `synthetic` or `external` cohort; otherwise `unknown`. CI does not imply synthetic traffic |
 | `KORTYX_USAGE_TELEMETRY_ID` | Optional random UUID for deliberate identity sharing between Node processes |
 
 Tests (`NODE_ENV=test`) disable the default Node integration. Consumer `NODE_ENV` and `CI` supply a coarse runtime mode; they do not select the maintainer project. Browser bootstrap receives only public configuration from Studio's server environment, so no `NEXT_PUBLIC_` variables are needed. Debug output contains approved dimensions and aggregate counts, never application payloads.
@@ -24,13 +25,28 @@ Node delivery lazily stores a random installation UUID in `~/.config/kortyx/usag
 
 The shared catalog allows fixed operation names, outcome enums, Kortyx release versions, provider families, storage/transport kinds, coarse HTTP status, known error codes and booleans. Arbitrary property keys are dropped; unrecognized enum values and error codes become `other`. The error projection reads only own data properties and does not invoke application getters.
 
+Sample chapters use the bounded `studio.sample` chapter/step catalog, including
+optional `error-diagnostics` and `evaluation-progress`. Optional starts/completions
+have their own chapter denominator and do not increase the six-chapter full-tour
+denominator. Completion is a displayed-guide outcome, not proof of learning.
+
+Studio page explanations use `studio.help` with finite `help_page`, `help_step`,
+`help_context` (real/sample), and `help_action` values. Starts count explanation
+attempts; step events count displayed steps, including Back. Completion, skip,
+exit, error, and example request/accept/cancel distinguish local help from a
+confirmed sample transfer. Compare completions against starts within the same
+page/context cohort, accounting for best-effort delivery and abandoned tabs.
+Record identifiers, URLs, selectors, setting values, and page content are excluded.
+Page-help facts are allowed in sample views while ordinary execution metrics remain
+suppressed; they do not measure SDK activation or customer execution outcomes.
+
 Prompts, inputs/outputs, messages, schemas, workflow/node/tool names, URLs, paths, custom application error codes, customer IDs, credentials, raw errors and stacks are excluded. Capture requests omit browser cookies and referrers, disable PostHog person profiles and geolocation enrichment, and send `$ip: null`. Network infrastructure still receives a request's source IP; the payload is not proof of legal anonymity. There is no autocapture, session replay or global exception handler.
 
 Records aggregate matching operations, outcomes and approved dimensions over a 60-second flush window. A scope finishes once. Successes and failures use the same counting policy. `usage.operation` records reliability and duration; `usage.feature` records deliberate adoption or explicit feature configuration. Background reads/refreshes have separate origins and must not be used as click counts.
 
 Each component client buffers at most 64 distinct groups and sends at most 120 groups per hour. A group can represent many attempts through `count`. Duration buckets are noncumulative: ≤10ms, (10,100], (100,1,000], (1,000,10,000], (10,000,60,000], >60,000ms. Query `duration_total_ms / count` for means or combine bucket counts for approximate distributions. Restrict latency queries to `usage.operation`.
 
-Delivery is best effort, single-flight, with a two-second transport timeout, no retries and unreferenced Node timers. SDK operations never await analytics delivery or change results because collection fails. The short-lived Studio setup CLI flushes after recording its outcome, so it can spend up to the transport timeout delivering its final batch. Browser visibility changes request a flush. Offline, disabled, terminated or short-lived processes can lose records; call the helper's `flush()` explicitly at an application-controlled shutdown boundary when appropriate. No disk event queue or crash detection is implied. `stats()` exposes local pending/dropped/delivered group counts. Budgets are per component and process, not an organization-wide spending guarantee; PostHog's $0 billing limits remain the financial control.
+Delivery is best effort, single-flight, with a two-second transport timeout, no retries and unreferenced Node timers. SDK operations never await analytics delivery or change results because collection fails. The CLI flushes registered component clients after command parsing/execution, waiting at most two seconds even for a custom transport. Setup stages and command invocations are separate measurements. Browser visibility changes request a flush. Offline, disabled, terminated or short-lived processes can lose records; call the helper's `flush()` explicitly at an application-controlled shutdown boundary when appropriate. No disk event queue or crash detection is implied. `stats()` exposes local pending/dropped/delivered group counts. Budgets are per component and process, not an organization-wide spending guarantee; PostHog's $0 billing limits remain the financial control.
 
 ## Reusable boundaries
 
@@ -78,14 +94,120 @@ aggregation and failure-independent delivery apply unchanged.
 
 SDK boundaries include initialization, registry resolution, direct execution/resume, chat admission, terminal graph execution, response completion, model and tool execution, structured output, explicit response completion, prompt resolution, eval outcomes, MCP connection/discovery/close, checkpoint operations and observability export attempts. Early response completion and later graph failure are separate outcomes. An eval quality failure is distinct from an execution error.
 
-Studio boundaries include CLI setup, API requests/ingest, server reads and parse failures, first-use guidance and observed data, navigation, tab/trace inspection, render failures/retries, prompt actions, eval launch/read/cancel and live refresh/reconnect. Prompt and eval browser actions measure request acceptance; they do not imply background worker completion. Some preflight errors before a selected boundary remain outside this slice.
+Studio boundaries include CLI setup, API requests/ingest, server reads and parse failures, first-use guidance and observed data, navigation, tab/trace inspection, render failures/retries, prompt actions, eval launch/read/cancel and live refresh/reconnect. Prompt and eval browser actions measure request acceptance. The built-in eval worker separately measures persisted terminal results; its attempt and legacy run units are described below. SDK eval configuration rejection is included with phase `configuration`.
 
-The [opportunity inventory](../../docs/audits/sdk-studio-usage-telemetry.md) covers remaining node/parallel/child execution, prompt cache/refresh, persistence internals, exporter queue loss, deeper filters/saved views, worker completion, reviews and upgrade phases. Reserved catalog operations are not evidence that these boundaries already emit. Feature-flag evaluation and exposure are a future addition.
+The [opportunity inventory](../../docs/audits/sdk-studio-usage-telemetry.md) covers remaining node/parallel/child execution, prompt cache/refresh, persistence internals, exporter queue loss, deeper filters/saved views, deeper worker phases, reviews and upgrade phases. Reserved catalog operations are not evidence that these boundaries already emit. Feature-flag evaluation and exposure are a future addition.
 
 Use `sum(properties.count)`, rather than raw PostHog event count, to measure attempts. For a comparable component/operation cohort, failure rate is `sum(count where outcome = failed) / sum(count for completed operation outcomes)`. Decide explicitly whether cancellations, denial and suspension belong in the denominator. Compare installation reach, repeated use and failure rate by release; do not rank features solely by workload volume. PostHog funnels over aggregates give coarse observed ordering (flush windows can obscure exact ordering), not exact user-session sequences or time since npm installation.
 
+## Measurement quality (version 2)
+
+The envelope remains `schema_version: 1`; additive approved properties carry
+`measurement_version: "2"`. Historical events without it retain their old coverage.
+Compare matching measurement versions, release, component, operation, origin,
+action, analytics environment and runtime mode. Never add up all SDK operations
+as workflow executions.
+
+| Unit (`measurement_kind`) | Counting boundary |
+| --- | --- |
+| `invocation` | One CLI command parse/action outcome or SDK eval `run()` call, including rejected configuration |
+| `execution` | SDK terminal graph execution; persisted legacy worker suite run |
+| `attempt` | Each model/export send attempt; each claimed eval worker case/repetition attempt |
+| `chunk`, `output`, `update` | Structured text deltas, final output, other updates; workload volume, not feature adoption |
+| `stage` | A Studio setup phase, separate from CLI command completion |
+| `poll` | Background Studio reads, including diagnostic prefetch; not deliberate inspection |
+| `request`, `call` | Studio request or other SDK API call outcome |
+| `feature`, `observation` | Deliberate/configured use versus server/background observation |
+
+`runtime_mode` uses Node `CI` / `NODE_ENV`, independently of the destination
+project. Node factories and Studio browser bootstrap carry the same safe context.
+`traffic_source` is explicit: `unknown` is not evidence of external customer use.
+Pseudonymous installation/browser/process reach is not a count of customers.
+
+Failures always have an approved `error_code` and `error_category`. Unclassified
+explicit failures use `UNCLASSIFIED_FAILURE`; inspected unknown errors use `other`.
+The classifier reads at most eight objects, three links deep, following own data
+fields `failure`, `cause`, and `error`. Known network codes, HTTP classes, framework
+failure descriptors and optional bounded `cause_code`/`cause_category`/`retryable`
+help distinguish provider, transport, validation and configuration problems. Eval
+failures also project the first issue's approved `eval_phase`; multiple issues in
+one run are not a complete failure census.
+Messages, stacks, diagnostic IDs and exception serialization remain excluded.
+Cancellation, suspension, limits, eval quality failures and ungraded outcomes stay
+separate from technical failures. A known category is not proof of a product bug.
+
+Exporter records add bucketed `batch_size`, `retry_attempt`, `retried` and
+`recovered`. Recovery means a successful send after a retry, not a correlated
+end-to-end delivery guarantee. Recursive split sends are separate attempts.
+
+Each successful normal delivery may carry `usage_dropped_observations` and
+`usage_failed_batches` once, as deltas since the last delivered health snapshot.
+They include cardinality/budget losses and identity/transport failures; local
+`stats().dropped` still counts groups. Health never bypasses budgets, creates its
+own event, or persists on disk. Permanently offline, crashed, disabled or restarted
+processes cannot report their lost tail. These counters expose observed loss;
+they cannot reconstruct unbiased feature shares after selective group dropping.
+A failed or ambiguous delivery may have reached ingestion: these counters describe
+client-observed loss, not verified server loss.
+
+## Eval and diagnostic boundaries
+
+Eval launch/settings requests record only booleans for explicit settings and
+case/run deadline overrides. The SDK `sdk.eval` boundary reports configuration
+rejections and safe execution failure codes. Neither timeout values nor judge,
+suite, case, input, criteria or output identities are usage dimensions.
+
+The built-in worker emits `studio.eval.execution` only **after** `store.finish`
+persists a result. A claimed case/repetition uses `attempt`; a legacy suite run
+uses `execution`. Cancellation, technical failure, quality failure, ungraded and
+success are separate. Lease loss, process shutdown and a failed persistence write
+do not imply completion. Retries/reclaims may execute again; no global deduplication
+or evaluation-level completion funnel is promised.
+
+A custom `EvalJobStore` keeps instrumentation ownership by default. Embedders may
+supply `EvalWorkerOptions.usage` to opt in, or `false` to disable worker emission.
+Choose one owner: either the existing store wrapper or this injected helper.
+Use the same replacement catalog in its helper. Existing custom store adapters do
+not acquire a second public emitter when upgraded. The dispatcher also accepts
+`usage` / `false`; the worker forwards the chosen client or disables dispatch
+collection for a custom store. Retry observations use `origin=server` and are
+separate from terminal results. Standalone dispatcher defaults use component
+`eval_dispatch`, correcting the formerly invalid component spelling.
+
+Studio emits a `studio.diagnostic` feature on deliberate error-detail inspection,
+and separately records background diagnostic reads and their failures. Shared table controls emit bounded
+`studio.table.layout` interaction actions, including resize initiation, without
+column names, sizes or saved layout content. Neither
+path sends the displayed error content. Customer-authenticated diagnostic storage
+continues to serve original errors independently.
+
 ## Proposal: replace direct capture with a Kortyx collector
 
 Keep direct EU capture for this slice. A later collector can accept the existing versioned record envelope, validate the catalog again, cap body/group/count sizes, rate-limit installations and total daily ingestion, deduplicate `event_id`, strip network metadata, and route approved releases to the intended PostHog project. Keep retention and operational logs bounded and avoid logging request bodies. Forward asynchronously with its own retry/deduplication policy.
 
 This moves abuse controls and global budgets to an owned boundary: a public PostHog capture key can be copied and spoofed, so client caps alone cannot protect ingestion or establish authenticity. Replace the transport or set `KORTYX_USAGE_TELEMETRY_ENDPOINT`; product instrumentation does not change. Endpoint deployment, global limits and project routing belong to that follow-up. Existing released SDK versions continue direct capture until upgraded, so migration must account for them.
+
+### Studio contextual help
+
+`studio.help` counts feature interactions using `help_action: open | documentation`
+and `help_placement: card | button`. Count closed-to-open card transitions once;
+count documentation navigation intent separately. Compare summed card-footer
+navigation counts with summed card-open counts to assess help use. Standalone
+button clicks are a separate population, and these aggregates do not establish
+unique users, successful page loads, or completed reading. Existing opt-out,
+budgets, aggregation and best-effort delivery apply. No help labels, descriptions,
+URLs, customer identifiers or execution content are collected. This measures
+maintainer product usage separately from customer execution observability.
+
+## Topology publication measurement
+
+`cli.command` counts each `topology.push` invocation once, including dry runs.
+The allowlisted `topology_strict` boolean records whether optional strict discovery
+was selected. A blocked publication reports `KTX_TOPOLOGY_INCOMPLETE`, validation
+category and `retryable: false`; the CLI's existing terminal outcome and opt-out,
+bounded delivery and exporter-failure behavior apply. Compare `sum(count)` for
+strict invocations with all topology-push invocations to measure adoption, and
+blocked strict invocations with all strict invocations to measure rejection rate.
+Do not interpret a successful invocation as complete discovery when strict mode
+is off or intentional dynamic calls are exempt. Paths, expressions, import traces,
+credentials and diagnostic messages are never usage properties.

package/dist/browser-node.d.ts

@@ -1,16 +1,24 @@
-import { U as UsageTelemetry, a as UsageOptions } from './client-rVwo4XAO.js';
-export { c as createUsageTelemetry, s as scale } from './client-rVwo4XAO.js';
+import { U as UsageTelemetry, a as UsageOptions } from './client-BPjAx5oh.js';
+export { c as createUsageTelemetry, s as scale } from './client-BPjAx5oh.js';
 
+/** Flush registered component replacements without adding another emitter. */
+declare function flushUsageTelemetry(): Promise<void>;
 declare function createNodeUsageTelemetry(options: Pick<UsageOptions, "context"> & Partial<Omit<UsageOptions, "context">>): UsageTelemetry;
 declare function configureUsageTelemetry(component: string, client: UsageTelemetry): void;
 declare function getUsageTelemetry(component: string, version?: string, product?: string): UsageTelemetry;
 declare function usageDisabled(): boolean;
+declare function runtimeUsageContext(): {
+    runtimeMode: "unknown";
+    trafficSource: "unknown";
+};
 declare function publicUsageConfiguration(): {
     enabled: boolean;
+    runtimeMode: "unknown";
+    trafficSource: "unknown";
     debug: boolean;
     environment: "production";
     key: string;
     host: string;
 };
 
-export { UsageTelemetry, configureUsageTelemetry, createNodeUsageTelemetry, getUsageTelemetry, publicUsageConfiguration, usageDisabled };
+export { UsageTelemetry, configureUsageTelemetry, createNodeUsageTelemetry, flushUsageTelemetry, getUsageTelemetry, publicUsageConfiguration, runtimeUsageContext, usageDisabled };