testerkit 0.4.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- testerkit/__init__.py +68 -0
- testerkit/_docs/_assets/operator-ui/channels/table.png +0 -0
- testerkit/_docs/_assets/operator-ui/dashboard/runs.png +0 -0
- testerkit/_docs/_assets/operator-ui/dashboard/stations.png +0 -0
- testerkit/_docs/_assets/operator-ui/events/filters.png +0 -0
- testerkit/_docs/_assets/operator-ui/events/table.png +0 -0
- testerkit/_docs/_assets/operator-ui/explore/chart.png +0 -0
- testerkit/_docs/_assets/operator-ui/explore/filters.png +0 -0
- testerkit/_docs/_assets/operator-ui/explore/plot-controls.png +0 -0
- testerkit/_docs/_assets/operator-ui/files/table.png +0 -0
- testerkit/_docs/_assets/operator-ui/launch/form.png +0 -0
- testerkit/_docs/_assets/operator-ui/metrics/filters.png +0 -0
- testerkit/_docs/_assets/operator-ui/metrics/pareto.png +0 -0
- testerkit/_docs/_assets/operator-ui/metrics/ppk.png +0 -0
- testerkit/_docs/_assets/operator-ui/metrics/yield.png +0 -0
- testerkit/_docs/_assets/operator-ui/results/detail-header.png +0 -0
- testerkit/_docs/_assets/operator-ui/results/detail-measurements.png +0 -0
- testerkit/_docs/_assets/operator-ui/results/detail-overview.png +0 -0
- testerkit/_docs/_assets/operator-ui/results/detail-steps.png +0 -0
- testerkit/_docs/_assets/operator-ui/results/stats.png +0 -0
- testerkit/_docs/_assets/operator-ui/results/table.png +0 -0
- testerkit/_docs/_assets/operator-ui/tour/designer.png +0 -0
- testerkit/_docs/_assets/operator-ui/tour/docs.png +0 -0
- testerkit/_docs/_assets/operator-ui/tour/fixtures.png +0 -0
- testerkit/_docs/_assets/operator-ui/tour/instruments.png +0 -0
- testerkit/_docs/_assets/operator-ui/tour/parts.png +0 -0
- testerkit/_docs/_assets/operator-ui/tour/profiles.png +0 -0
- testerkit/_docs/_assets/operator-ui/tour/stations.png +0 -0
- testerkit/_docs/_assets/operator-ui/tour/tests.png +0 -0
- testerkit/_docs/_assets/operator-ui/tour/uuts.png +0 -0
- testerkit/_docs/concepts/configuration/capabilities.md +447 -0
- testerkit/_docs/concepts/configuration/fixtures.md +335 -0
- testerkit/_docs/concepts/configuration/index.md +14 -0
- testerkit/_docs/concepts/configuration/parts.md +276 -0
- testerkit/_docs/concepts/configuration/stations.md +214 -0
- testerkit/_docs/concepts/data/data-stores.md +199 -0
- testerkit/_docs/concepts/data/event-log.md +185 -0
- testerkit/_docs/concepts/data/event-sourcing.md +76 -0
- testerkit/_docs/concepts/data/flight-streaming.md +58 -0
- testerkit/_docs/concepts/data/index.md +17 -0
- testerkit/_docs/concepts/data/sessions.md +78 -0
- testerkit/_docs/concepts/data/three-verbs.md +260 -0
- testerkit/_docs/concepts/execution/index.md +13 -0
- testerkit/_docs/concepts/execution/outcomes.md +199 -0
- testerkit/_docs/concepts/execution/step-hierarchy.md +170 -0
- testerkit/_docs/concepts/execution/step-manifest.md +115 -0
- testerkit/_docs/concepts/index.md +40 -0
- testerkit/_docs/concepts/overview/ai-integration.md +57 -0
- testerkit/_docs/concepts/overview/architecture.md +357 -0
- testerkit/_docs/concepts/overview/index.md +14 -0
- testerkit/_docs/concepts/overview/platform-vs-framework.md +147 -0
- testerkit/_docs/concepts/overview/pytest.md +53 -0
- testerkit/_docs/concepts/overview/tiers.md +46 -0
- testerkit/_docs/how-to/catalog/datasheet-to-test.md +169 -0
- testerkit/_docs/how-to/catalog/index.md +11 -0
- testerkit/_docs/how-to/configuration/configuring-stations.md +364 -0
- testerkit/_docs/how-to/configuration/custom-drivers.md +484 -0
- testerkit/_docs/how-to/configuration/index.md +13 -0
- testerkit/_docs/how-to/configuration/mock-mode.md +342 -0
- testerkit/_docs/how-to/data/benchmarking.md +120 -0
- testerkit/_docs/how-to/data/capture-an-artifact.md +84 -0
- testerkit/_docs/how-to/data/capture-waveform.md +110 -0
- testerkit/_docs/how-to/data/choosing-a-channel-verb.md +108 -0
- testerkit/_docs/how-to/data/compare-runs.md +120 -0
- testerkit/_docs/how-to/data/export-results.md +104 -0
- testerkit/_docs/how-to/data/find-flaky-tests.md +108 -0
- testerkit/_docs/how-to/data/grafana-dashboards.md +148 -0
- testerkit/_docs/how-to/data/index.md +23 -0
- testerkit/_docs/how-to/data/mcp-debug-failures.md +125 -0
- testerkit/_docs/how-to/data/mcp-query-runs.md +120 -0
- testerkit/_docs/how-to/data/querying-channels.md +117 -0
- testerkit/_docs/how-to/data/querying-events.md +85 -0
- testerkit/_docs/how-to/data/stream-live-channel.md +134 -0
- testerkit/_docs/how-to/execution/custom-operator-ui.md +136 -0
- testerkit/_docs/how-to/execution/index.md +21 -0
- testerkit/_docs/how-to/execution/limits.md +193 -0
- testerkit/_docs/how-to/execution/managing-sessions.md +100 -0
- testerkit/_docs/how-to/execution/multi-uut-testing.md +211 -0
- testerkit/_docs/how-to/execution/operator-prompts.md +151 -0
- testerkit/_docs/how-to/execution/profiles.md +304 -0
- testerkit/_docs/how-to/execution/spec-driven-testing.md +169 -0
- testerkit/_docs/how-to/execution/test-context.md +212 -0
- testerkit/_docs/how-to/execution/traceability.md +252 -0
- testerkit/_docs/how-to/execution/vector-expansion.md +394 -0
- testerkit/_docs/how-to/execution/writing-tests.md +363 -0
- testerkit/_docs/how-to/index.md +59 -0
- testerkit/_docs/how-to/overview/index.md +11 -0
- testerkit/_docs/how-to/overview/mcp-integration.md +168 -0
- testerkit/_docs/how-to/overview/operator-ui-tour.md +234 -0
- testerkit/_docs/integration/data/grafana.md +162 -0
- testerkit/_docs/integration/data/index.md +15 -0
- testerkit/_docs/integration/data/lakehouse-import.md +226 -0
- testerkit/_docs/integration/data/logging.md +131 -0
- testerkit/_docs/integration/data/results-api.md +51 -0
- testerkit/_docs/integration/index.md +20 -0
- testerkit/_docs/integration/runtime/harness.md +281 -0
- testerkit/_docs/integration/runtime/index.md +13 -0
- testerkit/_docs/integration/runtime/instruments.md +284 -0
- testerkit/_docs/integration/runtime/pytest-existing.md +315 -0
- testerkit/_docs/reference/catalog/cookbook.md +389 -0
- testerkit/_docs/reference/catalog/index.md +11 -0
- testerkit/_docs/reference/catalog/schema.md +375 -0
- testerkit/_docs/reference/cli.md +660 -0
- testerkit/_docs/reference/configuration.md +406 -0
- testerkit/_docs/reference/data/channels-schema.md +145 -0
- testerkit/_docs/reference/data/event-types.md +554 -0
- testerkit/_docs/reference/data/events-schema.md +120 -0
- testerkit/_docs/reference/data/files-schema.md +141 -0
- testerkit/_docs/reference/data/index.md +18 -0
- testerkit/_docs/reference/data/models.md +2229 -0
- testerkit/_docs/reference/data/outputs.md +82 -0
- testerkit/_docs/reference/data/parquet-schema.md +529 -0
- testerkit/_docs/reference/data/performance-limits.md +224 -0
- testerkit/_docs/reference/data/query-api.md +247 -0
- testerkit/_docs/reference/index.md +69 -0
- testerkit/_docs/reference/operator-ui/channels/detail.md +82 -0
- testerkit/_docs/reference/operator-ui/channels/list.md +70 -0
- testerkit/_docs/reference/operator-ui/dashboard.md +87 -0
- testerkit/_docs/reference/operator-ui/designer.md +167 -0
- testerkit/_docs/reference/operator-ui/events.md +89 -0
- testerkit/_docs/reference/operator-ui/files.md +74 -0
- testerkit/_docs/reference/operator-ui/fixtures.md +100 -0
- testerkit/_docs/reference/operator-ui/instruments.md +126 -0
- testerkit/_docs/reference/operator-ui/launch.md +79 -0
- testerkit/_docs/reference/operator-ui/live.md +102 -0
- testerkit/_docs/reference/operator-ui/measurements.md +159 -0
- testerkit/_docs/reference/operator-ui/metrics.md +217 -0
- testerkit/_docs/reference/operator-ui/parts.md +98 -0
- testerkit/_docs/reference/operator-ui/profiles.md +103 -0
- testerkit/_docs/reference/operator-ui/results/detail.md +214 -0
- testerkit/_docs/reference/operator-ui/results/list.md +106 -0
- testerkit/_docs/reference/operator-ui/stations.md +98 -0
- testerkit/_docs/reference/operator-ui/tests.md +173 -0
- testerkit/_docs/reference/operator-ui/uuts.md +67 -0
- testerkit/_docs/reference/overview/index.md +11 -0
- testerkit/_docs/reference/overview/pytest-native.md +142 -0
- testerkit/_docs/reference/overview/skills.md +177 -0
- testerkit/_docs/reference/pytest/fixtures.md +342 -0
- testerkit/_docs/reference/pytest/index.md +12 -0
- testerkit/_docs/reference/pytest/markers.md +257 -0
- testerkit/_docs/reference/runtime/api.md +246 -0
- testerkit/_docs/reference/runtime/client.md +280 -0
- testerkit/_docs/reference/runtime/connect.md +140 -0
- testerkit/_docs/reference/runtime/index.md +13 -0
- testerkit/_docs/tutorial/01-first-test.md +113 -0
- testerkit/_docs/tutorial/02-mock-instruments.md +88 -0
- testerkit/_docs/tutorial/03-fixtures.md +169 -0
- testerkit/_docs/tutorial/04-limits.md +139 -0
- testerkit/_docs/tutorial/05-configuration.md +194 -0
- testerkit/_docs/tutorial/06-specifications.md +273 -0
- testerkit/_docs/tutorial/07-real-instruments.md +245 -0
- testerkit/_docs/tutorial/08-capabilities.md +223 -0
- testerkit/_docs/tutorial/09-production.md +374 -0
- testerkit/_docs/tutorial/10-live-monitoring.md +156 -0
- testerkit/_docs/tutorial/11-waveforms-and-evidence.md +169 -0
- testerkit/_docs/tutorial/12-continuous-monitoring.md +145 -0
- testerkit/_docs/tutorial/13-parallel-testing.md +233 -0
- testerkit/_docs/tutorial/index.md +61 -0
- testerkit/_docs/tutorial/quickstart.md +282 -0
- testerkit/analysis/__init__.py +9 -0
- testerkit/analysis/_common.py +24 -0
- testerkit/analysis/measurement_facets.py +524 -0
- testerkit/analysis/measurements_query.py +1458 -0
- testerkit/analysis/metrics.py +389 -0
- testerkit/analysis/runs_query.py +434 -0
- testerkit/analysis/steps_query.py +413 -0
- testerkit/api/__init__.py +6 -0
- testerkit/api/_mime.py +71 -0
- testerkit/api/app.py +1171 -0
- testerkit/api/dialogs/__init__.py +36 -0
- testerkit/api/dialogs/manager.py +617 -0
- testerkit/api/dialogs/models.py +79 -0
- testerkit/api/models.py +80 -0
- testerkit/api/responses.py +205 -0
- testerkit/api/runner.py +222 -0
- testerkit/api/schemas.py +311 -0
- testerkit/benchmark/__init__.py +34 -0
- testerkit/benchmark/concurrency.py +248 -0
- testerkit/benchmark/core.py +232 -0
- testerkit/benchmark/runner.py +628 -0
- testerkit/benchmark/scenario.py +313 -0
- testerkit/benchmark/system.py +191 -0
- testerkit/benchmark/workloads.py +507 -0
- testerkit/catalog/__init__.py +6 -0
- testerkit/catalog/generic/generic_dmm.yaml +61 -0
- testerkit/catalog/generic/generic_eload.yaml +31 -0
- testerkit/catalog/generic/generic_fgen.yaml +56 -0
- testerkit/catalog/generic/generic_oscilloscope.yaml +45 -0
- testerkit/catalog/generic/generic_psu.yaml +77 -0
- testerkit/catalog/generic/generic_smu.yaml +73 -0
- testerkit/channels.py +538 -0
- testerkit/cli/__init__.py +32 -0
- testerkit/cli/_common.py +113 -0
- testerkit/cli/_time.py +179 -0
- testerkit/cli/benchmark_cmd.py +56 -0
- testerkit/cli/catalog_cmd.py +39 -0
- testerkit/cli/daemon.py +157 -0
- testerkit/cli/data_cmd.py +1040 -0
- testerkit/cli/discover_cmd.py +94 -0
- testerkit/cli/docs_cmd.py +100 -0
- testerkit/cli/instrument.py +172 -0
- testerkit/cli/mcp_cmd.py +37 -0
- testerkit/cli/metrics.py +366 -0
- testerkit/cli/project.py +312 -0
- testerkit/cli/root.py +14 -0
- testerkit/cli/runs.py +412 -0
- testerkit/cli/schema_cmd.py +87 -0
- testerkit/cli/serve_cmd.py +50 -0
- testerkit/cli/setup_cmd.py +604 -0
- testerkit/cli/station.py +294 -0
- testerkit/cli/validate.py +112 -0
- testerkit/client.py +463 -0
- testerkit/connect.py +635 -0
- testerkit/data/__init__.py +8 -0
- testerkit/data/_accumulator_pool.py +455 -0
- testerkit/data/_atomic.py +59 -0
- testerkit/data/_collection_indices.py +67 -0
- testerkit/data/_daemon_lifecycle.py +535 -0
- testerkit/data/_duckdb_daemon.py +695 -0
- testerkit/data/_duckdb_flight_server.py +813 -0
- testerkit/data/_event_filters.py +15 -0
- testerkit/data/_event_reader.py +101 -0
- testerkit/data/_flight_errors.py +199 -0
- testerkit/data/_flight_query.py +202 -0
- testerkit/data/_flight_retry.py +148 -0
- testerkit/data/_flight_subscribe.py +67 -0
- testerkit/data/_index_epoch.py +342 -0
- testerkit/data/_ipc_writer.py +185 -0
- testerkit/data/_json_safe.py +63 -0
- testerkit/data/_process.py +18 -0
- testerkit/data/_push_relay.py +124 -0
- testerkit/data/_runs_duckdb_daemon.py +2704 -0
- testerkit/data/_session_reaper.py +110 -0
- testerkit/data/_sql_helpers.py +30 -0
- testerkit/data/_store.py +24 -0
- testerkit/data/backends/__init__.py +7 -0
- testerkit/data/backends/_event_accumulator.py +957 -0
- testerkit/data/backends/_row_helpers.py +1172 -0
- testerkit/data/backends/parquet.py +1143 -0
- testerkit/data/backends/protocol.py +35 -0
- testerkit/data/channels/__init__.py +8 -0
- testerkit/data/channels/_flight_daemon.py +75 -0
- testerkit/data/channels/client.py +242 -0
- testerkit/data/channels/flight_manager.py +88 -0
- testerkit/data/channels/index.py +664 -0
- testerkit/data/channels/models.py +418 -0
- testerkit/data/channels/server.py +162 -0
- testerkit/data/channels/store.py +1481 -0
- testerkit/data/data_dir.py +61 -0
- testerkit/data/duckdb_manager.py +79 -0
- testerkit/data/event_log.py +384 -0
- testerkit/data/event_store.py +768 -0
- testerkit/data/events.py +1025 -0
- testerkit/data/exporters/__init__.py +6 -0
- testerkit/data/exporters/_helpers.py +40 -0
- testerkit/data/exporters/csv_exporter.py +168 -0
- testerkit/data/exporters/hdf5.py +234 -0
- testerkit/data/exporters/json_exporter.py +229 -0
- testerkit/data/exporters/mdf4.py +201 -0
- testerkit/data/exporters/stdf.py +255 -0
- testerkit/data/exporters/tdms.py +292 -0
- testerkit/data/files/__init__.py +71 -0
- testerkit/data/files/_backend.py +207 -0
- testerkit/data/files/_catalog_daemon.py +84 -0
- testerkit/data/files/catalog.py +218 -0
- testerkit/data/files/catalog_manager.py +325 -0
- testerkit/data/files/models.py +60 -0
- testerkit/data/files/serializers.py +414 -0
- testerkit/data/files/store.py +463 -0
- testerkit/data/files/streaming.py +783 -0
- testerkit/data/models.py +604 -0
- testerkit/data/ref.py +152 -0
- testerkit/data/retention.py +297 -0
- testerkit/data/run_store.py +367 -0
- testerkit/data/runs_duckdb_manager.py +103 -0
- testerkit/data/schema_dispatch.py +191 -0
- testerkit/data/schema_migrate.py +76 -0
- testerkit/data/schema_versions.py +85 -0
- testerkit/data/schemas.py +288 -0
- testerkit/data/subscribers/__init__.py +48 -0
- testerkit/data/subscribers/_base.py +20 -0
- testerkit/data/subscribers/_output_file.py +25 -0
- testerkit/data/subscribers/replay.py +55 -0
- testerkit/environment.py +84 -0
- testerkit/execution/__init__.py +11 -0
- testerkit/execution/_git.py +190 -0
- testerkit/execution/_state.py +740 -0
- testerkit/execution/accessors.py +54 -0
- testerkit/execution/audit.py +51 -0
- testerkit/execution/cascade.py +85 -0
- testerkit/execution/connections.py +491 -0
- testerkit/execution/harness.py +1931 -0
- testerkit/execution/instrument_events.py +42 -0
- testerkit/execution/limits.py +154 -0
- testerkit/execution/metadata.py +102 -0
- testerkit/execution/mocks.py +63 -0
- testerkit/execution/profiles.py +616 -0
- testerkit/execution/run_scope.py +1331 -0
- testerkit/execution/session_scope.py +196 -0
- testerkit/execution/sidecar.py +280 -0
- testerkit/execution/site_runner.py +705 -0
- testerkit/execution/sites.py +183 -0
- testerkit/execution/sync.py +294 -0
- testerkit/execution/uut_provider.py +231 -0
- testerkit/execution/vectors.py +205 -0
- testerkit/execution/verify.py +396 -0
- testerkit/expand.py +75 -0
- testerkit/files.py +235 -0
- testerkit/fixtures/__init__.py +6 -0
- testerkit/fixtures/manager.py +357 -0
- testerkit/grafana/__init__.py +1 -0
- testerkit/grafana/bootstrap.py +61 -0
- testerkit/grafana/cli.py +314 -0
- testerkit/grafana/dashboards/asset_utilization.json +92 -0
- testerkit/grafana/dashboards/channel_explorer.json +77 -0
- testerkit/grafana/dashboards/event_log.json +92 -0
- testerkit/grafana/dashboards/failure_pareto.json +104 -0
- testerkit/grafana/dashboards/measurement_distribution.json +97 -0
- testerkit/grafana/dashboards/measurement_trend.json +161 -0
- testerkit/grafana/dashboards/station_comparison.json +95 -0
- testerkit/grafana/dashboards/test_duration.json +104 -0
- testerkit/grafana/dashboards/unit_traceability.json +73 -0
- testerkit/grafana/dashboards/yield_overview.json +189 -0
- testerkit/grafana/provisioning/dashboards.yaml.j2 +12 -0
- testerkit/grafana/provisioning/datasources.yaml.j2 +17 -0
- testerkit/grafana/server.py +282 -0
- testerkit/init.py +724 -0
- testerkit/instruments/__init__.py +10 -0
- testerkit/instruments/base.py +82 -0
- testerkit/instruments/discovery/__init__.py +62 -0
- testerkit/instruments/discovery/_base.py +174 -0
- testerkit/instruments/discovery/lxi.py +152 -0
- testerkit/instruments/discovery/ni.py +95 -0
- testerkit/instruments/discovery/serial.py +73 -0
- testerkit/instruments/discovery/visa.py +98 -0
- testerkit/instruments/lifecycle.py +213 -0
- testerkit/instruments/loader.py +79 -0
- testerkit/instruments/locks.py +178 -0
- testerkit/instruments/mocks.py +230 -0
- testerkit/instruments/observer.py +270 -0
- testerkit/instruments/observers/__init__.py +36 -0
- testerkit/instruments/observers/_base.py +68 -0
- testerkit/instruments/observers/daqmx.py +70 -0
- testerkit/instruments/observers/descriptor.py +69 -0
- testerkit/instruments/observers/generic.py +92 -0
- testerkit/instruments/observers/lantz.py +31 -0
- testerkit/instruments/observers/modbus.py +96 -0
- testerkit/instruments/observers/motion.py +68 -0
- testerkit/instruments/observers/ni_modular.py +73 -0
- testerkit/instruments/observers/ophyd.py +80 -0
- testerkit/instruments/observers/pymeasure.py +120 -0
- testerkit/instruments/observers/qcodes.py +66 -0
- testerkit/instruments/observers/scpi.py +88 -0
- testerkit/instruments/observers/tektronix.py +57 -0
- testerkit/instruments/observers/visa.py +103 -0
- testerkit/instruments/pool.py +459 -0
- testerkit/instruments/proxy.py +59 -0
- testerkit/instruments/route_manager.py +329 -0
- testerkit/instruments/routed_proxy.py +88 -0
- testerkit/instruments/server.py +508 -0
- testerkit/instruments/switch.py +43 -0
- testerkit/instruments/visa.py +294 -0
- testerkit/matching/__init__.py +6 -0
- testerkit/matching/service.py +969 -0
- testerkit/mcp/__init__.py +6 -0
- testerkit/mcp/server.py +681 -0
- testerkit/mcp/tools.py +1868 -0
- testerkit/models/__init__.py +18 -0
- testerkit/models/capability.py +598 -0
- testerkit/models/catalog.py +76 -0
- testerkit/models/data_options.py +153 -0
- testerkit/models/enums.py +334 -0
- testerkit/models/instrument.py +152 -0
- testerkit/models/instrument_asset.py +25 -0
- testerkit/models/part.py +271 -0
- testerkit/models/part_manifest.py +96 -0
- testerkit/models/project.py +114 -0
- testerkit/models/station.py +140 -0
- testerkit/models/test_config.py +748 -0
- testerkit/ontology/__init__.py +39 -0
- testerkit/ontology/schema.py +188 -0
- testerkit/ontology/testerkit.yaml +1420 -0
- testerkit/parts/__init__.py +8 -0
- testerkit/parts/context.py +181 -0
- testerkit/parts/folder.py +251 -0
- testerkit/parts/loader.py +27 -0
- testerkit/prompts/__init__.py +17 -0
- testerkit/prompts/core.py +133 -0
- testerkit/pytest_plugin/__init__.py +1287 -0
- testerkit/pytest_plugin/autouse.py +369 -0
- testerkit/pytest_plugin/helpers.py +354 -0
- testerkit/pytest_plugin/hooks.py +1982 -0
- testerkit/pytest_plugin/markers.py +203 -0
- testerkit/pytest_plugin/retry.py +37 -0
- testerkit/pytest_plugin/sweeps.py +117 -0
- testerkit/queries.py +78 -0
- testerkit/reports/__init__.py +6 -0
- testerkit/reports/core.py +492 -0
- testerkit/reports/datasheet.py +831 -0
- testerkit/reports/templates/datasheet.html +383 -0
- testerkit/reports/templates/default.html +153 -0
- testerkit/sbom.py +142 -0
- testerkit/schema_export.py +101 -0
- testerkit/signals.py +63 -0
- testerkit/skills/README.md +58 -0
- testerkit/skills/templates/project-instructions.md +125 -0
- testerkit/skills/testerkit-analysis/SKILL.md +87 -0
- testerkit/skills/testerkit-capture/SKILL.md +114 -0
- testerkit/skills/testerkit-data/SKILL.md +138 -0
- testerkit/skills/testerkit-datasheets/SKILL.md +100 -0
- testerkit/skills/testerkit-datasheets/agents/scaffold-writer.md +115 -0
- testerkit/skills/testerkit-datasheets/agents/section-extractor.md +98 -0
- testerkit/skills/testerkit-datasheets/agents/section-reviewer.md +153 -0
- testerkit/skills/testerkit-datasheets/agents/section-splitter.md +89 -0
- testerkit/skills/testerkit-datasheets/agents/section-writer.md +226 -0
- testerkit/skills/testerkit-datasheets/references/catalog-pipeline.md +271 -0
- testerkit/skills/testerkit-datasheets/references/process-queue.md +42 -0
- testerkit/skills/testerkit-datasheets/references/scaffold.md +156 -0
- testerkit/skills/testerkit-datasheets/references/test-pipeline.md +312 -0
- testerkit/skills/testerkit-debug/SKILL.md +127 -0
- testerkit/skills/testerkit-interactive/SKILL.md +188 -0
- testerkit/skills/testerkit-interactive/references/live-ui-patterns.md +102 -0
- testerkit/skills/testerkit-mocks/SKILL.md +128 -0
- testerkit/skills/testerkit-parts/SKILL.md +126 -0
- testerkit/skills/testerkit-profiles/SKILL.md +139 -0
- testerkit/skills/testerkit-sites/SKILL.md +140 -0
- testerkit/skills/testerkit-stations/SKILL.md +146 -0
- testerkit/skills/testerkit-tests/SKILL.md +136 -0
- testerkit/store.py +1639 -0
- testerkit/ui/__init__.py +71 -0
- testerkit/ui/_asgi.py +162 -0
- testerkit/ui/app.py +32 -0
- testerkit/ui/components/__init__.py +13 -0
- testerkit/ui/components/artifact_viewer.py +250 -0
- testerkit/ui/components/channel_values.py +159 -0
- testerkit/ui/components/event_timeline.py +222 -0
- testerkit/ui/components/execution_gantt.py +210 -0
- testerkit/ui/components/file_streams.py +147 -0
- testerkit/ui/components/instrument_activity.py +146 -0
- testerkit/ui/components/session_table.py +118 -0
- testerkit/ui/pages/__init__.py +25 -0
- testerkit/ui/pages/channels/__init__.py +4 -0
- testerkit/ui/pages/channels/detail.py +932 -0
- testerkit/ui/pages/channels/list.py +515 -0
- testerkit/ui/pages/dashboard.py +236 -0
- testerkit/ui/pages/designer/__init__.py +5 -0
- testerkit/ui/pages/designer/graph.py +635 -0
- testerkit/ui/pages/designer/matching.py +663 -0
- testerkit/ui/pages/designer/page.py +593 -0
- testerkit/ui/pages/designer/properties.py +591 -0
- testerkit/ui/pages/designer/state.py +468 -0
- testerkit/ui/pages/docs/__init__.py +6 -0
- testerkit/ui/pages/docs/index.py +95 -0
- testerkit/ui/pages/docs/page.py +698 -0
- testerkit/ui/pages/events/__init__.py +3 -0
- testerkit/ui/pages/events/list.py +263 -0
- testerkit/ui/pages/explore.py +1108 -0
- testerkit/ui/pages/files/__init__.py +4 -0
- testerkit/ui/pages/files/detail.py +426 -0
- testerkit/ui/pages/files/list.py +447 -0
- testerkit/ui/pages/fixtures/__init__.py +8 -0
- testerkit/ui/pages/fixtures/detail.py +280 -0
- testerkit/ui/pages/fixtures/edit.py +370 -0
- testerkit/ui/pages/fixtures/list.py +180 -0
- testerkit/ui/pages/fixtures/new.py +195 -0
- testerkit/ui/pages/instruments/__init__.py +4 -0
- testerkit/ui/pages/instruments/detail.py +320 -0
- testerkit/ui/pages/instruments/edit.py +482 -0
- testerkit/ui/pages/instruments/list.py +249 -0
- testerkit/ui/pages/instruments/new.py +200 -0
- testerkit/ui/pages/launch.py +261 -0
- testerkit/ui/pages/live.py +124 -0
- testerkit/ui/pages/metrics_page.py +1229 -0
- testerkit/ui/pages/parts/__init__.py +11 -0
- testerkit/ui/pages/parts/detail.py +202 -0
- testerkit/ui/pages/parts/edit.py +310 -0
- testerkit/ui/pages/parts/list.py +160 -0
- testerkit/ui/pages/parts/new.py +170 -0
- testerkit/ui/pages/parts/requirements.py +153 -0
- testerkit/ui/pages/parts/stations.py +199 -0
- testerkit/ui/pages/profiles/__init__.py +4 -0
- testerkit/ui/pages/profiles/detail.py +113 -0
- testerkit/ui/pages/profiles/list.py +86 -0
- testerkit/ui/pages/results/__init__.py +4 -0
- testerkit/ui/pages/results/detail.py +631 -0
- testerkit/ui/pages/results/list.py +230 -0
- testerkit/ui/pages/stations/__init__.py +4 -0
- testerkit/ui/pages/stations/detail.py +256 -0
- testerkit/ui/pages/stations/edit.py +273 -0
- testerkit/ui/pages/stations/list.py +180 -0
- testerkit/ui/pages/stations/new.py +311 -0
- testerkit/ui/pages/tests/__init__.py +4 -0
- testerkit/ui/pages/tests/detail.py +152 -0
- testerkit/ui/pages/tests/list.py +194 -0
- testerkit/ui/pages/uuts/__init__.py +4 -0
- testerkit/ui/pages/uuts/list.py +97 -0
- testerkit/ui/shared/__init__.py +8 -0
- testerkit/ui/shared/components.py +1826 -0
- testerkit/ui/shared/dialogs.py +105 -0
- testerkit/ui/shared/event_binding.py +194 -0
- testerkit/ui/shared/layout.py +314 -0
- testerkit/ui/shared/services.py +1719 -0
- testerkit/ui/shared/timestamps.py +21 -0
- testerkit/ui/static/branding/favicon-16.png +0 -0
- testerkit/ui/static/branding/favicon-32.png +0 -0
- testerkit/ui/static/branding/favicon-48.png +0 -0
- testerkit/ui/static/branding/testerkit-icon-256.png +0 -0
- testerkit/ui/static/branding/testerkit-mark.svg +1 -0
- testerkit/ui/static/branding/testerkit-wordmark.svg +1 -0
- testerkit/ui/static/global.css +465 -0
- testerkit/utils/__init__.py +17 -0
- testerkit/utils/enum_meta.py +800 -0
- testerkit/utils/paths.py +75 -0
- testerkit/utils/ranges.py +257 -0
- testerkit/validation.py +82 -0
- testerkit/verbs.py +142 -0
- testerkit-0.4.0.dist-info/METADATA +250 -0
- testerkit-0.4.0.dist-info/RECORD +521 -0
- testerkit-0.4.0.dist-info/WHEEL +4 -0
- testerkit-0.4.0.dist-info/entry_points.txt +5 -0
- testerkit-0.4.0.dist-info/licenses/LICENSE +191 -0
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
# Tests
|
|
2
|
+
|
|
3
|
+
**URLs:** `/tests` (list), `/tests/{path}` (detail)
|
|
4
|
+
|
|
5
|
+
A test in TesterKit is a `def test_*` function that pytest discovers — either a
|
|
6
|
+
top-level function or a method inside a `class Test*`. The Tests pages give you
|
|
7
|
+
a file-level inventory of every `test_*.py` under `tests/`, cross-referenced
|
|
8
|
+
against run history, without importing or executing any test code.
|
|
9
|
+
|
|
10
|
+
## List — `/tests`
|
|
11
|
+
|
|
12
|
+
Files are grouped by directory. Each directory renders a folder-icon header
|
|
13
|
+
followed by one row per `test_*.py` file under it.
|
|
14
|
+
|
|
15
|
+
### File rows
|
|
16
|
+
|
|
17
|
+
Each row shows:
|
|
18
|
+
|
|
19
|
+
| Field | What it shows |
|
|
20
|
+
|---|---|
|
|
21
|
+
| Filename | The `test_*.py` filename, in monospace |
|
|
22
|
+
| Test count | Number of `def test_*` functions found in the file (top-level and inside `class Test*`) |
|
|
23
|
+
| Class count | Number of `class Test*` definitions, shown inline when non-zero |
|
|
24
|
+
| Vectors | A `~N vectors` badge, present when the file has parametrize, `testerkit_sweeps`, or similar list-valued markers. The count is a rough upper-bound estimate from the decorator arguments — not an exact cross-product |
|
|
25
|
+
| Markers | Up to four marker chips (decorator names, without the `pytest.mark.` prefix), then `+N` when there are more |
|
|
26
|
+
| Runs | Count of distinct `run_id` values in history where any step from this file executed; omitted when zero |
|
|
27
|
+
| Failed | Count of those runs with at least one failed step; omitted when there are no runs or no failures |
|
|
28
|
+
| Sidecar | A `sidecar` badge when a `.yaml` file with the same stem exists alongside the `.py`; a dash otherwise |
|
|
29
|
+
|
|
30
|
+
Clicking any row navigates to `/tests/{path}` for that file.
|
|
31
|
+
|
|
32
|
+
When a file cannot be parsed (syntax error or I/O problem), the row renders in
|
|
33
|
+
red with a warning icon and a truncated error message. The file is still
|
|
34
|
+
clickable — the detail page shows the full parse error.
|
|
35
|
+
|
|
36
|
+
### Badge
|
|
37
|
+
|
|
38
|
+
A `N files` badge in the page header shows the total number of `test_*.py`
|
|
39
|
+
files found.
|
|
40
|
+
|
|
41
|
+
### Observed in history (no matching source)
|
|
42
|
+
|
|
43
|
+
Below the file list, when run history contains step paths that do not match
|
|
44
|
+
any function in the current source, an **Observed in history (no matching
|
|
45
|
+
source)** section appears. This surfaces tests that were renamed, deleted, or
|
|
46
|
+
run from a code state that isn't checked out. The section shows the count of
|
|
47
|
+
orphaned paths as a warning badge.
|
|
48
|
+
|
|
49
|
+
The orphan table has these columns:
|
|
50
|
+
|
|
51
|
+
| Column | What it shows |
|
|
52
|
+
|---|---|
|
|
53
|
+
| Step Path | The `step_path` value from the steps table — bare function name or `Class/method` |
|
|
54
|
+
| Runs | Distinct run count |
|
|
55
|
+
| Passed | Runs with outcome `passed` |
|
|
56
|
+
| Failed | Runs with outcome `failed` |
|
|
57
|
+
| Last Run | Most recent step start timestamp, browser-local time |
|
|
58
|
+
|
|
59
|
+
Rows are sorted by Last Run descending so the most recently orphaned paths appear first.
|
|
60
|
+
|
|
61
|
+
### Empty state
|
|
62
|
+
|
|
63
|
+
When no `test_*.py` files are found and there are no orphan step paths, the
|
|
64
|
+
file list is replaced with a card:
|
|
65
|
+
|
|
66
|
+
> No test files found.
|
|
67
|
+
>
|
|
68
|
+
> Add test_*.py files under tests/.
|
|
69
|
+
|
|
70
|
+
### Data-testid anchors
|
|
71
|
+
|
|
72
|
+
| Element | `data-testid` |
|
|
73
|
+
|---|---|
|
|
74
|
+
| Directory-grouped file list container | `tests-table` |
|
|
75
|
+
| Orphaned step paths table | `tests-orphans-table` |
|
|
76
|
+
|
|
77
|
+
## Detail — `/tests/{path}`
|
|
78
|
+
|
|
79
|
+
The `{path}` segment carries the full relative path from the project root,
|
|
80
|
+
including the `tests/` prefix and the `.py` extension — for example,
|
|
81
|
+
`/tests/tests/test_power_rail.py`. The path is resolved against the server's
|
|
82
|
+
current working directory; only files inside the working tree are served.
|
|
83
|
+
|
|
84
|
+
### Header
|
|
85
|
+
|
|
86
|
+
The page header shows the file path, a `N tests` badge, a `sidecar` badge
|
|
87
|
+
when a sidecar exists, and a **Launch Test** button (top right) that navigates
|
|
88
|
+
to `/launch?test={path}`.
|
|
89
|
+
|
|
90
|
+
A note below the header reads:
|
|
91
|
+
|
|
92
|
+
> What actually runs depends on the active profile — sidecar < profile
|
|
93
|
+
> (last-wins). See Profiles.
|
|
94
|
+
|
|
95
|
+
"Profiles" is a link to `/profiles`. For the cascade order, see
|
|
96
|
+
[Profiles](../configuration.md#profile-blocks-under-profiles).
|
|
97
|
+
|
|
98
|
+
### Test functions table
|
|
99
|
+
|
|
100
|
+
A table with one row per `def test_*` function found in the file. Columns:
|
|
101
|
+
|
|
102
|
+
| Column | What it shows |
|
|
103
|
+
|---|---|
|
|
104
|
+
| Test | Bare function name (`test_foo`) |
|
|
105
|
+
| Class | Parent `class Test*` name, or `—` for top-level functions |
|
|
106
|
+
| Markers | Comma-separated decorator names (without `pytest.mark.` prefix), or `—` |
|
|
107
|
+
| Vectors | Estimated parametrize / `testerkit_sweeps` vector count, or `—` when zero |
|
|
108
|
+
| Sidecar | `✓` when this function (or its parent class) has an entry in the sidecar `tests:` block; `—` otherwise |
|
|
109
|
+
| Runs | Distinct run count from history for this step path; `0` when never run |
|
|
110
|
+
| Passed | Runs with outcome `passed` |
|
|
111
|
+
| Failed | Runs with outcome `failed` |
|
|
112
|
+
| Last Run | Most recent step start timestamp, browser-local time; `—` when never run |
|
|
113
|
+
|
|
114
|
+
The step path used to look up run history is `Class/method` when the function
|
|
115
|
+
is inside a class, or the bare function name for top-level functions. This
|
|
116
|
+
matches the `step_path` value recorded in run history.
|
|
117
|
+
|
|
118
|
+
The test functions table is omitted when the file has a parse error.
|
|
119
|
+
|
|
120
|
+
Data-testid anchor: `test-functions-table`.
|
|
121
|
+
|
|
122
|
+
### Tabs
|
|
123
|
+
|
|
124
|
+
Below the test functions table, two tabs are available:
|
|
125
|
+
|
|
126
|
+
| Tab | Content |
|
|
127
|
+
|---|---|
|
|
128
|
+
| Code | The `.py` file rendered as a read-only Python code block |
|
|
129
|
+
| Sidecar YAML | The colocated `.yaml` file rendered as a read-only YAML code block; present only when a sidecar exists |
|
|
130
|
+
|
|
131
|
+
When no sidecar exists, only the **Code** tab appears.
|
|
132
|
+
|
|
133
|
+
### Parse error state
|
|
134
|
+
|
|
135
|
+
When the file cannot be parsed, a red card appears at the top of the content
|
|
136
|
+
area with the full error message. The Code tab still renders the raw file
|
|
137
|
+
contents.
|
|
138
|
+
|
|
139
|
+
### Not-found state
|
|
140
|
+
|
|
141
|
+
When `{path}` does not resolve to an existing `.py` file inside the working
|
|
142
|
+
tree, the page shows a card:
|
|
143
|
+
|
|
144
|
+
> Test file '{path}' not found.
|
|
145
|
+
|
|
146
|
+
with a "← Back to Tests" link.
|
|
147
|
+
|
|
148
|
+
## Underlying data
|
|
149
|
+
|
|
150
|
+
The file list is built by reading every `test_*.py` under `tests/` directly —
|
|
151
|
+
without importing or running any test code — to extract its function names,
|
|
152
|
+
class names, and decorators. Files that fail to parse are still listed,
|
|
153
|
+
flagged with the parse error.
|
|
154
|
+
|
|
155
|
+
Run history (Runs / Passed / Failed / Last Run) is tallied from run history
|
|
156
|
+
for each test, by step path. For the full step record schema, see
|
|
157
|
+
[Parquet schema](../data/parquet-schema.md).
|
|
158
|
+
|
|
159
|
+
A test shows the `sidecar` badge when a `.yaml` file with the same stem sits
|
|
160
|
+
alongside the `.py`. On the detail page, a function gets the `✓` when it (or
|
|
161
|
+
its class) has an entry in the sidecar's `tests:` block.
|
|
162
|
+
|
|
163
|
+
## See also
|
|
164
|
+
|
|
165
|
+
- [How-to → Writing Tests](../../how-to/execution/writing-tests.md) — pytest
|
|
166
|
+
classes, sidecar YAML, the `verify` pattern
|
|
167
|
+
- [pytest markers reference](../pytest/markers.md) — every `@testerkit.mark.*`
|
|
168
|
+
decorator that appears in the Markers column
|
|
169
|
+
- [Profiles](profiles.md) — how profiles override sidecar config at session
|
|
170
|
+
start (the cascade note on the detail page)
|
|
171
|
+
- [Launch Test](launch.md) — start a run targeting one of these test files
|
|
172
|
+
- [Results](results/list.md) — per-run view; the step-level detail that feeds
|
|
173
|
+
the Runs / Passed / Failed counts on this page
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# UUTs
|
|
2
|
+
|
|
3
|
+
**URLs:** `/uuts`
|
|
4
|
+
|
|
5
|
+
Every distinct UUT serial that has appeared in run history gets one row
|
|
6
|
+
here. A UUT isn't declared in YAML — it's identified at run time by the
|
|
7
|
+
serial number the operator (or automation) supplies when starting a run.
|
|
8
|
+
|
|
9
|
+
## List — `/uuts`
|
|
10
|
+
|
|
11
|
+
A table with one row per distinct UUT serial observed in run history.
|
|
12
|
+
Rows are ordered by most-recent run first.
|
|
13
|
+
|
|
14
|
+
| Column | What it shows |
|
|
15
|
+
|---|---|
|
|
16
|
+
| Serial | The `uut_serial_number` value recorded on the run |
|
|
17
|
+
| Part Number | The `uut_part_number` recorded on the most recent run for this serial |
|
|
18
|
+
| Lot | The `uut_lot_number` recorded on the most recent run for this serial |
|
|
19
|
+
| Runs | Total run count across all outcomes for this serial |
|
|
20
|
+
| Passed | Run count with outcome `passed` |
|
|
21
|
+
| Failed | Run count with outcome `failed` |
|
|
22
|
+
| Last Run | Most recent run start timestamp, browser-local time |
|
|
23
|
+
|
|
24
|
+
A badge in the page header shows the total count of observed UUTs.
|
|
25
|
+
|
|
26
|
+
The table has no filter row. All rows are observed-only — there is no
|
|
27
|
+
Configured / Observed distinction because UUTs have no backing YAML.
|
|
28
|
+
|
|
29
|
+
There is no detail page at `/uuts/{serial}`. The list is the only view.
|
|
30
|
+
|
|
31
|
+
### Empty state
|
|
32
|
+
|
|
33
|
+
When no runs with a non-empty `uut_serial_number` are present, the table is
|
|
34
|
+
replaced with a card:
|
|
35
|
+
|
|
36
|
+
> No UUTs observed yet.
|
|
37
|
+
>
|
|
38
|
+
> Run a test against a station to populate this list. Every distinct UUT
|
|
39
|
+
> serial that appears in run history shows up here.
|
|
40
|
+
|
|
41
|
+
### Outcome columns
|
|
42
|
+
|
|
43
|
+
**Passed** and **Failed** count only runs whose outcome is exactly
|
|
44
|
+
`passed` or `failed`. Runs with other outcomes (for example `aborted`)
|
|
45
|
+
contribute to the **Runs** total but are not reflected in either column.
|
|
46
|
+
|
|
47
|
+
## Underlying data
|
|
48
|
+
|
|
49
|
+
Each column maps to a field on the run:
|
|
50
|
+
|
|
51
|
+
| Column | Run field |
|
|
52
|
+
|---|---|
|
|
53
|
+
| Serial | `uut_serial_number` |
|
|
54
|
+
| Part Number | `uut_part_number` |
|
|
55
|
+
| Lot | `uut_lot_number` |
|
|
56
|
+
|
|
57
|
+
For the full set of run fields, see
|
|
58
|
+
[Parquet schema](../data/parquet-schema.md).
|
|
59
|
+
|
|
60
|
+
## See also
|
|
61
|
+
|
|
62
|
+
- [Parquet schema](../data/parquet-schema.md) — the `uut_serial_number`,
|
|
63
|
+
`uut_part_number`, and `uut_lot_number` run columns
|
|
64
|
+
- [Results](results/list.md) — per-run view; its Serial column shows
|
|
65
|
+
which runs belong to a UUT
|
|
66
|
+
- [Launch Test](launch.md) — where the operator supplies the serial
|
|
67
|
+
number that populates this page
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# Reference — Overview
|
|
2
|
+
|
|
3
|
+
The foundational reference pages: pytest mechanics every TesterKit test uses, and the AI / MCP skills the platform ships.
|
|
4
|
+
|
|
5
|
+
- [pytest-native](pytest-native.md) — TesterKit tests against pytest's own collection / fixtures / markers / `conftest.py` / CLI flags. The baseline every other reference page builds on. Generated from source.
|
|
6
|
+
- [Skills](skills.md) — the 11 Agent Skills that ship with TesterKit for AI-assisted authoring, and how `testerkit setup <tool>` installs them into each AI tool.
|
|
7
|
+
|
|
8
|
+
## See also
|
|
9
|
+
|
|
10
|
+
- [Concepts → Overview](../../concepts/overview/index.md) — the platform-vs-framework framing
|
|
11
|
+
- [pytest plugin reference](../pytest/) — the fixtures and markers the plugin adds on top of pytest-native
|
|
@@ -0,0 +1,142 @@
|
|
|
1
|
+
# pytest-native reference
|
|
2
|
+
|
|
3
|
+
TesterKit is a hardware test **platform**; pytest is its primary test-runner integration. The bundled pytest plugin slots into a stock pytest install with zero configuration — every pytest concept (collection, fixtures, markers, plugins, `conftest.py`, command-line flags) works unchanged. This page is the map of what pytest gives you natively and what the plugin layers on top. For the plugin's own surface see [TesterKit fixtures](../pytest/fixtures.md) and [TesterKit markers](../pytest/markers.md). Other runner integrations (OpenHTF, LabVIEW / TestStand via the results API) live under [Integrations](../../integration/).
|
|
4
|
+
|
|
5
|
+
## Collection
|
|
6
|
+
|
|
7
|
+
The bundled pytest plugin uses pytest's default collection — no custom file collectors.
|
|
8
|
+
|
|
9
|
+
| Convention | Default |
|
|
10
|
+
|---|---|
|
|
11
|
+
| Test files | `tests/test_*.py` or `tests/*_test.py` |
|
|
12
|
+
| Test classes | classes named `Test*` (no `__init__`) |
|
|
13
|
+
| Test functions / methods | functions named `test_*` |
|
|
14
|
+
| Override | standard pytest `python_files` / `python_classes` / `python_functions` in `pyproject.toml` |
|
|
15
|
+
|
|
16
|
+
What TesterKit adds at collection time:
|
|
17
|
+
|
|
18
|
+
- A `pytest_collection_modifyitems` hook merges per-test sidecar YAML (`tests/test_<module>.yaml`) into each item's marker set. This expands `testerkit_sweeps` into one pytest case per row exactly as if you had written `@pytest.mark.parametrize` — pytest still owns the case multiplication.
|
|
19
|
+
- Profiles (`--test-profile=<name>`) add `pytest.mark.skip` to items they exclude. The selection is visible in `pytest --collect-only -q`.
|
|
20
|
+
|
|
21
|
+
The sidecar is recursive: top-level keys apply to every test in the file; `tests: { ClassName: { ... } }` scopes per class; `tests: { ClassName: { tests: { test_method: { ... } } } }` scopes per method. See [Sidecar configuration](../configuration.md#sidecar-yaml).
|
|
22
|
+
|
|
23
|
+
## Fixtures
|
|
24
|
+
|
|
25
|
+
pytest's fixture model is unchanged.
|
|
26
|
+
|
|
27
|
+
- **All four scopes work.** `function`, `class`, `module`, `session` — choose the one that matches your resource cost.
|
|
28
|
+
- **Resolution by name.** Take a fixture in the test signature; pytest resolves it from the nearest `conftest.py` upward, then from registered plugins (TesterKit included).
|
|
29
|
+
- **Yield fixtures, finalizers, request injection** all work as pytest documents them.
|
|
30
|
+
- **`autouse=True`** works. TesterKit uses an autouse-session fixture internally so every test records measurements without taking a recording fixture as an argument.
|
|
31
|
+
|
|
32
|
+
You can write your own fixtures in `conftest.py` alongside TesterKit's. A common pattern is a project-local `uut` factory wrapping the TesterKit `uut` session fixture, or a per-class hardware-setup fixture that takes `instruments` as a dependency.
|
|
33
|
+
|
|
34
|
+
The fixtures the TesterKit plugin contributes are documented in [TesterKit fixtures](../pytest/fixtures.md).
|
|
35
|
+
|
|
36
|
+
## Markers
|
|
37
|
+
|
|
38
|
+
pytest's marker mechanism is unchanged. All of these work on TesterKit tests as documented in the pytest manual:
|
|
39
|
+
|
|
40
|
+
| Marker | Use |
|
|
41
|
+
|---|---|
|
|
42
|
+
| `@pytest.mark.parametrize` | Generate one pytest case per row. Stack to cross-product. |
|
|
43
|
+
| `@pytest.mark.skip` / `@pytest.mark.skipif` | Skip the test (with optional condition). |
|
|
44
|
+
| `@pytest.mark.xfail` | Mark as expected-to-fail; surfaces as `XFAIL` / `XPASS`. |
|
|
45
|
+
| `@pytest.mark.usefixtures("a", "b")` | Require fixtures without taking them in the signature. |
|
|
46
|
+
| `@pytest.mark.filterwarnings` | Per-test warning filters. |
|
|
47
|
+
| Custom markers via `pytest.ini` / `pyproject.toml` | Register and filter with `-m`. |
|
|
48
|
+
|
|
49
|
+
The seven `@pytest.mark.testerkit_*` markers TesterKit adds are registered the same way as native markers and stack with them (with one constraint — see the [no-stacking rule](../pytest/markers.md#no-stacking-rule) for TesterKit markers). See [TesterKit markers](../pytest/markers.md) for full details.
|
|
50
|
+
|
|
51
|
+
`@pytest.mark.parametrize` and `@pytest.mark.testerkit_sweeps` interoperate: both feed the same `inputs` lane on the vector row (query with `FieldRef.input(name)`).
|
|
52
|
+
|
|
53
|
+
## conftest.py
|
|
54
|
+
|
|
55
|
+
Works as pytest documents it. Place a `conftest.py` in `tests/` (or a subdirectory) for fixtures and hooks that apply to that scope. Common uses with TesterKit:
|
|
56
|
+
|
|
57
|
+
- Project-local fixtures that wrap or extend TesterKit's (a typed `uut` accessor, a per-class measurement helper).
|
|
58
|
+
- `pytest_addoption` for project-specific CLI flags.
|
|
59
|
+
- `pytest_collection_modifyitems` for project-specific item filtering. (TesterKit's hook is in the plugin and runs alongside, not instead of, yours.)
|
|
60
|
+
- `pytest_runtest_setup` / `_teardown` for per-test setup beyond what fixtures express.
|
|
61
|
+
|
|
62
|
+
The TesterKit plugin loads via the standard pytest entry-point mechanism — no `conftest.py` manipulation needed.
|
|
63
|
+
|
|
64
|
+
## Command-line flags
|
|
65
|
+
|
|
66
|
+
All pytest flags work. The ones that matter most for hardware test work:
|
|
67
|
+
|
|
68
|
+
| Flag | Purpose |
|
|
69
|
+
|---|---|
|
|
70
|
+
| `-k "expr"` | Run tests whose nodeid matches the substring expression. |
|
|
71
|
+
| `-m "marker"` | Run tests whose markers match the boolean expression. |
|
|
72
|
+
| `-x` / `--maxfail=N` | Stop after the first failure / Nth failure. |
|
|
73
|
+
| `--lf` / `--ff` | Run last-failed / failed-first (uses pytest's cache). |
|
|
74
|
+
| `--collect-only -q` | Show what would run without running. |
|
|
75
|
+
| `-v` / `-q` | Verbose / quiet. |
|
|
76
|
+
| `--tb=short` / `--tb=line` / `--tb=no` | Traceback style. |
|
|
77
|
+
| `-p no:plugin` | Disable a specific plugin. |
|
|
78
|
+
| `--co` | Alias for `--collect-only`. |
|
|
79
|
+
|
|
80
|
+
TesterKit adds the flags below.
|
|
81
|
+
|
|
82
|
+
<!-- Generated from pytest_addoption; regenerate with
|
|
83
|
+
`uv run python scripts/generate_reference_docs.py pytest-native` after editing the hook. -->
|
|
84
|
+
|
|
85
|
+
<!-- GENERATED:pytest-plugin-flags:start -->
|
|
86
|
+
| Flag | Type | Default | Description |
|
|
87
|
+
|---|---|---|---|
|
|
88
|
+
| `--uut-serial` | `text` | `'UUT001'` | UUT serial number |
|
|
89
|
+
| `--uut-serials` | `text` | | Per-site UUT serials: 0=SN1,1=SN2 (by index) or left=SN1,right=SN2 (by name) |
|
|
90
|
+
| `--site` | `text` | | Physical fixture site for this single-process run (e.g. ``0``, ``1``, or a named site like ``left``). Use this when running a single UUT against a specific position in a multi-site fixture so the run records which site was exercised. Multi-site orchestration uses ``--uut-serials`` instead — supplying both is an error. |
|
|
91
|
+
| `--uut-part-number` | `text` | | UUT part number |
|
|
92
|
+
| `--uut-revision` | `text` | | UUT revision |
|
|
93
|
+
| `--uut-lot-number` | `text` | | UUT lot/batch number (mirrors TESTERKIT_UUT_LOT_NUMBER env var) |
|
|
94
|
+
| `--station` | `text` | | Station ID or YAML path. Bare id looks up ``stations/<id>.yaml``; a value with ``/`` or ``.yaml``/``.yml`` is used as an explicit path. When unset, the resolver tries hostname auto-match against stations/*.yaml ``hostname:`` fields, then falls back to ``ProjectConfig.default_station``. |
|
|
95
|
+
| `--operator` | `text` | | Operator name |
|
|
96
|
+
| `--data-dir` | `text` | *resolved at runtime* | Directory for Parquet results (default: platform data dir) |
|
|
97
|
+
| `--part` | `text` | | Part ID or YAML path. Bare id looks up ``parts/<id>.yaml``; a value with ``/`` or ``.yaml``/``.yml`` is used as an explicit path. |
|
|
98
|
+
| `--guardband` | `text` | `'0'` | Default guardband percentage |
|
|
99
|
+
| `--mock-instruments` | `flag` | `False` | Use mock instruments instead of real hardware. Resolution: this flag > TESTERKIT_MOCK_INSTRUMENTS env var > testerkit.yaml `mock_instruments:` > false. |
|
|
100
|
+
| `--no-mock-instruments` | `flag` (inverse) | `True` | Use real hardware (overrides TESTERKIT_MOCK_INSTRUMENTS env and testerkit.yaml `mock_instruments: true`). |
|
|
101
|
+
| `--fixture` | `text` | | Fixture ID or YAML path. Bare id looks up ``fixtures/<id>.yaml``; a value with ``/`` or ``.yaml``/``.yml`` is used as an explicit path. When unset, the resolver tries the active profile's ``fixture:`` field, then ``ProjectConfig.default_fixture``, then the single-file fallback in ``fixtures/``. |
|
|
102
|
+
| `--test-phase` | `text` | | Test phase (development, validation, characterization, production). Defaults to development; a non-development phase is demoted to development on a dirty git tree or under --mock-instruments. |
|
|
103
|
+
| `--strict-traceability` | `flag` | `False` | Fail tests whose measurements lack required traceability fields (run_id, step_name, and spec_ref/uut_pin when a spec is active). |
|
|
104
|
+
| `--test-profile` | `text` | *resolved at runtime* | Named profile from testerkit.yaml `profiles:` (overrides vectors, limits, markers, and filter for the session). |
|
|
105
|
+
| `--no-test-profile` | `flag` | `False` | Skip profile resolution. Use when profiles are declared but you want to run with bare project defaults (ad-hoc runs). |
|
|
106
|
+
|
|
107
|
+
Plus dynamic flags generated from `testerkit.yaml`: every `profiles[*].facets:` key becomes `--<facet-key>` and every `required_inputs:` key becomes `--<required-input-key>`. See [how-to/profiles.md](../../how-to/execution/profiles.md) for the resolution chain (CLI flag → env var → profile binding → `default_*`).
|
|
108
|
+
<!-- GENERATED:pytest-plugin-flags:end -->
|
|
109
|
+
|
|
110
|
+
## Plugins that interact with TesterKit
|
|
111
|
+
|
|
112
|
+
| Plugin | Notes |
|
|
113
|
+
|---|---|
|
|
114
|
+
| **pytest-rerunfailures** | Powers `@pytest.mark.testerkit_retry`. Install it if you use the marker; the translation happens in the plugin. |
|
|
115
|
+
| **pytest-xdist** | Parallel execution. Generally **not** appropriate for hardware tests on a single bench (instruments aren't reentrant). Fine for mock-only suites and CI lint passes. |
|
|
116
|
+
| **pytest-cov** | Code coverage. Works unchanged on test files; collects coverage on the test code, not on instrument drivers behind hardware. |
|
|
117
|
+
| **pytest-html** / **pytest-json-report** | Independent of TesterKit's own event log + parquet output. Run them alongside if you want pytest-flavored reports too. |
|
|
118
|
+
|
|
119
|
+
## Discovery vs activation
|
|
120
|
+
|
|
121
|
+
Two things to keep separate:
|
|
122
|
+
|
|
123
|
+
- **Discovery** is pytest's: what `tests/test_*.py` files match, what items they expose. TesterKit has no opinion here.
|
|
124
|
+
- **Activation** is TesterKit's: which station, part, fixture, profile is loaded for the session. Driven by the CLI flags above (or by `default_station:` / `default_profile:` in `testerkit.yaml`).
|
|
125
|
+
|
|
126
|
+
A test that runs on a bringup tier with no station YAML and a test that runs on a factory tier with full traceability are **collected identically**. The activation context decides what fixtures resolve to and what limits `verify` finds.
|
|
127
|
+
|
|
128
|
+
## See also
|
|
129
|
+
|
|
130
|
+
- [TesterKit fixtures](../pytest/fixtures.md) — all the fixtures the plugin contributes
|
|
131
|
+
- [TesterKit markers](../pytest/markers.md) — the seven `testerkit_*` markers and their sidecar equivalents
|
|
132
|
+
- [Sidecar configuration](../configuration.md#sidecar-yaml) — sidecar YAML merge semantics
|
|
133
|
+
- [CLI reference](../cli.md) — full flag list, including the non-pytest commands (`testerkit serve`, `testerkit runs`, etc.)
|
|
134
|
+
- [pytest documentation](https://docs.pytest.org/en/stable/) — canonical reference for everything in the "pytest-native" half of this page
|
|
135
|
+
|
|
136
|
+
|
|
137
|
+
**Related quadrants:**
|
|
138
|
+
|
|
139
|
+
- [Concepts → Overview](../../concepts/overview/index.md) — concepts entry point for this category
|
|
140
|
+
- [How-to → Overview](../../how-to/overview/index.md) — how-to entry point for this category
|
|
141
|
+
- [Integration](../../integration/index.md) — integration entry point for this category
|
|
142
|
+
- [Tutorial](../../tutorial/index.md) — tutorial entry point for this category
|
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
# Skills reference
|
|
2
|
+
|
|
3
|
+
TesterKit ships 12 **Agent Skills** — one directory per skill under
|
|
4
|
+
[`src/testerkit/skills/`](https://github.com/pragmatest-dev/testerkit/tree/main/src/testerkit/skills),
|
|
5
|
+
each a `SKILL.md` written to the
|
|
6
|
+
[Agent Skills open standard](https://github.blog/changelog/2025-12-18-github-copilot-now-supports-agent-skills/):
|
|
7
|
+
YAML frontmatter (`name`, `description`) an agent matches against the user's
|
|
8
|
+
request, then a body the agent reads once matched. This page is the
|
|
9
|
+
inventory — what each skill covers, how the depth behind it is sourced, and
|
|
10
|
+
how `testerkit setup <tool>` gets the files in front of your agent.
|
|
11
|
+
|
|
12
|
+
For motivation (why AI integration at all), see
|
|
13
|
+
[concepts/why-ai-integration](../../concepts/overview/ai-integration.md). For
|
|
14
|
+
registering the MCP server and instructions file per tool, see
|
|
15
|
+
[how-to/mcp-integration](../../how-to/overview/mcp-integration.md).
|
|
16
|
+
|
|
17
|
+
## The 12 skills
|
|
18
|
+
|
|
19
|
+
| Skill | Use when |
|
|
20
|
+
|---|---|
|
|
21
|
+
| `testerkit-tests` | Testing, measuring, or logging any hardware value — the front door. Starts at zero config, grows only as the request demands. |
|
|
22
|
+
| `testerkit-mocks` | Running tests without real hardware — dev-machine smoke tests, CI, a bench that's tied up, or pinning one instrument call's return value. |
|
|
23
|
+
| `testerkit-stations` | Setting up the bench — instruments, roles, a bring-your-own driver, discovery, fixture pin routing. |
|
|
24
|
+
| `testerkit-parts` | Specifying the DUT — documented characteristics, pin map, or a datasheet limit as reusable part YAML. |
|
|
25
|
+
| `testerkit-profiles` | Different limits, sweeps, mocks, or wiring per test phase or part variant, selected with a CLI flag at run time. |
|
|
26
|
+
| `testerkit-sites` | Testing multiple UUTs at once on one fixture — multi-site / multi-socket parallel production testing. |
|
|
27
|
+
| `testerkit-capture` | Capturing non-tabular evidence during a test — a waveform, a live sensor feed, a photo, a vendor capture file, a log. |
|
|
28
|
+
| `testerkit-data` | Reading, querying, or exporting existing test data — runs, steps, measurements, channels, files, events — via the models, CLI, Query API, or MCP. |
|
|
29
|
+
| `testerkit-analysis` | Computing a statistic across runs — yield, Pareto, Ppk, a trend, a retest rate, time-loss. |
|
|
30
|
+
| `testerkit-debug` | Triaging why a run failed, errored, looks wrong, or is missing. |
|
|
31
|
+
| `testerkit-interactive` | Pausing a test for operator input, building a custom live operator screen, or driving a station interactively outside pytest. |
|
|
32
|
+
| `testerkit-datasheets` | Importing an instrument or part datasheet PDF into catalog or part config. |
|
|
33
|
+
|
|
34
|
+
Each row above is a condensed reading of that skill's own `description`
|
|
35
|
+
frontmatter — the exact text an agent matches against. `testerkit-tests` is
|
|
36
|
+
the deliberate front door: it starts at a bare `def test_x(verify): ...`
|
|
37
|
+
with no station, no YAML, and only routes to a sibling skill (mocks,
|
|
38
|
+
stations, parts, profiles, sites, capture, data, analysis, debug,
|
|
39
|
+
interactive) once the request needs that layer.
|
|
40
|
+
|
|
41
|
+
(`testerkit-capture` writes non-tabular evidence, `testerkit-data` reads any store
|
|
42
|
+
back, `testerkit-analysis` computes statistics over the tabular data — three
|
|
43
|
+
distinct jobs on the data axis, no overlap.)
|
|
44
|
+
|
|
45
|
+
## Single-source design
|
|
46
|
+
|
|
47
|
+
A `SKILL.md` is not a standalone essay. It carries the **action
|
|
48
|
+
playbook** — the decision the agent has to make and the judgment calls
|
|
49
|
+
around it (which verb, which config rung, what to warn about) — in a body
|
|
50
|
+
short enough to read in full every time the skill matches. Anything that's
|
|
51
|
+
already documented at length lives in the shipped docs, and the skill
|
|
52
|
+
tells the agent to read it there instead of re-stating it:
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
## Deeper
|
|
56
|
+
Read the docs:
|
|
57
|
+
testerkit docs show concepts/overview/tiers
|
|
58
|
+
testerkit docs show how-to/execution/writing-tests
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
This keeps exactly one copy of any factual claim — the shipped `docs/`
|
|
62
|
+
tree — instead of a frozen paraphrase inside the skill that drifts the
|
|
63
|
+
next time the underlying behavior changes. `testerkit docs show <path>`
|
|
64
|
+
streams that same tree; see [The `testerkit docs` CLI](#the-testerkit-docs-cli)
|
|
65
|
+
below.
|
|
66
|
+
|
|
67
|
+
Two skills carry a `references/` subdirectory for content that has no
|
|
68
|
+
shipped-doc home:
|
|
69
|
+
|
|
70
|
+
- **`testerkit-datasheets`** — `references/catalog-pipeline.md`,
|
|
71
|
+
`test-pipeline.md`, `scaffold.md`, `process-queue.md` are the full
|
|
72
|
+
phase-by-phase orchestration contracts for its four import pipelines
|
|
73
|
+
(too long and too procedural for a `docs/` page). It also carries
|
|
74
|
+
`agents/` — five sub-agent prompts (`section-splitter`,
|
|
75
|
+
`scaffold-writer`, `section-extractor`, `section-writer`,
|
|
76
|
+
`section-reviewer`) the catalog pipeline spawns in sequence; each is
|
|
77
|
+
single-responsibility and not invoked directly by a user.
|
|
78
|
+
- **`testerkit-interactive`** — `references/live-ui-patterns.md` is the
|
|
79
|
+
decision table + anti-pattern catalog for building a custom NiceGUI
|
|
80
|
+
page against live channel data (subscribe-vs-raw-thread, when to
|
|
81
|
+
stop mutating from a callback).
|
|
82
|
+
|
|
83
|
+
Every other skill's `SKILL.md` is self-contained plus links out to `testerkit
|
|
84
|
+
docs show` — no `references/` directory.
|
|
85
|
+
|
|
86
|
+
## Install / cross-tool
|
|
87
|
+
|
|
88
|
+
`testerkit setup <tool>` copies the packaged skill directories into the
|
|
89
|
+
tool's **native** Agent-Skills path — the same `SKILL.md` file, read
|
|
90
|
+
directly, no per-tool adapter or reformatting:
|
|
91
|
+
|
|
92
|
+
| Tool | Setup command | Skills copied to | Instructions file | MCP registration |
|
|
93
|
+
|---|---|---|---|---|
|
|
94
|
+
| Claude Code | `testerkit setup claude-code` | `.claude/skills/` | `CLAUDE.md` | `claude mcp add testerkit -- <bin> mcp serve` |
|
|
95
|
+
| OpenAI Codex | `testerkit setup codex` | `.agents/skills/` | `AGENTS.md` | printed for `~/.codex/config.toml` (Codex's home config; TesterKit doesn't write another tool's home config) |
|
|
96
|
+
| Cursor | `testerkit setup cursor` | `.cursor/skills/` | `AGENTS.md` | `.cursor/mcp.json` |
|
|
97
|
+
| GitHub Copilot | `testerkit setup copilot` | `.github/skills/` | `.github/copilot-instructions.md` + `AGENTS.md` | `.vscode/mcp.json` |
|
|
98
|
+
|
|
99
|
+
All four read `SKILL.md` natively — Claude Code and Codex from the start;
|
|
100
|
+
Cursor since 2.4 (shipped 2026-01-22); GitHub Copilot since its Agent
|
|
101
|
+
Skills support shipped 2025-12-18 (stable in VS Code early January 2026),
|
|
102
|
+
across VS Code/JetBrains agent mode, Copilot CLI, and the coding agent.
|
|
103
|
+
There is no Cursor `.mdc` rules file and no Copilot `.prompt.md` wrapper
|
|
104
|
+
generated — the single `SKILL.md` is the whole artifact.
|
|
105
|
+
|
|
106
|
+
The instructions file (`CLAUDE.md` / `AGENTS.md` /
|
|
107
|
+
`.github/copilot-instructions.md`) is marker-managed: `testerkit setup`
|
|
108
|
+
writes the TesterKit section between `<!-- testerkit:start -->` /
|
|
109
|
+
`<!-- testerkit:end -->` markers, creating the file if it doesn't exist or
|
|
110
|
+
replacing only that section if it does, leaving the rest of the file
|
|
111
|
+
alone.
|
|
112
|
+
|
|
113
|
+
`testerkit setup claude-desktop` and `testerkit setup cline` are MCP-only —
|
|
114
|
+
neither projects a native skills directory. `claude-desktop` bundles the
|
|
115
|
+
skill files inside the `.mcpb` extension as reference material rather
|
|
116
|
+
than a loose, natively-read directory; `cline` writes only
|
|
117
|
+
`cline_mcp_settings.json`.
|
|
118
|
+
|
|
119
|
+
Every `testerkit setup <tool>` command accepts `--print-only` to preview all
|
|
120
|
+
of its side effects — skills copied, instructions file created/updated,
|
|
121
|
+
MCP config — without writing anything.
|
|
122
|
+
|
|
123
|
+
## The `testerkit docs` CLI
|
|
124
|
+
|
|
125
|
+
```cli
|
|
126
|
+
$ testerkit docs list reference
|
|
127
|
+
reference/catalog/schema
|
|
128
|
+
reference/cli
|
|
129
|
+
reference/overview/pytest-native
|
|
130
|
+
reference/overview/skills
|
|
131
|
+
...
|
|
132
|
+
|
|
133
|
+
$ testerkit docs show concepts/data/three-verbs
|
|
134
|
+
# Three verbs: configure, observe, verify
|
|
135
|
+
...
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
`testerkit docs list [section]` enumerates shipped doc pages (optionally
|
|
139
|
+
scoped to `concepts`, `how-to`, `reference`, `tutorial`, or `integration`);
|
|
140
|
+
`testerkit docs show <path>` prints one page to stdout, resolved with or
|
|
141
|
+
without a trailing `.md`. Both read from the same `docs/` tree that ships
|
|
142
|
+
in the installed package (a bundled copy in a wheel install, the repo's
|
|
143
|
+
`docs/` directory in an editable/source checkout) — the environment-stable
|
|
144
|
+
way for an agent (or a human) to read TesterKit documentation without baking
|
|
145
|
+
an absolute path into project config. This is the single source every
|
|
146
|
+
skill's "Deeper" section points at, and it replaces the removed `testerkit
|
|
147
|
+
refs` command — there is no separate curated reference corpus to keep in
|
|
148
|
+
sync.
|
|
149
|
+
|
|
150
|
+
## MCP vs skills
|
|
151
|
+
|
|
152
|
+
Two different jobs, not overlapping ones:
|
|
153
|
+
|
|
154
|
+
- **MCP** is live execution and introspection — `testerkit_project`,
|
|
155
|
+
`testerkit_run`, `testerkit_match`, `testerkit_discover`, `testerkit_runs`,
|
|
156
|
+
`testerkit_metrics`, and the rest of the tool surface the `testerkit mcp
|
|
157
|
+
serve` server exposes. Calling one of these does something or reads
|
|
158
|
+
something right now. Full per-tool detail: [API reference →
|
|
159
|
+
MCP tools](../runtime/api.md#tools).
|
|
160
|
+
- **Skills** are judgment and procedure — which verb to reach for, which
|
|
161
|
+
config rung a request actually needs, what to check before declaring a
|
|
162
|
+
pipeline done. A skill matching doesn't execute anything by itself; it
|
|
163
|
+
tells the agent which MCP tools or CLI commands to call and in what
|
|
164
|
+
order, and where to read further before it does.
|
|
165
|
+
|
|
166
|
+
A skill's playbook routes into MCP tools and CLI commands as the steps
|
|
167
|
+
require — `testerkit-tests` step 6 runs plain `pytest`; `testerkit-datasheets`
|
|
168
|
+
saves through `testerkit_project(action="save", ...)`; `testerkit-debug` reads
|
|
169
|
+
back through `testerkit_runs` / `testerkit_steps`. Neither layer substitutes for
|
|
170
|
+
the other.
|
|
171
|
+
|
|
172
|
+
## See also
|
|
173
|
+
|
|
174
|
+
- [Concepts: why AI integration](../../concepts/overview/ai-integration.md) — motivation
|
|
175
|
+
- [How-to: MCP integration](../../how-to/overview/mcp-integration.md) — registering the server and instructions file with each AI tool
|
|
176
|
+
- [Reference: MCP tools](../runtime/api.md#tools) — per-tool parameter detail
|
|
177
|
+
- [Reference: CLI](../cli.md) — `testerkit docs`, `testerkit setup`, and every other command
|