Self-test protocol

Recipe and configuration authors can embed test cases in a bundle. A self-test job runs those cases against the recipe and reports whether the recipe behaves as its author expected. This appendix defines the protocol side of self-test jobs: how a client selects cases, which asset queries are valid, and how workers report failures.

Start with the self-test documentation in the shared package. It explains what a self-test run is and how cases are compiled and judged. Read this page when you write client, broker, or worker code that carries a self-test job over the protocol.

Added in version 7.0.0.

TEST_RECIPE retains wire value 1 and runs recipe and configuration suites inside one job. VERIFY_PRODUCT retains value 2. All existing field numbers remain unchanged.

Admission and selection

CreateJobRequest.selfTest is a presence-aware SelfTestSelection message. For TEST_RECIPE, absent or empty selection chooses both available suites. suite accepts BOTH, RECIPE_SUITE, or CONFIG_SUITE. caseIds contains exact qualified IDs, such as recipe:correct and config:correct. The author portion is an ASCII XML-style name with at most 93 characters. The broker rejects unknown IDs, an explicitly requested missing configuration suite, and an empty final selection before scheduling.

For VERIFY_PRODUCT, omit selfTest. Even an explicitly empty selection is invalid. Recipe and configuration uploads are authoring sources; clients cannot upload a top-level product for a self-test job.

Deployment and mocked results

Deploy the broker and workers together with self-test support. Clients submit TEST_RECIPE jobs directly. Self-tests use the same worker resource and tool selection as ordinary verification.

The broker completes mocked tasks without assigning them to workers. Dependent tasks receive the mocked properties and files in ordinary result bundles.

Asset scopes

For self-test downloads, use the following matrix. Input queries never contain a step or task number. A task number always requires a step ID.

Download queries

Query

Client

Worker

RECIPE, no case

Original recipe

Shared runtime recipe without test-only content

CONFIG, no case

Original configuration, if supplied

Invalid

CONFIG, case

Effective case configuration

Effective assigned-case configuration, or explicitly absent

PRODUCT, case

Effective case product

Effective assigned-case product

RESULT, no case or step

Persisted suite report

Invalid

RESULT, case only

Ordinary merged case results

Invalid

RESULT, case and step/task

Ordinary case task result

Dependency result in the assigned case

Unscoped PRODUCT, case-scoped RECIPE, and RESULT with a step but no case are invalid. Unknown case IDs fail explicitly. Workers may upload only RESULT for the exact case, step, and task number of their current assignment. Clients may upload only unscoped RECIPE and CONFIG. Ordinary verification retains its input and result scopes and rejects every case ID.

validate_query checks structural rules. validate_query_scope additionally checks job type, caller role, upload direction, and worker case/task ownership. The broker must also verify job ownership, known cases, dependency membership, and accepted attempt identity.

Failure provenance and attempts

Each self-test assignment carries a fresh broker-issued TaskId.attemptId. Workers return that value as AssetData.taskAttemptId on every task result, including successful results. It is not part of the logical step/task identity.

AssetData.executionFailure carries an ExecutionFailure with a typed origin and diagnostic message. Origins cover executor failures, worker failures, cancellation, and unknown infrastructure failures. Presence always denotes a failure outside the checklet. An empty message or UNKNOWN_EXECUTION_FAILURE must never satisfy an expected checklet error. A checklet returning an ordinary error result has no such metadata.

Execution metadata is valid only on a task RESULT. Failure metadata requires an attempt ID. The broker accepts result identity, failure metadata, and completion atomically, only for the currently assigned attempt. Reject late results from superseded attempts even if their content hash matches a cached asset. Failure metadata belongs to the accepted result revision, not globally to an asset hash. Persist and restore it together with that revision.

send_asset accepts execution_failure and task_attempt_id keyword arguments for worker uploads.

Completion and cancellation

FINISHED, FAILED, and CANCELLED are all terminal job states. A fully evaluated suite can reach FINISHED with failed assertions; read the suite report to determine the test verdict. Execution failure and cancellation are never successful test outcomes. Jobs have no execution deadline; clients call endJob to cancel unfinished execution. endJob also releases job assets, so download the report before calling it. Existing client RPCs support submission, status observation, report download, and cancellation without a separate UI-specific protocol.