Self-test protocol¶
Recipe and configuration authors can embed test cases in a bundle. A self-test job runs those cases against the recipe and reports whether the recipe behaves as its author expected. This appendix defines the protocol side of self-test jobs: how a client selects cases, which asset queries are valid, and how workers report failures.
Start with the self-test documentation in the shared package. It explains what a self-test run is and how cases are compiled and judged. Read this page when you write client, broker, or worker code that carries a self-test job over the protocol.
Added in version 7.0.0.
TEST_RECIPE retains wire value 1 and runs recipe and configuration suites
inside one job. VERIFY_PRODUCT retains value 2. All existing field numbers
remain unchanged.
Admission and selection¶
CreateJobRequest.selfTest is a presence-aware SelfTestSelection message.
For TEST_RECIPE, absent or empty selection chooses both available suites.
suite accepts BOTH, RECIPE_SUITE, or CONFIG_SUITE. caseIds
contains exact qualified IDs, such as recipe:correct and config:correct.
The author portion is an ASCII XML-style name with at most 93 characters.
The broker rejects unknown IDs, an explicitly requested missing configuration
suite, and an empty final selection before scheduling.
For VERIFY_PRODUCT, omit selfTest. Even an explicitly empty selection
is invalid. Recipe and configuration uploads are authoring sources; clients
cannot upload a top-level product for a self-test job.
Deployment and mocked results¶
Deploy the broker and workers together with self-test support. Clients submit
TEST_RECIPE jobs directly. Self-tests use the same worker resource and tool
selection as ordinary verification.
The broker completes mocked tasks without assigning them to workers. Dependent tasks receive the mocked properties and files in ordinary result bundles.
Asset scopes¶
For self-test downloads, use the following matrix. Input queries never contain a step or task number. A task number always requires a step ID.
Query |
Client |
Worker |
|---|---|---|
RECIPE, no case |
Original recipe |
Shared runtime recipe without test-only content |
CONFIG, no case |
Original configuration, if supplied |
Invalid |
CONFIG, case |
Effective case configuration |
Effective assigned-case configuration, or explicitly absent |
PRODUCT, case |
Effective case product |
Effective assigned-case product |
RESULT, no case or step |
Persisted suite report |
Invalid |
RESULT, case only |
Ordinary merged case results |
Invalid |
RESULT, case and step/task |
Ordinary case task result |
Dependency result in the assigned case |
Unscoped PRODUCT, case-scoped RECIPE, and RESULT with a step but no case are invalid. Unknown case IDs fail explicitly. Workers may upload only RESULT for the exact case, step, and task number of their current assignment. Clients may upload only unscoped RECIPE and CONFIG. Ordinary verification retains its input and result scopes and rejects every case ID.
validate_query checks structural rules. validate_query_scope additionally
checks job type, caller role, upload direction, and worker case/task ownership.
The broker must also verify job ownership, known cases, dependency membership,
and accepted attempt identity.
Failure provenance and attempts¶
Each self-test assignment carries a fresh broker-issued TaskId.attemptId.
Workers return that value as AssetData.taskAttemptId on every task result,
including successful results. It is not part of the logical step/task identity.
AssetData.executionFailure carries an ExecutionFailure with a typed
origin and diagnostic message. Origins cover executor failures, worker
failures, cancellation, and unknown infrastructure failures.
Presence always denotes a failure outside the checklet. An empty message or
UNKNOWN_EXECUTION_FAILURE must never satisfy an expected checklet error.
A checklet returning an ordinary error result has no such metadata.
Execution metadata is valid only on a task RESULT. Failure metadata requires an attempt ID. The broker accepts result identity, failure metadata, and completion atomically, only for the currently assigned attempt. Reject late results from superseded attempts even if their content hash matches a cached asset. Failure metadata belongs to the accepted result revision, not globally to an asset hash. Persist and restore it together with that revision.
send_asset accepts execution_failure and task_attempt_id keyword
arguments for worker uploads.
Completion and cancellation¶
FINISHED, FAILED, and CANCELLED are all terminal job states. A fully
evaluated suite can reach FINISHED with failed assertions; read the suite report
to determine the test verdict. Execution failure and cancellation are never
successful test outcomes. Jobs have no execution deadline; clients call
endJob to cancel unfinished execution. endJob also
releases job assets, so download the report before calling it. Existing client
RPCs support submission, status observation, report download, and cancellation
without a separate UI-specific protocol.