momotor.shared.selftest

Recipe authors can embed self-tests in a recipe or configuration bundle. Each test case supplies a product fixture, optionally a configuration fixture and mocked results, and a list of expectations about the results the recipe should produce. The XML syntax for declaring cases is documented in momotor-bundles:selftests. This module is the other half: it turns those declarations into something executable, and it turns execution results into a verdict.

The module does not execute anything itself. It has no scheduler, no executor, and no transport. Provide the code that runs a recipe, and this module does the planning before and the judging after. The momotor-engine-cli-selftest test command and the broker’s self-test job handler are the two adapters that exist today.

Added in version 3.0.0.

Installation

pip install 'momotor-engine-shared[selftest]'

The selftest extra pulls in momotor-bundles and momotor-engine-options. The base package does not depend on either, and importing anything else from momotor.shared does not import the self-test module, so existing installations are unaffected.

The three-step flow

A self-test run is always the same shape.

  1. Compile. compile_suite() reads the test declarations from the recipe (and configuration) and produces a SuitePlan. Every case in the plan has its own frozen runtime inputs, its own task graph, and its expectations resolved to concrete task IDs.

  2. Execute. For every CasePlan, run case.recipe with case.config and case.product, collect the results into a ResultsBundle, and note anything that went wrong.

  3. Evaluate. evaluate_case() compares what was observed with what the case expected and produces a CaseReport. build_report() assembles the case reports into a TestResultBundle.

Put together, the smallest working adapter looks like this:

from momotor.shared.selftest import (
    CaseObservation, TestSelection, build_report, compile_suite, evaluate_case,
)

with compile_suite(recipe, config, TestSelection(suite='both')) as plan:
    reports = []
    for case in plan.cases:
        # Your code. Run case.recipe with case.config and case.product in a
        # clean working directory and collect a ResultsBundle.
        observed = execute(case)
        observation = CaseObservation(observed.results, failures=observed.failures)
        reports.append(evaluate_case(case, observation))
    report = build_report(plan, reports)
    report.to_file('self-tests.zip')

Everything happens inside the with block. The plan owns temporary copies of the bundles, and those copies disappear when the context closes. Compile, execute, evaluate, and export the report before leaving the block.

Step 1: compiling a suite

with compile_suite(recipe, config, selection, max_cases=1000) as plan:
    ...

recipe is the RecipeBundle under test. config is the ConfigBundle it would normally run with, or None. The compiler takes snapshots of both immediately, the originals can be closed as soon as the context has been entered.

Which cases end up in the plan

Both the recipe and the configuration can declare test cases. The compiler calls these the two suites, recipe and config. Case IDs are qualified with their suite, so the recipe’s smoke case is recipe:smoke and the configuration’s smoke case is config:smoke. They are different cases and can coexist.

TestSelection controls what gets compiled:

  • suite is 'both' (the default), 'recipe', or 'config'.

  • cases is a tuple of qualified IDs. Empty means every case in the selected suites.

Selecting 'config' without passing a configuration is an error. Naming a case that does not exist is an error. Ending up with zero cases is also an error. Zero tests never counts as a passing run.

Which configuration each case runs with

A recipe case runs with the configuration fixture embedded in the test case, or with no configuration at all if the case has none. It never picks up the config argument that was passed to compile_suite(). A configuration case runs with its enclosing configuration, which is the config argument, with the test declarations stripped out.

What a compiled case contains

Each CasePlan in plan.cases has:

id

The qualified ID, for example recipe:smoke.

recipe, config, product

Runtime projections (see runtime_projection()). These are ordinary bundles with all test declarations, expected files, and mock payloads removed, which can be handed to a scheduler or executor without leaking fixture data into the run. config is None when the case has no configuration.

graph

The full expanded task graph, mapping each StepTaskId to the tasks it depends on. This is the same graph the scheduler would compute for these inputs (it comes from get_full_task_dependencies()). Every task in it must produce a result for the case to complete.

assertions

The case’s expectations, each with an id and a tuple of targets: the concrete task IDs the expectation applies to. See Result queries below.

input_hashes

SHA-256 hashes of the runtime recipe, configuration, product, and mocks.

mock_results

A ResultsBundle of pre-authored results for tasks the case wants to skip, or None. See Mocked results.

Admission limits

compile_suite() refuses suites that are too large, before doing any expensive work. The limits are keyword arguments:

  • max_cases (default 1000): number of selected cases.

  • max_tasks (default 10000): total tasks across all case graphs. The compiler checks declared task counts before expanding a graph, so an oversized suite fails fast.

  • max_fixture_bytes (default 256 MiB): uncompressed size of all runtime inputs, expected files, and expanded mock payloads.

All limit violations raise ValueError with limit in the message.

Result queries

An expectation’s step attribute is not a step ID. It is a result query, the same syntax the options package uses to look up results, and it resolves against the case’s expanded graph. run matches a step with one task, run.* matches every task of a multi-task step, run.0,run.1 matches two of them, and build.*,test.* spans steps.

The compiler resolves these once, at compile time, and freezes the resulting targets. A few rules apply:

  • Every comma-separated component must match at least one task. A component that matches nothing fails compilation, even if the other components match.

  • Overlapping components are fine. The compiler deduplicates and orders the targets the way they appear in the graph.

  • Relative references ($-prefixed, which mean “relative to the current task”) are rejected. An expectation has no current task.

Mock results use the same query language for their step selector.

Step 2: executing a case

Run the case in isolation

Use case.recipe, case.config, and case.product as the job inputs. Give each case a fresh working directory. Do not share state between cases.

Collect every result

Put the Result for each task into a ResultsBundle. The evaluator compares the result step IDs against case.graph. A planned task without a result makes the case an error. A result for a task that is not in the graph also makes the case an error.

Report failures as typed failures, not messages

When the executor crashes, a worker disappears, or an artifact cannot be read, record an ExecutionFailure with the appropriate origin and pass it in CaseObservation.failures. The evaluator treats any failure as an error, regardless of what the assertions say. It does not parse result messages to infer failures, so if a failure is not reported explicitly, it did not happen as far as the report is concerned.

This matters most for expected errors. A test can declare outcome="error" for a task and pass when the checklet reports an error. But if the executor failed, that is not the checklet erroring, and the case must not pass. Reporting the executor failure in failures is what makes the distinction visible to the evaluator.

Honor mocked results

If case.mock_results is not None, it contains one ready-made Result for each mocked task, already keyed by concrete task ID. For each such task, do not run the checklet. Skip preflight, tool selection, environment setup, all of it. Feed the mocked result into the normal completion path so dependent tasks see it exactly as they would see a real result. Then include the mocked results in the ResultsBundle passed to the evaluator, since they are planned tasks and must be accounted for.

Unmocked tasks still run. The graph is not pruned, so a mocked task’s own dependencies execute normally before the mock is released.

Report the execution state

CaseObservation.state is one of:

completed

Every task reached a terminal state. Missing results are now errors.

cancelled

The run was interrupted. The case gets a cancelled verdict and any assertions without a result are marked cancelled.

pending or running

The case has not finished. It gets no verdict and the report cannot be complete. Use this when reporting partial progress.

duration (seconds) and provenance (a mapping of strings, for things like {'backend': 'local'}) are passed through to the report unchanged.

Step 3: evaluating and reporting

evaluate_case() is a pure function of the plan and the observation. Same inputs, same report, every time.

How a verdict is reached

The evaluator checks, in this order:

  1. If state is cancelled, the verdict is cancelled.

  2. If state is pending or running, there is no verdict.

  3. If there is any failure, the verdict is error. Failures include the ones supplied plus the ones the evaluator adds: a planned task with no result, a result for an unplanned task, an unreadable artifact, or a checklet error outcome on a task that no expectation declared as outcome="error".

  4. If any assertion did not pass, the verdict is failed.

  5. Otherwise, passed.

Every check the evaluator runs produces an AssertionReport in report.assertions, with the expected and actual values (truncated to 2048 characters, and flagged when truncated) and a short reason code such as property_value_or_type or file_content. The original execution results are kept in report.results so a reader can look at what actually happened.

Comparison rules

Property comparisons are strict about types. 1 does not equal "1", and True does not equal 1. Ordering comparisons (lt, le, gt, ge) require a numeric expected value, and a non-numeric or non-finite actual value fails them. A declared None matches a property whose value is null, and does not match a property with empty content.

When a result has several properties with the same name, values on the expectation decides what happens. The default one fails unless exactly one property matches the name. all requires every one to satisfy the comparison, any requires at least one. Both fail on an empty set.

File comparisons stream the content and compare SHA-256 digests, so large files do not get loaded into memory and the report shows the two digests rather than the content. bytes compares raw bytes. text decodes with the expectation’s charset (default UTF-8) and only normalizes newline sequences when normalize_newlines is set. A file expectation must match exactly one file. Use class to disambiguate, and member to select a file inside a directory attachment.

Building the suite report

report = build_report(plan, reports, run_id=job_id)

build_report() puts the case reports in plan order and fills in a pending record for every selected case not reported on, so the report always accounts for the whole selection. Passing a report for a case that is not in the plan, or two reports for the same case, raises ValueError.

The result is a TestResultBundle. Its complete flag is true only when every case has a terminal verdict. A partial report cannot pass; it has no overall verdict until every case is done.

Mocked results

Added in version 3.0.0.

A test case can declare <mock step="..."> elements with a full result payload (the authoring side is covered in momotor-bundles:selftests). The compiler resolves each selector against the graph, copies the payload once per matched task with the concrete task ID filled in, and freezes the lot into case.mock_results. Two mocks may not target the same task; that fails compilation. The expanded payloads count toward max_fixture_bytes.

The mocked bundle gets its own entry in case.input_hashes['mocks'], and the plan manifest lists the mocked task IDs under mocked_tasks. The case report records the same list as a JSON string in report.provenance['mocked_tasks']. Recording it as provenance rather than as a result property keeps the mock from influencing any downstream result query.

A mocked error outcome behaves like a real checklet error. If no expectation declares outcome="error" for that task, the case is an error.

Storing a plan for later

A broker that schedules self-test jobs across restarts needs to know that the plan it resumes is the plan it started. plan_manifest() returns a JSON-safe dict with everything that was decided at compile time: source and input hashes, the full graph per case, resolved assertion targets, normalized comparison values, expected-file digests, and mocked task IDs.

Store the manifest with the immutable input assets before scheduling. When restoring, compile again from the stored sources and compare the new manifest with the stored one. A mismatch means the inputs changed, and the job should be refused rather than continued.

The manifest is deterministic. Compiling the same sources twice yields equal manifests.

API reference

Compile immutable, case-specific execution inputs and assertion targets.

class momotor.shared.selftest.planning.CasePlan(id, recipe, config, product, graph, assertions, input_hashes, *, mock_results=None)

One complete graph and isolated runtime inputs for a qualified case.

class momotor.shared.selftest.planning.PlannedExpectation(id, declaration, targets)

A declaration with stable identity and graph-resolved targets.

class momotor.shared.selftest.planning.SuitePlan(cases, source_hashes)

Ordered selected cases with frozen input and source identities.

class momotor.shared.selftest.planning.TestSelection(suite='both', cases=())

Select available suites and exact qualified case IDs.

momotor.shared.selftest.planning.bundle_hash(bundle)

Hash exported bundle members independently of ZIP timestamps.

Parameters:

bundle (Bundle) – Source whose content and attachments are hashed.

Return type:

str

Returns:

SHA-256 content identity.

momotor.shared.selftest.planning.compile_suite(recipe, config=None, selection=None, *, max_cases=1000, max_tasks=10000, max_fixture_bytes=268435456)

Freeze selected suites, their owned inputs, graphs, and assertions.

Parameters:
  • recipe (RecipeBundle) – Recipe under test, including its self-contained cases.

  • config (ConfigBundle | None) – Optional enclosing configuration for configuration cases.

  • selection (TestSelection | None) – Suite/case selectors, defaulting to both available suites.

  • max_cases (int) – Admission limit on selected cases.

  • max_tasks (int) – Admission limit on total expanded tasks.

  • max_fixture_bytes (int) – Admission limit on serialized effective inputs.

Yield:

Plan with artifacts valid until this context closes.

Raises:

ValueError – For invalid selections, declarations, or limits.

Return type:

Iterator[SuitePlan]

Added in version 3.0.0.

momotor.shared.selftest.planning.plan_manifest(plan)

Describe every frozen execution and assertion choice for durable storage.

Parameters:

plan (SuitePlan) – Compiled suite whose inputs remain open.

Return type:

dict

Returns:

JSON-safe manifest suitable for exact restoration checks.

Evaluate frozen self-test assertions without transport or scheduling state.

class momotor.shared.selftest.evaluation.CaseObservation(results, state='completed', failures=(), duration=None, provenance=<factory>)

Accepted execution results and failures for one case.

completed means every task terminated, including tasks without a result. Pending/running cases have no final verdict. Failures are accepted-attempt observations supplied by the execution adapter, never inferred from messages.

momotor.shared.selftest.evaluation.build_report(plan, reports, *, run_id=None)

Build an ordered suite report including pending cases omitted by adapters.

Parameters:
  • plan (SuitePlan) – Frozen suite selection and source identities.

  • reports (Iterable[CaseReport]) – Available case reports, each from the selected case set.

  • run_id (str | None) – Optional transport-independent run identifier.

Return type:

TestResultBundle

Returns:

Canonical report bundle whose counts and verdict derive from cases.

Raises:

ValueError – For duplicate or unknown cases.

momotor.shared.selftest.evaluation.evaluate_case(plan, observation)

Evaluate all checks on all frozen targets and account for every task.

Parameters:
  • plan (CasePlan) – Compiled case with graph-resolved expectations.

  • observation (CaseObservation) – Accepted results, execution state, and typed failures.

Return type:

CaseReport

Returns:

Deterministic report retaining ordinary execution results.

Added in version 3.0.0.