momotor.shared.selftest¶
Recipe authors can embed self-tests in a recipe or configuration bundle. Each test case supplies a product fixture, optionally a configuration fixture and mocked results, and a list of expectations about the results the recipe should produce. The XML syntax for declaring cases is documented in momotor-bundles:selftests. This module is the other half: it turns those declarations into something executable, and it turns execution results into a verdict.
The module does not execute anything itself. It has no scheduler, no executor,
and no transport. Provide the code that runs a recipe, and this module does
the planning before and the judging after. The momotor-engine-cli-selftest
test command and the broker’s self-test job handler are the two adapters
that exist today.
Added in version 3.0.0.
Installation¶
pip install 'momotor-engine-shared[selftest]'
The selftest extra pulls in momotor-bundles and
momotor-engine-options. The base package does not depend on either, and
importing anything else from momotor.shared does not import the self-test
module, so existing installations are unaffected.
The three-step flow¶
A self-test run is always the same shape.
Compile.
compile_suite()reads the test declarations from the recipe (and configuration) and produces aSuitePlan. Every case in the plan has its own frozen runtime inputs, its own task graph, and its expectations resolved to concrete task IDs.Execute. For every
CasePlan, runcase.recipewithcase.configandcase.product, collect the results into aResultsBundle, and note anything that went wrong.Evaluate.
evaluate_case()compares what was observed with what the case expected and produces aCaseReport.build_report()assembles the case reports into aTestResultBundle.
Put together, the smallest working adapter looks like this:
from momotor.shared.selftest import (
CaseObservation, TestSelection, build_report, compile_suite, evaluate_case,
)
with compile_suite(recipe, config, TestSelection(suite='both')) as plan:
reports = []
for case in plan.cases:
# Your code. Run case.recipe with case.config and case.product in a
# clean working directory and collect a ResultsBundle.
observed = execute(case)
observation = CaseObservation(observed.results, failures=observed.failures)
reports.append(evaluate_case(case, observation))
report = build_report(plan, reports)
report.to_file('self-tests.zip')
Everything happens inside the with block. The plan owns temporary copies of
the bundles, and those copies disappear when the context closes. Compile,
execute, evaluate, and export the report before leaving the block.
Step 1: compiling a suite¶
with compile_suite(recipe, config, selection, max_cases=1000) as plan:
...
recipe is the RecipeBundle under test.
config is the ConfigBundle it would normally
run with, or None. The compiler takes
snapshots of both immediately, the originals can be closed as soon as the
context has been entered.
Which cases end up in the plan¶
Both the recipe and the configuration can declare test cases. The compiler
calls these the two suites, recipe and config. Case IDs are
qualified with their suite, so the recipe’s smoke case is recipe:smoke
and the configuration’s smoke case is config:smoke. They are different
cases and can coexist.
TestSelection controls what gets compiled:
suiteis'both'(the default),'recipe', or'config'.casesis a tuple of qualified IDs. Empty means every case in the selected suites.
Selecting 'config' without passing a configuration is an error. Naming a
case that does not exist is an error. Ending up with zero cases is also an
error. Zero tests never counts as a passing run.
Which configuration each case runs with¶
A recipe case runs with the
configuration fixture embedded in the test case, or with no configuration at
all if the case has none. It never picks up the config argument that was passed
to compile_suite(). A configuration case runs with its enclosing
configuration, which is the config argument, with the test declarations
stripped out.
What a compiled case contains¶
Each CasePlan in plan.cases has:
idThe qualified ID, for example
recipe:smoke.recipe,config,productRuntime projections (see
runtime_projection()). These are ordinary bundles with all test declarations, expected files, and mock payloads removed, which can be handed to a scheduler or executor without leaking fixture data into the run.configisNonewhen the case has no configuration.graphThe full expanded task graph, mapping each
StepTaskIdto the tasks it depends on. This is the same graph the scheduler would compute for these inputs (it comes fromget_full_task_dependencies()). Every task in it must produce a result for the case to complete.assertionsThe case’s expectations, each with an
idand a tuple oftargets: the concrete task IDs the expectation applies to. See Result queries below.input_hashesSHA-256 hashes of the runtime recipe, configuration, product, and mocks.
mock_resultsA
ResultsBundleof pre-authored results for tasks the case wants to skip, orNone. See Mocked results.
Admission limits¶
compile_suite() refuses suites that are too large, before doing any
expensive work. The limits are keyword arguments:
max_cases(default 1000): number of selected cases.max_tasks(default 10000): total tasks across all case graphs. The compiler checks declared task counts before expanding a graph, so an oversized suite fails fast.max_fixture_bytes(default 256 MiB): uncompressed size of all runtime inputs, expected files, and expanded mock payloads.
All limit violations raise ValueError with limit in the message.
Result queries¶
An expectation’s step attribute is not a step ID. It is a
result query, the same syntax the
options package uses to look up results, and it resolves against the case’s
expanded graph. run matches a step with one task,
run.* matches every task of a multi-task step, run.0,run.1 matches two
of them, and build.*,test.* spans steps.
The compiler resolves these once, at compile time, and freezes the resulting targets. A few rules apply:
Every comma-separated component must match at least one task. A component that matches nothing fails compilation, even if the other components match.
Overlapping components are fine. The compiler deduplicates and orders the targets the way they appear in the graph.
Relative references (
$-prefixed, which mean “relative to the current task”) are rejected. An expectation has no current task.
Mock results use the same query language for their step selector.
Step 2: executing a case¶
Run the case in isolation¶
Use case.recipe, case.config, and case.product as the job inputs.
Give each case a fresh working directory. Do not share state between cases.
Collect every result¶
Put the Result for each task into
a ResultsBundle. The evaluator
compares the result step IDs against case.graph. A planned task without a
result makes the case an error. A result for a task that is not in the graph
also makes the case an error.
Report failures as typed failures, not messages¶
When the executor crashes, a worker disappears, or an artifact cannot be read,
record an ExecutionFailure
with the appropriate origin and pass it in CaseObservation.failures.
The evaluator treats any failure as an error,
regardless of what the assertions say. It does not parse result messages to
infer failures, so if a failure is not reported explicitly, it did not
happen as far as the report is concerned.
This matters most for expected errors. A test can declare
outcome="error" for a task and pass when the checklet reports an error.
But if the executor failed, that is not the checklet erroring, and the case
must not pass. Reporting the executor failure in failures is what makes the
distinction visible to the evaluator.
Honor mocked results¶
If case.mock_results is not None, it contains one ready-made
Result for each mocked task,
already keyed by concrete task ID. For each such task,
do not run the checklet. Skip preflight, tool selection, environment setup,
all of it. Feed the mocked result into the normal completion path so dependent
tasks see it exactly as they would see a real result. Then include the mocked
results in the ResultsBundle passed to the evaluator, since they are
planned tasks and must be accounted for.
Unmocked tasks still run. The graph is not pruned, so a mocked task’s own dependencies execute normally before the mock is released.
Report the execution state¶
CaseObservation.state is one of:
completedEvery task reached a terminal state. Missing results are now errors.
cancelledThe run was interrupted. The case gets a
cancelledverdict and any assertions without a result are markedcancelled.pendingorrunningThe case has not finished. It gets no verdict and the report cannot be complete. Use this when reporting partial progress.
duration (seconds) and provenance (a mapping of strings, for things
like {'backend': 'local'}) are passed through to the report unchanged.
Step 3: evaluating and reporting¶
evaluate_case() is a pure function of the plan and the observation.
Same inputs, same report, every time.
How a verdict is reached¶
The evaluator checks, in this order:
If
stateiscancelled, the verdict iscancelled.If
stateispendingorrunning, there is no verdict.If there is any failure, the verdict is
error. Failures include the ones supplied plus the ones the evaluator adds: a planned task with no result, a result for an unplanned task, an unreadable artifact, or a checkleterroroutcome on a task that no expectation declared asoutcome="error".If any assertion did not pass, the verdict is
failed.Otherwise,
passed.
Every check the evaluator runs produces an
AssertionReport in
report.assertions, with the expected and actual values (truncated to 2048
characters, and flagged when truncated) and a short reason code such as
property_value_or_type or file_content. The original execution results
are kept in report.results so a reader can look at what actually happened.
Comparison rules¶
Property comparisons are strict about types. 1 does not equal "1", and
True does not equal 1. Ordering comparisons (lt, le, gt,
ge) require a numeric expected value, and a non-numeric or non-finite
actual value fails them. A declared None matches a property whose value is
null, and does not match a property with empty content.
When a result has several properties with the same name, values on the
expectation decides what happens. The default one fails unless exactly one
property matches the name. all requires every one to satisfy the
comparison, any requires at least one. Both fail on an empty set.
File comparisons stream the content and compare SHA-256 digests, so large
files do not get loaded into memory and the report shows the two digests
rather than the content. bytes compares raw bytes. text decodes with
the expectation’s charset (default UTF-8) and only normalizes newline
sequences when normalize_newlines is set. A file expectation must match
exactly one file. Use class to disambiguate, and member to select a
file inside a directory attachment.
Building the suite report¶
report = build_report(plan, reports, run_id=job_id)
build_report() puts the case reports in plan order and fills in a
pending record for every selected case not reported on, so the
report always accounts for the whole selection. Passing a report for a case
that is not in the plan, or two reports for the same case, raises
ValueError.
The result is a TestResultBundle. Its
complete flag is true only when
every case has a terminal verdict. A partial report cannot pass; it has no
overall verdict until every case is done.
Mocked results¶
Added in version 3.0.0.
A test case can declare <mock step="..."> elements with a full result
payload (the authoring side is covered in momotor-bundles:selftests).
The compiler resolves each selector against the graph, copies the
payload once per matched task with the concrete task ID filled in, and
freezes the lot into case.mock_results. Two mocks may not target the same
task; that fails compilation. The expanded payloads count toward
max_fixture_bytes.
The mocked bundle gets its own entry in case.input_hashes['mocks'], and
the plan manifest lists the mocked task IDs under mocked_tasks. The case
report records the same list as a JSON string in
report.provenance['mocked_tasks']. Recording it as provenance rather than
as a result property keeps the mock from influencing any downstream result
query.
A mocked error outcome behaves like a real checklet error. If no
expectation declares outcome="error" for that task, the case is an error.
Storing a plan for later¶
A broker that schedules self-test jobs across restarts needs to know that the
plan it resumes is the plan it started. plan_manifest() returns a
JSON-safe dict with everything that was decided at compile time: source and
input hashes, the full graph per case, resolved assertion targets, normalized
comparison values, expected-file digests, and mocked task IDs.
Store the manifest with the immutable input assets before scheduling. When restoring, compile again from the stored sources and compare the new manifest with the stored one. A mismatch means the inputs changed, and the job should be refused rather than continued.
The manifest is deterministic. Compiling the same sources twice yields equal manifests.
API reference¶
Compile immutable, case-specific execution inputs and assertion targets.
- class momotor.shared.selftest.planning.CasePlan(id, recipe, config, product, graph, assertions, input_hashes, *, mock_results=None)¶
One complete graph and isolated runtime inputs for a qualified case.
- class momotor.shared.selftest.planning.PlannedExpectation(id, declaration, targets)¶
A declaration with stable identity and graph-resolved targets.
- class momotor.shared.selftest.planning.SuitePlan(cases, source_hashes)¶
Ordered selected cases with frozen input and source identities.
- class momotor.shared.selftest.planning.TestSelection(suite='both', cases=())¶
Select available suites and exact qualified case IDs.
- momotor.shared.selftest.planning.bundle_hash(bundle)¶
Hash exported bundle members independently of ZIP timestamps.
- momotor.shared.selftest.planning.compile_suite(recipe, config=None, selection=None, *, max_cases=1000, max_tasks=10000, max_fixture_bytes=268435456)¶
Freeze selected suites, their owned inputs, graphs, and assertions.
- Parameters:
recipe (
RecipeBundle) – Recipe under test, including its self-contained cases.config (
ConfigBundle|None) – Optional enclosing configuration for configuration cases.selection (
TestSelection|None) – Suite/case selectors, defaulting to both available suites.max_cases (
int) – Admission limit on selected cases.max_tasks (
int) – Admission limit on total expanded tasks.max_fixture_bytes (
int) – Admission limit on serialized effective inputs.
- Yield:
Plan with artifacts valid until this context closes.
- Raises:
ValueError – For invalid selections, declarations, or limits.
- Return type:
Added in version 3.0.0.
- momotor.shared.selftest.planning.plan_manifest(plan)¶
Describe every frozen execution and assertion choice for durable storage.
Evaluate frozen self-test assertions without transport or scheduling state.
- class momotor.shared.selftest.evaluation.CaseObservation(results, state='completed', failures=(), duration=None, provenance=<factory>)¶
Accepted execution results and failures for one case.
completedmeans every task terminated, including tasks without a result. Pending/running cases have no final verdict. Failures are accepted-attempt observations supplied by the execution adapter, never inferred from messages.
- momotor.shared.selftest.evaluation.build_report(plan, reports, *, run_id=None)¶
Build an ordered suite report including pending cases omitted by adapters.
- Parameters:
- Return type:
- Returns:
Canonical report bundle whose counts and verdict derive from cases.
- Raises:
ValueError – For duplicate or unknown cases.
- momotor.shared.selftest.evaluation.evaluate_case(plan, observation)¶
Evaluate all checks on all frozen targets and account for every task.
- Parameters:
plan (
CasePlan) – Compiled case with graph-resolved expectations.observation (
CaseObservation) – Accepted results, execution state, and typed failures.
- Return type:
CaseReport- Returns:
Deterministic report retaining ordinary execution results.
Added in version 3.0.0.