Self-test declarations and reports

Added in version 9.0.0.

A self-test checks a recipe or a configuration against a known product. You declare the product to process and the outcome you expect, for example that a step passes with a particular score. The Momotor Engine runs the steps and compares the actual results with your expectations.

This package stores self-test declarations in recipe and configuration bundles and stores the verdicts in a TestResultBundle. It does not run the tests itself. Read this page when you write self-tests in bundle XML or in Python, or when you read a self-test report.

RecipeBundle.tests and ConfigBundle.tests are immutable keyed tuples. They support ordered iteration, integer indexing, and lookup by author ID. An ID starts with an ASCII letter or underscore and contains at most 93 ASCII letters, digits, underscores, periods, or hyphens. IDs are unique within a suite. The runtime identity adds recipe: or config: to the author ID.

Authoring XML

A test contains optional metadata, an optional configuration fixture, one explicit product fixture, optional mock results, and at least one expectation. Only recipe tests can contain a configuration fixture. Embedded configurations cannot contain tests. An omitted recipe configuration fixture means no configuration. The test does not inherit a course configuration. Configuration tests use their enclosing configuration unchanged.

<recipe xmlns="http://momotor.org/1.0">
  <steps><step id="run"/></steps>
  <tests>
    <test id="correct">
      <config><options><option name="count" type="integer" value="2"/></options></config>
      <product><files><file name="answer.txt" type="text/plain">4</file></files></product>
      <expect step="run" outcome="pass">
        <property name="score" comparison="eq" type="integer" value="1"/>
        <file name="stdout.txt" comparison="text" src="expected/correct.txt"
              normalize-newlines="true"/>
      </expect>
    </test>
  </tests>
</recipe>

The step attribute contains a result query, including numbered targets, wildcards, and comma-separated queries. It is not an XML ID reference. The execution planner validates queries against the expanded graph. There is no separate task attribute.

Each expectation contains an outcome, properties, files, or a combination. Outcomes are pass, fail, skip, and error. A property comparison is exists, absent, eq, ne, lt, le, gt, or ge. Property values defaults to one; all and any allow repeated actual properties. Ordered comparisons require finite numeric expected values. Equality also supports strings, bytes, booleans, and explicit nulls. <none/> declares null, while an absent expected value is invalid for a value comparison. type="decimal" preserves exact decimal expectations and ordinary result properties. Declared scalar types must match the decoded expected value. Supported type names are string, integer or int, float, decimal, and boolean or bool. Encoded strings use UTF-8. Encoded content without a declared string type remains bytes.

File comparisons are exists, absent, bytes, or text. name identifies the actual output. src identifies expected content within the bundle declaring the test. Expected content can also be inline, including ordinary base64 or quoted-printable encoding. An empty content comparison expects an empty file. Presence checks cannot contain expected content.

class narrows the actual file selection. member selects a relative file inside a directory attachment. Text comparisons use charset="utf-8" by default. normalize-newlines is a boolean and defaults to false. Comparison evaluation belongs to momotor.shared.selftest; this package owns the declarations, attachments, and report format.

Mock results

Added in version 9.0.0.

Add mock elements after the product fixture and before the expectations to replace selected tasks with authored results. This lets a recipe test exercise downstream conditions, artifact routing, or aggregation without running the producer’s checklet.

<test id="forward-output">
  <product/>
  <mock step="run.0" outcome="pass">
    <properties>
      <property name="status" value="Completed"/>
      <property name="exitcode" type="integer" value="0"/>
    </properties>
    <files><file name="stdout.txt" class="stdout" type="text/plain">42</file></files>
  </mock>
  <expect step="evaluate.0" outcome="pass"/>
</test>

The mock payload uses the ordinary result format: a required outcome and optional properties, options, and files groups. Properties carry result metadata such as scores, status, and visibility. Files support inline text, base64 content, external src attachments, directory attachments, classes, and executable flags. Source attachments are resolved relative to the bundle declaring the test and copied with it.

step uses the same context-free query syntax as expectations. A selector can match several numbered tasks. Each receives an independent copy of the same payload with its concrete task ID. The compiler rejects unmatched selectors and overlapping mock declarations before execution. Repeating a target within one selector does not create multiple results.

The full dependency graph remains in place. A mock applies when its task becomes eligible after its dependencies complete, replacing the entire task execution, including preflight checks, tool requirements, environment setup, and the checklet call. Ancestor tasks still run. Unmocked dependants perform their normal preflight checks against the supplied outcome and consume its properties and artifacts normally. Expectations remain independent checks. Declaring a mock does not replace them.

Python creation and ownership

from momotor.bundles import ProductBundle, RecipeBundle
from momotor.bundles.elements.tests import Expectation, TestCase
from momotor.bundles.elements.steps import Step

recipe = RecipeBundle()
recipe.create(
    steps=[Step(recipe).create(id='run')],
    tests=[TestCase('empty', ProductBundle().create(), (Expectation('run', 'pass'),))],
)

TestCase, Expectation, PropertyExpectation, and FileExpectation are frozen dataclasses. FileExpectation.expected is an ordinary File containing the expected bytes. TestCase.product and TestCase.config are ordinary bundles. TestCase.recreate(target_bundle) copies nested attachments into independent storage. Creating a recipe or configuration also takes an independent copy of its supplied cases. Source ZIP files can then be closed or removed without invalidating the copied cases.

TestCase(..., mocks=(result, ...)) accepts ordinary Result elements. mocks is an immutable tuple, defaults to empty, and is keyword-only so existing positional calls remain valid. A mock’s Result.step_id stores its selector; the execution planner rewrites it to the concrete task ID when materializing results. Mock file contents remain available after their source bundles close.

runtime_projection() returns an independently owned bundle of the same type. Recipe and configuration projections contain runtime data without suite declarations, expected files, mock artifacts, or other test fixtures. Product projections contain only the selected product fixture. snapshot_bundle() copies all authoring content. export_bundle() returns deterministic ZIP bytes without closing the source. Close returned bundles when finished, for example with contextlib.closing().

Reports

TestResultBundle.create(cases=...) accepts frozen CaseReport records from momotor.bundles.elements.testreports. Each record keeps the suite owner and author ID separate, exposes a qualified case_id, and contains ordinary ResultsBundle execution results. Attachments are copied into independent storage under case-specific paths. Reports can be recreated into a new TestResultBundle after their source bundles close.

Reports accept optional root Meta through create(meta=...) and expose it as report.meta, like other bundle types. Metadata survives XML, ZIP, directory, and recreation round trips. Recreation copies descriptions into the destination bundle. Reports without metadata remain valid. The writer’s generator_name option adds generator metadata to reports too.

AssertionReport records keep the assertion ID, original query, resolved target, comparison, verdict, reason code, expected and actual diagnostics, and truncation flag. XML stores diagnostics as JSON scalar literals inside the expected and actual elements, preserving null, boolean, numeric, and string distinctions, including XML control characters. ExecutionFailure records keep the failure origin, reason, task target, and accepted attempt. Input hashes and execution provenance are immutable mappings from names to values. CaseReport.files retains full expected artifacts as ordinary File elements. The report takes independent ownership of those attachments; diagnostic strings can refer to their stable logical names.

The report also stores the format version, completeness, run ID, source hashes, evaluator version, and the ordered selected case IDs. Every selected case must have a record. States are pending, running, completed, and cancelled. Terminal verdicts are passed, failed, error, and cancelled. A complete report cannot contain unfinished cases. Counts and the overall verdict are derived from case records. An empty complete report has verdict error. An incomplete report has no terminal verdict. Cancellation takes precedence, followed by error, failure, and pass.

Self-test declarations and reports are separate from ordinary product and result bundles. Those formats keep their existing meaning. A report only adds case verdicts and diagnostics around ordinary execution results.

Changed in version 9.0.0: This declaration syntax and case-record report replace the expectedResult syntax and grouped-results report that earlier releases shipped as not yet implemented and subject to change. There is no legacy XML migration. See Migration.

API reference

These classes describe tests and reports. They do not execute tests. Fields beginning with an underscore are internal bookkeeping fields.

Immutable self-test declarations and independently owned runtime inputs.

Added in version 9.0.0.

class momotor.bundles.elements.tests.Expectation(step, outcome=None, properties=(), files=(), id=None)

Checks applied independently to every result selected by step.

Parameters:

  • step (str) – Result query, such as run or run.0. The execution planner resolves this query.

  • outcome (str | None) – Optional expected outcome: pass, fail, skip, or error.

  • properties (tuple[PropertyExpectation, ...]) – Property checks for each selected result.

  • files (tuple[FileExpectation, ...]) – File checks for each selected result.

  • id (str | None) – Optional assertion identifier, unique within the test case.

Raises:

ValueError – If the query is empty, the outcome is invalid, or there are no checks.

class momotor.bundles.elements.tests.FileExpectation(name, comparison='exists', expected=None, class_=None, member=None, charset='utf-8', normalize_newlines=False)

A comparison against a result file or a directory member.

Parameters:

  • name (str) – Logical name of the actual output file.

  • comparison (str) – exists, absent, bytes, or text.

  • expected (File | None) – File containing expected content. Required only for bytes and text comparisons.

  • class – Optional class used to select actual files.

  • member (str | None) – Relative member path inside an actual directory attachment. Parent traversal is forbidden.

  • charset (str) – Text decoding charset. Defaults to UTF-8.

  • normalize_newlines (bool) – Whether text comparison normalizes line endings. Defaults to false.

Raises:

ValueError – If the content, path, or text settings do not match the comparison.

class momotor.bundles.elements.tests.PropertyExpectation(name, comparison='eq', value=<object object>, type_=None, values='one', _source=None)

A comparison against a result property.

Parameters:

  • name (str) – Name of the actual property.

  • comparison (str) – exists, absent, eq, ne, lt, le, gt, or ge.

  • value (Any) – Expected scalar. None means explicit null. Presence checks omit the value.

  • type – Optional scalar type name. The Python value must match this type.

  • values (str) – one requires one actual value. all and any allow repeated properties.

Raises:

ValueError – If a comparison, type, or expected value is invalid. Ordered comparisons require finite numbers.

class momotor.bundles.elements.tests.TestCase(id, product, expectations, config=None, meta=None, _owner=None, *, mocks=())

A self-test with fixtures, expectations, and optional mock results.

Creating a recipe or configuration copies the supplied cases into independent storage. recreate() also copies fixture and mock attachments.

Parameters:

  • id (str) – Author ID, unique within the suite. See validate_test_id().

  • product (ProductBundle) – Required product fixture. An empty product bundle is allowed.

  • expectations (tuple[Expectation, ...]) – At least one expectation.

  • config (ConfigBundle | None) – Optional configuration fixture for recipe tests only. It cannot contain tests.

  • meta (Meta | None) – Optional case metadata.

  • mocks (tuple[Result, ...]) – Keyword-only tuple of ordinary results. Each result’s step_id selects tasks to replace.

Raises:

ValueError – If the ID, fixtures, expectations, or mock declarations are invalid.

recreate(target_bundle)

Copy this case and every attachment into independent ownership.

Parameters:

target_bundle (Bundle) – Authoring bundle receiving the copied case.

Return type:

TestCase

Returns:

An immutable case with no dependency on source storage.

momotor.bundles.elements.tests.export_bundle(bundle)

Copy a bundle and its attachments into independent in-memory ownership.

Parameters:

bundle (Bundle) – Source bundle, which remains usable.

Return type:

bytes

Returns:

Deterministic ZIP bytes including every attachment.

momotor.bundles.elements.tests.runtime_projection(bundle)

Create an owned runtime input without test declarations or fixtures.

Parameters:

bundle (TypeVar(BT, bound= Bundle)) – Recipe, configuration, or product authoring bundle.

Return type:

TypeVar(BT, bound= Bundle)

Returns:

Independent input containing only ordinary runtime data.

momotor.bundles.elements.tests.snapshot_bundle(bundle)

Copy a bundle into independent ownership without closing its source.

Parameters:

bundle (TypeVar(BT, bound= Bundle)) – Source bundle.

Return type:

TypeVar(BT, bound= Bundle)

Returns:

A bundle owning its complete ZIP content.

momotor.bundles.elements.tests.validate_test_id(value)

Validate an author ID fitting a qualified 100-character runtime ID.

Parameters:

value (str) – Unqualified author ID.

Raises:

ValueError – If the ID is not an ASCII identifier of at most 93 characters.

Return type:

None

class momotor.bundles.elements.testreports.CaseReport(owner, id, state, verdict=None, duration=None, assertions=(), results=None, failures=(), input_hashes=<factory>, provenance=<factory>, files=())

Execution state, verdict, and evidence for one self-test case.

Parameters:

  • owner (str) – recipe or config.

  • id (str) – Unqualified author ID.

  • state (str) – pending, running, completed, or cancelled.

  • verdict (str | None) – None for pending or running cases. Otherwise passed, failed, error, or cancelled.

  • duration (float | None) – Optional finite, nonnegative duration in seconds.

  • assertions (tuple[AssertionReport, ...]) – Assertion diagnostics in report order.

  • results (ResultsBundle | None) – Optional ordinary execution results bundle.

  • failures (tuple[ExecutionFailure, ...]) – Execution failures.

  • input_hashes (Mapping[str, str]) – Hashes of this case’s inputs.

  • provenance (Mapping[str, str]) – Execution metadata keyed by name.

  • files (tuple[File, ...]) – Full diagnostic artifacts as ordinary file elements.

Raises:

ValueError – If identity, state, verdict, duration, or evidence is inconsistent. A passed case cannot contain failures.

recreate()

Return a case report owning an independent copy of every result attachment.

Return type:

CaseReport

property case_id: str

Qualified runtime identity, preserving owner and author ID separately.

class momotor.bundles.elements.testreports.AssertionReport(id, step, target, kind, comparison, verdict, reason='', expected=None, actual=None, truncated=False)

Verdict and scalar diagnostics for one resolved assertion target.

Parameters:

  • id (str) – Assertion identifier.

  • step (str) – Original result query.

  • target (str) – Resolved task target.

  • kind (str) – Assertion kind recorded by the evaluator.

  • comparison (str) – Comparison recorded by the evaluator.

  • verdict (str) – passed, failed, error, pending, or cancelled.

  • reason (str) – Evaluator reason code, or an empty string.

  • expected (str | int | float | bool | None) – Expected diagnostic value: string, integer, finite float, boolean, or null.

  • actual (str | int | float | bool | None) – Actual diagnostic value with the same supported types as expected.

  • truncated (bool) – Whether diagnostics were shortened. Full content can be stored in case files.

Raises:

ValueError – If the verdict is invalid or diagnostics contain unsupported values.

class momotor.bundles.elements.testreports.ExecutionFailure(origin, reason, target=None, attempt=None)

An execution failure associated with a task attempt when known.

Parameters:

  • origin (str) – Nonempty identifier for the failure source.

  • reason (str) – Nonempty failure reason.

  • target (str | None) – Optional task target.

  • attempt (str | None) – Optional accepted attempt identifier.

Raises:

ValueError – If origin or reason is empty.