evaluate step

step evaluate

Compare two files and optionally generate a report with a side-by-side diff depending on the level of difference.

See also

Package mtrchk-org-momotor-evaluate-compare

Python package containing the checklet implementing this step. The recipe requires version ~=2.0, this document is based on version 2.0.0.

preflight@scheduler option

A preflight check handled by the scheduler. This allows recipes to indicate situations in which the step does not have to be executed.

See Scheduler preflight option for the documentation of this option.

Type:

string

Required:

False

Multiple:

True

All:

True

Value:

%notall pass => fail-hidden { "status": "Not executed due to error", "reason": ${prop[@result#run.$@:reason]} }

tasks@scheduler option

Enable multiple tasks for this step. If not provided, a single task is generated for this step.

See Scheduler tasks option for the documentation of this option.

Type:

string

Required:

False

Multiple:

False

Value:

?

secret option

Dynamic secret property value for the result. Momotor LTI will not show steps with a secret property set to true to learners, but will still show that step to reviewers.

The option value can be one of the following:

  • always or true: adds a secret property with value true to the result.

  • false: do not add a secret property to the result.

  • on-fail: adds a secret property with value true to the result if the step fails.

  • on-pass: adds a secret property with value true to the result if the step passed.

  • default: inherits the value from the next provider. If the last provider is ‘default’ no property will be added.

Note that this only adds a secret property with value true, but never one with value false. This ensures that a default defined in a recipe will be used unless the step or config explicitly override it.

The value can contain reference placeholders to inherit the value from another step’s options or properties.

Type:

Any

Required:

False

Multiple:

False

Default:

true

label option

Label for this step. Momotor LTI uses labels as the header of the step in the result view. If a step does not have a label, Momotor LTI will use the step id as the label.

Can contain reference placeholders and task id placeholders.

Type:

Any

Required:

False

Multiple:

False

Default:

Evaluate test case $1#

feedback-pass option

Feedback to prepend to the report property when this step passes.

Can contain reference placeholders and task id placeholders.

Type:

Any

Required:

False

Multiple:

False

Default:

No default

feedback-fail option

Feedback to prepend to the report property when this step fails.

Can contain reference placeholders and task id placeholders.

Type:

Any

Required:

False

Multiple:

False

Default:

No default

font option

Font name(s) to use for the diff output. Multiple fonts can be specified, separated by commas

Type:

Any

Required:

False

Multiple:

False

Default:

Consolas,"Courier New",monospace

left-header option

Diff table left header

Type:

Any

Required:

False

Multiple:

False

Default:

Left

right-header option

Diff table right header

Type:

Any

Required:

False

Multiple:

False

Default:

Right

diff-tab-size option

Expand tabs in the diff to this many spaces

Type:

integer

Required:

False

Multiple:

False

Default:

4

diff-columns option

Diff table column width

Type:

integer

Required:

False

Multiple:

False

Default:

60

diff-context-lines option

Amount of context to show on the diff. -1 for full diff

Type:

integer

Required:

False

Multiple:

False

Default:

4

ignore-final-newline option

Ignore difference in final newline

Type:

boolean

Required:

False

Multiple:

False

Default:

False

ignore-trailing-whitespace option

Ignore difference in trailing whitespace for all lines

Type:

boolean

Required:

False

Multiple:

False

Default:

False

sort-left option

Sort the left file before comparison. Either a boolean value (yes, no, true, false, etc.), or human, nat, os, real to sort using a natural sort algorithm.

See the documentation of the natsort package for details on the natural sort algorithms.

Note

Sorting requires the entire file to be read into memory, which may be undesirable for large files.

Type:

string

Required:

False

Multiple:

False

Default:

False

sort-right option

Sort the right file before comparison. Either a boolean value (yes, no, true, false, etc.), or human, nat, os, real to sort using a natural sort algorithm.

See the documentation of the natsort package for details on the natural sort algorithms.

Note

Sorting requires the entire file to be read into memory, which may be undesirable for large files.

Type:

string

Required:

False

Multiple:

False

Default:

False

pass-level option

When to pass the compare:

  • identical: Contents must be fully identical

  • case: Contents may differ in whitespace, blank lines and casing

  • whitespace: Contents may differ in whitespace only (excluding blank lines)

  • blank: Contents may differ in whitespace and blank lines

  • different: Contents may differ completely. This will make this step always pass.

Later listed passing levels include earlier listed levels, e.g. if the detected level is whitespace, the files may also include differences in case.

Type:

Any

Required:

False

Multiple:

False

Default:

identical

score option

Score when passing a level. A (possibly empty) comma separated list of scores for each level:

[ [ identical-score ]
    [ "," [ case-score ]
        [ "," [ whitespace-score ]
            [ "," [ blank-score ]
                [ "," [ different-score ] ]
            ]
        ]
    ]
]

Missing values at the end will be the same as the last value in the list, an empty value means no score will be given for that level.

For example, a score definition of 10,9,8 will give a score of 10 when the files are identical, 9 when they differ only in case, and 8 otherwise.

Type:

Any

Required:

False

Multiple:

False

Default:

Empty string

score-fail option

Score when failing to meet the required level. Either a constant score value, string constant none to indicate no score when failing, or keep to keep the score from the score list.

Type:

Any

Required:

False

Multiple:

False

Default:

none

fail-with-reason option

When failing, set the reason property. If false the reason is listed as the status

Type:

boolean

Required:

False

Multiple:

False

Default:

True

left-file-ref option

Reference to the left-side file of the comparison. Should resolve to exactly one file, otherwise a CheckletError is raised.

The reference is parsed with find_files(), see that method for details.

Type:

Any

Required:

True

Multiple:

False

Default:

@result#run.$@:stdout

right-file-ref option

Reference to the right-side file of the comparison. Should resolve to exactly one file, otherwise a CheckletError is raised.

The reference is parsed with find_files(), see that method for details.

Type:

Any

Required:

True

Multiple:

False

Default:

No default

show-diff-level option

Level of difference to add a diff report, either a compare level (identical, case, whitespace, blank, different), fail to show diff on fail outcome, or none to never show diff

Type:

Any

Required:

False

Multiple:

False

Default:

fail

missing-file-reason option

Fail reason when a file is missing

Type:

Any

Required:

False

Multiple:

False

Default:

Unable to evaluate