Checklet

checklet mtrchk.org.momotor.evaluate.compare.EvaluateCompare(recipe, product, config, results, task_id, work_dir=None)

Compare two files and optionally generate a report with a side-by-side diff depending on the level of difference.

tasks@scheduler option

Enable multiple tasks for this step. If not provided, a single task is generated for this step.

See Scheduler tasks option for the documentation of this option.

Type:

string

Required:

False

Multiple:

False

Location:

config, recipe, step

Default:

No default

Provided by: StepTasksCheckletMixin

secret option

Dynamic secret property value for the result. Momotor LTI will not show steps with a secret property set to true to learners, but will still show that step to reviewers.

The option value can be one of the following:

  • always or true: adds a secret property with value true to the result.

  • false: do not add a secret property to the result.

  • on-fail: adds a secret property with value true to the result if the step fails.

  • on-pass: adds a secret property with value true to the result if the step passed.

  • default: inherits the value from the next provider. If the last provider is ‘default’ no property will be added.

Note that this only adds a secret property with value true, but never one with value false. This ensures that a default defined in a recipe will be used unless the step or config explicitly override it.

The value can contain reference placeholders to inherit the value from another step’s options or properties.

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

default

Provided by: SecretOptionMixin

label option

Label for this step. Momotor LTI uses labels as the header of the step in the result view. If a step does not have a label, Momotor LTI will use the step id as the label.

Can contain reference placeholders and task id placeholders.

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

No default

Provided by: LabelOptionMixin

feedback-pass option

Feedback to prepend to the report property when this step passes.

Can contain reference placeholders and task id placeholders.

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

No default

Provided by: FeedbackOptionMixin

feedback-fail option

Feedback to prepend to the report property when this step fails.

Can contain reference placeholders and task id placeholders.

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

No default

Provided by: FeedbackOptionMixin

font option

Font name(s) to use for the diff output. Multiple fonts can be specified, separated by commas

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

Consolas,"Courier New",monospace

Provided by: RenderDiffMixin

left-header option

Diff table left header

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

Left

Provided by: RenderDiffMixin

right-header option

Diff table right header

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

Right

Provided by: RenderDiffMixin

diff-tab-size option

Expand tabs in the diff to this many spaces

Type:

integer

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

4

Provided by: RenderDiffMixin

diff-columns option

Diff table column width

Type:

integer

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

60

Provided by: RenderDiffMixin

diff-context-lines option

Amount of context to show on the diff. -1 for full diff

Type:

integer

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

4

Provided by: RenderDiffMixin

ignore-final-newline option

Ignore difference in final newline

Type:

boolean

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

False

Provided by: CompareMixin

ignore-trailing-whitespace option

Ignore difference in trailing whitespace for all lines

Type:

boolean

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

False

Provided by: CompareMixin

sort-left option

Sort the left file before comparison. Either a boolean value (yes, no, true, false, etc.), or human, nat, os, real to sort using a natural sort algorithm.

See the documentation of the natsort package for details on the natural sort algorithms.

Note

Sorting requires the entire file to be read into memory, which may be undesirable for large files.

Type:

string

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

False

Provided by: CompareMixin

sort-right option

Sort the right file before comparison. Either a boolean value (yes, no, true, false, etc.), or human, nat, os, real to sort using a natural sort algorithm.

See the documentation of the natsort package for details on the natural sort algorithms.

Note

Sorting requires the entire file to be read into memory, which may be undesirable for large files.

Type:

string

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

False

Provided by: CompareMixin

pass-level option

When to pass the compare:

  • identical: Contents must be fully identical

  • case: Contents may differ in whitespace, blank lines and casing

  • whitespace: Contents may differ in whitespace only (excluding blank lines)

  • blank: Contents may differ in whitespace and blank lines

  • different: Contents may differ completely. This will make this step always pass.

Later listed passing levels include earlier listed levels, e.g. if the detected level is whitespace, the files may also include differences in case.

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

identical

Provided by: CompareMixin

score option

Score when passing a level. A (possibly empty) comma separated list of scores for each level:

[ [ identical-score ] 
    [ "," [ case-score ] 
        [ "," [ whitespace-score ] 
            [ "," [ blank-score ]  
                [ "," [ different-score ] ] 
            ] 
        ] 
    ] 
]

Missing values at the end will be the same as the last value in the list, an empty value means no score will be given for that level.

For example, a score definition of 10,9,8 will give a score of 10 when the files are identical, 9 when they differ only in case, and 8 otherwise.

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

Empty string

Provided by: CompareMixin

score-fail option

Score when failing to meet the required level. Either a constant score value, string constant none to indicate no score when failing, or keep to keep the score from the score list.

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

none

Provided by: CompareMixin

fail-with-reason option

When failing, set the reason property. If false the reason is listed as the status

Type:

boolean

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

True

Provided by: CompareMixin

left-file-ref option

Reference to the left-side file of the comparison. Should resolve to exactly one file, otherwise a CheckletError is raised.

The reference is parsed with find_files(), see that method for details.

Type:

Any

Required:

True

Multiple:

False

Location:

config, step, recipe

Default:

No default

right-file-ref option

Reference to the right-side file of the comparison. Should resolve to exactly one file, otherwise a CheckletError is raised.

The reference is parsed with find_files(), see that method for details.

Type:

Any

Required:

True

Multiple:

False

Location:

config, step, recipe

Default:

No default

show-diff-level option

Level of difference to add a diff report, either a compare level (identical, case, whitespace, blank, different), fail to show diff on fail outcome, or none to never show diff

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

fail

missing-file-reason option

Fail reason when a file is missing

Type:

Any

Required:

False

Multiple:

False

Location:

config, step, recipe

Default:

Nothing to compare