Momotor lti-stepreport checklet¶

The mtrchk-org-momotor-lti-stepreport package provides a checklet to generate a report base on results of individual steps.

Checklet¶

checklet mtrchk.org.momotor.lti.stepreport.RenderStepReport(recipe, product, config, results, task_id, work_dir=None)¶

Renders a report to summarize other steps, and collects the scores of these step to generate a final score.

Steps to include in the report need to be grouped using the group option. The report uses information from the step outcome and properties for the report. The following step result properties are used:

label: Used as the description in the report. If no label property exists, but a label option is provided, that is used as the description. If neither exists, the step id is used.
secret: By default, secret steps are not shown in reports.
score: Score of this step.
max-score: Maximum possible score for this step. This is used and needed when scaling the score.
reason: Reason for failure.

With the group option, it is possible to override the score, max-score and whether to include secret steps and some other options for each group.

label option¶

Label for this step. Momotor LTI uses labels as the header of the step in the result view. If a step does not have a label, Momotor LTI will use the step id as the label.

Can contain reference placeholders and task id placeholders.

Type:: Any
Required:: False
Multiple:: False
Location:: config, step, recipe
Default:: No default

Provided by: LabelOptionMixin

secret option¶

Dynamic secret property value for the result. Momotor LTI will not show steps with a secret property set to true to learners, but will still show that step to reviewers.

The option value can be one of the following:

always or true: adds a secret property with value true to the result.
false: do not add a secret property to the result.
on-fail: adds a secret property with value true to the result if the step fails.
on-pass: adds a secret property with value true to the result if the step passed.
default: inherits the value from the next provider. If the last provider is ‘default’ no property will be added.

Note that this only adds a secret property with value true, but never one with value false. This ensures that a default defined in a recipe will be used unless the step or config explicitly override it.

The value can contain reference placeholders to inherit the value from another step’s options or properties.

Type:: Any
Required:: False
Multiple:: False
Location:: config, step, recipe
Default:: default

Provided by: SecretOptionMixin

group option¶

The groups of steps to report on. Multiple groups can be defined, each group consisting of one or more steps. Format:

step_groups    ::= queries [ "=>" group_settings ]
group_settings ::= key_value_pair [ " " key_value_pair ]
key_value_pair ::= key [ "=" [ value ] ]
key            ::= "name" | "pass-score" | "fail-score" | "max-score"
                   | "show-step-score" | "group-max-score"
                   | "group-scaled-score" | "group-passing-score"
                   | "include-skipped" | "include-secret"
                   | "include-status"
value          ::= "true" | "false" | "null" | "none" | number
                   | "'" string "'" | '"' string '"' | string

With step_groups containing a result query to select one or more step results to include in this group, optionally followed by a space separated list of key value pairs with options for the group. String values must be quoted if they contain a space. Both single and double quotation marks can be used.

The following keys are used:

name: group name. Used as the description field in the test report
pass-score: score for passing steps. If not provided, the score property of the step itself is used.
fail-score: score for failing steps. If not provided, the score property of the step itself is used.
max-score: maximum score for steps. If not provided, the max-score property of the step itself is used.
show-step-score: true to show scores for steps. By default, scores are hidden when group score scaling is used
group-max-score: maximum score for the group
group-scaled-score: scaled score for the group
group-passing-score: score required for a pass
include-skipped: true to include skipped steps in the group. Overrides include-skipped-results option for this group
include-secret: true to include secret steps in the group. Overrides include-secret-results option for this group
include-status: true to include the status property in the step description. For failed steps, a status property overrides reason if both are present.

A key that’s not followed by an = is treated as a boolean option with value true. A key that’s followed by an = but no value is treated as none.

For example:

evaluate.* => name="Evaluate" pass-score= fail-score=0 max-score=6 
              scaled-score=10 include-secret group-passing-score=6

Note

Earlier versions of this checklet used a result selector instead of a result query for step_groups. This syntax is deprecated and will be removed in a future version.

To convert, change the selector to a query by removing the result[#...] part, i.e. result[#step] becomes step.

Type:: Any
Required:: False
Multiple:: True
All:: False
Location:: config, step, recipe
Default:: Empty string

base-score option¶

Base score. Any score in the report will be added to this score

Type:: integer
Required:: False
Multiple:: False
Location:: config, step, recipe
Default:: No default

expected-max-score option¶

Expected maximum score. Overrules the maximum score provided by the report

Type:: integer
Required:: False
Multiple:: False
Location:: config, step, recipe
Default:: No default

scaled-max-score option¶

Maximum score after scaling. If not provided, no scaling is applied

Type:: integer
Required:: False
Multiple:: False
Location:: config, step, recipe
Default:: No default

pass-with-score option¶

Minimum score (after scaling) required to PASS. If undefined, always passes

Type:: integer
Required:: False
Multiple:: False
Location:: config, step, recipe
Default:: No default

pass-required-sel option¶

One ore more selectors that must all resolve to true for a pass

Type:: Any
Required:: False
Multiple:: True
All:: False
Location:: config, step, recipe
Default:: No default

include-secret-results option¶

If true, include secret results in the report

Type:: boolean
Required:: False
Multiple:: False
Location:: config, step, recipe
Default:: False

include-skipped-results option¶

If true, include skipped results in the report

Type:: boolean
Required:: False
Multiple:: False
Location:: config, step, recipe
Default:: True

adjust-scale-for-secrets option¶

If true, adjust the scaled score for secret results

Type:: boolean
Required:: False
Multiple:: False
Location:: config, step, recipe
Default:: True

mirror-reasons option¶

If true and all steps have the same reason, use that reason as the final reason

Type:: boolean
Required:: False
Multiple:: False
Location:: config, step, recipe
Default:: True