Utility classes and functions

class mtrchk.org.momotor.mixin.exec.capture.Capture(cap_stdin=True, cap_stdout=True, cap_stderr=True, *, limits=None, max_size=1048576)

Send and capture I/O on stdin/stdout/stderr and records timestamps when each line was sent or received.

Truncates stream if it gets too big. If a stream is truncated, the last record for that stream will contain the boolean value False.

Spools to a buffer in memory until total data size gets bigger than max_size

any_truncated()

Whether any stream was truncated

Return type:

bool

lines(t)

The total number of lines captured for the stream with id t, where t is one of STDIN, STDOUT or STDERR

Return type:

int

playback()

Start playback of I/O on stdin/stdout/stderr

Return type:

Iterable[dict]

record()

Start recording I/O on stdin/stdout/stderr

Return type:

CaptureRecorderBase

size(t)

The total number of bytes captured for the stream with id t, where t is one of STDIN, STDOUT or STDERR

Return type:

int

truncated(t)

Whether the stream with id t was truncated, where t is one of STDIN, STDOUT or STDERR

Return type:

bool

class mtrchk.org.momotor.mixin.exec.capture.CaptureLimits(max_stdout_lines=None, max_stdout_bytes=None, max_stderr_lines=None, max_stderr_bytes=None)

Limits for capturing stdout/stderr. The stream will be truncated (no more output captured) if any of the limits are exceeded.

max_stderr_bytes: int | None = None

Maximum number of bytes to capture from stderr

max_stderr_lines: int | None = None

Maximum number of lines to capture from stderr

max_stdout_bytes: int | None = None

Maximum number of bytes to capture from stdout

max_stdout_lines: int | None = None

Maximum number of lines to capture from stdout

class mtrchk.org.momotor.mixin.exec.capture.CapturePlayback(file)

A class for playing back I/O on stdin/stdout/stderr from a tempfile.SpooledTemporaryFile as recorded by CaptureRecorder.

class mtrchk.org.momotor.mixin.exec.capture.CapturePlaybackBase(file)

A base class for playing back I/O on stdin/stdout/stderr.

mtrchk.org.momotor.mixin.exec.capture.CaptureRecord

TypeAlias for a capture record. The dictionary contains the following keys:

  • @ (str): The timestamp in seconds since execution start. Contains a Decimal timestamp.

  • 0 (int): The data on stdin. Contains a str or bytes value.

  • 1 (int): The data on stdout. Contains a str or bytes value, or the boolean value False if the stream was truncated at this timestamp

  • 2 (int): The data on stderr. Contains a str or bytes value, or the boolean value False if the stream was truncated at this timestamp

class mtrchk.org.momotor.mixin.exec.capture.CaptureRecorder(file, capture, size, lines, trunc, limits)

A class for recording I/O on stdin/stdout/stderr to a tempfile.SpooledTemporaryFile.

Parameters:

  • file (SpooledTemporaryFile) – The file to write the capture to

  • capture (tuple[bool, bool, bool]) – A tuple of booleans indicating which streams to capture

  • size (list[int]) – A list of 3 integers where the size of the captured data for each stream is recorded

  • lines (list[int]) – A list of 3 integers where the number of lines of the captured data for each stream is recorded

  • trunc (list[bool]) – A list of 3 booleans indicating whether the stream was truncated

  • limits (CaptureLimits) – The limits for capturing stdout/stderr

add(*args)

Add data to the capture.

Expected to be called with 3 arguments, one for each stream. If a stream is not captured, the argument should be None.

TIMESTAMP_PRECISION: ClassVar[Decimal] = Decimal('0.001')

The precision to use for timestamps

class mtrchk.org.momotor.mixin.exec.capture.CaptureRecorderBase(file, capture, size, lines, trunc, limits)

A base class for recording I/O on stdin/stdout/stderr. The base class only records the size and number of lines of the captured data for each stream, and checks capture limits. It does not record the actual data.

Parameters:

  • file – The file to write the capture to

  • capture (tuple[bool, bool, bool]) – A tuple of booleans indicating which streams to capture

  • size (list[int]) – A list of 3 integers where the size of the captured data for each stream is recorded

  • lines (list[int]) – A list of 3 integers where the number of lines of the captured data for each stream is recorded

  • trunc (list[bool]) – A list of 3 booleans indicating whether the stream was truncated

  • limits (CaptureLimits) – The limits for capturing stdout/stderr

add(*args)

Add data to the capture.

Expected to be called with 3 arguments, one for each stream. If a stream is not captured, the argument should be None.

class mtrchk.org.momotor.mixin.exec.capture.JSONCaptureEncoder(*, ensure_ascii=False, sort_keys=False, **kwargs)

JSON encoder to convert CaptureRecord objects to JSON. Differs from the json.JSONEncoder in that it:

  • Generates more compact JSON, by overriding the item_separator and key_separator attributes

  • ensure_ascii defaults to False

  • sort_keys defaults to False

  • Converts decimal.Decimal objects to floats

default(o)

Convert the record to JSON

Return type:

Any

mtrchk.org.momotor.mixin.exec.capture.STDERR: Literal[0, 1, 2] = 2

Constant to reference stderr

mtrchk.org.momotor.mixin.exec.capture.STDIN: Literal[0, 1, 2] = 0

Constant to reference stdin

mtrchk.org.momotor.mixin.exec.capture.STDOUT: Literal[0, 1, 2] = 1

Constant to reference stdout

class mtrchk.org.momotor.mixin.exec.monitor.ChildUsage(utime=0.0, stime=0.0, mem=0)
mtrchk.org.momotor.mixin.exec.monitor.Limits

deprecated alias of RunLimits

class mtrchk.org.momotor.mixin.exec.monitor.Reason(value)

An enum for the various reasons a process may be terminated

class mtrchk.org.momotor.mixin.exec.monitor.ResourceMonitor(parent_pid, process, limits, shell, *, mem_maps_filter=True)

A thread to monitor another process and enforce runtime limits.

join(timeout=None)

Wait until the thread terminates.

This blocks the calling thread until the thread whose join() method is called terminates – either normally or through an unhandled exception or until the optional timeout occurs.

When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call is_alive() after join() to decide whether a timeout happened – if the thread is still alive, the join() call timed out.

When the timeout argument is not present or None, the operation will block until the thread terminates.

A thread can be join()ed many times.

join() raises a RuntimeError if an attempt is made to join the current thread as that would cause a deadlock. It is also an error to join() a thread before it has been started and attempts to do so raises the same exception.

Return type:

None

run()

Check usage of child processes

stop()

Stop the monitor and kill the monitored processes

Return type:

None

reason: Reason | None

Termination reason

usage: Usage

Current usage of the process and it’s children

class mtrchk.org.momotor.mixin.exec.monitor.RunLimits(timeLimit=None, blockLimit=None, memoryLimit=None, processLimit=None)

Runtime limits for a process executed with MPopen

blockLimit: int | None = None

Maximum time a process may be blocked (in seconds)

memoryLimit: int | None = None

Maximum memory usage (in bytes)

processLimit: int | None = None

Maximum number of processes that can be spawned

timeLimit: int | None = None

Maximum time process is allowed to run (in seconds)

class mtrchk.org.momotor.mixin.exec.monitor.Usage(wctime=0.0, utime=0.0, stime=0.0, mem=0, proc=0)
mem: int = 0

Memory usage

proc: int = 0

Process count

stime: float = 0.0

CPU system time

utime: float = 0.0

CPU user time

wctime: float = 0.0

Wall clock time

mtrchk.org.momotor.mixin.exec.monitor.verbose_reason: dict[Reason, str] = {Reason.BLOCKLIMIT_EXCEEDED: 'Blocked or deadlocked', Reason.MEMORYLIMIT_EXCEEDED: 'Memory usage limit exceeded', Reason.PROCESSLIMIT_EXCEEDED: 'Too many processes started', Reason.TIMELIMIT_EXCEEDED: 'Time limit exceeded'}

Verbose termination reason strings for use in reports and as reason property

class mtrchk.org.momotor.mixin.exec.mpopen.MPopen(args, limits, mem_maps_filter=True, shell=False, **kwargs)

A monitored version of subprocess.Popen

Checklets should use the ExecutionMixin or CapturedExecutionMixin mixin instead of this class directly.

communicate(input_file, *, input_delay_ms=None)

A generator variant of subprocess.Popen.communicate() that keeps the interleaving on stdin/stdout/stderr intact.

Input will be read from input_stream, output on stdout and stderr will be collected.

Return type:

Generator[tuple[AnyStr, AnyStr, AnyStr], None, None]

communicate_readonly()

A generator variant of subprocess.Popen.communicate() that keeps the interleaving of output on stdout/stderr intact.

No input will be provided on stdin, output on stdout and stderr will be collected.

Return type:

Generator[tuple[AnyStr, AnyStr], None, None]

status_properties()

Get status to use in checklet properties.

Return type:

dict[str, Union[str, bool, None, NewType(NoContentType, object), bytes, int, float, Decimal, Sequence[str], set[str]]]

Returns:

A dict with the returncode, usage and reason values usable as checklet properties.

property outcome: Outcome

An Outcome value based on the reason and returncode.

property reason: Reason | None

The termination reason, if any

property returncode: int | None

The process return code as reported by subprocess.Popen

property usage: Usage | None

The process usage, or None if the monitor hasn’t run yet