momotor.shared.resources

Resources match jobs with workers. A job states the resources it needs. A worker states the resources it offers. Either side can also reject a resource. The broker uses the match result to pick a worker for each task.

A resource definition is a set of named groups. Each group holds items. For example, the group language can hold the items python and java. Items are tags: a worker either offers a tag or it does not. Numeric values, such as an amount of memory, are not supported yet.

Match a job with a worker

Parse each definition with from_string(). Then call match() on the job resources with the worker resources as the argument.

from momotor.shared.resources import NoMatch, Resources

job = Resources.from_string('language: python; python: 3.12')
worker = Resources.from_string('language: ?python, ?java; python: ?3.11, ?3.12')

try:
    score = job.match(worker)
except NoMatch:
    print('this worker cannot run the job')
else:
    print(score)

The example prints -1.0. The match returns a number:

  • A lower number is a better match.

  • Negative infinity is the strongest possible match.

  • Positive infinity is the weakest possible match.

  • NoMatch means the worker cannot run the job at all.

The ? prefix in the worker definition marks a tag as optional. The worker offers the tag, but a job does not need to ask for it. A tag without a prefix is required. A tag with a ~ prefix is excluded. The sections below define the syntax and every matching rule.

Combine and subtract definitions

union() merges definitions. difference() removes the items of the later definitions from the first one:

all_languages = Resources.from_string('language: python, java')
java_only = Resources.from_string('language: java')
print(Resources.difference(all_languages, java_only).as_str())

The example prints language: python.

Resource definition syntax

A worker defines the resources it can provide, whereas a job defines the resources it requires. The syntax for both these definitions is the same:

group[":"]item[","item]*

The group is the name of a group, consisting of a series of alphanumeric ASCII characters, dashes (-), underscores (_), dots (.) and/or whitespace. Any other character can be escaped with a backslash character (\). It is also possible to use quotes, where both single (') and double quotes (") can be used. Within quotes, any character is valid. Quotes can start and end anywhere in the string, but need to be matched in pairs. Whitespace is normally stripped from the beginning and end of group names, but using quotes it will be retained.

The following are all valid group names:

  • name - Nothing special here

  • dotted.name - The dot is valid

  • spaced name - Whitespace is also valid as part of the group name

  • "email@example.com" - The @ is not an allowed character, so the string needs to be quoted

  • email"@"example.com - Only quoting the @

  • email\@example.com - Escaping the @

  • backslashed\\name - The backslash itself also needs to be escaped

  • "trailing whitespace " - The whitespace at the end is retained

The first character that is not a valid character for a group name ends the group name field, so for example a colon (:), a tilde (~) or a less-than (<) character all end the group name field.

A group name field is followed by an optional colon (:) and a list of items. The colon is only needed if otherwise the transition from the group name field to the item list is indeterminate.

Items are separated by commas (,). At least one item is required. A semi-colon (;), newline character or end of string ends the list of items.

The item is the definition for an item. Items can contain any character except commas (,), colons (:), semi-colons (;), single (') and double quotes ("), and backslashes (\). Any of these characters can be escaped using the backslash (\), or included in quotes. Just like with group names, whitespace is stripped, but will be retained within quotes.

The following are all valid items:

  • string - Nothing special here

  • < 1000 - The < and space are both valid characters within an item

  • five<6 - So this is valid too

  • list",of,strings" - We need to quote the string to use commas in an item, item value will be list,of,strings

  • list\,of\,strings - The commas can also be escaped, item value will also be list,of,strings

Multiple groups can be separated by a newline character or a semi-colon (;)

These are all valid full resource definitions:

  • lang: python, java - A group named lang, containing two items python and java The colon is needed to separate the group name from the items here

  • memory < 1 GiB - A group named memory, containing one item < 1 GiB A colon separating the group name from the item is not needed here, but would be allowed

  • version <3, >=5 - A group named version, containing two items <3 and >=5

  • test case: "one; two, or more" - A group named test case, containing one item one; two, or more

  • one: 1; two: 2 - Two groups, one named one containing the item 1, and one named two containing the item 2

  • one: 1
    two: 2
    

    Identical to one: 1; two: 2

Tag type resources

There are three types of tags:

  • required tags: defined as a name without prefix: tag

  • optional tags: prefixed with a question mark: ?tag

  • excluded tags: prefixed with a tilde: ~tag

Workers provide tag resources. The worker can indicate tags it requires from the jobs, tags that are optional, and excluded tags that it will not accept.

Jobs require tag resources. The job can indicate tags it requires from the worker, tags that are optional, and excluded tags that it does not want from workers.

Matches have a strength, indicated as a floating point number ranging from negative infinity to positive infinity, making it possible to compare them. The strongest possible match is represented as negative infinity, and the weakest match is positive infinity.

If both the worker and job define the same required tag, this is considered the strongest possible match. If both worker and job define the tag as optional the match is the weakest possible match.

If a tag is excluded by either of the parties, while the other requires or optionally defines the tag, the match is rejected. If both exclude the tag, the tag is ignored.

The following table describes all possible matches:

task \ worker

-

tag

?tag

~tag

-

NoMatch

tag

NoMatch

STRONGEST

STRONG

NoMatch

?tag

WEAK

WEAKEST

NoMatch

~tag

NoMatch

NoMatch

  • -: worker or task does not have this tag defined

  • blank cell: ignore, no change in match value

  • NoMatch: indicates the NoMatch exception will be raised

  • any other value: corresponding value for the match.

If every resource tag match is a blank cell in the above table, the final match will be NEUTRAL which is weaker than a strong match, but stronger than a weak match.

If multiple tags within the same resource group match, the strongest match is returned.

If multiple resource groups match, the weakest group determines the final value.

Note that the constants are defined the other way around: WEAKEST > STRONGEST. The reason for these counter-intuitive values is the planned handling of value type resources, where a lower valued match will indicate a better match.

Example: Matching programming languages

The worker defines the following tags

language: ?java, ?python
java: ?8, ?11, ?12, ?13
python: ?3.6, ?3.7

This worker indicates it can process Java and Python, and which versions for each

The jobs:

  • language: java
    java: 12
    

    This job indicates it wants Java 12. The language group matches STRONG, the java group matches NEUTRAL, so the total match is NEUTRAL, because the weakest group defines the total match. Since the job does not have a python requirement, and the worker defines all python group tags as optional, this is not considered at all for the match.

  • language: java
    java: 14
    

    This job indicates it wants Java 14. The language group matches STRONG, however, the java group produces a NoMatch

  • language: java
    java: ?14
    

    This job indicates it would like Java 14. The language group match matches STRONG again, the java group matches NEUTRAL, so the total match is NEUTRAL again

  • arch: x86
    

    This job does not mention any programming language, it just asks for an arch tag of x86. The worker does not define a arch resource at all, so this will be a NoMatch.

  • arch: ?x86
    

    This job also does not mention any programming language, it just asks for an optional arch tag of x86. The worker does not define a arch resource group at all, it does not exclude it either. The final match will be NEUTRAL

  • arch: ~x86
    

    This job explicitly rejects arch of x86. However, since our worker does not define an arch resource, this will also result in a NEUTRAL match

Value type resources

Not implemented yet.

This could be used to match dynamic resources like memory or disk space.

An example of a required values definition would be:

  • ram >= 1 GiB - request a worker with at least 1 GiB of free RAM

An example of a provided values definition would be:

  • ram = 2147483648 - the worker has this amount of available RAM

API reference

class momotor.shared.resources.Resources(definition=None)

A set of resource groups for a task or a worker.

Create instances with from_string(), from_dict(), or from_key_value(). Compare a task’s resources with a worker’s resources with match().

Items are boolean tags:

  • tag: required tag

  • ?tag: optional tag

  • ~tag: excluded tag

The narrative documentation for this module has the full table of match values for every combination of task and worker tags.

Parameters:

definition (dict[str, ResourceGroup] | None) – A dictionary of group names to resource groups

classmethod difference(*resources)

Subtract Resources objects from the first one

Return type:

Self

Returns:

The resulting resources

Added in version 2.1.0.

classmethod from_dict(definition)

Factory to create a Resources object from a dict of group names to group string definitions

Return type:

Self

classmethod from_key_value(key, value)

Factory to create a Resources object from a key and values

Parameters:
  • key (str) – Group name

  • value (str | Iterable[str]) – Group definition string or list of strings

Return type:

Self

classmethod from_string(value)

Factory to create a Resources object from one or more strings

Parameters:

value (str | Iterable[str]) – A string or iterable of strings containing resource definitions including a group name

Return type:

Self

Returns:

The parsed resources

Raises:

ValueError if the string is an invalid resources definition

classmethod union(*resources)

Merge multiple Resources objects into a new one

Return type:

Self

Returns:

The merged resources

as_str(*, multiline=False, compact=False)

Convert these resources into a string that can be parsed back into a Resources object using from_string()

Parameters:
  • multiline (bool) – if True, use newlines to split groups, otherwise uses semicolons

  • compact (bool) – if True, returns a compact representation, otherwise adds more whitespace

Return type:

str

Returns:

string representation of the resources

as_str_tuples()
Return type:

Iterable[tuple[str, str]]

as_tuples()
Return type:

Iterable[tuple[str, tuple[str, ...]]]

get(key)

Get the ResourceGroup by name

Parameters:

key (str) – Group name

Return type:

ResourceGroup

Returns:

The resource group matching the group name

group_names()

Returns a list of group names

Return type:

list[str]

Returns:

group names

match(worker_resource)

Returns the match between these resources and the provided worker’s resources.

Parameters:

worker_resource (Resources) – The worker’s resources

Return type:

float

Returns:

match value, the lower the value, the better the match is (ranges from -inf to +inf). If nothing matches, but also nothing is excluded, will return NEUTRAL

Raises:

NoMatch if no match can be made (for example due to an exclude)

class momotor.shared.resources.group.ResourceGroup(items=None)

A group of resources.

classmethod create(value)
Return type:

Self

classmethod difference(*groups)

Return a new resource group with items that are in the first group but not in the others

Added in version 2.1.0.

Return type:

Self

classmethod union(*groups)

Merge multiple resource groups into a new one

Return type:

Self

as_str()
Return type:

str

as_str_tuple()
Return type:

tuple[str, ...]

match(worker_group)

Returns the match between this resource group and the provided worker’s resource group

Parameters:

worker_group (Self | None) – The worker’s resource group

Return type:

float | None

Returns:

match value, the lower the value, the better the match is (ranges from -inf to +inf)

Raises:

NoMatch if there are missing or excluded tags

property items: tuple[ResourceItem, ...]
class momotor.shared.resources.item.ResourceItem(value, required, excluded)
classmethod compare_missing(worker)

. worker task

required

optional

excluded

-

NoMatch

Return type:

float | None

classmethod create(value)
Return type:

Iterable[ResourceItem]

as_str()
Return type:

str

comparable(worker)
Return type:

bool

compare(worker)

. worker task

required

optional

excluded

required

STRONGEST

STRONG

NoMatch

optional

WEAK

WEAKEST

NoMatch

excluded

NoMatch

NoMatch

Return type:

float | None

class momotor.shared.resources.tag.Tag(value)
classmethod compare_missing(worker)

. worker task

required

optional

excluded

-

NoMatch

Return type:

float | None

classmethod create(value)
Return type:

Iterable[ResourceItem]

as_str()
Return type:

str

comparable(worker)
Return type:

bool

compare(worker)

. worker task

required

optional

excluded

required

STRONGEST

STRONG

NoMatch

optional

WEAK

WEAKEST

NoMatch

excluded

NoMatch

NoMatch

Return type:

float | None

class momotor.shared.resources.NoMatch

Raised when no match is possible

momotor.shared.resources.const.NEUTRAL = 0.0

Match value for tags. NEUTRAL indicates there are no matching tags between task and worker.

momotor.shared.resources.const.STRONG = -1.0

Match value for tags. STRONG indicates a good match: the task requires this tag, for the worker it is optional

momotor.shared.resources.const.STRONGEST = -inf

Match value for tags. STRONGEST indicates the best possible match: both task and worker require this tag

momotor.shared.resources.const.WEAK = 1.0

Match value for tags. WEAK indicates a slightly worst match: the worker requires the tag, for the task it is optional

momotor.shared.resources.const.WEAKEST = inf

Match value for tags. WEAKEST indicates a worst possible match: for both worker and task the tag is optional