`momotor.shared.resources`#

Resources are used to match jobs with workers. A job can request resources, workers provide certain resources. Either job and workers can also explicitly reject certain resources.

The momotor.shared.resources module contains the code to match requested resources with provided resources.

Currently, resources are boolean tags (ie. a worker can indicate it provides the resource “Python 3”). A future update could make it possible to use resources with numeric values (eg. a worker can indicate it can provide up to 4 GiB of memory for the job)

Resource definition syntax#

A worker defines the resources it can provide, whereas a job defines the resources it requires. The syntax for both these definitions is the same:

group[":"]item[","item]*

The group is the name of a group, consisting of a series of alphanumeric ASCII characters, dashes (-), underscores (_), dots (.) and/or whitespace. Any other character can be escaped with a backslash character (\). It is also possible to use a quotes, where both single (') and double quotes (") can be used. Within quotes, any character is valid. Quotes can start and end anywhere in the string, but need to be matched in pairs. Whitespace is normally stripped from the beginning and end of group names, but using quotes it will be retained.

The following are all valid group names:

name - Nothing special here
dotted.name - The dot is valid
spaced name - Whitespace is also valid as part of the group name
"email@example.com" - The @ is not an allowed character, so the string needs to be quoted
email"@"example.com - Only quoting the @
email\@example.com - Escaping the @
backslashed\\name - The backslash itself also needs to be escaped
"trailing whitespace " - The whitespace at the end is retained

The first character that is not a valid character for a group name ends the group name field, so for example a colon (:), a tilde (~) or a less-than (<) character all end the group name field.

A group name field is followed by an optional colon (:) and a list of items. The colon is only needed if otherwise the transition from the group name field to the item list is indeterminate.

Items are separated by comma’s (,). At least one item is required. A semi-colon (;), newline character or end of string ends the list of items.

The item is the definition for an item. Items can contain any character except comma’s (,), colons (:), semi-colons (;), single (') and double quotes ("), and backslashes (\). Any of these characters can be escaped using the backslash (\), or included in quotes. Just like with group names, whitespace is stripped, but will be retained within quotes.

The following are all valid items:

string - Nothing special here
< 1000 - The < and space are both valid characters within an item
five<6 - So this is valid too
list",of,strings" - We need to quote the string to use comma’s in an item, item value will be list,of,strings
list\,of\,strings - The comma’s can also be escaped, item value will also be list,of,strings

Multiple groups can be separated by a newline character or a semi-colon (;)

These are all valid full resource definitions:

lang: python, java - A group named lang, containing two items python and java The colon is needed to separate the group name from the items here
memory < 1 GiB - A group named memory, containing one item < 1 GiB A colon separating the group name from the item is not needed here, but would be allowed
version <3, >=5 - A group named version, containing two items <3 and >=5
test case: "one; two, or more" - A group named test case, containing one item one; two, or more
one: 1; two: 2 - Two groups, one named one containing the item 1, and one named two containing the item 2
```
one: 1
two: 2
```
Identical to one: 1; two: 2

Tag type resources#

There are three types of tags:

required tags: defined as a name without prefix: tag
optional tags: prefixed with a question mark: ?tag
excluded tags: prefixed with a tilde: ~tag

Workers provide tag resource. The worker can indicate tags it requires from the jobs, tags that are optional, and excluded tags that it will not accept.

Jobs require tag resources. The job can indicate tags it requires from the worker, tags that are optional, and excluded tags that it does not want from workers.

Matches have a strength, indicated as a floating point number ranging from negative infinity to positive infinity, making it possible to compare them. The strongest possible match is represented as negative infinity, and the weakest match is positive infinity.

If both the worker and job define the same required tag, this is considered the strongest possible match. If both worker and job define the tag as optional the match is the weakest possible match.

If a tag is excluded by either of the parties, while the other requires or optionally defines the tag, the match is rejected. If both exclude the tag, the tag is ignored.

The following table describes all possible matches:

task \ worker	-	`tag`	`?tag`	`~tag`
-		`NoMatch`
`tag`	`NoMatch`	`STRONGEST`	`STRONG`	`NoMatch`
`?tag`		`WEAK`	`WEAKEST`	`NoMatch`
`~tag`		`NoMatch`	`NoMatch`

-: worker or task does not have this tag defined
blank cell: ignore, no change in match value
NoMatch: indicates the NoMatch exception will be raised
any other value: corresponding value for the match.

If every resource tag match is a blank cell in the above table, the final match will be NEUTRAL which is weaker than a strong match, but stronger than a weak match.

If multiple tags within the same resource group match, the strongest match is returned.

If multiple resource groups match, the weakest group determines the final value.

Note that the constants are defined the other way around: WEAKEST > STRONGEST. The reason for these counter intuitive values is the planned handling of value type resources, where a lower valued match will indicate a better match.

Example: Matching programming languages#

The worker defines the following tags

language: ?java, ?python
java: ?8, ?11, ?12, ?13
python: ?3.6, ?3.7

This worker indicates it can process Java and Python, and which versions for each

The jobs:

```
language: java
java: 12
```
This job indicates it wants Java 12. The language group matches STRONG, the java group matches NEUTRAL, so the total match is NEUTRAL, because the weakest group defines the total match. Since the job does not have a python requirement, and the worker defines all python group tags as optional, this is not considered at all for the match.
```
language: java
java: 14
```
This job indicates it wants Java 14. The language group matches STRONG, however, the java group produces a NoMatch
```
language: java
java: ?14
```
This job indicates it would like Java 14. The language group match matches STRONG again, the java group matches NEUTRAL, so the total match is NEUTRAL again
```
arch: x86
```
This job does not mention any programming language, it just asks for an arch tag of x86. The worker does not define a arch resource at all, so this will be a NoMatch.
```
arch: ?x86
```
This job also does not mention any programming language, it just asks for an optional arch tag of x86. The worker does not define a arch resource group at all, it does not exclude it either. The final match will be NEUTRAL
```
arch: ~x86
```
This job explicitly rejects arch of x86. However, since our worker does not define an arch resource, this will also result in a NEUTRAL match

Value type resources#

Not implemented yet.

This could be used to match dynamic resources like memory or disk space.

An example of a required values definition would be:

ram >= 1 GiB - request a worker with at least 1 GiB of free RAM

An example of a provided values definition would be:

ram = 2147483648 - the worker has this amount of available RAM

Class documentation#

class momotor.shared.resources.Resources(definition=None)#

Manage resources for a task and worker.

resource value handling:

boolean items:

tag: required tag
?tag: optional tag
~tag: excluded tag

The following table indicates the match value for boolean tags of given type, NoMatch indicates that the NoMatch exception is raised, any other value indicates that that value is returned.

-: worker or task does not have this tag defined
blank cell: ignore, no change in match value

TODO numeric items

Parameters:: definition (dict[str, ResourceGroup]) – A dictionary of group names to resource groups

as_str(*, multiline=False, compact=False)#

Convert these resources into a string that can be parsed back into a Resources object using from_string()

Parameters:

multiline (bool) – if True, use newlines to split groups, otherwise uses semicolons
compact (bool) – if True, returns a compact representation, otherwise adds more whitespace

Return type:

str

Returns:

string representation of the resources

as_str_tuples()#

Return type:: Iterable[tuple[str, str]]

as_tuples()#

Return type:: Iterable[tuple[str, tuple[str]]]

classmethod from_dict(definition)#

Factory to create a Resources object from a dict of group names to group string definitions

Return type:: Self

classmethod from_key_value(key, value)#

Factory to create a Resources object from a key and values

Parameters:

key (str) – Group name
value (str | Iterable[str]) – Group definition string or list of strings

Return type:

Self

classmethod from_string(value)#

Factory to create a Resources object from one or more strings

Parameters:: value (str | Iterable[str]) – A string or iterable of strings containing resource definitions including a group name
Return type:: Self
Returns:: The parsed resources
Raises:: ValueError if the string is an invalid resources definition

get(key)#

Get the ResourceGroup by name

Parameters:: key (str) – Group name
Return type:: ResourceGroup
Returns:: The resource group matching the group name

group_names()#

Returns a list of group names

Return type:: list[str]
Returns:: group names

match(worker_resource)#

Returns the match between these resources and the provided worker’s resources.

Parameters:: worker_resource (Resources) – The worker’s resources
Return type:: float
Returns:: match value, the lower the value, the better the match is (ranges from -inf to +inf). If nothing matches, but also nothing is excluded, will return NEUTRAL
Raises:: NoMatch if no match can be made (eg. due to an exclude)

classmethod union(*resources)#

Merge multiple Resources objects into a new one

Return type:: Self
Returns:: The merged resources

class momotor.shared.resources.group.ResourceGroup(items=None)#

A group of resources.

as_str()#

Return type:: str

as_str_tuple()#

Return type:: tuple[str, ...]

classmethod create(value)#

Return type:: Self

match(worker_group)#

Returns the match between this resource group and the provided worker’s resource group

Parameters:: worker_group (Optional[ResourceGroup]) – The worker’s resource group
Return type:: float | None
Returns:: match value, the lower the value, the better the match is (ranges from -inf to +inf)
Raises:: NoMatch if there are missing or excluded tags

classmethod union(*groups)#

Merge multiple resource groups into a new one

Return type:: Self

property items: tuple[ResourceItem]#

class momotor.shared.resources.item.ResourceItem(value, required, excluded)#

as_str()#

Return type:: str

comparable(worker)#

Return type:: bool

compare(worker)#

. worker task	required	optional	excluded
required	`STRONGEST`	`STRONG`	`NoMatch`
optional	`WEAK`	`WEAKEST`	`NoMatch`
excluded	`NoMatch`	`NoMatch`

Return type:: float | None

classmethod compare_missing(worker)#

. worker task	required	optional	excluded
-	`NoMatch`

Return type:: float | None

classmethod create(value)#

Return type:: Iterable[Self]

class momotor.shared.resources.tag.Tag(value)#

as_str()#

Return type:: str

comparable(worker)#

Return type:: bool

compare(worker)#

. worker task	required	optional	excluded
required	`STRONGEST`	`STRONG`	`NoMatch`
optional	`WEAK`	`WEAKEST`	`NoMatch`
excluded	`NoMatch`	`NoMatch`

Return type:: float | None

classmethod compare_missing(worker)#

. worker task	required	optional	excluded
-	`NoMatch`

Return type:: float | None

classmethod create(value)#

Return type:: Iterable[Self]

class momotor.shared.resources.NoMatch#: Raised when no match is possible

momotor.shared.resources.const.NEUTRAL = 0.0#: Match value for tags. NEUTRAL indicates there are no matching tags between task and worker.

momotor.shared.resources.const.STRONG = -1.0#: Match value for tags. STRONG indicates a good match: the task requires this tag, for the worker it is optional

momotor.shared.resources.const.STRONGEST = -inf#: Match value for tags. STRONGEST indicates the best possible match: both task and worker require this tag

momotor.shared.resources.const.WEAK = 1.0#: Match value for tags. WEAK indicates a slightly worst match: the worker requires the tag, for the task it is optional

momotor.shared.resources.const.WEAKEST = inf#: Match value for tags. WEAKEST indicates a worst possible match: for both worker and task the tag is optional

momotor.shared.resources#

Resource definition syntax#

Tag type resources#

Example: Matching programming languages#

Value type resources#

Class documentation#

`momotor.shared.resources`#