momotor.shared.resources¶
Resources match jobs with workers. A job states the resources it needs. A worker states the resources it offers. Either side can also reject a resource. The broker uses the match result to pick a worker for each task.
A resource definition is a set of named groups. Each group holds items. For
example, the group language can hold the items python and java.
Items are tags: a worker either offers a tag or it does not. Numeric values,
such as an amount of memory, are not supported yet.
Match a job with a worker¶
Parse each definition with
from_string(). Then call
match() on the job resources
with the worker resources as the argument.
from momotor.shared.resources import NoMatch, Resources
job = Resources.from_string('language: python; python: 3.12')
worker = Resources.from_string('language: ?python, ?java; python: ?3.11, ?3.12')
try:
score = job.match(worker)
except NoMatch:
print('this worker cannot run the job')
else:
print(score)
The example prints -1.0. The match returns a number:
A lower number is a better match.
Negative infinity is the strongest possible match.
Positive infinity is the weakest possible match.
NoMatchmeans the worker cannot run the job at all.
The ? prefix in the worker definition marks a tag as optional. The
worker offers the tag, but a job does not need to ask for it. A tag without
a prefix is required. A tag with a ~ prefix is excluded. The sections
below define the syntax and every matching rule.
Combine and subtract definitions¶
union() merges definitions.
difference() removes the items
of the later definitions from the first one:
all_languages = Resources.from_string('language: python, java')
java_only = Resources.from_string('language: java')
print(Resources.difference(all_languages, java_only).as_str())
The example prints language: python.
Resource definition syntax¶
A worker defines the resources it can provide, whereas a job defines the resources it requires. The syntax for both these definitions is the same:
group[":"]item[","item]*
The group is the name of a group, consisting of a series of alphanumeric ASCII characters, dashes (-),
underscores (_), dots (.) and/or whitespace. Any other character can be escaped with a backslash
character (\). It is also possible to use quotes, where both single (') and double quotes (")
can be used. Within quotes, any character is valid. Quotes can start and end anywhere in
the string, but need to be matched in pairs. Whitespace is normally stripped from the beginning
and end of group names, but using quotes it will be retained.
The following are all valid group names:
name- Nothing special heredotted.name- The dot is validspaced name- Whitespace is also valid as part of the group name"email@example.com"- The@is not an allowed character, so the string needs to be quotedemail"@"example.com- Only quoting the@email\@example.com- Escaping the@backslashed\\name- The backslash itself also needs to be escaped"trailing whitespace "- The whitespace at the end is retained
The first character that is not a valid character for a group name ends the group name field, so for example
a colon (:), a tilde (~) or a less-than (<) character all end the group name field.
A group name field is followed by an optional colon (:) and a list of items. The colon is only needed
if otherwise the transition from the group name field to the item list is indeterminate.
Items are separated by commas (,). At least one item is required. A semi-colon (;), newline character
or end of string ends the list of items.
The item is the definition for an item. Items can contain any character except commas (,),
colons (:), semi-colons (;), single (') and double quotes ("), and backslashes (\).
Any of these characters can be escaped using the backslash (\), or included in quotes.
Just like with group names, whitespace is stripped, but will be retained within quotes.
The following are all valid items:
string- Nothing special here< 1000- The<and space are both valid characters within an itemfive<6- So this is valid toolist",of,strings"- We need to quote the string to use commas in an item, item value will belist,of,stringslist\,of\,strings- The commas can also be escaped, item value will also belist,of,strings
Multiple groups can be separated by a newline character or a semi-colon (;)
These are all valid full resource definitions:
lang: python, java- A group namedlang, containing two itemspythonandjavaThe colon is needed to separate the group name from the items herememory < 1 GiB- A group namedmemory, containing one item< 1 GiBA colon separating the group name from the item is not needed here, but would be allowedversion <3, >=5- A group namedversion, containing two items<3and>=5test case: "one; two, or more"- A group namedtest case, containing one itemone; two, or moreone: 1; two: 2- Two groups, one namedonecontaining the item1, and one namedtwocontaining the item2one: 1 two: 2
Identical to
one: 1; two: 2
Tag type resources¶
There are three types of tags:
required tags: defined as a name without prefix:
tagoptional tags: prefixed with a question mark:
?tagexcluded tags: prefixed with a tilde:
~tag
Workers provide tag resources. The worker can indicate tags it requires from the jobs, tags that are optional, and excluded tags that it will not accept.
Jobs require tag resources. The job can indicate tags it requires from the worker, tags that are optional, and excluded tags that it does not want from workers.
Matches have a strength, indicated as a floating point number ranging from negative infinity to positive infinity, making it possible to compare them. The strongest possible match is represented as negative infinity, and the weakest match is positive infinity.
If both the worker and job define the same required tag, this is considered the strongest possible match. If both worker and job define the tag as optional the match is the weakest possible match.
If a tag is excluded by either of the parties, while the other requires or optionally defines the tag, the match is rejected. If both exclude the tag, the tag is ignored.
The following table describes all possible matches:
task \ worker |
- |
|
|
|
- |
||||
|
||||
|
||||
|
-: worker or task does not have this tag defined
blank cell: ignore, no change in match value
NoMatch: indicates theNoMatchexception will be raisedany other value: corresponding value for the match.
If every resource tag match is a blank cell in the above table, the final match will be
NEUTRAL which is weaker than a strong match, but stronger than a weak match.
If multiple tags within the same resource group match, the strongest match is returned.
If multiple resource groups match, the weakest group determines the final value.
Note that the constants are defined the other way around:
WEAKEST > STRONGEST.
The reason for these counter-intuitive values is the planned handling of value type resources, where a lower valued
match will indicate a better match.
Example: Matching programming languages¶
The worker defines the following tags
language: ?java, ?python
java: ?8, ?11, ?12, ?13
python: ?3.6, ?3.7
This worker indicates it can process Java and Python, and which versions for each
The jobs:
language: java java: 12
This job indicates it wants Java 12. The
languagegroup matchesSTRONG, thejavagroup matchesNEUTRAL, so the total match isNEUTRAL, because the weakest group defines the total match. Since the job does not have apythonrequirement, and the worker defines allpythongroup tags as optional, this is not considered at all for the match.language: java java: 14
This job indicates it wants Java 14. The
languagegroup matchesSTRONG, however, thejavagroup produces aNoMatchlanguage: java java: ?14
This job indicates it would like Java 14. The
languagegroup match matchesSTRONGagain, thejavagroup matchesNEUTRAL, so the total match isNEUTRALagainarch: x86
This job does not mention any programming language, it just asks for an
archtag ofx86. The worker does not define aarchresource at all, so this will be aNoMatch.arch: ?x86
This job also does not mention any programming language, it just asks for an optional
archtag ofx86. The worker does not define aarchresource group at all, it does not exclude it either. The final match will beNEUTRALarch: ~x86
This job explicitly rejects
archofx86. However, since our worker does not define anarchresource, this will also result in aNEUTRALmatch
Value type resources¶
Not implemented yet.
This could be used to match dynamic resources like memory or disk space.
An example of a required values definition would be:
ram >= 1 GiB- request a worker with at least 1 GiB of free RAM
An example of a provided values definition would be:
ram = 2147483648- the worker has this amount of available RAM
API reference¶
- class momotor.shared.resources.Resources(definition=None)¶
A set of resource groups for a task or a worker.
Create instances with
from_string(),from_dict(), orfrom_key_value(). Compare a task’s resources with a worker’s resources withmatch().Items are boolean tags:
tag: required tag
?tag: optional tag
~tag: excluded tag
The narrative documentation for this module has the full table of match values for every combination of task and worker tags.
- Parameters:
definition (
dict[str,ResourceGroup] |None) – A dictionary of group names to resource groups
- classmethod difference(*resources)¶
Subtract
Resourcesobjects from the first one- Return type:
Self- Returns:
The resulting resources
Added in version 2.1.0.
- classmethod from_dict(definition)¶
Factory to create a
Resourcesobject from a dict of group names to group string definitions- Return type:
Self
- classmethod from_string(value)¶
Factory to create a
Resourcesobject from one or more strings- Parameters:
value (
str|Iterable[str]) – A string or iterable of strings containing resource definitions including a group name- Return type:
Self- Returns:
The parsed resources
- Raises:
ValueErrorif the string is an invalid resources definition
- classmethod union(*resources)¶
Merge multiple
Resourcesobjects into a new one- Return type:
Self- Returns:
The merged resources
- as_str(*, multiline=False, compact=False)¶
Convert these resources into a string that can be parsed back into a
Resourcesobject usingfrom_string()
- get(key)¶
Get the
ResourceGroupby name- Parameters:
key (
str) – Group name- Return type:
- Returns:
The resource group matching the group name
- match(worker_resource)¶
Returns the match between these resources and the provided worker’s resources.
- Parameters:
worker_resource (
Resources) – The worker’s resources- Return type:
- Returns:
match value, the lower the value, the better the match is (ranges from
-infto+inf). If nothing matches, but also nothing is excluded, will returnNEUTRAL- Raises:
NoMatchif no match can be made (for example due to an exclude)
- class momotor.shared.resources.group.ResourceGroup(items=None)¶
A group of
resources.- classmethod create(value)¶
- Return type:
Self
- classmethod difference(*groups)¶
Return a new resource group with items that are in the first group but not in the others
Added in version 2.1.0.
- Return type:
Self
- classmethod union(*groups)¶
Merge multiple resource groups into a new one
- Return type:
Self
- match(worker_group)¶
Returns the match between this resource group and the provided worker’s resource group
- property items: tuple[ResourceItem, ...]¶
- class momotor.shared.resources.item.ResourceItem(value, required, excluded)¶
- classmethod compare_missing(worker)¶
. worker task
required
optional
excluded
-
- classmethod create(value)¶
- Return type:
- class momotor.shared.resources.tag.Tag(value)¶
- classmethod compare_missing(worker)¶
. worker task
required
optional
excluded
-
- classmethod create(value)¶
- Return type:
- class momotor.shared.resources.NoMatch¶
Raised when no match is possible
- momotor.shared.resources.const.NEUTRAL = 0.0¶
Match value for tags.
NEUTRALindicates there are no matching tags between task and worker.
- momotor.shared.resources.const.STRONG = -1.0¶
Match value for tags.
STRONGindicates a good match: the task requires this tag, for the worker it is optional
- momotor.shared.resources.const.STRONGEST = -inf¶
Match value for tags.
STRONGESTindicates the best possible match: both task and worker require this tag
- momotor.shared.resources.const.WEAK = 1.0¶
Match value for tags.
WEAKindicates a slightly worst match: the worker requires the tag, for the task it is optional
- momotor.shared.resources.const.WEAKEST = inf¶
Match value for tags.
WEAKESTindicates a worst possible match: for both worker and task the tag is optional