Tool registry¶
Tools like Java and Python are not always installed in the same location on different installations.
The tool registry provides a way for Momotor checklets to find the external tools needed in a system independent way. The registry also is used to indicate to the scheduler which tools a worker has available. This allows use of workers with different configurations without running the risk of a job getting scheduled on a worker which does not have a specific tool (or version of that tool) available.
Registry structure¶
The registry itself is a directory structure containing text files, making it easy to generate these files during setup of the tools. The registry also supports multiple versions of tools to be installed and selectable, and also further variants within these versions.
Example of a registry structure:
├── anaconda3/
│ ├── 2021.05/
│ │ ├── base
│ │ ├── python38 ➔ base
│ │ └── _default ➔ base
│ ├── 2021.11/
│ │ ├── base
│ │ ├── python38
│ │ └── _default ➔ base
│ └── _default/ ➔ 2021.11/
├── java/
│ ├── 8
│ ├── 17
│ ├── 18
│ ├── latest ➔ 18
│ ├── lts ➔ 17
│ └── _default ➔ 17
└── python/
├── 2.7.18
├── 3.8.10
├── 3.8.11
└── 3.9.7
The first level of the registry contains the tool names as they will be used by the checklets. The next level
contains version numbers and the (optional) third directory level contains variants. It’s possible to have
even more directory levels. Files contain the information about a tool, which is documented
below.
Soft links (shown in the example above using ➔) are allowed and create an alias.
The most basic way to select a tool is to use the path of the tool file name, e.g. anaconda3/2021.11/python38
If parts of the path are not provided, the _default file (or, preferably, link) will be used if this is available,
otherwise the highest available version will be selected. So, for the example above, tool name java will
select Java 17, whereas tool name python will select Python 3.9.7.
For tools with multiple sub-directories, an underscore can be used to indicated the default,
e.g. anaconda3/_/python38 will select the python38 variant of the default anaconda3 installation.
Dotted version numbers can be abbreviated, and if multiple versions match, the highest version is selected, i.e.
tool name python/3.8 will select Python 3.8.11 in the example above, python/2 selects Python 2.7.18,
and python/3 selects Python 3.9.7. Version number abbreviation is checked by “dot”, i.e. 3.8.1 would
not match 3.8.10 or 3.8.11.
It’s possible to create named versions, e.g. java/lts will select Java 17. Numeric versions have priority over
named versions, so in case of the python tree above, if there would have been a named version, 3.9.7 would still
be the default. When a directory only contains names, no numeric versions and no _default, the alphabetically
highest name will be considered the default. Version numbers are correctly ordered numerically, so if the _default
file did not exist in the java tree, Java 18 would have been the default version, not Java 8
File and directory names starting with a dot (.) or ending with a tilde (~) are ignored while scanning
the registry.
Registry file¶
A registry file contains environment variables required for the tool, and the path to the tool itself. The tool path is usually the executable, but it could also be a directory. How the environment variables and tool path are interpreted is up to the checklets using the tool registry.
An example registry file is:
PYTHONHOME=${HOME}/python38
PYTHONPATH=${PYTHONHOME}/extra-packages/
${PYTHONHOME}/bin/python
As shown in the example above, environment variables can refer to other variables in the current environment, including variables defined on earlier lines in the tool definition file. They will be resolved when the tool file is read.
The tool path is always on the last non-empty line of the file, and the other lines should be valid environment variable definitions.
If the tool path or the value of an environment variable is quoted, any text after the end quote is ignored:
SOMEVARIABLE='this text is part of the variable' this text is ignored
'/quoted/path' this text is also ignored
The value of SOMEVARIABLE will be this text is part of the variable, and the path will be /quoted/path.
Both single and double quotes are supported.
Variable expansion uses Python Template strings ($-strings), not a shell.
Both $NAME and ${NAME} expand, including inside quoted values. Unknown
variables remain unchanged. Commands and shell expressions are not executed.
Registry location¶
Registry lookup is available on POSIX systems. The default directories, in
priority order, are ~/.toolregistry.d and /etc/toolregistry.d. The first
entry for a tool name wins.
TOOLREGISTRY replaces these defaults with a colon-separated list of paths.
An empty TOOLREGISTRY disables the default paths. Explicit paths passed to
registered_tools() or
resolve_tool() come first.
include_default_paths=False uses only those explicit paths.
Registry scans are cached for the most recent argument combination. Editing a
registry file does not refresh a cached scan in a running process. Missing tool
paths are excluded unless registered_tools(..., include_missing=True) is used.
Existence checks do not verify executable permissions.
Usage in bundles¶
To configure a checklet to use a certain tool version, two options are needed: the
tools@scheduler option in each step that requires external tools,
to indicate to the scheduler which tools are required by the step’s checklet, and options in the
tools domain to define the actual tool versions to use.
The options in the tools domain can be defined in the recipe or config bundles, and can contain placeholders. This makes it possible to define the tool version based on a property generated by earlier executed steps, for example:
<options domain="tools">
<option name="java" value="${prop[#build-java:exectool]}" />
</options>
where the build-java step generates a property exectool which contains a proper tool name to use to
execute the generated class files.
Tool and registry functions and classes¶
- momotor.options.tools.registry.registered_tools(paths=None, *, include_default_paths=True, include_missing=False)¶
Return registered tools and aliases, keyed by registry name.
Earlier registry paths take priority. The most recent scan is cached by arguments, so changing a registry file does not refresh that cached result. Invalid files are logged and skipped.
If include_default_paths is True (default), this reads the tool registry from .toolregistry.d in the current user’s home directory and /etc/toolregistry.d. If paths is provided, registry will be read from all paths in the path list as well.
- Parameters:
paths (
Iterable[str|PathLike] |None) – paths to read the tool registry from. prepended to the default pathsinclude_default_paths (
bool) – Append paths fromTOOLREGISTRY, or standard paths when unset. An empty environment variable adds no paths.include_missing (
bool) – include tools that are registered but the executable does not actually exist
- Return type:
- Returns:
a mapping from tool name to tool dataclass
Only available on Posix systems, does not work on Windows
- momotor.options.tools.registry.resolve_tool(name, *, paths=None, include_default_paths=True)¶
Resolve a tool name to a
Tooldataclass.If include_default_paths is True (default), this reads the tool registry from .toolregistry.d in the current user’s home directory and /etc/toolregistry.d. If paths is provided, registry will be read from all paths in the path list as well.
- Parameters:
name (
str|PathLike|ToolName) – Name of the tool to resolvepaths (
Iterable[str|PathLike] |None) – paths to read the tool registry from. prepended to the default pathsinclude_default_paths (
bool) – Append paths fromTOOLREGISTRY, or standard paths when unset. An empty environment variable adds no paths.
- Return type:
- Returns:
The tool info object.
- Raises:
FileNotFoundError – if the name could not be resolved
Only available on Posix systems, does not work on Windows
- momotor.options.tools.registry.tool_registry_paths(paths=None, include_default_paths=True)¶
Yield existing registry paths in lookup order.
Explicit paths come first. When defaults are enabled,
TOOLREGISTRYsupplies additional colon-separated paths. An unset variable uses~/.toolregistry.dand/etc/toolregistry.d. An empty variable adds no paths.
- class momotor.options.tools.tool.Tool(name, environment, path)¶
Data class representing the contents of a tool registry file.
- classmethod from_file_factory(registry_path, tool_file_path)¶
Read a tool registry file file and return a populated
Tooldataclass.
- exists()¶
Shortcut for
path.exists(). Result is cached.- Return type:
- Returns:
True if the tool exists.
- class momotor.options.tools.tool.ToolName(name)¶
Represents a tool name as a tuple of
ToolVersionobjectsInstantiated from a tool file name (either a
str,pathlib.PurePath, or anotherToolName), it splits all the parts of the tool name and represents each part as aToolVersion, and allows these names to be compared and ordered.>>> ToolName('test') == ToolName('test/_') True
>>> ToolName('test/1.0') < ToolName('test/2.0') True
- classmethod factory(name, *parts)¶
Helper factory to create a
ToolNamefrom astr,pathlib.PurePath, anotherToolName, or a sequence ofstrorToolVersionelements.If name is a
ToolName, returns name unmodified, otherwise instantiates a newToolNameobject for the given name.- Return type:
Self
- is_partial(other)¶
Checks if all elements of
self.versionsare the same or a partial version ofother.versions.- Return type:
>>> ToolName('test').is_partial(ToolName('test/1.0')) True
>>> ToolName('test/1').is_partial(ToolName('test/1.0')) True
>>> ToolName('test/1.0').is_partial(ToolName('test/1.0')) True
>>> ToolName('test/1.0').is_partial(ToolName('test')) False
>>> ToolName('test').is_partial(ToolName('test.1')) False
- property versions: tuple[ToolVersion, ...]¶
A tuple representing the
namesplit on theSEPARATORand each part converted to aToolVersion.
- momotor.options.tools.tool.match_tool(name, tools)¶
Match tool name with
DEFAULTplaceholders and (partial) version numbers to a tool name in the tools container.Returns the most specific matched name from tools, or None if no match could be made.
- momotor.options.tools.tool.match_tool_requirements(requirements, toolset)¶
Match each requirement to an available tool using
match_tool().Alternatives are tried in the iteration order of each requirement’s collection. Sets do not preserve the order from the original option value.
- Parameters:
- Return type:
- Returns:
mapping of requirement name (key of the requirements mapping) to matched tool name
- Raises:
ValueError – when requirements cannot be fulfilled
- class momotor.options.tools.version.ToolVersion(value)¶
Represents a tool version string (i.e., a string with dotted version-number parts) and makes it possible to order these. Also handles sub-version suffixes like 1.0-0, 1.0-1, 1.0-2, etc.
ToolVersion.DEFAULTis a version constant which is always better than any other version number.>>> _test = lambda x, y: (x < y, x == y, x > y)
>>> _test(ToolVersion('1'), ToolVersion('2')) (True, False, False)
>>> _test(ToolVersion('1'), ToolVersion('1.1')) (True, False, False)
>>> _test(ToolVersion('1.0'), ToolVersion('1.1')) (True, False, False)
>>> _test(ToolVersion('1.0'), ToolVersion('1.x')) (False, False, True)
>>> _test(ToolVersion('1.x'), ToolVersion('1.x')) (False, True, False)
>>> _test(ToolVersion('1.0'), ToolVersion('1.0')) (False, True, False)
>>> _test(ToolVersion('1.1'), ToolVersion('1.0')) (False, False, True)
>>> _test(ToolVersion('1.1'), ToolVersion('2')) (True, False, False)
>>> _test(ToolVersion('1.00'), ToolVersion('1.0')) (False, True, False)
>>> _test(ToolVersion('1.09'), ToolVersion('1.10')) (True, False, False)
>>> _test(ToolVersion('1.0'), ToolVersion('1.0-0')) (True, False, False)
>>> _test(ToolVersion('1.0'), ToolVersion('1.0-1')) (True, False, False)
>>> _test(ToolVersion('1.0-0'), ToolVersion('1.0-1')) (True, False, False)
>>> _test(ToolVersion('1.0-0'), ToolVersion('1.1')) (True, False, False)
>>> _test(ToolVersion(ToolVersion.DEFAULT), ToolVersion('1')) (False, False, True)
>>> _test(ToolVersion('1'), ToolVersion(ToolVersion.DEFAULT)) (True, False, False)
>>> _test(ToolVersion(ToolVersion.DEFAULT), ToolVersion(ToolVersion.DEFAULT)) (False, True, False)
- is_partial(other)¶
Returns True if self is equal to or a partial version of other.
- Return type:
>>> ToolVersion('1').is_partial(ToolVersion('1')) True
>>> ToolVersion('1').is_partial(ToolVersion('1.0')) True
>>> ToolVersion('1').is_partial(ToolVersion('1.0-0')) True
>>> ToolVersion('1.0').is_partial(ToolVersion('1.0-0')) True
>>> ToolVersion('1.0').is_partial(ToolVersion('1')) False
>>> ToolVersion('2').is_partial(ToolVersion('1.0')) False
>>> ToolVersion('1').is_partial(default_tool_version) False
>>> default_tool_version.is_partial(ToolVersion('1')) True
>>> default_tool_version.is_partial(default_tool_version) True
- momotor.options.tools.version.default_tool_version = ToolVersion(value='_')¶
Constant
ToolVersion(ToolVersion.DEFAULT)