Tool registry

Tools like Java and Python are not always installed in the same location on different installations.

The tool registry provides a way for Momotor checklets to find the external tools needed in a system independent way. Additionally, the registry is used to indicate to the scheduler which tools a worker has available. This allows use of workers with different configurations without running the risk of a job getting scheduled on a worker which does not have a specific tool (or version of that tool) available.

Registry structure

The registry itself is a directory structure containing text files, making it easy to generate these files during setup of the tools. The registry also supports multiple versions of tools to be installed and selectable, and also further variants within these versions.

Example of a registry structure:

├── anaconda3/
│   ├── 2021.05/
│   │   ├── base
│   │   ├── python38 ➔ base
│   │   └── _default ➔ base
│   ├── 2021.11/
│   │   ├── base
│   │   ├── python38
│   │   └── _default ➔ base
│   └── _default/ ➔ 2021.11/
├── java/
│   ├── 8
│   ├── 17
│   ├── 18
│   ├── latest ➔ 18
│   ├── lts ➔ 17
│   └── _default ➔ 17
└── python/
    ├── 2.7.18
    ├── 3.8.10
    ├── 3.8.11
    └── 3.9.7

The first level of the registry contains the tool names as they will be used by the checklets. The next level contains version numbers and the (optional) third directory level contains variants. It’s possible to have even more directory levels. Files contain the information about a tool, which is documented below. Soft links (shown in the example above using ➔) are allowed and create an alias.

The most basic way to select a tool is to use the path of the tool file name, e.g. anaconda3/2021.11/python38

If parts of the path are not provided, the _default file (or, preferably, link) will be used if this is available, otherwise the highest available version will be selected. So, for the example above, tool name java will select Java 17, whereas tool name python will select Python 3.9.7.

For tools with multiple sub-directories, an underscore can be used to indicated the default, e.g. anaconda/_/python38 will select the python38 variant of the default anaconda3 installation.

Dotted version numbers can be abbreviated, and if multiple versions match, the highest version is selected, i.e. tool name python/3.8 will select Python 3.8.11 in the example above, python/2 selects Python 2.7.18, and python/3 selects Python 3.9.7. Version number abbreviation is checked by “dot”, i.e. 3.8.1 would not match 3.8.10 or 3.8.11.

It’s possible to create named versions, e.g. java/lts will select Java 17. Numeric versions have priority over named versions, so in case of the python tree above, if there would have been a named version, 3.9.7 would still be the default. When a directory only contains names, no numeric versions and no _default, the alphabetically highest name will be considered the default. Version numbers are correctly ordered numerically, so if the _default file did not exist in the java tree, Java 18 would have been the default version, not Java 8

File and directory names starting with a dot (.) or ending with a tilde (~) are ignored while scanning the registry.

Registry file

A registry file contains environment variables required for the tool, and the path to the tool itself. The tool path is usually the executable, but it could also be a directory. How the environment variables and tool path are interpreted is up to the checklets using the tool registry.

An example registry file is:

PYTHONHOME=${HOME}/python38
PYTHONPATH=${PYTHONHOME}/extra-packages/
${PYTHONHOME}/bin/python

As shown in the example above, environment variables can refer to other variables in the current environment, including variables defined on earlier lines in the tool definition file. They will be resolved when the tool file is read.

The tool path is always on the last non-empty line of the file, and the other lines should be valid environment variable definitions.

If the tool path or the value of an environment variable is quoted, any text after the end quote is ignored:

SOMEVARIABLE='this text is part of the variable' this text is ignored
'/quoted/path' this text is also ignored

The value of SOMEVARIABLE will be this text is part of the variable, and the path will be /quoted/path. Both single and double quotes are supported.

The variable expansion is done using Python Template strings.

Registry location

By default, the registry is read from the /etc/toolregistry.d and ~/toolregistry.d directories, where entries in the latter override entries in the first.

The environment variable TOOLREGISTRY can be used to change the defaults. It is a colon (:) separated list of paths, similar to the standard PATH environment variable.

The registered_tools() and resolve_tool() functions allow extending or overriding the defaults.

Usage in bundles

To configure a checklet to use a certain tool version, two options are needed: the tools@scheduler option in each step that requires external tools, to indicate to the scheduler which tools are required by the step’s checklet, and options in the tools domain to define the actual tool versions to use.

The options in the tools domain can be defined in the recipe or config bundles, and can contain placeholders. This makes it possible to define the tool version based a on property generated by earlier executed steps, for example:

<options domain="tools">
  <option name="java" value="${prop[#build-java:exectool]}" />
</options>

where the build-java step generates a property exectool which contains a proper tool name to use to execute the generated class files.

Tool and registry functions and classes

momotor.options.tools.registry.registered_tools(paths=None, *, include_default_paths=True, include_missing=False)

Return a mapping with all locally installed tools.

If include_default_paths is True (default), this reads the tool registry from .toolregistry.d in the current user’s home directory and /etc/toolregistry.d. If paths is provided, registry will be read from all paths in the path list as well.

Parameters:

  • paths (Optional[Iterable[Union[str, PathLike]]]) – paths to read the tool registry from. prepended to the default paths

  • include_default_paths (bool) – include the default paths

  • include_missing (bool) – include tools that are registered but the executable does not actually exist

Return type:

dict[ToolName, Tool]

Returns:

a mapping from tool name to tool dataclass

Only available on Posix systems, does not work on Windows

momotor.options.tools.registry.resolve_tool(name, *, paths=None, include_default_paths=True)

Resolve a tool name to a Tool dataclass.

If include_default_paths is True (default), this reads the tool registry from .toolregistry.d in the current user’s home directory and /etc/toolregistry.d. If paths is provided, registry will be read from all paths in the path list as well.

Parameters:
Return type:

Tool

Returns:

The tool info object.

Raises:

FileNotFoundError – if the name could not be resolved

Only available on Posix systems, does not work on Windows

momotor.options.tools.registry.tool_registry_paths(paths=None, include_default_paths=True)

Collect the tool registry paths

Return type:

Iterable[Path]

class momotor.options.tools.tool.Tool(name, environment, path)

Data class representing the contents of a tool registry file.

exists()

Shortcut for path.exists(). Result is cached.

Return type:

bool

Returns:

True if the tool exists.

classmethod from_file_factory(registry_path, tool_file_path)

Read a tool registry file file and return a populated Tool dataclass.

Parameters:

  • registry_path (Path) – path to the registry

  • tool_file_path (PurePath) – path to the tool file, relative to registry_path

Return type:

Tool

Returns:

the tool

environment: Mapping[str, str]

Environment variables for the tool as indicated by the tool file

name: ToolName

Canonical name of the tool after resolving soft links

path: Path

Path to the tool as indicated by the tool file

class momotor.options.tools.tool.ToolName(name)

Represents a tool name as a tuple of ToolVersion objects

Instantiated from a tool file name (either a str, pathlib.PurePath, or another ToolName), it splits all the parts of the tool name and represents each part as a ToolVersion, and allows these names to be compared and ordered.

>>> ToolName('test') == ToolName('test/_')
True

>>> ToolName('test/1.0') < ToolName('test/2.0')
True
classmethod factory(name, *parts)

Helper factory to create a ToolName from a str, pathlib.PurePath, another ToolName, or a sequence of str or ToolVersion elements.

If name is a ToolName, returns name unmodified, otherwise instantiates a new ToolName object for the given name.

Return type:

TypeVar(TN, bound= ToolName)

is_partial(other)

Checks if all elements of self.versions are the same or a partial version of other.versions.

Return type:

bool

>>> ToolName('test').is_partial(ToolName('test/1.0'))
True

>>> ToolName('test/1').is_partial(ToolName('test/1.0'))
True

>>> ToolName('test/1.0').is_partial(ToolName('test/1.0'))
True

>>> ToolName('test/1.0').is_partial(ToolName('test'))
False

>>> ToolName('test').is_partial(ToolName('test.1'))
False
DEFAULT_FILENAME: ClassVar[str] = '_default'

Constant for the default file name

SEPARATOR: ClassVar[str] = '/'

Constant for the separator between name elements

name: str

The tool name

property versions: tuple[ToolVersion, ...]

A tuple representing the name split on the SEPARATOR and each part converted to a ToolVersion.

momotor.options.tools.tool.match_tool(name, tools)

Match tool name with DEFAULT placeholders and (partial) version numbers to a tool name in the tools container.

Returns the most specific matched name from tools, or None if no match could be made.

Parameters:
Return type:

Union[str, PathLike, ToolName, None]

Returns:

The matched name from tools, or None

momotor.options.tools.tool.match_tool_requirements(requirements, toolset)

Match tools in requirements with tools in tools using match_tool() and return a sequence with the most specific matched.

Parameters:
Return type:

dict[str, Union[str, PathLike, ToolName]]

Returns:

mapping of requirement name (key of the requirements mapping) to matched tool name

Raises:

ValueError – when requirements cannot be fulfilled

class momotor.options.tools.version.ToolVersion(value)

Represents a tool version string (i.e., a string with dotted version-number parts) and makes it possible to order these. Also handles sub-version suffixes like 1.0-0, 1.0-1, 1.0-2, etc.

ToolVersion.DEFAULT is a version constant which is always better than any other version number.

>>> _test = lambda x, y: (x < y, x == y, x > y)

>>> _test(ToolVersion('1'), ToolVersion('2'))
(True, False, False)

>>> _test(ToolVersion('1'), ToolVersion('1.1'))
(True, False, False)

>>> _test(ToolVersion('1.0'), ToolVersion('1.1'))
(True, False, False)

>>> _test(ToolVersion('1.0'), ToolVersion('1.x'))
(False, False, True)

>>> _test(ToolVersion('1.x'), ToolVersion('1.x'))
(False, True, False)

>>> _test(ToolVersion('1.0'), ToolVersion('1.0'))
(False, True, False)

>>> _test(ToolVersion('1.1'), ToolVersion('1.0'))
(False, False, True)

>>> _test(ToolVersion('1.1'), ToolVersion('2'))
(True, False, False)

>>> _test(ToolVersion('1.00'), ToolVersion('1.0'))
(False, True, False)

>>> _test(ToolVersion('1.09'), ToolVersion('1.10'))
(True, False, False)

>>> _test(ToolVersion('1.0'), ToolVersion('1.0-0'))
(True, False, False)

>>> _test(ToolVersion('1.0'), ToolVersion('1.0-1'))
(True, False, False)

>>> _test(ToolVersion('1.0-0'), ToolVersion('1.0-1'))
(True, False, False)

>>> _test(ToolVersion('1.0-0'), ToolVersion('1.1'))
(True, False, False)

>>> _test(ToolVersion(ToolVersion.DEFAULT), ToolVersion('1'))
(False, False, True)

>>> _test(ToolVersion('1'), ToolVersion(ToolVersion.DEFAULT))
(True, False, False)

>>> _test(ToolVersion(ToolVersion.DEFAULT), ToolVersion(ToolVersion.DEFAULT))
(False, True, False)
is_default()
Return type:

bool

is_partial(other)

Returns True if self is equal to or a partial version of other.

Return type:

bool

>>> ToolVersion('1').is_partial(ToolVersion('1'))
True

>>> ToolVersion('1').is_partial(ToolVersion('1.0'))
True

>>> ToolVersion('1').is_partial(ToolVersion('1.0-0'))
True

>>> ToolVersion('1.0').is_partial(ToolVersion('1.0-0'))
True

>>> ToolVersion('1.0').is_partial(ToolVersion('1'))
False

>>> ToolVersion('2').is_partial(ToolVersion('1.0'))
False

>>> ToolVersion('1').is_partial(default_tool_version)
False

>>> default_tool_version.is_partial(ToolVersion('1'))
True

>>> default_tool_version.is_partial(default_tool_version)
True
DEFAULT: ClassVar[str] = '_'

Constant indicating a default version

value: str

The original value

property version: tuple[str | tuple[int, ...]]

A tuple representing the value split on the dot and dash characters (., -) and each part converted to int if possible, and otherwise a str.

The special value DEFAULT is converted into an empty tuple.

momotor.options.tools.version.default_tool_version = ToolVersion(value='_')

Constant ToolVersion(ToolVersion.DEFAULT)