Assets

An asset is a file that belongs to a job: its recipe, the product under test, or a result. This module moves assets between a client or worker and the broker. The broker stores each asset once, identified by the hash of its content. See Hash.

Upload and download

Callers name an asset with an AssetQuery. The query states the category of the asset, such as recipe or result. A query for a result also states the step and task that produced it.

send_asset() uploads a file:

from momotor.rpc.asset.remote import send_asset
from momotor.rpc.proto.asset_pb2 import RECIPE, AssetQuery

await send_asset(
    stub, job_id,
    AssetQuery(category=RECIPE),
    'recipe.zip', server_info,
)

The function hashes the file first. If the broker already stores that content, the broker says so and no file data crosses the network. Small files ride inside their hash and are not sent either. Otherwise the file is streamed in chunks.

receive_asset() downloads a file. Its optional exists callback lets the caller skip a download when it already has the content locally.

Workers that upload a task result can attach the identity of the task attempt and, when the failure happened outside the checklet, an execution failure record. Self-test protocol defines these fields.

Added in version 7.0.0: The execution_failure and task_attempt_id parameters of send_asset().

Server side

Broker code uses the stream helpers in momotor.rpc.asset.server. They are the receiving and sending counterparts of the functions above. Client and worker code does not call them.

Errors

Asset transfers can fail in two separate ways, and code that handles errors must catch both.

The classes in momotor.rpc.asset.exceptions report problems that the transfer code in this package detects itself: the received size or hash does not match the announced asset, or the stream ends before the asset is complete. Each of these classes derives directly from Exception. None of them derives from the classes in Exception, so catch them separately.

Broker-reported asset errors are different. The broker reports those as AssetException, a subclass of RPCException in momotor.rpc.exception. Catching RPCException covers AssetException. It does not cover the transfer errors above.

API reference

Remote

async momotor.rpc.asset.remote.receive_asset(stub, job_id, query, path, exists=None, process_executor=None, timeout=None)

Function for use by a client or worker to request and receive an asset from the broker.

Produces log messages on the momotor.rpc.asset.remote logger.

Parameters:
Return type:

tuple[AssetData, bool]

Returns:

A tuple with the AssetData identifying the asset, and a boolean indicating whether the asset already exists locally.

async momotor.rpc.asset.remote.send_asset(stub, job_id, query, path, server_info, *, process_executor=None, timeout=None, execution_failure=None, task_attempt_id=None)

Function for use by a client or worker to send an asset to the broker server

Produces log messages on the momotor.rpc.asset.remote logger.

Parameters:
  • stub (ClientStub | WorkerStub) – The connected stub to the broker.

  • job_id (str) – Id of the job

  • query (AssetQuery) – Query for the asset

  • path (str | Path) – Local file path of the file to send

  • server_info (ServerInfoResponse) – The server’s info response

  • process_executor (Executor | None) – An asyncio executor to execute long running CPU bound tasks

  • timeout (float | None) – Timeout

  • execution_failure (ExecutionFailure | None) – Infrastructure failure accompanying this task result.

  • task_attempt_id (str | None) – Broker-issued identity of the result’s task attempt.

Return type:

None

Added in version 7.0.0: The execution_failure and task_attempt_id parameters.

Server

async momotor.rpc.asset.server.download_asset_stream(data, chunks, stream, *, timeout=None)

Function for use by the broker to handle an asset download stream from a client or worker.

This communicates with a remote’s momotor.rpc.asset.remote.receive_asset()

Parameters:
async momotor.rpc.asset.server.upload_asset_stream(creator, stream, *, timeout=None)

Function for use by the broker to handle an asset upload stream to a client or worker.

This communicates with a remote’s momotor.rpc.asset.remote.send_asset()

Parameters:
  • creator (AsyncGenerator[int, bytes | None]) – A generator that processes the chunks and writes them to a file or other storage, and yields the number of bytes remaining.

  • stream (Stream) – The gPRC stream to the client or worker

  • timeout – Timeout

Exceptions

exception momotor.rpc.asset.exceptions.AssetHashMismatchError
exception momotor.rpc.asset.exceptions.AssetSizeMismatchError
exception momotor.rpc.asset.exceptions.UnexpectedEndOfStream

Utils

async momotor.rpc.asset.utils.file_reader(path, mode, queue, *, chunk_size=8388608)

Helper to read chunks from a file into a queue

Parameters:
  • path (Path) – file to read

  • mode (str) – file read mode

  • queue (Queue) – queue to put the chunks into. Should be a one-place queue

  • chunk_size (int) – maximum size of the chunks

async momotor.rpc.asset.utils.file_writer(path, mode, queue)

Helper to write chunks from a queue into a file

Parameters:
  • path (Path) – file to write

  • mode (str) – file write mode

  • queue (Queue) – queue to get the chunks from. Should be a one-place queue

momotor.rpc.asset.utils.get_file_hash(func_code, path)

Calculate a hash of the given file’s content

Parameters:
  • func_code (str | int) – a multihash code (string or integer)

  • path (str | Path) – path of the file to hash

Return type:

bytes

Returns:

unencoded hash digest generated by the hash function

momotor.rpc.asset.utils.get_file_multihash(path, server_info=None)

Get multihash value for file in path

If the file is small enough to fit, the entire contents will be encoded in the multihash instead of an actual hash. A boolean in the result will indicate whether the result is a real hash or encoded content

If a server_info object is provided, the resulting hash will be compatible with the server, otherwise the hash generated will use the capabilities of the local system.

Parameters:
Return type:

tuple[bytes, bool]

Returns:

tuple of base58 encoded multihash, and boolean indicating if identity encoding was used