Assets¶
An asset is a file that belongs to a job: its recipe, the product under test, or a result. This module moves assets between a client or worker and the broker. The broker stores each asset once, identified by the hash of its content. See Hash.
Upload and download¶
Callers name an asset with an
AssetQuery. The query states the
category of the asset, such as recipe or result. A query for a result also
states the step and task that produced it.
send_asset() uploads a file:
from momotor.rpc.asset.remote import send_asset
from momotor.rpc.proto.asset_pb2 import RECIPE, AssetQuery
await send_asset(
stub, job_id,
AssetQuery(category=RECIPE),
'recipe.zip', server_info,
)
The function hashes the file first. If the broker already stores that content, the broker says so and no file data crosses the network. Small files ride inside their hash and are not sent either. Otherwise the file is streamed in chunks.
receive_asset() downloads a file. Its
optional exists callback lets the caller skip a download when it
already has the content locally.
Workers that upload a task result can attach the identity of the task attempt and, when the failure happened outside the checklet, an execution failure record. Self-test protocol defines these fields.
Added in version 7.0.0: The execution_failure and task_attempt_id parameters of
send_asset().
Server side¶
Broker code uses the stream helpers in
momotor.rpc.asset.server. They are the receiving and sending
counterparts of the functions above. Client and worker code does not call
them.
Errors¶
Asset transfers can fail in two separate ways, and code that handles errors must catch both.
The classes in momotor.rpc.asset.exceptions report problems
that the transfer code in this package detects itself: the received
size or hash does not match the announced asset, or the stream ends
before the asset is complete. Each of these classes derives directly
from Exception. None of them derives from the classes in
Exception, so catch them separately.
Broker-reported asset errors are different. The broker reports those as
AssetException, a subclass of
RPCException in
momotor.rpc.exception. Catching RPCException covers
AssetException. It does not cover the transfer errors above.
API reference¶
Remote¶
- async momotor.rpc.asset.remote.receive_asset(stub, job_id, query, path, exists=None, process_executor=None, timeout=None)¶
Function for use by a client or worker to request and receive an asset from the broker.
Produces log messages on the
momotor.rpc.asset.remotelogger.- Parameters:
stub (
ClientStub|WorkerStub) – The connected stub to the broker.job_id (
str) – Id of the jobquery (
AssetQuery) – Query for the assetpath (
str|Path) – Local file path where the file is to be storedexists (
Callable[[AssetData],Awaitable[bool]] |None) – A function that checks if the file is already known locallyprocess_executor (
Executor|None) – An asyncio executor to execute long running CPU bound tasks
- Return type:
- Returns:
A tuple with the
AssetDataidentifying the asset, and a boolean indicating whether the asset already exists locally.
- async momotor.rpc.asset.remote.send_asset(stub, job_id, query, path, server_info, *, process_executor=None, timeout=None, execution_failure=None, task_attempt_id=None)¶
Function for use by a client or worker to send an asset to the broker server
Produces log messages on the
momotor.rpc.asset.remotelogger.- Parameters:
stub (
ClientStub|WorkerStub) – The connected stub to the broker.job_id (
str) – Id of the jobquery (
AssetQuery) – Query for the assetserver_info (
ServerInfoResponse) – The server’s info responseprocess_executor (
Executor|None) – An asyncio executor to execute long running CPU bound tasksexecution_failure (
ExecutionFailure|None) – Infrastructure failure accompanying this task result.task_attempt_id (
str|None) – Broker-issued identity of the result’s task attempt.
- Return type:
Added in version 7.0.0: The execution_failure and task_attempt_id parameters.
Server¶
- async momotor.rpc.asset.server.download_asset_stream(data, chunks, stream, *, timeout=None)¶
Function for use by the broker to handle an asset download stream from a client or worker.
This communicates with a remote’s
momotor.rpc.asset.remote.receive_asset()- Parameters:
data (
AssetData) – Anmomotor.rpc.proto.asset_pb2.AssetDataobject identifying the requested assetchunks (
AsyncGenerator[bytes,None] |None) – A generator producing chunks of the requested assetstream (
Stream) – The gPRC stream to the client or workertimeout – Timeout
- async momotor.rpc.asset.server.upload_asset_stream(creator, stream, *, timeout=None)¶
Function for use by the broker to handle an asset upload stream to a client or worker.
This communicates with a remote’s
momotor.rpc.asset.remote.send_asset()- Parameters:
creator (
AsyncGenerator[int,bytes|None]) – A generator that processes the chunks and writes them to a file or other storage, and yields the number of bytes remaining.stream (
Stream) – The gPRC stream to the client or workertimeout – Timeout
Exceptions¶
- exception momotor.rpc.asset.exceptions.AssetHashMismatchError¶
- exception momotor.rpc.asset.exceptions.AssetSizeMismatchError¶
- exception momotor.rpc.asset.exceptions.UnexpectedEndOfStream¶
Utils¶
- async momotor.rpc.asset.utils.file_reader(path, mode, queue, *, chunk_size=8388608)¶
Helper to read chunks from a file into a queue
- async momotor.rpc.asset.utils.file_writer(path, mode, queue)¶
Helper to write chunks from a queue into a file
- momotor.rpc.asset.utils.get_file_hash(func_code, path)¶
Calculate a hash of the given file’s content
- momotor.rpc.asset.utils.get_file_multihash(path, server_info=None)¶
Get multihash value for file in path
If the file is small enough to fit, the entire contents will be encoded in the multihash instead of an actual hash. A boolean in the result will indicate whether the result is a real hash or encoded content
If a server_info object is provided, the resulting hash will be compatible with the server, otherwise the hash generated will use the capabilities of the local system.