Assets#
The momotor.rpc.asset
module contains functions and classes to transfer asset files to and from the
Momotor broker.
Remote#
- async momotor.rpc.asset.remote.receive_asset(stub, job_id, query, path, exists=None, process_executor=None, timeout=None)#
Function for use by a client or worker to request and receive an asset from the broker.
Produces log messages on the
momotor.rpc.asset.remote
logger.- Parameters:
stub (
Union
[ClientStub
,WorkerStub
]) – The connected stub to the broker.job_id (
str
) – Id of the jobquery (
AssetQuery
) – Query for the assetpath (
str
|Path
) – Local file path where the file is to be storedexists (
Callable
[[AssetData
],Awaitable
[bool
]]) – A function that checks if the file is already known locallyprocess_executor – An asyncio executor to execute long running CPU bound tasks
timeout – Timeout
- Return type:
- Returns:
A tuple with the
AssetData
identifying the asset, and a boolean indicating whether the asset already exists locally.
- async momotor.rpc.asset.remote.send_asset(stub, job_id, query, path, server_info, *, process_executor=None, timeout=None)#
Function for use by a client or worker to send an asset to the broker server
Produces log messages on the
momotor.rpc.asset.remote
logger.- Parameters:
stub (
Union
[ClientStub
,WorkerStub
]) – The connected stub to the broker.job_id (
str
) – Id of the jobquery (
AssetQuery
) – Query for the assetserver_info (
ServerInfoResponse
) – The server’s info responseprocess_executor – An asyncio executor to execute long running CPU bound tasks
timeout – Timeout
- Return type:
Server#
- async momotor.rpc.asset.server.download_asset_stream(data, chunks, stream, *, timeout=None)#
Function for use by the broker to handle an asset download stream from a client or worker.
This communicates with a remote’s
momotor.rpc.asset.remote.receive_asset()
- Parameters:
data (
AssetData
) – Anmomotor.rpc.proto.asset_pb2.AssetData
object identifying the requested assetchunks (
AsyncGenerator
[bytes
,None
] |None
) – A generator producing chunks of the requested assetstream (
Stream
) – The gPRC stream to the client or workertimeout – Timeout
- async momotor.rpc.asset.server.upload_asset_stream(creator, stream, *, timeout=None)#
Function for use by the broker to handle an asset upload stream to a client or worker.
This communicates with a remote’s
momotor.rpc.asset.remote.send_asset()
- Parameters:
creator (
AsyncGenerator
[int
,bytes
|None
]) – A generator that processes the chunks and writes them to a file or other storage, and yields the number of bytes remaining.stream (
Stream
) – The gPRC stream to the client or workertimeout – Timeout
Exceptions#
- exception momotor.rpc.asset.exceptions.AssetHashMismatchError#
- exception momotor.rpc.asset.exceptions.AssetSizeMismatchError#
- exception momotor.rpc.asset.exceptions.UnexpectedEndOfStream#
Utils#
- async momotor.rpc.asset.utils.file_reader(path, mode, queue, *, chunk_size=8388608)#
Helper to read chunks from a file into a queue
- async momotor.rpc.asset.utils.file_writer(path, mode, queue)#
Helper to write chunks from a queue into a file
- momotor.rpc.asset.utils.get_file_hash(func_code, path)#
Calculate a hash of the given file’s content
- momotor.rpc.asset.utils.get_file_multihash(path, server_info=None)#
Get multihash value for file in path
If the file is small enough to fit, the entire contents will be encoded in the multihash instead of an actual hash. A boolean in the result will indicate whether the result is a real hash or encoded content
If a server_info object is provided, the resulting hash will be compatible with the server, otherwise the hash generated will use the capabilities of the local system.