Assets¶
The momotor.rpc.asset
module contains functions and classes to transfer asset files to and from the
Momotor broker.
Remote¶
- async momotor.rpc.asset.remote.send_asset(stub, job_id, query, path, server_info, *, process_executor=None, timeout=None)¶
Function for use by a client or worker to send an asset to the broker server
Produces log messages on the
momotor.rpc.asset.remote
logger.- Parameters
stub (
Union
[ClientStub
,WorkerStub
]) – The connected stub to the broker.job_id (
str
) – Id of the jobquery (
AssetQuery
) – Query for the assetpath (
Union
[str
,Path
]) – Local file path of the file to sendserver_info (
ServerInfoResponse
) – The server’s info responseprocess_executor – An asyncio executor to execute long running CPU bound tasks
timeout – Timeout
- Return type
- async momotor.rpc.asset.remote.receive_asset(stub, job_id, query, path, exists=None, process_executor=None, timeout=None)¶
Function for use by a client or worker to request and receive an asset from the broker.
Produces log messages on the
momotor.rpc.asset.remote
logger.- Parameters
stub (
Union
[ClientStub
,WorkerStub
]) – The connected stub to the broker.job_id (
str
) – Id of the jobquery (
AssetQuery
) – Query for the assetpath (
Union
[str
,Path
]) – Local file path where the file is to be storedexists (
Optional
[Callable
[[AssetData
],Awaitable
[bool
]]]) – A function that checks if the file is already known locallyprocess_executor – An asyncio executor to execute long running CPU bound tasks
timeout – Timeout
- Return type
- Returns
A tuple with the
AssetData
identifying the asset, and a boolean indicating whether the asset already exists locally.
Server¶
- async momotor.rpc.asset.server.download_asset_stream(data, chunks, stream, *, timeout=None)¶
Function for use by the broker to handle an asset download stream from a client or worker.
This communicates with a remote’s
momotor.rpc.asset.remote.receive_asset()
- Parameters
data (
AssetData
) – Anmomotor.rpc.proto.asset_pb2.AssetData
object identifying the requested assetchunks (
Optional
[AsyncGenerator
[bytes
,None
]]) – A generator producing chunks of the requested assetstream (
Stream
) – The gPRC stream to the client or workertimeout – Timeout
- async momotor.rpc.asset.server.upload_asset_stream(creator, stream, *, timeout=None)¶
Function for use by the broker to handle an asset upload stream to a client or worker.
This communicates with a remote’s
momotor.rpc.asset.remote.send_asset()
- Parameters
creator (
AsyncGenerator
[int
,Optional
[bytes
]]) – A generator that processes the chunks and writes them to a file or other storage, and yields the number of bytes remaining.stream (
Stream
) – The gPRC stream to the client or workertimeout – Timeout
Exceptions¶
- exception momotor.rpc.asset.exceptions.AssetSizeMismatchError¶
- exception momotor.rpc.asset.exceptions.AssetHashMismatchError¶
- exception momotor.rpc.asset.exceptions.UnexpectedEndOfStream¶
Utils¶
- momotor.rpc.asset.utils.get_file_multihash(path, server_info=None)¶
Get multihash value for file in path
If the file is small enough to fit, the entire contents will be encoded in the multihash instead of an actual hash. A boolean in the result will indicate whether the result is a real hash or encoded content
If a server_info object is provided, the resulting hash will be compatible with the server, otherwise the hash generated will use the capabilities of the local system.
- momotor.rpc.asset.utils.get_file_hash(func_code, path)¶
Calculate a hash of the given file’s content
- async momotor.rpc.asset.utils.file_reader(path, mode, queue, *, chunk_size=8388608)¶
Helper to read chunks from a file into a queue