Hash

The broker stores each asset once, no matter how many jobs use it. To do that, it identifies an asset by a hash of its content. Two uploads with the same content get the same hash, so the second upload sends no data.

Momotor uses the multihash format, encoded with base58. A multihash carries the code of the hashing algorithm next to the digest. That lets a client and a broker agree on an algorithm they both support, and it lets a future version move to a stronger algorithm without changing the format. SUPPORTED_HASH_FUNCS lists the algorithm codes this version can use, most preferred first.

Very short content, shorter than MAX_IDENTITY_LENGTH bytes, is its own hash. The content bytes are stored inside the hash value, in what the multihash format calls an identity hash. The broker then needs no separate transfer for such assets.

You rarely call these functions directly. The asset helpers in Assets hash files for you. Use the functions here when you need the hash of a file before any transfer, for example to check whether the broker must receive it at all:

import hashlib

from momotor.rpc.hash.funcs import get_algorithm_by_code, get_code_by_hash_obj

algorithm = get_algorithm_by_code('sha2-256')
digest = algorithm(b'content')
print(get_code_by_hash_obj(digest) == get_code_by_hash_obj(hashlib.sha256()))

API reference

Functions

momotor.rpc.hash.funcs.get_algorithm_by_code(code)

Get a PEP 452 compatible hashing object by its multihash code or name

Parameters:

code (str | int) – multihash code or name

Return type:

Callable

Returns:

PEP 452 compatible hashing object

momotor.rpc.hash.funcs.get_code_by_hash_obj(hash_obj)

Get the multihash code for a PEP 452 compatible hashing object

Parameters:

hash_obj – PEP 452 compatible hashing object

Return type:

int

Returns:

multihash code

momotor.rpc.hash.funcs.SUPPORTED_HASH_FUNCS = [20, 21, 19, 22, 23, 18, 17]

Available multihash algorithm codes, from most to least preferred.

Identity hash

A PEP 452 compatible implementation of an identity hashing object.

The identity hash stores the literal content as a hash:

>>> IdentityHash(b'any random data').digest() == b'any random data'
True
class momotor.rpc.hash.identity.IdentityHash(data=None)
copy()

Create a copy of this hash object

Return type:

IdentityHash

Returns:

Copy of the object

digest()

Get the current data

Return type:

bytes

Returns:

the current data

hexdigest()

Get the current data as hex-encoded digest

Return type:

bytes

Returns:

the hex-encoded current data

update(data)

Extend the hash with data

Parameters:

data (bytes) – data to concatenate to the existing data

property block_size
property digest_size
name = 'identity'

identifier for the hashing function

momotor.rpc.hash.identity.new(data=None)

Create a identity hash object

Parameters:

data (bytes | memoryview | None) – initial data for the hash

Return type:

IdentityHash

Returns:

the identity hash object

momotor.rpc.hash.identity.digest_size = None

Digest size is variable