Sequence filtering

The momotor.bundles.utils.filters package provides two filterable sequences, the FilterableList and FilterableTuple. Both add the same functions to the standard python list and tuple types.

The interface is loosely based on Django QuerySets field lookups.

The examples in this documentation use the following filterable lists:

>>> data_str = FilterableList([ItemA(a='abc'), ItemA(a='ABC'), ItemA(a='def')])
>>> data_int = FilterableList([ItemA(a=1), ItemA(a=2), ItemA(a=3)])
>>> data_set = FilterableList([ItemA(a={1, 2}), ItemA(a={2, 3}), ItemA(a={3, 4})])
>>> data_ab = FilterableList([ItemAB(a=1, b=1), ItemAB(a=1, b=2), ItemAB(a=2, b=2), ItemAB(a=3, b=3)])

Where ItemA and ItemAB are datatypes having both an a attribute, and ItemAB also has a b attribute.

Filtering and excluding

The most basic lookup looks like this:

>>> data_ab.filter(a=1)  # filter all items with a==1
[ItemAB(a=1, b=1), ItemAB(a=1, b=2)]

This filters data_ab for any objects containing an a attribute with value equal to 1.

The result of filter() is itself another filterable type, so filters can be chained:

>>> data_ab.filter(a=1).filter(b=2)  # filter all items with a==1, the filter the result for a==2
[ItemAB(a=1, b=2)]

FilterableList.exclude(...) excludes items:

>>> data_ab.exclude(a=1)  # exclude all items where a==1
[ItemAB(a=2, b=2), ItemAB(a=3, b=3)]

Providing multiple lookups in filter() or exclude() will filter or exclude objects matching all the lookups:

>>> data_ab.filter(a=1, b=2)  # filter all items with a==1 and b==2
[ItemAB(a=1, b=2)]

>>> data_ab.exclude(a=1, b=2)  # exclude all items with a==1 and b==2
[ItemAB(a=1, b=1), ItemAB(a=2, b=2), ItemAB(a=3, b=3)]

Not() object

Lookups can be negated using a Not object:

>>> data_ab.filter(Not(a=1))  # filter all items where `a` is not 1
[ItemAB(a=2, b=2), ItemAB(a=3, b=3)]

FilterableList.exclude(...) is a shortcut for FilterableList.filter(Not(...))

All() object

The All object can be used to make lookups with multiple arguments more explicit:

>>> data_ab.filter(All(a=1, b=2))  # filter all items with a==1 and b==2
[ItemAB(a=1, b=2)]

Any() object

Lookups can also be combined using the Any object to find an object which matches any of the requested lookups:

>>> data_ab.filter(Any(a=1, b=2))  # filter all items with a==1 OR b==2
[ItemAB(a=1, b=1), ItemAB(a=1, b=2), ItemAB(a=2, b=2)]

Combining filter objects

Lookup objects can be combined, for example:

>>> data_ab.filter(Any(All(a=1, b=2), b=3))  # Look for items with (a==1 AND b==2) OR b==3
[ItemAB(a=1, b=2), ItemAB(a=3, b=3)]

F() object

Using keyword arguments it is not possible to filter for multiple values of the same property, since filter(a=1, a=3) has a repeated keyword a:

>>> data_ab.filter(a=1, a=3)  # this is a SyntaxError
Traceback (most recent call last):
...
SyntaxError: keyword argument repeated

To look for data_ab values for the same attribute, use the F object to wrap individual queries:

>>> data_ab.filter(F(a=1), F(a=3))  # filter all items with a==1 AND a==3 (which is nonsensical indeed, but this is just an example)
[]

This can also be combined with the other filter objects like Any

>>> data_ab.filter(Any(F(a=1), F(a=3)))  # filter items with a==1 OR a==3
[ItemAB(a=1, b=1), ItemAB(a=1, b=2), ItemAB(a=3, b=3)]

F objects can also be used to easily pass on filter queries as arguments to functions:

>>> def filter_data(f: F):
...     return data_ab.filter(f)

>>> for f in [F(a=1), F(a=2)]:
...     print(filter_data(f))
[ItemAB(a=1, b=1), ItemAB(a=1, b=2)]
[ItemAB(a=2, b=2)]

Any, All and Not are subclasses of F.

When mixing F arguments and keyword arguments, Python syntax requires the positional arguments to be provided before the keyword arguments. The following is invalid:

>>> data_ab.filter(a=1, Any(a=2, b=3))  # positional argument follows keyword argument
Traceback (most recent call last):
...
SyntaxError: positional argument follows keyword argument

Using an F object solves this issue:

>>> data_ab.filter(F(a=1), Any(a=2, b=3))
[]

Negation

The Any and All classes support negation using the ~ operator, for example ~All(…) is the same as Not(All(…))

Lookup types

Additionally to filtering or excluding exact values, there are several other lookup types. A lookup type is added to the lookup by separating the attribute name and the lookup with a double underscore __, eg. a__ne applies the not-equal lookup to the a attribute.

The following filter lookup operations are available:

Lookup type	Case sensitive lookup	Case insensitive lookup
Is-equal	(none) / eq / is	ieq / iis
Not-equal	ne	ine
Contains	contains	icontains
In	in	iin
Starts-with	startswith	istartswith
Ends-with	endswith	iendswith
Glob	glob	iglob
Regular expression	re	ire

Note: All case insensitive lookups only accept strings or a sequence of strings. Any other value will raise a TypeError:

>>> data_int.filter(a__ine=1)
Traceback (most recent call last):
...
TypeError: Expected a string or sequence of strings, got <class 'int'>

Is-equal lookup

Filters exact values.

Operator: No operator, is or eq

Examples:

>>> data_str.filter(a='abc')
[ItemA(a='abc')]

>>> data_str.filter(a__is='ABC')
[ItemA(a='ABC')]

>>> data_str.filter(a__eq='def')
[ItemA(a='def')]

>>> data_int.filter(a=1)
[ItemA(a=1)]

>>> data_int.filter(a='abc')
[]

Is-equal lookup (case insensitive)

Filters string attributes case insensitive.

Lookup operator: ieq / iis

Examples:

>>> data_str.filter(a__ieq='abc')
[ItemA(a='abc'), ItemA(a='ABC')]

>>> data_int.filter(a__iis='abc')
[]

Not-equal lookup

Lookup operator: ne

Usage example:

>>> data_str.filter(a__ne='abc')
[ItemA(a='ABC'), ItemA(a='def')]

>>> data_int.filter(a__ne=1)
[ItemA(a=2), ItemA(a=3)]

Not-equal lookup (case insensitive)

Lookup operator: ine

Usage example:

>>> data_str.filter(a__ine='abc')
[ItemA(a='def')]

Contains lookup

Lookup operator: contains

Works on Python types implementing object.__contains__() like str, sequences and sets.

Usage example:

>>> data_str.filter(a__contains='a')
[ItemA(a='abc')]

>>> data_set.filter(a__contains=2)
[ItemA(a={1, 2}), ItemA(a={2, 3})]

Contains lookup (case insensitive)

Lookup operator: icontains

Works on Python str and sets and sequences containing strings.

Usage example:

>>> data_str.filter(a__icontains='a')
[ItemA(a='abc'), ItemA(a='ABC')]

In lookup

Lookup operator: in

The reverse of contains. The lookup value must implement object.__contains__()

Usage example:

>>> data_str.filter(a__in={'abc', 'def', 'ghi'})
[ItemA(a='abc'), ItemA(a='def')]

>>> data_int.filter(a__in={0, 1, 2})
[ItemA(a=1), ItemA(a=2)]

In lookup (case insensitive)

Lookup operator: iin

Usage example:

>>> data_str.filter(a__iin={'abc', 'def', 'ghi'})
[ItemA(a='abc'), ItemA(a='ABC'), ItemA(a='def')]

Starts-with lookup

Lookup operator: startswith

Usage example:

>>> data_str.filter(a__startswith='a')
[ItemA(a='abc')]

Starts-with lookup (case insensitive)

Lookup operator: istartswith

Usage example:

>>> data_str.filter(a__istartswith='a')
[ItemA(a='abc'), ItemA(a='ABC')]

Ends-with lookup

Lookup operator: endswith

Usage example:

>>> data_str.filter(a__endswith='c')
[ItemA(a='abc')]

Ends-with lookup (case insensitive)

Lookup operator: iendswith

Usage example:

>>> data_str.filter(a__iendswith='c')
[ItemA(a='abc'), ItemA(a='ABC')]

Glob lookup

Lookup operator: glob

Uses the fnmatchcase() function.

Usage example:

>>> data_str.filter(a__glob='a*')
[ItemA(a='abc')]

Glob lookup (case insensitive)

Lookup operator: iglob

Uses the fnmatchcase() function.

Usage example:

>>> data_str.filter(a__iglob='a*')
[ItemA(a='abc'), ItemA(a='ABC')]

Regular expression lookup

Lookup operator: re

Usage example:

>>> data_str.filter(a__re=r'.b.')
[ItemA(a='abc')]

Regular expression lookup (case insensitive)

Lookup operator: ire

Usage example:

>>> data_str.filter(a__ire=r'.b.')
[ItemA(a='abc'), ItemA(a='ABC')]

Additional methods

FilterableList and FilterableTuple have several more functions:

ifilter(...)

Same as filter(), but returning an iterable of items.

iexclude(...)

Same as exclude(), but returning an iterable of items.

filter_with(func: Callable[[Any], bool])

Filter the sequence using a filter function. The function receives an item and should return a boolean indicating if the item should be included in the result.