Helpers

Helpers are filter functions generated according to a specification. They can be used in multiple string filter scenarios, such as glob filtering. These functions are designed to be used with the decorators above, but can be used in other scenarios if needed.

Date Range Filter

kedro_partitioned.pipeline.decorators.helper_factory.date_range_filter(min_date=Timestamp('1677-09-21 00:12:43.145224193'), max_date=Timestamp('2262-04-11 23:47:16.854775807'), format='%Y-%m-%d')

Generates a date_range filter function.

Parameters:

• min_date (str, optional) – Defaults to pd.Timestamp.min.

• max_date (str, optional) – Defaults to pd.Timestamp.max.

• format (str, optional) – Defaults to DATE_FORMAT_ISO.

Returns:

A function that takes a string as input, extracts

a regex match of the specified format and returns True whether a string is in date range (both inclusive), otherwise False.

Return type:

IsFunction[str]

Example

>>> upper = date_range_filter(max_date='2020-02-02', format='%Y-%m-%d')
>>> upper('2020-02-03')
False
>>> upper('2020-02-02')
True
>>> upper('random/string/2020-01-01/suffix')
True

>>> lower = date_range_filter(min_date='20200202', format='%Y%m%d')
>>> lower('20200201')
False
>>> lower('20200202')
True
>>> lower('random/string/20200301/suffix')
True

>>> bounded = date_range_filter(min_date='02-02/2020',
...     max_date='02-04/2020', format='%d-%m/%Y')
>>> bounded('prefix/02-04/2020/suffix/')
True
>>> bounded('prefix-02-02/2020-suffix')
True
>>> bounded('prefix/24-03/2020-suffix')
True
>>> bounded('prefix-13-01/2020/suffix')
False
>>> bounded('prefix25-05/2020suffix')
False

Warning

the specified format format must not be ambiguous. for example, if a string format is ‘%Y%m%d’, and another number of 8 digits appear, it will be recognized as date. if it is a path for example, this can be avoided by using ‘/%Y%m%d/’ as the format pattern

Regex Filter

kedro_partitioned.pipeline.decorators.helper_factory.regex_filter(pattern)

Converts a regex pattern into a boolean filter.

Parameters:

pattern (str) –

Returns:

A function that takes a string as input and checks

if it matches a pattern, if matches return True, otherwise False.

Return type:

IsFunction[str]

Example

>>> fn = regex_filter(r'ab?')
>>> fn('ad')
True
>>> fn('ab')
True
>>> fn('db')
False

Not Filter

kedro_partitioned.pipeline.decorators.helper_factory.not_filter(fn)

Inverts a function return.

Parameters:

fn (IsFunction[str]) –

Return type:

Callable[[str], bool]

Returns:

IsFunction[str]

Example

>>> def fn(x: str) -> bool: return x == 'a'
>>> nfn = not_filter(fn)
>>> fn('a')
True
>>> fn('b')
False
>>> nfn('a')
False
>>> nfn('b')
True