Package matchops

initialize matchops package

Sub-modules

matchops.constructor

module containing generic resources for constructing dataframes used in data operations.

matchops.constructor_tools

generic specs and func defs for data matching operations

matchops.matcher

module containing generic resources for constructing dataframes used in data operations.

matchops.standard_matchers

Base matchers for use with standard claimmaker data structures

Classes

class AddColumn (new_column_name: str, source_columns: list[str], apply_function: collections.abc.Callable[..., str], axis: int = 1)

defines a new column built from existing columns in a dataframe using the .apply() pandas function with axis=1.

Class variables

var apply_function : collections.abc.Callable[..., str]
var axis : int
var new_column_name : str
var source_columns : list[str]
class DataFrameConstructor (constructor: collections.abc.Callable[..., pandas.core.frame.DataFrame] = <function read_csv>, converters: dict[str, collections.abc.Callable[..., str]] = <factory>, add_columns: list[AddColumn] = <factory>, kwargs: dict[str, typing.Any] = <factory>, post_process: collections.abc.Callable[[pandas.core.frame.DataFrame], pandas.core.frame.DataFrame] | None = None)

stores pandas methods, arguments, and data transformations required to construct dataframes from arbitrary datasources from a common api.

Attributes

constructor : Callable[…, pd.DataFrame]
pandas method used to construct the dataframe
converters : dict[str, Callable[…, str]]
dictionary of column names and functions to apply to the column values during construction.
add_columns : list[AddColumn]
list of AddColumn dataclasses defining new columns to be added to the dataframe.
kwargs : dict[str, Any]
keyword arguments passed to constructor().
post_process : Callable[[pd.DataFrame], pd.DataFrame] | None
function to apply to the dataframe after construction.

Class variables

var add_columns : list[AddColumn]
var converters : dict[str, collections.abc.Callable[..., str]]
var kwargs : dict[str, typing.Any]
var post_process : collections.abc.Callable[[pandas.core.frame.DataFrame], pandas.core.frame.DataFrame] | None

Methods

def get_frame(self, data: Any, *, static_cols: collections.abc.Mapping[str, str] | None = None, reqd: collections.abc.Sequence[str] = ()) ‑> pandas.core.frame.DataFrame

Construct a dataframe using the supplied data

Args

data : Any
data to be passed to constructor

KwArgs

static_cols
a mapping representing a name and static value to be added as a column to the dataframe
reqd
list of column names that must be present in the dataframe. if not present, add the column with a default value of "".
class DataFrameMatcher (left_source: Any = b'', left_constructor: DataFrameConstructor = <factory>, left_static_cols: collections.abc.Mapping[str, str] = <factory>, right_source: Any = b'', right_constructor: DataFrameConstructor = <factory>, right_static_cols: collections.abc.Mapping[str, str] = <factory>, joins: list[JoinSpec] = <factory>, dq_not_match_cols: list[tuple[str, str]] = <factory>, dq_on_match_cols: list[tuple[str, str]] = <factory>, output_index: list[str] = <factory>, output_mapping: collections.abc.Mapping[str, str] = <factory>, sort_ascending: bool = True, keep_unmatched: bool = True, log_unmatched: bool = False, log_duplicates: bool = False, matcher_type: str = 'dataframe_matcher', output_dir: str = '')

Settings and functions required to process a facility schedule and match it to data found in a demographics export and the data extracted from the PDFs

Args

left_source : io.BytesIO, bytes, or str
source data for left (primary) dataframe. All types are converted to io.BytesIO and passed to left_construtor to be loaded by pandas
left_constructor : DataFrameConstrutor
DataFrameConstructor instance defining pandas loading routine and data transformations for the left_source.
left_static_cols : Mapping[str, str]
a mapping representing a name and static value to be added as a column to left (primary) dataframe
right_source : io.BytesIO, bytes, or str
source data for left (matching) dataframe. All types are converted to io.BytesIO and passed to right_constructor to be loaded by pandas
right_constructor : DataFrameConstrutor
DataFrameConstructor instance defining pandas loading routine and data transformations for the right_source.
right_static_cols : Mapping[str, str]
a mapping representing a name and static value to be added as a column to right (secondary) df
joins : list[JoinSpec]
list of successive join operations to call in sequence until all rows in left have been matched to a row in right.
dq_not_match_cols : list[tuple[str, str]]
if columns are populated for both left and right, the values must also match for a join to occur.
dq_on_match_cols : list[tuple[str, str]]
reverse of above, i.e. drop the join if either column is NOT populated or values match.
output_index : list[str]
column or columns used to construct the index of the output dataframe
output_mapping : Mapping[str, list[str]]
default={}, optional mapping applied when converting the dataframe output to a dictionary. (See to_dict() below). Entries should be of form: Key: column name from output dataframe Value: list of keys in the final dict in which the column value will be stored.
sort_ascending : bool
default=True; sort output by index in ascending order. sort will be descending if False is passed.
keep_unmatched : bool
default=True; if true, merge the remaining records in left dataframe with the output dataframe and populate nan values with "". If false, do not include unmatched records in output.
log_unmatched : bool
default=False; if true, save a dictionary of all unmatched records to the log. Otherwise, only print the index for all unmatched records.
log_duplicates : bool
default=False; if true, save a dictionary of duplicate records to the log. Otherwise, print a list of duplicate indices.

Attributes

left : pandas.DataFrame
primary dataframe. when a row in left is matched with a row in right the result is stored in output and the original row is removed from left. this process continues until all rows from left have been matched or all defined joins have been performed.
right : pandas.DataFrame
matching dataframe. source data that is being matched with entries in left.
duplicates : pandas.DataFrame
dataframe containing row data for duplicated indices in output.
output : pandas.DataFrame
joined output dataframe

Class variables

var dq_not_match_cols : list[tuple[str, str]]
var dq_on_match_cols : list[tuple[str, str]]
var joins : list[JoinSpec]
var keep_unmatched : bool
var left_constructorDataFrameConstructor
var left_source : Any
var left_static_cols : collections.abc.Mapping[str, str]
var log_duplicates : bool
var log_unmatched : bool
var matcher_type : str
var output_dir : str
var output_index : list[str]
var output_mapping : collections.abc.Mapping[str, str]
var right_constructorDataFrameConstructor
var right_source : Any
var right_static_cols : collections.abc.Mapping[str, str]
var sort_ascending : bool

Static methods

def as_bytesio(source: _io.BytesIO | bytes | str) ‑> _io.BytesIO

convert all various source formats to io.BytesIO

Methods

def match_on(self, join_spec: JoinSpec) ‑> pandas.core.frame.DataFrame

set indices and perform join on left and right dataframes returning a dataframe of successfully matched records, and removing all successfully matched records from the left frame.

Args

join_spec : JoinSpec
dataclass object with columns and join type

Returns

pd.DataFrame
joined dataframe
def to_dict(self, valid_keys: collections.abc.Sequence[str] | None = None) ‑> dict[str, dict[str, utilities.v_str.vStr]]

convert output dataframe to dictionary.

def to_json(self, valid_keys: str | None = None) ‑> str

convert output dataframe to json saving a copy to path if supplied.

class JoinSpec (left_columns: list[str], right_columns: list[str] = <factory>, join_type: Literal['left', 'right', 'inner', 'outer'] = 'inner', right_suffix: str = '_RIGHT', joins_formatters: list[collections.abc.Callable[[str], str]] = <factory>)

Defines a list of columns from a left dataframe and a list of columns from a right dataframe that are to be used to join the two.

Args

left_columns
list of participating column names in the left dataframe
right_columns : optional
list of participating column names in the right dataframe. If not supplied, defaults to left_columns. If supplied, left_columns and right_columns MUST be of equal length.
join_type
"left", "right", "inner", or "outer". Default is "inner".
right_suffix
text appended to right column name in join result when the original column name exists in both left and right.
joins_formatters
list of callables for reformatting index values prior to performing join operation.

Raises

ValueError
if length of left_columns != length of supplied right_columns OR length of left columns != length of supplied joins_formatters.

Class variables

var join_type : Literal['left', 'right', 'inner', 'outer']
var joins_formatters : list[collections.abc.Callable[[str], str]]
var left_columns : list[str]
var right_columns : list[str]
var right_suffix : str