lakehouse.databricks.common.generate_expectation_results

lakehouse.databricks.common.generate_expectation_results(spark, mounting_point, file_path, lake_file_struct, file_type, expectations_mnt, expectations_file_path, expectation_params, pipeline_execution_id, dest_lake_zone, source_name, select_expr=None, complete_file_path=None, options=None)

A function to generate expectation results defined in a YAML definition

Parameters

sparkspark context

spark context passed from the calling spark instance

mounting_pointstring

string mounting point where data is located

file_pathstring

string file path to be appended to mount point

lake_file_structstring

string lake file structure to be appeneded to file path when querying nested data

file_typestring

string type of file to be read, delta, parquet, json, csv file types supported supported

expectations_mntstring

string the mount where the expectation yaml files are found

expectations_file_pathstring

string the path to the expecation yaml file

expectation_paramsjson

the expectation parameters to be used when performing the expectations

pipeline_execution_idstring

adf pipeline execution id

dest_lake_zonestring

data lake zone that expectation is being produced for

source_namestring

name of the expectation source

select_exprstring, default=None

hive query to get expectation data. Defaults to None.

complete_file_pathstring, default=None

complete file path of the expectation data if neccessary. Defaults to None.

Returns

dataframe:

dataframe with expectation results