lakehouse.databricks.transform_data.process_transform_definition

lakehouse.databricks.transform_data.process_transform_definition(spark, config_mount, def_path, execution_id)

This function processes a Data Lakehouse transformation defined by a YAML configuration file

Parameters

sparkspark context

spark context passed from the calling spark instance

config_mountstring

The mounting point where the configuration definition is found

def_pathstring

The path to the configuration file used to process the data

execution_idstring

the Azure Data Factory Excution id used to tie the function back to the calling ADF Pipeline

Raises:

Exception : any errors encoutered are bubbled up to the calling notebook or method

Returns

json :

the JSON representation of the results of the transformation, including the Expectation Statistics, Delta Load Statistics, Exepectation Error Behaviour and Syanapse view location and definition