lakehouse.databricks.stream.process_silver_stream_definition

lakehouse.databricks.stream.process_silver_stream_definition(spark, config_mount, def_path, execution_id)

This function processes a batched stream defined by a YAML configuration file

Parameters

sparkspark context

spark context passed from the calling spark instance

config_mountstring

The mounting point where the configuration definition is found

def_pathstring

The path to the configuration file used to process the data

execution_idstring

the Azure Data Factory Excution id used to tie the function back to the calling ADF Pipeline

Raises:

Exception : any errors encoutered are bubbled up to the calling notebook or method

Returns

json :

the JSON representation of the results of the transformation, including the Expectation Statistics, Delta Load Statistics, Exepectation Error Behaviour and Syanapse view location and definition