lakehouse.databricks.cdm.stream_cdm_table_silver

lakehouse.databricks.cdm.stream_cdm_table_silver(spark, src_mount, src_zone, dest_mnt, dst_zone, src_schema, dst_schema, entity_path, entity, execution_id)

This function streams Bronze CDM tables to the Silver Zone of the Data Lakehouse

Parameters

sparkspark context

spark context passed from the calling spark instance

src_mountstring

mounting point where the source data is found

src_zonestring

not utilized

dest_mntstring

mounting point where there data will be saved

dst_zonestring

Zone in the Data Lakehouse where the Delta Table will be stored

src_schemastring

database/schema in the Data Lakehouse where the Bronze Delta Table is stored

dst_schemastring

database/schema in the Data Lakehouse where the Delta Table will be stored

entity_pathstring

path to the bronze data lake table to be silvered

entitystring

name of the silver entity/table to merge

execution_idstring

exection_id of the Azure Data Factory Pipeline to tie back to the ADF execution

Raises:

Exception : bubbles up errors to the calling notebook or application

Returns

json :

returns the statistics associated with the Streaming of the bronze CDM data to the Silver zone of the Data Lakehouse, including, Expectation Statistics, and delta load statistics