Data Lakehouse File Streaming and data transformation Documentation

The common module provides generic helper functions to support a Delta Lakehouse implementation

Data Lakehouse Autoloader Streaming Transformation

process_silver_stream_definition(spark, ...)

This function processes a batched stream defined by a YAML configuration file

Data Lakehouse transformation

process_transform_definition(spark, ...)

This function processes a Data Lakehouse transformation defined by a YAML configuration file