Extract Transform Load (ETL)
ETL process is to fetch data from different types of systems, structure it and save it into the destination database.
In the case of a batch job, the query will be run on the data saved at
source-path and the transformed data will be saved at the destination
In the case of a streaming job, the query will run on the data continuously from
source-path and transformed data will be appended in the destination
dest-path again and again as data comes in.
Merging static data (DB) with streaming data
There might be use cases where you want to merge static data (e.g. MySQL) with the streaming data. You can do this as follows:
Executing the Job
With the planner’s logical plan, incremental execution plan is generated on top of it