Metadata checkpointing saves information defining the streaming operations, which are represented by a Directed Acyclic Graph (DAG) to the HDFS. This can be used to recover the DAG, if there is a failure and the application is restarted. The driver restarts and reads the metadata from HDFS, and rebuilds the DAG and recovers all the operational state before the crash.

Metadata includes the following:

• Configuration: the configuration that was used to create the streaming application
• DStream operations: the set of DStream operations that define the streaming application
• Incomplete batches: batches whose jobs are queued but have not completed yet