Spark SQL can save data to external storage systems such as files, Hive tables and JDBC databases through DataFrameWriter interface.
The format of the API call is dataframe.write.outputtype
- Parquet
- ORC
- Text
- Hive table
- JSON
- CSV
- JDBC
Let's look at a couple of examples of writing or saving a DataFrame to a CSV file:
scala> statesPopulationDF.write.option("header", "true").csv("statesPopulation_dup.csv")
scala> statesTaxRatesDF.write.option("header", "true").csv("statesTaxRates_dup.csv")