Data Sources

Spark SQL supports operating on a variety of data sources through the DataFrame interface. A DataFrame can be operated on using relational transformations and can also be used to create a temporary view. Registering a DataFrame as a temporary view allows you to run SQL queries over its data. This section describes the general methods for loading and saving data using the Spark Data Sources and then goes into specific options that are available for the built-in data sources.

• Generic Load/Save Functions
  • Manually Specifying Options
  • Run SQL on files directly
  • Save Modes
  • Saving to Persistent Tables
  • Bucketing, Sorting and Partitioning
• Generic File Source Options
  • Ignore Corrupt Files
  • Ignore Missing Files
  • Path Global Filter
  • Recursive File Lookup
• Parquet Files
  • Loading Data Programmatically
  • Partition Discovery
  • Schema Merging
  • Hive metastore Parquet table conversion
  • Configuration
• ORC Files
• JSON Files
• CSV Files
• Text Files
• Hive Tables
  • Specifying storage format for Hive tables
  • Interacting with Different Versions of Hive Metastore
• JDBC To Other Databases
• Avro Files
  • Deploying
  • Load and Save Functions
  • to_avro() and from_avro()
  • Data Source Option
  • Configuration
  • Compatibility with Databricks spark-avro
  • Supported types for Avro -> Spark SQL conversion
  • Supported types for Spark SQL -> Avro conversion
• Whole Binary Files
• Troubleshooting