User Guide

There are basic guides shared with other languages in Programming Guides at the Spark documentation as below:

• RDD Programming Guide

• Spark SQL, DataFrames and Datasets Guide

• Structured Streaming Programming Guide

• Spark Streaming Programming Guide

• Machine Learning Library (MLlib) Guide

PySpark specific user guide is as follows:

• Python Package Management
  • Using PySpark Native Features
  • Using Conda
  • Using Virtualenv
  • Using PEX
• Spark SQL
  • Apache Arrow in PySpark
• Pandas API on Spark
  • Options and settings
  • From/to pandas and PySpark DataFrames
  • Transform and apply a function
  • Type Support in Pandas API on Spark
  • Type Hints in Pandas API on Spark
  • From/to other DBMSes
  • Best Practices
  • FAQ

Quickstart: Pandas API on Spark Python Package Management