pyspark.conf.SparkConf

Class SparkConf

object --+ | SparkConf

Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.

Most of the time, you would create a SparkConf object with SparkConf(), which will load values from spark.* Java system properties as well. In this case, any parameters you set directly on the SparkConf object take priority over system properties.

For unit tests, you can also call SparkConf(false) to skip loading external settings and get the same configuration no matter what the system properties are.

All setter methods in this class support chaining. For example, you can write

conf.setMaster("local").setAppName("My 
  app")

Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user.

Instance Methods

__init__(self, loadDefaults=True, _jvm=None, _jconf=None)
Create a new Spark configuration.

source code

set(self, key, value)
Set a configuration property.

source code

setIfMissing(self, key, value)
Set a configuration property, if not already set.

source code

setMaster(self, value)
Set master URL to connect to.

source code

setAppName(self, value)
Set application name.

source code

setSparkHome(self, value)
Set path where Spark is installed on worker nodes.

source code

setExecutorEnv(self, key=None, value=None, pairs=None)
Set an environment variable to be passed to executors.

source code

setAll(self, pairs)
Set multiple parameters, passed as a list of key-value pairs.

source code

get(self, key, defaultValue=None)
Get the configured value for some key, or return a default otherwise.

source code

getAll(self)
Get all values as a list of key-value pairs.

source code

contains(self, key)
Does this configuration contain a given key?

source code

toDebugString(self)
Returns a printable version of the configuration, as a list of key=value pairs, one per line.

source code

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Properties

Inherited from object: __class__

Method Details

init(self, loadDefaults=True, _jvm=None, _jconf=None)
(Constructor)

source code

Create a new Spark configuration.

Parameters:

loadDefaults - whether to load values from Java system properties (True by default)
_jvm - internal parameter used to pass a handle to the Java VM; does not need to be set by users
_jconf - Optionally pass in an existing SparkConf handle to use its parameters

Overrides: object.__init__

setAll(self, pairs)

source code

Set multiple parameters, passed as a list of key-value pairs.

Parameters:

pairs - list of key-value pairs to set

Class SparkConf

__init__(self, loadDefaults=True, _jvm=None, _jconf=None) (Constructor)

setAll(self, pairs)

init(self, loadDefaults=True, _jvm=None, _jconf=None)
(Constructor)