R: Create a Spark Configuration for Livy

livy_config {sparklyr}

R Documentation

Create a Spark Configuration for Livy

Description

Create a Spark Configuration for Livy

Usage

livy_config(config = spark_config(), username = NULL, password = NULL,
  negotiate = FALSE, custom_headers = list(`X-Requested-By` = "sparklyr"),
  ...)

Arguments

`config`	Optional base configuration
`username`	The username to use in the Authorization header
`password`	The password to use in the Authorization header
`negotiate`	Whether to use gssnegotiate method or not
`custom_headers`	List of custom headers to append to http requests. Defaults to `list("X-Requested-By" = "sparklyr")`.
`...`	additional Livy session parameters

Details

Extends a Spark "spark_config" configuration with settings for Livy. For instance, "username" and "password" define the basic authentication settings for a Livy session.

The default value of "custom_headers" is set to list("X-Requested-By" = "sparklyr") in order to facilitate connection to Livy servers with CSRF protection enabled.

Additional parameters for Livy sessions are:

proxy_user: User to impersonate when starting the session
jars: jars to be used in this session
py_files: Python files to be used in this session
files: files to be used in this session
driver_memory: Amount of memory to use for the driver process
driver_cores: Number of cores to use for the driver process
executor_memory: Amount of memory to use per executor process
executor_cores: Number of cores to use for each executor
num_executors: Number of executors to launch for this session
archives: Archives to be used in this session
queue: The name of the YARN queue to which submitted
queue: The name of this session
heartbeat_timeout: Timeout in seconds to which session be orphaned

Value

Named list with configuration data

[Package sparklyr version 0.8.2 Index]