R: Create a cluster of R workers for parallel processing

makeClusterPSOCK {future} | R Documentation

## Create a cluster of R workers for parallel processing

### Description

The `makeClusterPSOCK()` function creates a cluster of R workers for parallel processing. These R workers may be background R sessions on the current machine, R sessions on external machines (local or remote), or a mix of such. For external workers, the default is to use SSH to connect to those external machines. This function works similarly to `makePSOCKcluster` of the parallel package, but provides additional and more flexibility options for controlling the setup of the system calls that launch the background R workers, and how to connect to external machines.

### Usage

    makeClusterPSOCK(workers, makeNode = makeNodePSOCK, port = c("auto",
      "random"), ..., verbose = getOption("future.debug", FALSE))

    makeNodePSOCK(worker = "localhost", master = NULL, port,
      connectTimeout = getOption("future.makeNodePSOCK.connectTimeout", 2 * 60),
      timeout = getOption("future.makeNodePSOCK.timeout", 30 * 24 * 60 * 60),
      rscript = NULL, homogeneous = NULL, rscript_args = NULL,
      methods = TRUE, useXDR = TRUE, outfile = "/dev/null",
      renice = NA_integer_, rshcmd = getOption("future.makeNodePSOCK.rshcmd",
      NULL), user = NULL, revtunnel = TRUE,
      rshopts = getOption("future.makeNodePSOCK.rshopts", NULL), rank = 1L,
      manual = FALSE, dryrun = FALSE, verbose = FALSE)

### Arguments

`workers` | The hostnames of workers (as a character vector) or the number of localhost workers (as a positive integer).
`makeNode` | A function that creates a `"SOCKnode"` or `"SOCK0node"` object, which represents a connection to a worker.
`port` | The port number of the master used to for communicating with all the workers (via socket connections). If an integer vector of ports, then a random one among those is chosen. If `"random"`, then a random port in `11000:11999` is chosen. If `"auto"` (default), then the default is taken from environment variable R_PARALLEL_PORT, otherwise `"random"` is used.
`...` | Optional arguments passed to `makeNode(workers[i], ..., rank = i)` where `i = seq_along(workers)`.
`verbose` | If TRUE, informative messages are outputted.
`worker` | The hostname or IP number of the machine where the worker should run.
`master` | The hostname or IP number of the master / calling machine, as known to the workers. If NULL (default), then the default is `Sys.info()[["nodename"]]` unless `worker` is localhost or `revtunnel = TRUE` in case it is `"localhost"`.
`connectTimeout` | The maximum time (in seconds) allowed for each socket connection between the master and a worker to be established (defaults to 2 minutes). See note below on current lack of support on Linux and macOS systems.
`timeout` | The maximum time (in seconds) allowed to pass without the master and a worker communicate with each other (defaults to 30 days).
`rscript, homogeneous` | The system command for launching `Rscript` on the worker and whether it is installed in the same path as the calling machine or not. For more details, see below.
`rscript_args` | Additional arguments to `Rscript` (as a character vector).
`methods` | If TRUE, then the methods package is also loaded.
`useXDR` | If TRUE, the communication between master and workers, which is binary, will use big-endian (XDR).
`outfile` | Where to direct the stdout and stderr connection output from the workers.
`renice` | A numerical 'niceness' (priority) to set for the worker processes.
`rshcmd, rshopts` | The command (character vector) to be run on the master to launch a process on another host and any additional arguments (character vector). These arguments are only applied if `machine` is not localhost. For more details, see below.
`user` | (optional) The user name to be used when communicating with another host.
`revtunnel` | If TRUE, a reverse SSH tunnel is set up for each worker such that the worker R process sets up a socket connection to its local port `(port - rank + 1)` which then reaches the master on port `port`. If FALSE, then the worker will try to connect directly to port `port` on `master`. For more details, see below.
`rank` | A unique one-based index for each worker (automatically set).
`manual` | If TRUE the workers will need to be run manually. The command to run will be displayed.
`dryrun` | If TRUE, nothing is set up, but a message suggesting how to launch the worker from the terminal is outputted. This is useful for troubleshooting.

### Value

An object of class `c("SOCKcluster", "cluster")` consisting of a list of `"SOCKnode"` or `"SOCK0node"` workers.

`makeNodePSOCK()` returns a `"SOCKnode"` or `"SOCK0node"` object representing an established connection to a worker.

### Definition of localhost

A hostname is considered to be localhost if it equals:

- `"localhost"`,

- `"127.0.0.1"`, or

- `Sys.info()[["nodename"]]`.

It is also considered localhost if it appears on the same line as the value of `Sys.info()[["nodename"]]` in file ‘/etc/hosts’.

### Default values of arguments `rshcmd` and `rshopts`

Arguments `rshcmd` and `rshopts` are only used when connecting to an external host.

The default method for connecting to an external host is via SSH and the system executable for this is given by argument `rshcmd`. The default is given by option `future.makeNodePSOCK.rshcmd` and if that is not set the default is either of `ssh` and `plink -ssh`. Most Unix-like systems, including macOS, have `ssh` preinstalled on the `PATH`. It is less common to find this command on Windows system, which are more likely to have the `PuTTY` software and its SSH client `plink` installed. Furthermore, when running R from RStudio on Windows, the `ssh` client that is distributed with RStudio will be used as a fallback if neither of the above two commands are available on the `PATH`. Note that on Windows 10, there is a beta version of OpenSSH that can be enabled in the Windows Settings, cf. https://www.thomasmaurer.ch/2017/11/install-ssh-on-windows-10-as-optional-feature/. If no SSH-client is found, an informative error message is produced. It is also possible to specify the absolute path to the SSH client. To do this for PuTTY, specify the absolute path in the first element and option `-ssh` in the second as in `rshcmd = c("C:/Path/PuTTY/plink.exe", "-ssh")`. This is because all elements of `rshcmd` are individually "shell" quoted and element `rshcmd[1]` must be on the system `PATH`.

Additional SSH options may be specified via argument `rshopts`, which defaults to option `future.makeNodePSOCK.rshopts`. For instance, a private SSH key can be provided as `rshopts = c("-i", "~/.ssh/my_private_key")`. PuTTY users should specify a PuTTY PPK file, e.g. `rshopts = c("-i", "C:/Users/joe/.ssh/my_keys.ppk")`. Contrary to `rshcmd`, elements of `rshopts` are not quoted.

### Reverse SSH tunneling

The default is to use reverse SSH tunneling (`revtunnel = TRUE`) for workers running on other machines. This avoids the complication of otherwise having to configure port forwarding in firewalls, which often requires static IP address as well as privilieges to edit the firewall, something most users don't have. It also has the advantage of not having to know the internal and / or the public IP address / hostname of the master. Yet another advantage is that there will be no need for a DNS lookup by the worker machines to the master, which may not be configured or is disabled on some systems, e.g. compute clusters.

### Default value of argument `rscript`

If `homogeneous` is FALSE, the `rscript` defaults to `"Rscript"`, i.e. it is assumed that the `Rscript` executable is available on the `PATH` of the worker. If `homogeneous` is TRUE, the `rscript` defaults to `file.path(R.home("bin"), "Rscript")`, i.e. it is basically assumed that the worker and the caller share the same file system and R installation.

### Default value of argument `homogeneous`

The default value of `homogeneous` is TRUE if and only if either of the following is fullfilled:

- `worker` is localhost

- `revtunnel` is FALSE and `master` is localhost

- `worker` is neither an IP number nor a fully qualified domain name (FQDN). A hostname is considered to be a FQDN if it contains one or more periods

### Connection time out

Argument `connectTimeout` does not work properly on Unix and macOS due to limitation in R itself. For more details on this, please see R-devel thread 'BUG?: On Linux setTimeLimit() fails to propagate timeout error when it occurs (works on Windows)' on 2016-10-26 (https://stat.ethz.ch/pipermail/r-devel/2016-October/073309.html). When used, the timeout will eventually trigger an error, but it won't happen until the socket connection timeout `timeout` itself happens.

### Communication time out

If there is no communication between the master and a worker within the `timeout` limit, then the corresponding socket connection will be closed automatically. This will eventually result in an error in code trying to access the connection.

### Examples

    ## Setup of three R workers on two remote machines are set up
    workers <- c("n1.remote.org", "n2.remote.org", "n1.remote.org")
    cl <- makeClusterPSOCK(workers, dryrun = TRUE)

    ## Same setup when the two machines are on the local network and
    ## have identical software setups
    cl <- makeClusterPSOCK(
      workers,
      revtunnel = FALSE, homogeneous = TRUE,
      dryrun = TRUE
    )

    ## Setup of remote worker with more detailed control on
    ## authentication and reverse SSH tunnelling
    cl <- makeClusterPSOCK(
      "remote.server.org", user = "johnny",
      ## Manual configuration of reverse SSH tunnelling
      revtunnel = FALSE,
      rshopts = c("-v", "-R 11000:gateway:11942"),
      master = "gateway", port = 11942,
      ## Run Rscript nicely and skip any startup scripts
      rscript = c("nice", "/path/to/Rscript"),
      rscript_args = c("--vanilla"),
      dryrun = TRUE
    )

    ## Setup of 2 Docker workers running rocker/r-base
    ## (requires installation of future package)
    cl <- makeClusterPSOCK(
      rep("localhost", times = 2L),
      ## Launch Rscript inside Docker container
      rscript = c(
        "docker", "run", "--net=host", "rocker/r-base",
        "Rscript"
      ),
      ## Install future package
      rscript_args = c(
        "-e", shQuote("install.packages('future')")
      ),
      dryrun = TRUE
    )

    ## Setup of a single udocker.py worker running rocker/r-base
    ## (requires installation of future package and extra quoting)
    cl <- makeClusterPSOCK(
      "localhost",
      ## Launch Rscript inside Docker container (using udocker)
      rscript = c(
        "udocker.py", "run", "rocker/r-base",
        "Rscript"
      ),
      ## Install future package and manually launch parallel workers
      ## (need double shQuote():s because udocker.py drops one level)
      rscript_args = c(
        "-e", shQuote(shQuote("install.packages('future')")),
        "-e", shQuote(shQuote("parallel:::.slaveRSOCK()"))
      ),
      dryrun = TRUE
    )

    ## Launching worker on Amazon AWS EC2 running one of the
    ## Amazon Machine Images (AMI) provided by RStudio
    ## (http://www.louisaslett.com/RStudio_AMI/)
    public_ip <- "1.2.3.4"
    ssh_private_key_file <- "~/.ssh/my-private-aws-key.pem"
    cl <- makeClusterPSOCK(
      ## Public IP number of EC2 instance
      public_ip,
      ## User name (always 'ubuntu')
      user = "ubuntu",
      ## Use private SSH key registered with AWS
      rshopts = c(
        "-o", "StrictHostKeyChecking=no",
        "-o", "IdentitiesOnly=yes",
        "-i", ssh_private_key_file
      ),
      ## Set up .libPaths() for the 'ubuntu' user and
      ## install future package
      rscript_args = c(
        "-e", shQuote("local({
          p <- Sys.getenv('R_LIBS_USER')
          dir.create(p, recursive = TRUE, showWarnings = FALSE)
          .libPaths(p)
        })"),
        "-e", shQuote("install.packages('future')")
      ),
      dryrun = TRUE
    )

    ## Launching worker on Google Cloud Engine (GCE) running a
    ## container based VM (with a #cloud-config specification)
    public_ip <- "1.2.3.4"
    user <- "johnny"
    ssh_private_key_file <- "~/.ssh/google_compute_engine"
    cl <- makeClusterPSOCK(
      ## Public IP number of GCE instance
      public_ip,
      ## User name (== SSH key label (sic!))
      user = user,
      ## Use private SSH key registered with GCE
      rshopts = c(
        "-o", "StrictHostKeyChecking=no",
        "-o", "IdentitiesOnly=yes",
        "-i", ssh_private_key_file
      ),
      ## Launch Rscript inside Docker container
      rscript = c(
        "docker", "run", "--net=host", "rocker/r-base",
        "Rscript"
      ),
      ## Install future package
      rscript_args = c(
        "-e", shQuote("install.packages('future')")
      ),
      dryrun = TRUE
    )

    ## Connect to remote Unix machine 'remote.server.org' on port 2200
    ## as user 'bob' from a Windows machine with PuTTY installed
    cl <- makeClusterPSOCK(
      "remote.server.org", user = "bob",
      rshopts = c("-P", 2200, "-i", "C:/Users/bobby/.ssh/putty.ppk"),
      dryrun = TRUE
    )

[Package future version 1.8.0 Index]