R: Async Multi Download

multi {curl} | R Documentation

## Async Multi Download

### Description

AJAX style concurrent requests, possibly using HTTP/2 multiplexing. Results are only available via callback functions. Advanced use only!

### Usage

    multi_add(handle, done = NULL, fail = NULL, data = NULL, pool = NULL)

    multi_run(timeout = Inf, poll = FALSE, pool = NULL)

    multi_set(total_con = 50, host_con = 6, multiplex = TRUE, pool = NULL)

    multi_list(pool = NULL)

    multi_cancel(handle)

    new_pool(total_con = 100, host_con = 6, multiplex = TRUE)

    multi_fdset(pool = NULL)

### Arguments

`handle` | a curl handle with preconfigured `url` option.
`done` | callback function for completed request. Single argument with response data in same structure as curl_fetch_memory.
`fail` | callback function called on failed request. Argument contains error message.
`data` | callback function or open connection object for receiving data. If `NULL` the entire response content gets buffered and is returned in the `done` callback.
`pool` | a multi handle created by new_pool. Default uses a global pool.
`timeout` | max time in seconds to wait for results. Use `0` to poll for results without waiting at all.
`poll` | If `TRUE` then return immediately after any of the requests has completed. May also be an integer in which case it returns after n requests have completed.
`total_con` | max total concurrent connections.
`host_con` | max concurrent connections per host.
`multiplex` | enable HTTP/2 multiplexing if supported by host and client.

### Details

Requests are created in the usual way using a curl handle and added to the scheduler with multi_add. This function returns immediately and does not perform the request yet. The user needs to call multi_run which performs all scheduled requests concurrently. It returns when all requests have completed, or case of a `timeout` or `SIGINT` (e.g. if the user presses `ESC` or `CTRL+C` in the console). In case of the latter, simply call multi_run again to resume pending requests.

When the request succeeded, the `done` callback gets triggered with the response data. The structure if this data is identical to curl_fetch_memory. When the request fails, the `fail` callback is triggered with an error message. Note that failure here means something went wrong in performing the request such as a connection failure, it does not check the http status code. Just like curl_fetch_memory, the user has to implement application logic.

Raising an error within a callback function stops execution of that function but does not affect other requests.

A single handle cannot be used for multiple simultaneous requests. However it is possible to add new requests to a pool while it is running, so you can re-use a handle within the callback of a request from that same handle. It is up to the user to make sure the same handle is not used in concurrent requests.

The multi_cancel function can be used to cancel a pending request. It has no effect if the request was already completed or canceled.

The multi_fdset function returns the file descriptors curl is polling currently, and also a timeout parameter, the number of milliseconds an application should wait (at most) before proceeding. It is equivalent to the `curl_multi_fdset` and `curl_multi_timeout` calls. It is handy for applications that is expecting input (or writing output) through both curl, and other file descriptors.

### Examples

    results <- list()
    success <- function(x){
      results <<- append(results, list(x))
    }
    failure <- function(str){
      cat(paste("Failed request:", str), file = stderr())
    }
    # This handle will take longest (3sec)
    h1 <- new_handle(url = "https://eu.httpbin.org/delay/3")
    multi_add(h1, done = success, fail = failure)

    # This handle writes data to a file
    con <- file("output.txt", open = "wb")
    h2 <- new_handle(url = "https://eu.httpbin.org/post", postfields = "bla bla")
    multi_add(h2, done = success, fail = failure, data = con)

    # This handle raises an error
    h3 <- new_handle(url = "https://urldoesnotexist.xyz")
    multi_add(h3, done = success, fail = failure)

    # Actually perform the requests
    multi_run(timeout = 2)
    multi_run()

    # Check the file
    close(con)
    readLines("output.txt")
    unlink("output.txt")

[Package curl version 3.2 Index]