R: Estimate Remaining Runtimes

estimateRuntimes {batchtools} | R Documentation

## Estimate Remaining Runtimes

### Description

Estimates the runtimes of jobs using the random forest implemented in ranger. Observed runtimes are retrieved from the `Registry` and runtimes are predicted for unfinished jobs.

The estimated remaining time is calculated in the `print` method. You may also pass `n` here to determine the number of parallel jobs which is then used in a simple Longest Processing Time (LPT) algorithm to give an estimate for the parallel runtime.

### Usage

    estimateRuntimes(tab, ..., reg = getDefaultRegistry())

    ## S3 method for class 'RuntimeEstimate'
    print(x, n = 1L, ...)

### Arguments

`tab` | [`data.table`] Table with column “job.id” and additional columns to predict the runtime. Observed runtimes will be looked up in the registry and serve as dependent variable. All columns in `tab` except “job.id” will be passed to `ranger` as independent variables to fit the model.
`...` | [ANY] Additional parameters passed to `ranger`. Ignored for the `print` method.
`reg` | [`Registry`] Registry. If not explicitly passed, uses the default registry (see `setDefaultRegistry`).
`x` | [`RuntimeEstimate`] Object to print.
`n` | [`integer(1)`] Number of parallel jobs to assume for runtime estimation.

### Value

[`RuntimeEstimate`] which is a `list` with two named elements: “runtimes” is a `data.table` with columns “job.id”, “runtime” (in seconds) and “type” (“estimated” if runtime is estimated, “observed” if runtime was observed). The other element of the list named “model”] contains the fitted random forest object.

### See Also

`binpack` and `lpt` to chunk jobs according to their estimated runtimes.

### Examples

    # Create a simple toy registry
    set.seed(1)
    tmp = makeExperimentRegistry(file.dir = NA, make.default = FALSE, seed = 1)
    addProblem(name = "iris", data = iris, fun = function(data, ...) nrow(data), reg = tmp)
    addAlgorithm(name = "nrow", function(instance, ...) nrow(instance), reg = tmp)
    addAlgorithm(name = "ncol", function(instance, ...) ncol(instance), reg = tmp)
    addExperiments(algo.designs = list(nrow = CJ(x = 1:50, y = letters[1:5])), reg = tmp)
    addExperiments(algo.designs = list(ncol = CJ(x = 1:50, y = letters[1:5])), reg = tmp)

    # We use the job parameters to predict runtimes
    tab = unwrap(getJobPars(reg = tmp))

    # First we need to submit some jobs so that the forest can train on some data.
    # Thus, we just sample some jobs from the registry while grouping by factor variables.
    ids = tab[, .SD[sample(nrow(.SD), 5)], by = c("problem", "algorithm", "y")]
    setkeyv(ids, "job.id")
    submitJobs(ids, reg = tmp)
    waitForJobs(reg = tmp)

    # We "simulate" some more realistic runtimes here to demonstrate the functionality:
    # - Algorithm "ncol" is 5 times more expensive than "nrow"
    # - x has no effect on the runtime
    # - If y is "a" or "b", the runtimes are really high
    runtime = function(algorithm, x, y) {
      ifelse(algorithm == "nrow", 100L, 500L) + 1000L * (y %in% letters[1:2])
    }
    tmp$status[ids, done := done + tab[ids, runtime(algorithm, x, y)]]
    rjoin(sjoin(tab, ids), getJobStatus(ids, reg = tmp)[, c("job.id", "time.running")])

    # Estimate runtimes:
    est = estimateRuntimes(tab, reg = tmp)
    print(est)
    rjoin(tab, est$runtimes)
    print(est, n = 10)

    # Submit jobs with longest runtime first:
    ids = est$runtimes[type == "estimated"][order(runtime, decreasing = TRUE)]
    print(ids)
    ## Not run:
    submitJobs(ids, reg = tmp)

    ## End(Not run)

    # Group jobs into chunks with runtime < 1h
    ids = est$runtimes[type == "estimated"]
    ids[, chunk := binpack(runtime, 3600)]
    print(ids)
    print(ids[, list(runtime = sum(runtime)), by = chunk])
    ## Not run:
    submitJobs(ids, reg = tmp)

    ## End(Not run)

    # Group jobs into 10 chunks with similar runtime
    ids = est$runtimes[type == "estimated"]
    ids[, chunk := lpt(runtime, 10)]
    print(ids[, list(runtime = sum(runtime)), by = chunk])

[Package batchtools version 0.9.11 Index]