Author: Duncan Murdoch
Date: 2018-01-12

# User Interaction in WebGL

#### Duncan Murdoch

#### January 12, 2018

## Introduction

This document describes how to embed `rgl` scenes in HTML documents and use embedded Javascript to control a WebGL display in an HTML document. For more general information about `rgl`, see rgl Overview.

We assume that the HTML document is produced from R markdown source using `knitr` or `rmarkdown`. This format mixes text with Markdown markup with chunks of R code. There is a limited amount of discussion of other methods.

There are two ways to embed an `rgl` scene in the document. The recommended one is to call `rglwidget` to produce a “widget” which can be embedded into your document by printing it.

The older method is described in the Legacy WebGL Methods document. It is likely to be supported for some time, but is not recommended for new projects, as the widget method is easier for me to maintain.

I have conducted experiments on a third method. This is intended to be similar to the way standard 2D graphics are included by `knitr`, i.e. it will detect the fact that you’ve drawn something, and just include it automatically. At present it is not recommended, but that may change in the future.

## Browser support

Most browsers now support WebGL, but in some browsers it may be disabled by default. See http://get.webgl.org for help on a number of different browsers.

## Examples

We start with a simple plot of the iris data. We insert a code chunk and call the `rglwidget` function with optional argument `elementId`. This allows later Javascript code to refer to the image. We also save the object ids from the plot, so that they can be manipulated later.

    library(rgl)
    plotids <- with(iris, plot3d(Sepal.Length, Sepal.Width, Petal.Length,
                      type="s", col=as.numeric(Species)))
    rglwidget(elementId = "plot3drgl")

Next we insert a button to toggle the display of the data.

    toggleWidget(sceneId = "plot3drgl", ids = plotids["data"], label = "Data")

The `sceneId` is the same as the `elementId` we used in `rglwidget()`, the `ids` are the object ids of the objects that we’d like to toggle, and the `label` is the label shown on the button. To find the names in the `plotids` variable, apply `names()` or `unclass()`:

    names(plotids)

    ## [1] "data" "axes" "xlab" "ylab" "zlab"

    unclass(plotids)

    ## data axes xlab ylab zlab
    ##    7    8    9   10   11

## Using `magrittr` pipes

It can be error-prone to set the `elementId` in the `rglwidget()` to match the `sceneId` in the `toggleWidget()` (or `playwidget()`, described below). In the usual case where both are intended to appear together, `magrittr`-style pipes can be used quite flexibly: the first argument of the control widget accepts the result of `rglwidget()` (or other control widgets), and the `controllers` argument of `rglwidget()` accepts control widgets. For example,

    rglwidget() %>%
    toggleWidget(ids = plotids["data"], label = "Data")

Normally the sizing of the `rglwidget()` matches the chunk’s `fig.width` and `fig.height` settings. Sometimes the controls take up too much space, and you’ll want to reduce the size of the display. You can use the `figWidth` and `figHeight` in the `width` and `height` arguments to `rglwidget()` to compute alternate heights. For example, the chunk below requests a regular size figure in the chunk header, but shrinks the `rglwidget()` using the `height` argument. It also swaps the order of button and scene to illustrate how to put the control ahead of the scene:

    toggleWidget(NA, ids = plotids["data"], label = "Data") %>%
    rglwidget(controllers = ., height = 0.8*figHeight())

## Controls

We have seen how to change the contents of the plot using `toggleWidget`. We can do more elaborate displays. For example, we can redo the previous plot, but with the three species as separate “spheres” objects and buttons to toggle them:

    clear3d() # Remove the earlier display

    setosa <- with(subset(iris, Species == "setosa"),
         spheres3d(Sepal.Length, Sepal.Width, Petal.Length,
                      col=as.numeric(Species),
                      radius = 0.211))
    versicolor <- with(subset(iris, Species == "versicolor"),
         spheres3d(Sepal.Length, Sepal.Width, Petal.Length,
                   col=as.numeric(Species),
                   radius = 0.211))
    virginica <- with(subset(iris, Species == "virginica"),
         spheres3d(Sepal.Length, Sepal.Width, Petal.Length,
                   col=as.numeric(Species),
                   radius = 0.211))
    aspect3d(1,1,1)
    axesid <- decorate3d()
    rglwidget() %>%
    toggleWidget(ids = setosa) %>%
    toggleWidget(ids = versicolor) %>%
    toggleWidget(ids = virginica) %>%
    toggleWidget(ids = axesid)

Since we skipped the `label` argument, the buttons are labelled with the name of the variable passed as `ids`.

`toggleWidget()` is actually a convenient wrapper for two functions: `playwidget` and `subsetControl`. `playwidget()` adds the button to the web page (and can also add sliders, do animations, etc.), while `subsetControl()` chooses a subset of objects to display.

### `subsetControl`

For a more general example, we could use a slider to select several subsets of the data in the iris display. For example,

    rglwidget() %>%
    playwidget(start = 0, stop = 3, interval = 1,
           subsetControl(1, subsets = list(
                     Setosa = setosa,
                     Versicolor = versicolor,
                     Virginica = virginica,
                     All = c(setosa, versicolor, virginica)
                     )))

There are several other “control” functions.

### `par3dinterpControl`

`par3dinterpControl` approximates the result of `par3dinterp`.

For example, the following code (similar to the `play3d` example) rotates the scene in a complex way.

    M <- r3dDefaults$userMatrix
    fn <- par3dinterp(time = (0:2)*0.75, userMatrix = list(M,
                                          rotate3d(M, pi/2, 1, 0, 0),
                                          rotate3d(M, pi/2, 0, 1, 0)) )
    rglwidget() %>%
    playwidget(par3dinterpControl(fn, 0, 3, steps=15),
           step = 0.01, loop = TRUE, rate = 0.5)

Some things to note: The generated Javascript slider has 300 increments, so that motion appears smooth. However, storing 300 `userMatrix` values would take up a lot of space, so we use interpolation in the Javascript code. However, the Javascript code can only do linear interpolation, not the more complex spline-based SO(3) interpolation done by `par3dinterp`. Because of this, we need to output 15 steps from `par3dinterpControl` so that the distortions of linear interpolation are not visible.

### `propertyControl`

`propertyControl` is a more general function to set the value of properties of the scene. Currently most properties are supported, but use does require knowledge of the internal implementation.

### `clipplaneControl`

`clipplaneControl` allows the user to control the location of a clipping plane by moving a slider.

### `vertexControl`

Less general than `propertyControl` is `vertexControl`. This function sets attributes of individual vertices in a scene. For example, to set the x-coordinate of the closest point in the setosa group, and modify its colour from black to white,

    setosavals <- subset(iris, Species == "setosa")
    which <- which.min(setosavals$Sepal.Width)
    init <- setosavals$Sepal.Length[which]
    rglwidget() %>%
    playwidget(
      vertexControl(values = matrix(c(init,   0,      0,       0,
                                           8,    1,      1,       1), nrow = 2, byrow = TRUE),
                           attributes = c("x", "red", "green", "blue"),
                 vertices = which, objid = setosa),
        step=0.01)

### `ageControl`

A related function is `ageControl`, though it uses a very different specification of the attributes. It is used when the slider controls the “age” of the scene, and attributes of vertices change with their age.

To illustrate we will show a point moving along a curve. We give two `ageControl` calls in a list; the first one controls the colour of the trail, the second controls the position of the point:

    time <- 0:500
    xyz <- cbind(cos(time/20), sin(time/10), time)
    lineid <- plot3d(xyz, type="l", col = "black")["data"]
    sphereid <- spheres3d(xyz[1, , drop=FALSE], radius = 8, col = "red")
    rglwidget() %>%
    playwidget(list(
      ageControl(births = time, ages = c(0, 0, 50),
            colors = c("gray", "red", "gray"), objids = lineid),
      ageControl(births = 0, ages = time,
            vertices = xyz, objids = sphereid)),
      start = 0, stop = max(time) + 20, rate = 50,
      components = c("Reverse", "Play", "Slower", "Faster",
                     "Reset", "Slider", "Label"),
      loop = TRUE)

### `matrixControl`

The final function of this type is the not-yet-implemented `matrixControl`, for setting up multiple controls to modify a matrix, typically `userMatrix`. This is used when complex manipulation of a matrix requires several controls.

### `rglMouse`

While not exactly a control in the sense of the other functions in this section, the `rglMouse` function is used to add an HTML control to a display to allow the user to select the mouse mode.

For example, the display below initially allows selection of particular points, but the mouse mode may be changed to let the user rotate the display for a another view of the scene.

     ids <- with(iris, plot3d(Sepal.Length, Sepal.Width, Petal.Length,
                      type="s", col=as.numeric(Species)))
     par3d(mouseMode = "selecting")
     rglwidget(shared = rglShared(ids["data"])) %>% rglMouse()

The `rglShared()` call used here is described in the next section.

## Integration with `crosstalk`

The `crosstalk` package allows widgets to communicate with each other. Currently it supports selection and filtering of observations.

`rgl` can send, receive and display these messages. An `rgl` display may have several subscenes, each displaying different datasets. Each object in the scene is potentially a shared dataset in the `crosstalk` sense.

The linking depends on the `rglShared` function. Calling `rglShared(id)`, where `id` is the `rgl` id value for an object in the current scene, creates a shared data object containing the coordinates of the vertices of the `rgl` object. This object is passed to `rglwidget` in the `shared` argument. It can also be passed to other widgets that accept shared data, linking the two displays.

If a shared data object has been created in some other way, it can be linked to a particular `rgl` `id` value by copying its `key` and `group` properties as shown in the example below.

    library(crosstalk)
    sd <- SharedData$new(mtcars)
    ids <- plot3d(sd$origData())
    # Copy the key and group from existing shared data
    rglsd <- rglShared(ids["data"], key = sd$key(), group = sd$groupName())
    htmltools::tagList(rglwidget(shared = rglsd),
               filter_checkbox("cylinderselector",
                       "Cylinders", sd, ~ cyl, inline = TRUE))

Cylinders
4 6 8

If multiple objects in the `rgl` scene need to be considered as shared data, you can pass the results of several `rglShared()` calls in a list, i.e. `rglwidget(shared = <list>)`. The key values will be assumed to be shared across datasets; if this is not wanted, use a prefix or some other means to make sure they differ between objects.

If the same `rgl` id is used in more than one `rglShared()` object, it will respond to messages from all of them. This may lead to undesirable behaviour as one message cancels the previous one.

## Low level controls

We repeat the initial plot from this document:

    plotids <- with(iris, plot3d(Sepal.Length, Sepal.Width, Petal.Length,
                      type="s", col=as.numeric(Species)))
    subid <- currentSubscene3d()
    rglwidget(elementId="plot3drgl2")

We might like a button on the web page to cause a change to the display, e.g. a rotation of the plot. First we add buttons, with the “onclick” event set to a function described below:

    <button type="button" onclick="rotate(10)">Forward</button>
    <button type="button" onclick="rotate(-10)">Backward</button>

which produces these buttons:

We stored the subscene number that is currently active in `subid` in the code chunk above, and use it as ```r subid``` in the script below. `knitr` substitutes the value 1 when it processes the document.

The `rotate()` function uses the Javascript function `document.getElementById` to retrieve the `<div>` component of the web page containing the scene. It will have a component named `rglinstance` which contains information about the scene that we can modify:

    <script type="text/javascript">
    var rotate = function(angle) {
      var rgl = document.getElementById("plot3drgl2").rglinstance;
      rgl.getObj(`r subid`).par3d.userMatrix.rotate(angle, 0,1,0);
      rgl.drawScene();
    };
    </script>

If we had used `webGL=TRUE` in the chunk header, the `knitr` WebGL support would create a global object with a name of the form `<chunkname>rgl`. For example, if the code chunk was named `plot3d2`, the object would be called `plot3d2rgl`, and this code would work:

    <script type="text/javascript">
    var rotate = function(angle) {
      plot3d2rgl.getObj(`r subid`).par3d.userMatrix.rotate(angle, 0,1,0);
      plot3d2rgl.drawScene();
    };
    </script>

## Index

The following functions are described in this document:

ageControl | figWidth | propertyControl | subsetControl |
clipplaneControl | par3dinterpControl | rglMouse | toggleWidget |
figHeight | playwidget | rglShared | vertexControl |