R: Combine pages into a single data frame

rbind_pages {jsonlite} | R Documentation

## Combine pages into a single data frame

### Description

The `rbind_pages` function is used to combine a list of data frames into a single data frame. This is often needed when working with a JSON API that limits the amount of data per request. If we need more data than what fits in a single request, we need to perform multiple requests that each retrieve a fragment of data, not unlike pages in a book. In practice this is often implemented using a `page` parameter in the API. The `rbind_pages` function can be used to combine these pages back into a single dataset.

### Usage

    rbind_pages(pages)

### Arguments

`pages` | a list of data frames, each representing a page of data

### Details

The `rbind_pages` function generalizes `base::rbind` and `plyr::rbind.fill` with added support for nested data frames. Not each column has to be present in each of the individual data frames; missing columns will be filled up in `NA` values.

### Examples

    # Basic example
    x <- data.frame(foo = rnorm(3), bar = c(TRUE, FALSE, TRUE))
    y <- data.frame(foo = rnorm(2), col = c("blue", "red"))
    rbind_pages(list(x, y))

    ## Not run:
    baseurl <- "http://projects.propublica.org/nonprofits/api/v1/search.json"
    pages <- list()
    for(i in 0:20){
      mydata <- fromJSON(paste0(baseurl, "?order=revenue&sort_order=desc&page=", i))
      message("Retrieving page ", i)
      pages[[i+1]] <- mydata$filings
    }
    filings <- rbind_pages(pages)
    nrow(filings)
    colnames(filings)

    ## End(Not run)

[Package jsonlite version 1.5 Index]