Author: Matt Dancho
Date: 2020-03-04

# Scaling and Modeling with tidyquant

#### Matt Dancho

#### 2020-03-04

> Designed for the data science workflow of the `tidyverse`

# Overview

The greatest benefit to `tidyquant` is the ability to apply the data science workflow to easily model and scale your financial analysis as described in R for Data Science. Scaling is the process of creating an analysis for one asset and then extending it to multiple groups. This idea of scaling is incredibly useful to financial analysts because typically one wants to compare many assets to make informed decisions. Fortunately, the `tidyquant` package integrates with the `tidyverse` making scaling super simple!

All `tidyquant` functions return data in the `tibble` (tidy data frame) format, which allows for interaction within the `tidyverse`. This means we can:

- Seamlessly scale data retrieval and mutations
- Use the pipe (`%>%`) for chaining operations
- Use `dplyr` and `tidyr`: `select`, `filter`, `group_by`, `nest`/`unnest`, `spread`/`gather`, etc
- Use `purrr`: mapping functions with `map`
- Model financial analysis using the data science workflow in R for Data Science

We’ll go through some useful techniques for getting and manipulating groups of data.

# Prerequisites

Load the `tidyquant` package to get started.

    # Loads tidyquant, lubridate, xts, quantmod, TTR, and PerformanceAnalytics
    library(tidyverse)
    library(tidyquant)

# 1.0 Scaling the Getting of Financial Data

A very basic example is retrieving the stock prices for multiple stocks. There are three primary ways to do this:

## Method 1: Map a character vector with multiple stock symbols

    c("AAPL", "GOOG", "FB") %>%
        tq_get(get = "stock.prices", from = "2016-01-01", to = "2017-01-01")

    ## # A tibble: 756 x 8
    ##    symbol date        open  high   low close   volume adjusted
    ##    <chr>  <date>     <dbl> <dbl> <dbl> <dbl>    <dbl>    <dbl>
    ##  1 AAPL   2016-01-04 103.  105.  102   105.  67649400     98.2
    ##  2 AAPL   2016-01-05 106.  106.  102.  103.  55791000     95.8
    ##  3 AAPL   2016-01-06 101.  102.   99.9 101.  68457400     93.9
    ##  4 AAPL   2016-01-07  98.7 100.   96.4  96.4 81094400     89.9
    ##  5 AAPL   2016-01-08  98.6  99.1  96.8  97.0 70798000     90.4
    ##  6 AAPL   2016-01-11  99.0  99.1  97.3  98.5 49739400     91.9
    ##  7 AAPL   2016-01-12 101.  101.   98.8 100.  49154200     93.2
    ##  8 AAPL   2016-01-13 100.  101.   97.3  97.4 62439600     90.8
    ##  9 AAPL   2016-01-14  98.0 100.   95.7  99.5 63170100     92.8
    ## 10 AAPL   2016-01-15  96.2  97.7  95.4  97.1 79833900     90.6
    ## # … with 746 more rows

The output is a single level tibble with all or the stock prices in one tibble. The auto-generated column name is “symbol”, which can be pre-emptively renamed by giving the vector a name (e.g. `stocks <- c("AAPL", "GOOG", "FB")`) and then piping to `tq_get`.

## Method 2: Map a tibble with stocks in first column

First, get a stock list in data frame format either by making the tibble or retrieving from `tq_index` / `tq_exchange`. The stock symbols must be in the first column.

### Method 2A: Make a tibble

    stock_list <- tibble(stocks = c("AAPL", "JPM", "CVX"),
                         industry = c("Technology", "Financial", "Energy"))
    stock_list

    ## # A tibble: 3 x 2
    ##   stocks industry
    ##   <chr>  <chr>
    ## 1 AAPL   Technology
    ## 2 JPM    Financial
    ## 3 CVX    Energy

Second, send the stock list to `tq_get`. Notice how the symbol and industry columns are automatically expanded the length of the stock prices.

    stock_list %>%
        tq_get(get = "stock.prices", from = "2016-01-01", to = "2017-01-01")

    ## # A tibble: 756 x 9
    ##    stocks industry   date        open  high   low close   volume adjusted
    ##    <chr>  <chr>      <date>     <dbl> <dbl> <dbl> <dbl>    <dbl>    <dbl>
    ##  1 AAPL   Technology 2016-01-04 103.  105.  102   105.  67649400     98.2
    ##  2 AAPL   Technology 2016-01-05 106.  106.  102.  103.  55791000     95.8
    ##  3 AAPL   Technology 2016-01-06 101.  102.   99.9 101.  68457400     93.9
    ##  4 AAPL   Technology 2016-01-07  98.7 100.   96.4  96.4 81094400     89.9
    ##  5 AAPL   Technology 2016-01-08  98.6  99.1  96.8  97.0 70798000     90.4
    ##  6 AAPL   Technology 2016-01-11  99.0  99.1  97.3  98.5 49739400     91.9
    ##  7 AAPL   Technology 2016-01-12 101.  101.   98.8 100.  49154200     93.2
    ##  8 AAPL   Technology 2016-01-13 100.  101.   97.3  97.4 62439600     90.8
    ##  9 AAPL   Technology 2016-01-14  98.0 100.   95.7  99.5 63170100     92.8
    ## 10 AAPL   Technology 2016-01-15  96.2  97.7  95.4  97.1 79833900     90.6
    ## # … with 746 more rows

### Method 2B: Use index or exchange

Get an index…

    tq_index("DOW")

    ## # A tibble: 30 x 8
    ##    symbol company    identifier sedol  weight sector  shares_held local_currency
    ##    <chr>  <chr>      <chr>      <chr>   <dbl> <chr>         <dbl> <chr>
    ##  1 AAPL   Apple Inc. 03783310   20462… 0.0757 Inform…     5264000 USD
    ##  2 BA     Boeing Co… 09702310   21086… 0.0734 Indust…     5264000 USD
    ##  3 UNH    UnitedHea… 91324P10   29177… 0.0684 Health…     5264000 USD
    ##  4 HD     Home Depo… 43707610   24342… 0.0596 Consum…     5264000 USD
    ##  5 GS     Goldman S… 38141G10   24079… 0.0532 Financ…     5264000 USD
    ##  6 MCD    McDonald'… 58013510   25507… 0.0522 Consum…     5264000 USD
    ##  7 V      Visa Inc.… 92826C83   B2PZN… 0.0486 Inform…     5264000 USD
    ##  8 MSFT   Microsoft… 59491810   25881… 0.0430 Inform…     5264000 USD
    ##  9 MMM    3M Company 88579Y10   25957… 0.0380 Indust…     5264000 USD
    ## 10 JNJ    Johnson &… 47816010   24758… 0.0355 Health…     5264000 USD
    ## # … with 20 more rows

…or, get an exchange.

    tq_exchange("NYSE")

Send the index or exchange to `tq_get`. Important Note: This can take several minutes depending on the size of the index or exchange, which is why only the first three stocks are evaluated in the vignette.

    tq_index("DOW") %>%
        slice(1:3) %>%
        tq_get(get = "stock.prices")

    ## # A tibble: 7,674 x 15
    ##    symbol company identifier sedol weight sector shares_held local_currency
    ##    <chr>  <chr>   <chr>      <chr>  <dbl> <chr>        <dbl> <chr>
    ##  1 AAPL   Apple … 03783310   2046… 0.0757 Infor…     5264000 USD
    ##  2 AAPL   Apple … 03783310   2046… 0.0757 Infor…     5264000 USD
    ##  3 AAPL   Apple … 03783310   2046… 0.0757 Infor…     5264000 USD
    ##  4 AAPL   Apple … 03783310   2046… 0.0757 Infor…     5264000 USD
    ##  5 AAPL   Apple … 03783310   2046… 0.0757 Infor…     5264000 USD
    ##  6 AAPL   Apple … 03783310   2046… 0.0757 Infor…     5264000 USD
    ##  7 AAPL   Apple … 03783310   2046… 0.0757 Infor…     5264000 USD
    ##  8 AAPL   Apple … 03783310   2046… 0.0757 Infor…     5264000 USD
    ##  9 AAPL   Apple … 03783310   2046… 0.0757 Infor…     5264000 USD
    ## 10 AAPL   Apple … 03783310   2046… 0.0757 Infor…     5264000 USD
    ## # … with 7,664 more rows, and 7 more variables: date <date>, open <dbl>,
    ## #   high <dbl>, low <dbl>, close <dbl>, volume <dbl>, adjusted <dbl>

You can use any applicable “getter” to get data for every stock in an index or an exchange! This includes: “stock.prices”, “key.ratios”, “key.stats”, and more.

# 2.0 Scaling the Mutation of Financial Data

Once you get the data, you typically want to do something with it. You can easily do this at scale. Let’s get the yearly returns for multiple stocks using `tq_transmute`. First, get the prices. We’ll use the `FANG` data set, but you typically will use `tq_get` to retrieve data in “tibble” format.

    FANG

    ## # A tibble: 4,032 x 8
    ##    symbol date        open  high   low close    volume adjusted
    ##    <chr>  <date>     <dbl> <dbl> <dbl> <dbl>     <dbl>    <dbl>
    ##  1 FB     2013-01-02  27.4  28.2  27.4  28    69846400     28
    ##  2 FB     2013-01-03  27.9  28.5  27.6  27.8  63140600     27.8
    ##  3 FB     2013-01-04  28.0  28.9  27.8  28.8  72715400     28.8
    ##  4 FB     2013-01-07  28.7  29.8  28.6  29.4  83781800     29.4
    ##  5 FB     2013-01-08  29.5  29.6  28.9  29.1  45871300     29.1
    ##  6 FB     2013-01-09  29.7  30.6  29.5  30.6 104787700     30.6
    ##  7 FB     2013-01-10  30.6  31.5  30.3  31.3  95316400     31.3
    ##  8 FB     2013-01-11  31.3  32.0  31.1  31.7  89598000     31.7
    ##  9 FB     2013-01-14  32.1  32.2  30.6  31.0  98892800     31.0
    ## 10 FB     2013-01-15  30.6  31.7  29.9  30.1 173242600     30.1
    ## # … with 4,022 more rows

Second, use `group_by` to group by stock symbol. Third, apply the mutation. We can do this in one easy workflow. The `periodReturns` function is applied to each group of stock prices, and a new data frame was returned with the annual returns in the correct periodicity.

    FANG_returns_yearly <- FANG %>%
        group_by(symbol) %>%
        tq_transmute(select     = adjusted,
                     mutate_fun = periodReturn,
                     period     = "yearly",
                     col_rename = "yearly.returns")

Last, we can visualize the returns.

    FANG_returns_yearly %>%
        ggplot(aes(x = year(date), y = yearly.returns, fill = symbol)) +
        geom_bar(position = "dodge", stat = "identity") +
        labs(title = "FANG: Annual Returns",
             subtitle = "Mutating at scale is quick and easy!",
             y = "Returns", x = "", color = "") +
        scale_y_continuous(labels = scales::percent) +
        coord_flip() +
        theme_tq() +
        scale_fill_tq()

# 3.0 Modeling Financial Data using purrr

Eventually you will want to begin modeling (or more generally applying functions) at scale! One of the best features of the `tidyverse` is the ability to map functions to nested tibbles using `purrr`. From the Many Models chapter of “R for Data Science”, we can apply the same modeling workflow to financial analysis. Using a two step workflow:

1. Model a single stock
2. Scale to many stocks

Let’s go through an example to illustrate.

## Example: Applying a Regression Model to Detect a Positive Trend

In this example, we’ll use a simple linear model to identify the trend in annual returns to determine if the stock returns are decreasing or increasing over time.

### Analyze a Single Stock

First, let’s collect stock data with `tq_get()`

    AAPL <- tq_get("AAPL", from = "2007-01-01", to = "2016-12-31")
    AAPL

    ## # A tibble: 2,518 x 8
    ##    symbol date        open  high   low close    volume adjusted
    ##    <chr>  <date>     <dbl> <dbl> <dbl> <dbl>     <dbl>    <dbl>
    ##  1 AAPL   2007-01-03  12.3  12.4  11.7  12.0 309579900     10.4
    ##  2 AAPL   2007-01-04  12.0  12.3  12.0  12.2 211815100     10.6
    ##  3 AAPL   2007-01-05  12.3  12.3  12.1  12.2 208685400     10.5
    ##  4 AAPL   2007-01-08  12.3  12.4  12.2  12.2 199276700     10.6
    ##  5 AAPL   2007-01-09  12.4  13.3  12.2  13.2 837324600     11.5
    ##  6 AAPL   2007-01-10  13.5  14.0  13.4  13.9 738220000     12.0
    ##  7 AAPL   2007-01-11  13.7  13.8  13.6  13.7 360063200     11.9
    ##  8 AAPL   2007-01-12  13.5  13.6  13.3  13.5 328172600     11.7
    ##  9 AAPL   2007-01-16  13.7  13.9  13.6  13.9 311019100     12.0
    ## 10 AAPL   2007-01-17  13.9  13.9  13.5  13.6 411565000     11.8
    ## # … with 2,508 more rows

Next, come up with a function to help us collect annual log returns. The function below mutates the stock prices to period returns using `tq_transmute()`. We add the `type = "log"` and `period = "monthly"` arguments to ensure we retrieve a tibble of monthly log returns. Last, we take the mean of the monthly returns to get MMLR.

    get_annual_returns <- function(stock.returns) {
        stock.returns %>%
            tq_transmute(select     = adjusted,
                         mutate_fun = periodReturn,
                         type       = "log",
                         period     = "yearly")
    }

Let’s test `get_annual_returns` out. We now have the annual log returns over the past ten years.

    AAPL_annual_log_returns <- get_annual_returns(AAPL)
    AAPL_annual_log_returns

    ## # A tibble: 10 x 2
    ##    date       yearly.returns
    ##    <date>              <dbl>
    ##  1 2007-12-31         0.860
    ##  2 2008-12-31        -0.842
    ##  3 2009-12-31         0.904
    ##  4 2010-12-31         0.426
    ##  5 2011-12-30         0.228
    ##  6 2012-12-31         0.282
    ##  7 2013-12-31         0.0776
    ##  8 2014-12-31         0.341
    ##  9 2015-12-31        -0.0306
    ## 10 2016-12-30         0.118

Let’s visualize to identify trends. We can see from the linear trend line that AAPL’s stock returns are declining.

    AAPL_annual_log_returns %>%
        ggplot(aes(x = year(date), y = yearly.returns)) +
        geom_hline(yintercept = 0, color = palette_light()[[1]]) +
        geom_point(size = 2, color = palette_light()[[3]]) +
        geom_line(size = 1, color = palette_light()[[3]]) +
        geom_smooth(method = "lm", se = FALSE) +
        labs(title = "AAPL: Visualizing Trends in Annual Returns",
             x = "", y = "Annual Returns", color = "") +
        theme_tq()

Now, we can get the linear model using the `lm()` function. However, there is one problem: the output is not “tidy”.

    mod <- lm(yearly.returns ~ year(date), data = AAPL_annual_log_returns)
    mod

    ##
    ## Call:
    ## lm(formula = yearly.returns ~ year(date), data = AAPL_annual_log_returns)
    ##
    ## Coefficients:
    ## (Intercept)   year(date)
    ##    58.86282     -0.02915

We can utilize the `broom` package to get “tidy” data from the model. There’s three primary functions:

1. `augment`: adds columns to the original data such as predictions, residuals and cluster assignments
2. `glance`: provides a one-row summary of model-level statistics
3. `tidy`: summarizes a model’s statistical findings such as coefficients of a regression

We’ll use `tidy` to retrieve the model coefficients.

    library(broom)
    tidy(mod)

    ## # A tibble: 2 x 5
    ##   term        estimate std.error statistic p.value
    ##   <chr>          <dbl>     <dbl>     <dbl>   <dbl>
    ## 1 (Intercept)  58.9     113.         0.520   0.617
    ## 2 year(date)   -0.0291    0.0562    -0.518   0.618

Adding to our workflow, we have the following:

    get_model <- function(stock_data) {
        annual_returns <- get_annual_returns(stock_data)
        mod <- lm(yearly.returns ~ year(date), data = annual_returns)
        tidy(mod)
    }

Testing it out on a single stock. We can see that the “term” that contains the direction of the trend (the slope) is “year(date)”. The interpetation is that as year increases one unit, the annual returns decrease by 3%.

    get_model(AAPL)

    ## # A tibble: 2 x 5
    ##   term        estimate std.error statistic p.value
    ##   <chr>          <dbl>     <dbl>     <dbl>   <dbl>
    ## 1 (Intercept)  58.9     113.         0.520   0.617
    ## 2 year(date)   -0.0291    0.0562    -0.518   0.618

Now that we have identified the trend direction, it looks like we are ready to scale.

### Scale to Many Stocks

Once the analysis for one stock is done scale to many stocks is simple. For brevity, we’ll randomly sample ten stocks from the S&P500 with a call to `dplyr::sample_n()`.

    set.seed(10)
    stocks_tbl <- tq_index("SP500") %>%
        sample_n(5)
    stocks_tbl

    ## # A tibble: 5 x 8
    ##   symbol company     identifier sedol  weight sector  shares_held local_currency
    ##   <chr>  <chr>       <chr>      <chr>   <dbl> <chr>         <dbl> <chr>
    ## 1 IPGP   IPG Photon… 44980X10   2698… 1.81e-4 Inform…      379621 USD
    ## 2 EL     Estee Laud… 51843910   2320… 1.69e-3 Consum…     2391881 USD
    ## 3 INCY   Incyte Cor… 45337C10   2471… 5.52e-4 Health…     1894981 USD
    ## 4 EXPD   Expeditors… 30213010   2325… 4.83e-4 Indust…     1837518 USD
    ## 5 BEN    Franklin R… 35461310   2350… 2.65e-4 Financ…     3026109 USD

We can now apply our analysis function to the stocks using `dplyr::mutate` and `purrr::map`. The `mutate()` function adds a column to our tibble, and the `map()` function maps our custom `get_model` function to our tibble of stocks using the `symbol` column. The `tidyr::unnest` function unrolls the nested data frame so all of the model statistics are accessable in the top data frame level. The `filter`, `arrange` and `select` steps just manipulate the data frame to isolate and arrange the data for our viewing.

    stocks_model_stats <- stocks_tbl %>%
        select(symbol, company) %>%
        tq_get(from = "2007-01-01", to = "2016-12-31") %>%

        # Nest
        group_by(symbol, company) %>%
        nest() %>%

        # Apply the get_model() function to the new "nested" data column
        mutate(model = map(data, get_model)) %>%

        # Unnest and collect slope
        unnest(model) %>%
        filter(term == "year(date)") %>%
        arrange(desc(estimate)) %>%
        select(-term)

    stocks_model_stats

    ## # A tibble: 5 x 7
    ## # Groups:   symbol, company [5]
    ##   symbol company                data        estimate std.error statistic p.value
    ##   <chr>  <chr>                  <list>         <dbl>     <dbl>     <dbl>   <dbl>
    ## 1 IPGP   IPG Photonics Corpora… <tibble [2…  2.74e-2    0.0377   0.726     0.489
    ## 2 INCY   Incyte Corporation     <tibble [2…  2.10e-2    0.0682   0.308     0.766
    ## 3 EXPD   Expeditors Internatio… <tibble [2…  1.34e-2    0.0246   0.542     0.602
    ## 4 BEN    Franklin Resources In… <tibble [2… -1.50e-4    0.0392  -0.00383   0.997
    ## 5 EL     Estee Lauder Companie… <tibble [2… -1.26e-2    0.0299  -0.422     0.684

We’re done! We now have the coefficient of the linear regression that tracks the direction of the trend line. We can easily extend this type of analysis to larger lists or stock indexes. For example, the entire S&P500 could be analyzed removing the `sample_n()` following the call to `tq_index("SP500")`.

# 4.0 Error Handling when Scaling

Eventually you will run into a stock index, stock symbol, FRED data code, etc that cannot be retrieved. Possible reasons are:

- An index becomes out of date
- A company goes private
- A stock ticker symbol changes
- Yahoo / FRED just doesn’t like your stock symbol / FRED code

This becomes painful when scaling if the functions return errors. So, the `tq_get()` function is designed to handle errors gracefully. What this means is an `NA` value is returned when an error is generated along with a gentle error warning.

    tq_get("XYZ", "stock.prices")

    ## [1] NA

## Pros and Cons to Built-In Error-Handling

There are pros and cons to this approach that you may not agree with, but I believe helps in the long run. Just be aware of what happens:

- Pros: Long running scripts are not interrupted because of one error

- Cons: Errors can be inadvertently handled or flow downstream if the users does not read the warnings

## Bad Apples Fail Gracefully, tq_get

Let’s see an example when using `tq_get()` to get the stock prices for a long list of stocks with one `BAD APPLE`. The argument `complete_cases` comes in handy. The default is `TRUE`, which removes “bad apples” so future analysis have complete cases to compute on. Note that a gentle warning stating that an error occurred and was dealt with by removing the rows from the results.

    c("AAPL", "GOOG", "BAD APPLE") %>%
        tq_get(get = "stock.prices", complete_cases = TRUE)

    ## Warning: x = 'BAD APPLE', get = 'stock.prices': Error in getSymbols.yahoo(Symbols = "BAD APPLE", env = <environment>, : Unable to import "BAD APPLE".
    ## BAD APPLE download failed after two attempts. Error message:
    ## HTTP error 400.
    ##  Removing BAD APPLE.

    ## # A tibble: 5,116 x 8
    ##    symbol date        open  high   low close    volume adjusted
    ##    <chr>  <date>     <dbl> <dbl> <dbl> <dbl>     <dbl>    <dbl>
    ##  1 AAPL   2010-01-04  30.5  30.6  30.3  30.6 123432400     26.5
    ##  2 AAPL   2010-01-05  30.7  30.8  30.5  30.6 150476200     26.6
    ##  3 AAPL   2010-01-06  30.6  30.7  30.1  30.1 138040000     26.2
    ##  4 AAPL   2010-01-07  30.2  30.3  29.9  30.1 119282800     26.1
    ##  5 AAPL   2010-01-08  30.0  30.3  29.9  30.3 111902700     26.3
    ##  6 AAPL   2010-01-11  30.4  30.4  29.8  30.0 115557400     26.1
    ##  7 AAPL   2010-01-12  29.9  30.0  29.5  29.7 148614900     25.8
    ##  8 AAPL   2010-01-13  29.7  30.1  29.2  30.1 151473000     26.1
    ##  9 AAPL   2010-01-14  30.0  30.1  29.9  29.9 108223500     26.0
    ## 10 AAPL   2010-01-15  30.1  30.2  29.4  29.4 148516900     25.5
    ## # … with 5,106 more rows

Now switching `complete_cases = FALSE` will retain any errors as `NA` values in a nested data frame. Notice that the error message and output change. The error message now states that the `NA` values exist in the output and the return is a “nested” data structure.

    c("AAPL", "GOOG", "BAD APPLE") %>%
        tq_get(get = "stock.prices", complete_cases = FALSE)

    ## Warning: x = 'BAD APPLE', get = 'stock.prices': Error in getSymbols.yahoo(Symbols = "BAD APPLE", env = <environment>, : Unable to import "BAD APPLE".
    ## BAD APPLE download failed after two attempts. Error message:
    ## HTTP error 400.

    ## Warning in value[[3L]](cond): Returning as nested data frame.

    ## # A tibble: 3 x 2
    ##   symbol    stock.prices
    ##   <chr>     <list>
    ## 1 AAPL      <tibble [2,558 × 7]>
    ## 2 GOOG      <tibble [2,558 × 7]>
    ## 3 BAD APPLE <lgl [1]>

In both cases, the prudent user will review the warnings to determine what happened and whether or not this is acceptable. In the `complete_cases = FALSE` example, if the user attempts to perform downstream computations at scale, the computations will likely fail grinding the analysis to a hault. But, the advantage is that the user will more easily be able to filter to the problem childs to determine what happened and decide whether this is acceptable or not.