R: Fast Between (Averaging) and (Quasi-)Within (Centering)...

fbetween-fwithin {collapse} | R Documentation

## Fast Between (Averaging) and (Quasi-)Within (Centering) Transformations

### Description

`fbetween` and `fwithin` are S3 generics to efficiently obtain between-transformed (averaged) or (quasi-)within-transformed (demeaned) data. These operations can be performed groupwise and/or weighted. `B` and `W` are wrappers around `fbetween` and `fwithin` representing the 'between-operator' and the 'within-operator'.

(`B` / `W` provide more flexibility than `fbetween` / `fwithin` when applied to data frames (i.e. column subsetting, formula input, auto-renaming and id-variable-preservation capabilities...), but are otherwise identical.)

### Usage

    fbetween(x, ...)
     fwithin(x, ...)
           B(x, ...)
           W(x, ...)

    ## Default S3 method:
    fbetween(x, g = NULL, w = NULL, na.rm = TRUE, fill = FALSE, ...)
    ## Default S3 method:
    fwithin(x, g = NULL, w = NULL, na.rm = TRUE, mean = 0, theta = 1, ...)
    ## Default S3 method:
    B(x, g = NULL, w = NULL, na.rm = TRUE, fill = FALSE, ...)
    ## Default S3 method:
    W(x, g = NULL, w = NULL, na.rm = TRUE, mean = 0, theta = 1, ...)

    ## S3 method for class 'matrix'
    fbetween(x, g = NULL, w = NULL, na.rm = TRUE, fill = FALSE, ...)
    ## S3 method for class 'matrix'
    fwithin(x, g = NULL, w = NULL, na.rm = TRUE, mean = 0, theta = 1, ...)
    ## S3 method for class 'matrix'
    B(x, g = NULL, w = NULL, na.rm = TRUE, fill = FALSE, stub = "B.", ...)
    ## S3 method for class 'matrix'
    W(x, g = NULL, w = NULL, na.rm = TRUE, mean = 0, theta = 1, stub = "W.", ...)

    ## S3 method for class 'data.frame'
    fbetween(x, g = NULL, w = NULL, na.rm = TRUE, fill = FALSE, ...)
    ## S3 method for class 'data.frame'
    fwithin(x, g = NULL, w = NULL, na.rm = TRUE, mean = 0, theta = 1, ...)
    ## S3 method for class 'data.frame'
    B(x, by = NULL, w = NULL, cols = is.numeric, na.rm = TRUE,
      fill = FALSE, stub = "B.", keep.by = TRUE, keep.w = TRUE, ...)
    ## S3 method for class 'data.frame'
    W(x, by = NULL, w = NULL, cols = is.numeric, na.rm = TRUE,
      mean = 0, theta = 1, stub = "W.", keep.by = TRUE, keep.w = TRUE, ...)

    # Methods for compatibility with plm:

    ## S3 method for class 'pseries'
    fbetween(x, effect = 1L, w = NULL, na.rm = TRUE, fill = FALSE, ...)
    ## S3 method for class 'pseries'
    fwithin(x, effect = 1L, w = NULL, na.rm = TRUE, mean = 0, theta = 1, ...)
    ## S3 method for class 'pseries'
    B(x, effect = 1L, w = NULL, na.rm = TRUE, fill = FALSE, ...)
    ## S3 method for class 'pseries'
    W(x, effect = 1L, w = NULL, na.rm = TRUE, mean = 0, theta = 1, ...)

    ## S3 method for class 'pdata.frame'
    fbetween(x, effect = 1L, w = NULL, na.rm = TRUE, fill = FALSE, ...)
    ## S3 method for class 'pdata.frame'
    fwithin(x, effect = 1L, w = NULL, na.rm = TRUE, mean = 0, theta = 1, ...)
    ## S3 method for class 'pdata.frame'
    B(x, effect = 1L, w = NULL, cols = is.numeric, na.rm = TRUE,
      fill = FALSE, stub = "B.", keep.ids = TRUE, keep.w = TRUE, ...)
    ## S3 method for class 'pdata.frame'
    W(x, effect = 1L, w = NULL, cols = is.numeric, na.rm = TRUE,
      mean = 0, theta = 1, stub = "W.", keep.ids = TRUE, keep.w = TRUE, ...)

    # Methods for grouped data frame / compatibility with dplyr:

    ## S3 method for class 'grouped_df'
    fbetween(x, w = NULL, na.rm = TRUE, fill = FALSE,
             keep.group_vars = TRUE, keep.w = TRUE, ...)
    ## S3 method for class 'grouped_df'
    fwithin(x, w = NULL, na.rm = TRUE, mean = 0, theta = 1,
            keep.group_vars = TRUE, keep.w = TRUE, ...)
    ## S3 method for class 'grouped_df'
    B(x, w = NULL, na.rm = TRUE, fill = FALSE,
      stub = "B.", keep.group_vars = TRUE, keep.w = TRUE, ...)
    ## S3 method for class 'grouped_df'
    W(x, w = NULL, na.rm = TRUE, mean = 0, theta = 1,
      stub = "W.", keep.group_vars = TRUE, keep.w = TRUE, ...)

### Arguments

`x` | a numeric vector, matrix, data frame, panel series (class `pseries` of package `plm`), panel data frame (`plm::pdata.frame`) or grouped data frame (class 'grouped_df').
`g` | a factor, `GRP` object, atomic vector (internally converted to factor) or a list of vectors / factors (internally converted to a `GRP` object) used to group `x`.
`by` | B and W data.frame method: Same as g, but also allows one- or two-sided formulas i.e. `~ group1` or `var1 + var2 ~ group1 + group2`. See Examples.
`w` | a numeric vector of (non-negative) weights. `B`/`W` data frame and `pdata.frame` methods also allow a one-sided formula i.e. `~ weightcol`. The `grouped_df` (dplyr) method supports lazy-evaluation. See Examples.
`cols` | data.frame method: Select columns to center/average using a function, column names, indices or a logical vector. Default: All numeric variables. Note: `cols` is ignored if a two-sided formula is passed to `by`.
`na.rm` | logical. Skip missing values in `x` and `w` when computing averages. If `na.rm = FALSE` and a `NA` or `NaN` is encountered, the average for that group will be `NA`, and all data points belonging to that group in the output vector will also be `NA`.
`effect` | plm methods: Select which panel identifier should be used as grouping variable. 1L takes the first variable in the `plm::index`, 2L the second etc. Index variables can also be called by name using a character string. If more than one variable is supplied, the corresponding index-factors are interacted.
`stub` | a prefix or stub to rename all transformed columns. `FALSE` will not rename columns.
`fill` | option to `fbetween`/`B`: Logical. `TRUE` will overwrite missing values in `x` with the respective average. By default missing values in `x` are preserved.
`mean` | option to `fwithin`/`W`: The mean to center on, default is 0, but a different mean can be supplied and will be added to the data after the centering is performed. A special option when performing grouped centering is `mean = "overall.mean"`. In that case the overall mean of the data will be added after subtracting out group means.
`theta` | option to `fwithin`/`W`: Double. An optional scalar parameter for quasi-demeaning i.e. `x - theta * xi.`. This is useful for variance components ('random-effects') estimators. see Details.
`keep.by, keep.ids, keep.group_vars` | B and W data.frame, pdata.frame and grouped_df methods: Logical. Retain grouping / panel-identifier columns in the output. For data frames this only works if grouping variables were passed in a formula.
`keep.w` | B and W data.frame, pdata.frame and grouped_df methods: Logical. Retain column containing the weights in the output. Only works if `w` is passed as formula / lazy-expression.
`...` | arguments to be passed to or from other methods.

### Details

Without groups, `fbetween`/`B` replaces all data points in `x` with their mean or weighted mean (if `w` is supplied). Similarly `fwithin/W` subtracts the (weighted) mean from all data points i.e. centers the data on the mean.

With groups supplied to `g`, the replacement / centering performed by `fbetween/B` | `fwithin/W` becomes groupwise. In terms of panel data notation: If `x` is a vector in such a panel dataset, `xit` denotes a single data-point belonging to group `i` in time-period `t` (`t` need not be a time-period). Then `xi.` denotes `x`, averaged over `t`. `fbetween`/`B` now returns `xi.` and `fwithin`/`W` returns `x - xi.`. Thus for any data `x` and any grouping vector `g`: `B(x,g) + W(x,g) = xi. + x - xi. = x`. In terms of variance, `fbetween/B` only retains the variance between group averages, while `fwithin`/`W`, by subtracting out group means, only retains the variance within those groups.

The data replacement performed by `fbetween`/`B` can keep (default) or overwrite missing values (option `fill = TRUE`) in `x`. `fwithin/W` can center data simply (default), or add back a mean after centering (option `mean = value`), or add the overall mean in groupwise computations (option `mean = "overall.mean"`). Let `x..` denote the overall mean of `x`, then `fwithin`/`W` with `mean = "overall.mean"` returns `x - xi. + x..` instead of `x - xi.`. This is useful to get rid of group-differences but preserve the overall level of the data. In regression analysis, centering with `mean = "overall.mean"` will only change the constant term. See Examples.

If `theta != 1`, `fwithin`/`W` performs quasi-demeaning `x - theta * xi.`. If `mean = "overall.mean"`, `x - theta * xi. + theta * x..` is returned, so that the mean of the partially demeaned data is still equal to the overall data mean `x..`. A numeric value passed to `mean` will simply be added back to the quasi-demeaned data i.e. `x - theta * xi. + mean`.

Now in the case of a linear panel model y_{it} = β_0 + β_1 X_{it} + u_{it} with u_{it} = α_i + ε_{it}. If α_i \neq α = const. (there exists individual heterogeneity), then pooled OLS is at least inefficient and inference on β_1 is invalid. If E[α_i|X_{it}] = 0 (mean independence of individual heterogeneity α_i), the variance components or 'random-effects' estimator provides an asymptotically efficient FGLS solution by estimating a transformed model y_{it}-θ y_{i.} = β_0 + β_1 (X_{it} - θ X_{i.}) + (u_{it} - θ u_{i.}), where θ = 1 - \frac{σ_α}{√(σ^2_α + T σ^2_ε)}. An estimate of θ can be obtained from the an estimate of \hat{u}_{it} (the residuals from the pooled model). If E[α_i|X_{it}] \neq 0, pooled OLS is biased and inconsistent, and taking θ = 1 gives an unbiased and consistent fixed-effects estimator of β_1. See Examples.

### Value

`fbetween`/`B` returns `x` with every element replaced by its (groupwise) mean (`xi.`). Missing values are preserved if `fill = FALSE` (the default). `fwithin/W` returns `x` where every element was subtracted its (groupwise) mean (`x - theta * xi. + mean` or, if `mean = "overall.mean"`, `x - theta * xi. + theta * x..`). See Details.

### References

Mundlak, Yair. 1978. On the Pooling of Time Series and Cross Section Data. Econometrica 46 (1): 69-85.

### See Also

`fhdbetween/HDB and fhdwithin/HDW`, `fscale/STD`, `TRA`, Data Transformations, Collapse Overview

### Examples

    ## Simple centering and averaging
    head(fbetween(mtcars))
    head(B(mtcars))
    head(fwithin(mtcars))
    head(W(mtcars))
    all.equal(fbetween(mtcars) + fwithin(mtcars), mtcars)

    ## Groupwise centering and averaging
    head(fbetween(mtcars, mtcars$cyl))
    head(fwithin(mtcars, mtcars$cyl))
    all.equal(fbetween(mtcars, mtcars$cyl) + fwithin(mtcars, mtcars$cyl), mtcars)

    head(W(wlddev, ~ iso3c, cols = 9:13))    # Center the 5 series in this dataset by country
    head(cbind(get_vars(wlddev,"iso3c"),     # Same thing done manually using fwithin..
          add_stub(fwithin(get_vars(wlddev,9:13), wlddev$iso3c), "W.")))

    ## Using B() and W() for fixed-effects regressions:

    # Several ways of running the same regression with cyl-fixed effects
    lm(W(mpg,cyl) ~ W(carb,cyl), data = mtcars)                     # Centering each individually
    lm(mpg ~ carb, data = W(mtcars, ~ cyl, stub = FALSE))           # Centering the entire data
    lm(mpg ~ carb, data = W(mtcars, ~ cyl, stub = FALSE,            # Here only the intercept changes
                            mean = "overall.mean"))
    lm(mpg ~ carb + B(carb,cyl), data = mtcars)                     # Procedure suggested by
    # ..Mundlak (1978) - partialling out group averages amounts to the same as demeaning the data

    plm::plm(mpg ~ carb, mtcars, index = "cyl", model = "within")   # "Proof"..

    # This takes the interaction of cyl, vs and am as fixed effects
    lm(W(mpg,list(cyl,vs,am)) ~ W(carb,list(cyl,vs,am)), data = mtcars)
    lm(mpg ~ carb, data = W(mtcars, ~ cyl + vs + am, stub = FALSE))
    lm(mpg ~ carb + B(carb,list(cyl,vs,am)), data = mtcars)

    # Now with cyl fixed effects weighted by hp:
    lm(W(mpg,cyl,hp) ~ W(carb,cyl,hp), data = mtcars)
    lm(mpg ~ carb, data = W(mtcars, ~ cyl, ~ hp, stub = FALSE))
    lm(mpg ~ carb + B(carb,cyl,hp), data = mtcars)       # WRONG ! Gives a different coefficient!!

    ## Manual variance components (random-effects) estimation
    res <- HDW(mtcars, mpg ~ carb)[[1]]  # Get residuals from pooled OLS
    sig2_u <- fvar(res)
    sig2_e <- fvar(fwithin(res, mtcars$cyl))
    T <- length(res) / fndistinct(mtcars$cyl)
    sig2_alpha <- sig2_u - sig2_e
    theta <- 1 - sqrt(sig2_alpha) / sqrt(sig2_alpha + T * sig2_e)
    lm(mpg ~ carb, data = W(mtcars, ~ cyl, theta = theta, mean = "overall.mean", stub = FALSE))

    # A slightly different method to obtain theta...
    plm::plm(mpg ~ carb, mtcars, index = "cyl", model = "random")

[Package collapse version 1.7.6 Index]