R: Fast (Grouped, Weighted) Summary Statistics for...

qsu {collapse} | R Documentation

## Fast (Grouped, Weighted) Summary Statistics for Cross-Sectional and Panel Data

### Description

`qsu`, shorthand for quick-summary, is an extremely fast summary command inspired by the (xt)summarize command in the STATA statistical software.

It computes a set of 7 statistics (nobs, mean, sd, min, max, skewness and kurtosis) using a numerically stable one-pass method generalized from Welford's Algorithm. Statistics can be computed weighted, by groups, and also within-and between entities (for panel data, see Details).

### Usage

    qsu(x, ...)

    ## Default S3 method:
    qsu(x, g = NULL, pid = NULL, w = NULL, higher = FALSE,
        array = TRUE, stable.algo = TRUE, ...)

    ## S3 method for class 'matrix'
    qsu(x, g = NULL, pid = NULL, w = NULL, higher = FALSE,
        array = TRUE, stable.algo = TRUE, ...)

    ## S3 method for class 'data.frame'
    qsu(x, by = NULL, pid = NULL, w = NULL, cols = NULL, higher = FALSE,
        array = TRUE, vlabels = FALSE, stable.algo = TRUE, ...)

    # Methods for compatibility with plm:

    ## S3 method for class 'pseries'
    qsu(x, g = NULL, w = NULL, effect = 1L, higher = FALSE,
        array = TRUE, stable.algo = TRUE, ...)

    ## S3 method for class 'pdata.frame'
    qsu(x, by = NULL, w = NULL, cols = NULL, effect = 1L, higher = FALSE,
        array = TRUE, vlabels = FALSE, stable.algo = TRUE, ...)

    # Methods for compatibility with sf:

    ## S3 method for class 'sf'
    qsu(x, by = NULL, pid = NULL, w = NULL, cols = NULL, higher = FALSE,
        array = TRUE, vlabels = FALSE, stable.algo = TRUE, ...)

    ## S3 method for class 'qsu'
    print(x, digits = 4, nonsci.digits = 9, na.print = "-",
          return = FALSE, print.gap = 2, ...)

### Arguments

`x` | a vector, matrix, data frame, panel series (`plm::pseries`) or panel data frame (`plm::pdata.frame`).
`g` | a factor, `GRP` object, atomic vector (internally converted to factor) or a list of vectors / factors (internally converted to a `GRP` object) used to group `x`.
`by` | (p)data.frame method: Same as `g`, but also allows one- or two-sided formulas i.e. `~ group1 + group2` or `var1 + var2 ~ group1 + group2`. See Examples.
`pid` | same input as `g/by`: Specify a panel-identifier to also compute statistics on between- and within- transformed data. Data frame method also supports one- or two-sided formulas. Transformations are taken independently from grouping with `g/by` (grouped statistics are computed on the transformed data if `g/by` is also used). However, passing any LHS variables to `pid` will overwrite any `LHS` variables passed to `by`.
`w` | a vector of (non-negative) weights. Adding weights will compute the weighted mean, sd, skewness and kurtosis, and transform the data using weighted individual means if `pid` is used.
`cols` | select columns to summarize using column names, indices, a logical vector or a function (e.g. `is.numeric`). Two-sided formulas passed to `by` or `pid` overwrite `cols`.
`higher` | logical. Add higher moments (skewness and kurtosis).
`array` | logical. If computations have more than 2 dimensions (up to a maximum of 4D: variables, statistics, groups and panel-decomposition) output to array, else output (nested) list of matrices.
`stable.algo` | logical. `FALSE` uses a faster but less stable method to calculate the standard deviation (see Details of `fsd`). Only available if `w = NULL` and `higher = FALSE`.
`vlabels` | logical. Use variable labels in the summary. See `vlabels`.
`effect` | plm methods: Select which panel identifier should be used for between and within transformations of the data. 1L takes the first variable in the `plm::index`, 2L the second etc.. Index variables can also be called by name using a character string. More than one variable can be supplied.
`...` | arguments to be passed to or from other methods.
`digits` | the number of digits to print after the comma/dot.
`nonsci.digits` | the number of digits to print before resorting to scientific notation (default is to print out numbers with up to 9 digits and print larger numbers scientifically).
`na.print` | character string to substitute for missing values.
`return` | logical. Don't print but instead return the formatted object.
`print.gap` | integer. Spacing between printed columns. Passed to `print.default`.

### Details

The algorithm used to compute statistics is well described here [see sections Welford's online algorithm, Weighted incremental algorithm and Higher-order statistics. Skewness and kurtosis are calculated as described in Higher-order statistics and are mathematically identical to those implemented in the moments package. Just note that `qsu` computes the kurtosis (like `momens::kurtosis`), not the excess-kurtosis (= kurtosis - 3) defined in Higher-order statistics. The Weighted incremental algorithm described can easily be generalized to higher-order statistics].

Grouped computations specified with `g/by` are carried out extremely efficiently as in `fsum` (in a single pass, without splitting the data).

If `pid` is used, `qsu` performs a panel-decomposition of each variable and computes 3 sets of statistics: Statistics computed on the 'Overall' (raw) data, statistics computed on the 'Between' - transformed (pid - averaged) data, and statistics computed on the 'Within' - transformed (pid - demeaned) data.

More formally, let `x` (bold) be a panel vector of data for `N` individuals indexed by `i`, recorded for `T` periods, indexed by `t`. `xit` then denotes a single data-point belonging to individual `i` in time-period `t` (`t/T` must not represent time). Then `xi.` denotes the average of all values for individual `i` (averaged over `t`), and by extension `xN.` is the vector (length `N`) of such averages for all individuals. If no groups are supplied to `g/by`, the 'Between' statistics are computed on `xN.`, the vector of individual averages. (This means that for a non-balanced panel or in the presence of missing values, the 'Overall' mean computed on `x` can be slightly different than the 'Between' mean computed on `xN.`, and the variance decomposition is not exact). If groups are supplied to `g/by`, `xN.` is expanded to the vector `xi.` (length `N x T`) by replacing each value `xit` in `x` with `xi.`, while preserving missing values in `x`. Grouped Between-statistics are then computed on `xi.`, with the only difference that the number of observations ('Between-N') reported for each group is the number of distinct non-missing values of `xi.` in each group (not the total number of non-missing values of `xi.` in each group, which is already reported in 'Overall-N'). See Examples.

'Within' statistics are always computed on the vector `x - xi. + x..`, where `x..` is simply the 'Overall' mean computed from `x`, which is added back to preserve the level of the data. The 'Within' mean computed on this data will always be identical to the 'Overall' mean. In the summary output, `qsu` reports not 'N', which would be identical to the 'Overall-N', but 'T', the average number of time-periods of data available for each individual obtained as 'T' = 'Overall-N / 'Between-N'. See Examples.

Apart from 'N/T' and the extrema, the standard-deviations ('SD') computed on between- and within- transformed data are extremely valuable because they indicate how much of the variation in a panel-variable is between-individuals and how much of the variation is within-individuals (over time). At the extremes, variables that have common values across individuals (such as the time-variable(s) 't' in a balanced panel), can readily be identified as individual-invariant because the 'Between-SD' on this variable is 0 and the 'Within-SD' is equal to the 'Overall-SD'. Analogous, time-invariant individual characteristics (such as the individual-id 'i') have a 0 'Within-SD' and a 'Between-SD' equal to the 'Overall-SD'. See Examples.

`qsu` comes with it's own print method which by default writes out up to 9 digits at 4 decimal places. Larger numbers are printed in scientific format. for numbers between 7 and 9 digits, an apostrophe (') is placed after the 6th digit to designate the millions. Missing values are printed using '-'.

The sf method simply ignores the geometry column.

### Value

A vector, matrix, array or list of matrices of summary statistics. All matrices and arrays have a class 'qsu' and a class 'table' attached.

### Note

In weighted summaries, observations with missing or zero weights are skipped, and thus do not affect any of the calculated statistics, including the observation count. This also implies that a logical vector passed to `w` can be used to efficiently summarize a subset of the data.

### Note

If weights `w` are used together with `pid`, transformed data is computed using weighted individual means i.e. weighted `xi.` and weighted `x..`. Weighted statistics are subsequently computed on this weighted-transformed data.

### References

Welford, B. P. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics. 4 (3): 419-420. doi:10.2307/1266577.

### See Also

`descr`, Summary Statistics, Fast Statistical Functions, Collapse Overview

### Examples

    ## World Development Panel Data
    # Simple Summaries -------------------------
    qsu(wlddev)                                 # Simple summary
    qsu(wlddev, vlabels = TRUE)                 # Display variable labels
    qsu(wlddev, higher = TRUE)                  # Add skewness and kurtosis

    # Grouped Summaries ------------------------
    qsu(wlddev, ~ region, vlabels = TRUE)       # Statistics by World Bank Region
    qsu(wlddev, PCGDP + LIFEEX ~ income)        # Summarize GDP per Capita and Life Expectancy by
    stats <- qsu(wlddev, ~ region + income,     # World Bank Income Level
                 cols = 9:10, higher = TRUE)    # Same variables, by both region and income
    aperm(stats)                                # A different perspective on the same stats

    # Panel Data Summaries ---------------------
    qsu(wlddev, pid = ~ iso3c, vlabels = TRUE)   # Adding between and within countries statistics
    # -> They show amongst other things that year and decade are individual-invariant,
    # that we have GINI-data on only 161 countries, with only 8.42 observations per country on average,
    # and that GDP, LIFEEX and GINI vary more between-countries, but ODA received varies more within
    # countries over time.

    # Let's do this manually for PCGDP:
    x <- wlddev$PCGDP
    g <- wlddev$iso3c

    # This is the exact variance decomposion
    all.equal(fvar(x), fvar(B(x, g)) + fvar(W(x, g)))

    # What qsu does is calculate
    r <- rbind(Overall = qsu(x),
               Between = qsu(fmean(x, g)), # Aggregation instead of between-transform
               Within = qsu(fwithin(x, g, mean = "overall.mean"))) # Same as qsu(W(x, g) + fmean(x))
    r[3, 1] <- r[1, 1] / r[2, 1]
    print.qsu(r)
    # Proof:
    qsu(x, pid = g)

    # Using plm:
    pwlddev <- plm::pdata.frame(wlddev,         # Creating a Panel Data Frame frame from this data
                      index = c("iso3c","year"))
    qsu(pwlddev)                                # Summary for pdata.frame -> qsu(wlddev, pid = ~ iso3c)
    qsu(pwlddev$PCGDP)                          # Default summary for Panel Series  (class pseries)
    qsu(G(pwlddev$PCGDP))                       # Summarizing GDP growth, see also ?G

    # Grouped Panel Data Summaries -------------
    qsu(wlddev, ~ region, ~ iso3c, cols = 9:12) # Panel-Statistics by region
    psr <- qsu(pwlddev, ~ region, cols = 9:12)  # Same on plm pdata.frame
    psr                                         # -> Gives a 4D array
    psr[,"N/T",,]                               # Checking out the number of observations:
    # In North america we only have 3 countries, for the GINI we only have 3.91 observations on average
    # for 45 Sub-Saharan-African countries, etc..
    psr[,"SD",,]                                # Considering only standard deviations
    # -> In all regions variations in inequality (GINI) between countries are greater than variations
    # in inequality within countries. The opposite is true for Life-Expectancy in all regions apart
    # from Europe, etc..

    # Again let's do this manually for PDGCP:
    d <- cbind(Overall = x,
               Between = fbetween(x, g),
               Within = fwithin(x, g, mean = "overall.mean"))

    r <- qsu(d, g = wlddev$region)
    r[,"N","Between"] <- fndistinct(g[!is.na(x)], wlddev$region[!is.na(x)])
    r[,"N","Within"] <- r[,"N","Overall"] / r[,"N","Between"]
    r

    # Proof:
    qsu(wlddev, PCGDP ~ region, ~ iso3c)

    # Same as above, but output as nested list
    psrl <- qsu(wlddev, ~ region, ~ iso3c,
                cols = 9:12, array = FALSE)
    psrl                                        # We can use unlist2d to create a tidy data.frame
    head(unlist2d(psrl, c("Variable","Trans"),
                        row.names = "Region"))

    # Weighted Summaries -----------------------
    n <- nrow(wlddev)
    weights <- abs(rnorm(n))                    # Generate random weights
    qsu(wlddev, w = weights, higher = TRUE)     # Computed weighted mean, SD, skewness and kurtosis
    weightsNA <- weights                        # Weights may contain missing values.. inserting 1000
    weightsNA[sample.int(n, 1000)] <- NA
    qsu(wlddev, w = weightsNA, higher = TRUE)   # But now these values are removed from all variables

    # Grouped and panel-summaries can also be weighted in the same manor

[Package collapse version 1.7.6 Index]