%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%% PREAMBLE %%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%

%\VignetteIndexEntry{rrapply: revisiting R-base rapply}
%\VignetteAuthor{Joris Chau}
%\VignetteKeyword{introduction}
%\VignetteKeyword{rapply}
%\VignetteKeyword{rrapply}
%\VignetteEncoding{UTF-8}

\documentclass[10pt]{article}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{xcolor}
\definecolor{light-gray}{gray}{0.95}
\pdfstringdefDisableCommands{%
\def\\{}%
\def\texttt#1{<#1>}%
}
\newcommand{\code}[1]{\colorbox{light-gray}{\texttt{#1}}}

\addtolength{\oddsidemargin}{-.6in}%
\addtolength{\evensidemargin}{-.6in}%
\addtolength{\textwidth}{1.2in}%
\addtolength{\textheight}{1.3in}%
\addtolength{\topmargin}{-.8in}%

\newenvironment{remark}[1][Remark.]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}]}{\end{trivlist}}
\newcommand{\bs}[1]{\boldsymbol{#1}}
\newcommand{\te}[1]{\textnormal{#1}}
\makeatletter
\newcommand*\rel@kern[1]{\kern#1\dimexpr\macc@kerna}
\newcommand*\widebar[1]{%
\begingroup
\def\mathaccent##1##2{%
\rel@kern{0.8}%
\overline{\rel@kern{-0.8}\macc@nucleus\rel@kern{0.2}}%
\rel@kern{-0.2}%
}%
\macc@depth\@ne
\let\math@bgroup\@empty \let\math@egroup\macc@set@skewchar
\mathsurround\z@ \frozen@everymath{\mathgroup\macc@group\relax}%
\macc@set@skewchar\relax
\let\mathaccentV\macc@nested@a
\macc@nested@a\relax111{#1}%
\endgroup
}
\makeatother

%%%%%%%%%%%%%%%%%%%%
%%%%%%% BODY %%%%%%%
%%%%%%%%%%%%%%%%%%%%

\begin{document}
\SweaveOpts{concordance=TRUE}

\def\spacingset#1{\renewcommand{\baselinestretch}%
{#1}\small\normalsize} \spacingset{1}


<<echo=false>>=
op <- options(width = 60, continue = "  ")
@

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%% TITLE + ABSTRACT %%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\title{\texttt{rrapply}: revisiting \textsf{R}-base \texttt{rapply}}
\author{Joris Chau}
\date{\today}
\maketitle

\spacingset{1.25} 

\begin{abstract} 
\noindent 
The minimal \texttt{rrapply}-package contains a single function \code{rrapply()}, providing an extended implementation of \textsf{R}-base's \code{rapply()} function. Base \code{rapply()} applies a function \code{f} to all elements of a list recursively. The \code{rrapply()} function extends base \code{rapply()} by including a condition or predicate function for the application of \code{f} and the option to prune or aggregate list elements from the result. In addition, special arguments \code{.xname}, \code{.xpos}, \code{.xparents} and \code{.xsiblings} can be used inside the \code{f} and \code{condition} functions to access the name, location, parents and siblings in the nested list of the list element under evaluation. \code{rrapply()} builds upon \code{rapply()}'s native \textsf{C} implementation and for this reason requires no external \textsf{R}-package dependencies.
\end{abstract}

\spacingset{1.45} 

\section{Quick review of \texttt{rapply()}}

The dataset \code{renewable\_energy\_by\_country} included in the \texttt{rrapply}-package lists the share of renewable energy as a percentage in the total energy consumption per country in 2016. The dataset is publicly available at the United Nations Open SDG Data Hub (\href{https://unstats-undesa.opendata.arcgis.com/datasets/indicator-7-2-1-renewable-energy-share-in-the-total-final-energy-consumption-percent-3}{UNSD-SDG07}). The 249 countries and areas are structured as a nested list based on their geographical location according to the United Nations M49 standard (\href{https://unstats.un.org/unsd/methodology/m49/}{UNSD-M49}). The numeric values listed for each country are percentages, if no data is available the country's value is \code{NA}.
<<>>=
library(rrapply)
data("renewable_energy_by_country")
## display list structure (only first two elements of each node)
str(renewable_energy_by_country, list.len = 2, give.attr = FALSE)
@
For convenience, we subset only the values for countries and areas in \code{Oceania},
<<>>=
renewable_oceania <- renewable_energy_by_country[["World"]]["Oceania"]
str(renewable_oceania, list.len = 3, give.attr = FALSE)
@
Using base \code{rapply()}, we can apply a function \code{f} to each leaf element or leaf elements of a particular class or type. By a leaf element, we refer to any element of the list which is not itself list-like, in this case the numeric country percentages. For instance, we can replace all \code{NA}'s by zeros using an \code{ifelse} statement in the \code{f} function, 
<<>>=
na_zero_oceania_unlist <- rapply(
  renewable_oceania, 
  f = function(x) ifelse(is.na(x), 0, x)
)
head(na_zero_oceania_unlist)
@
By default, the result is returned \emph{unlisted}. The original list structure can be preserved via the arguments \code{how = "replace"} or \code{how = "list"}. Conceptually, \code{how = "replace"} makes a complete copy of the input list and recursively replaces the leaf elements with a class in \code{classes} by the result of applying \code{f}. \code{how = "list"} recursively makes copies of the list-like elements of the input list, replacing leaf elements with a class in \code{classes} by the result of applying \code{f}, and replacing any other leaf elements by the value of \code{deflt}. \code{how = "unlist"} calls \code{unlist()} with argument \code{recursive = TRUE} on the initial result obtained by \code{how = "list"}, thus allowing the use of the \code{deflt} argument.\\[3mm]
By making use of the fact that the \code{NA}'s are of \code{logical} type and the non-\code{NA}'s are of \code{numeric} type, another way of replacing \code{NA}'s by zeros is via the \code{classes} argument:
<<>>=
na_zero_oceania_replace <- rapply(
  renewable_oceania, 
  f = function(x) 0, 
  classes = "logical", 
  how = "replace"
)
str(na_zero_oceania_replace, list.len = 3, give.attr = FALSE)
@
Or, by combining the \code{classes} and \code{deflt} arguments together with \code{how = "list"} or \code{how = "unlist"}, 
<<>>=
na_zero_oceania_list <- rapply(
  renewable_oceania, 
  f = function(x) x, 
  classes = "numeric", 
  deflt = 0, 
  how = "list"
)
str(na_zero_oceania_list, list.len = 3, give.attr = FALSE)
@
Each list element in \code{renewable\_energy\_by\_country} contains an \code{"M49-code"} attribute with the ``UN Standard Country or Area Codes for Statistical Use (Series M, No. 49)''. In order to keep this attribute when replacing \code{NA}'s by zeros, we could modify the above call with \code{how = "replace"} to,
<<>>=
na_zero_oceania_replace_attr <- rapply(
  renewable_oceania, 
  f = function(x) replace(x, is.na(x), 0), 
  how = "replace"
)
str(na_zero_oceania_replace_attr, list.len = 2)
@
With \code{how = "list"}, intermediate list attributes --excluding the leaf elements-- are in general not preserved. For this reason, it is probably best to use \code{how = "replace"} whenever possible if list attributes are present and must be preserved. 
<<>>=
na_zero_oceania_list_attr <- rapply(
  renewable_oceania, 
  f = function(x) replace(x, is.na(x), 0), 
  how = "list"
)
## this preserves all list attributes
str(na_zero_oceania_replace_attr, max.level = 2)
## this does not preserves all attributes!
str(na_zero_oceania_list_attr, max.level = 2)
@

\section{When to use \texttt{rrapply()}} 

\subsection{List pruning and unnesting}

With base \code{rapply()} there is no convenient way to prune or filter leaf elements from the input list. Using the \code{deflt} argument, we could set all leaf elements that are not subject to application of \code{f} to e.g. \code{NA} or \code{NULL}, but we cannot drop these leaf elements altogether from the resulting list.\\[3mm]
The \code{rrapply()} function adds an option to set the \code{how} argument to \code{how = "prune"}, in which case all leaf elements that are not subject to application of \code{f} are pruned from the list. The original list structure is retained, similar to the non-pruned options \code{how = "replace"} or \code{how = "list"}. \\[3mm]
Using \code{how = "prune"}, we can drop all \code{NA} elements while preserving the original list structure:
<<>>=
na_drop_oceania_list <- rrapply(
  renewable_oceania, 
  f = function(x) x, 
  classes = "numeric", 
  how = "prune"
)
str(na_drop_oceania_list, list.len = 3, give.attr = FALSE)
@
Instead, we can set \code{how = "flatten"} to return a flattened unnested version of the pruned list. This is more efficient than first returning the pruned list with \code{how = "prune"} and unlisting or flattening the list in a subsequent step. 
<<>>=
na_drop_oceania_flat <- rrapply(
  renewable_oceania, 
  f = function(x) x, 
  classes = "numeric", 
  how = "flatten"
)
head(na_drop_oceania_flat, n = 10)
@
Another option is to set \code{how = "melt"} to return a melted data.frame of the pruned list similar in format to \code{reshape2::melt()} applied to a nested list. The rows of the melted data.frame contain the node paths of the elements in the pruned list. The \code{"value"} column is a list-column with the values of the terminal nodes analogous to the flattened list returned by \code{how = "flatten"}. 
<<>>=
na_drop_melt <- rrapply(
  renewable_oceania,
  f = function(x) x,
  classes = "numeric",
  how = "melt"
)
head(na_drop_melt)
@
If no names are present in a sublist of the input list, \code{how = "melt"} replaces the names in the melted data.frame by the list element indices \code{"1"}, \code{"2"}, and so on:
<<>>=
## Remove all names at L2 (these arguments are explained in the following sections)
oceania_unnamed <- rrapply(
  renewable_oceania,
  classes = "list",
  condition = function(x, .xname) .xname == "Oceania",
  f = unname
)

na_drop_melt2 <- rrapply(
  oceania_unnamed,
  f = function(x) x,
  classes = "numeric",
  how = "melt"
)
head(na_drop_melt2)
@
The dataset \code{pokedex} also included in the \texttt{rrapply}-package is a nested list containing 151 sublists with property values of each Pok\'emon in Pok\'emon GO. The \code{pokedex} dataset differs from \code{renewable\_energy\_by\_country} in the sense that \code{pokedex} contains 151 \emph{repeated} observations represented as individual sublists. \code{how = "bind"} unnests nested lists containing repeated observations and returns a wide data.frame, where each individual observation (i.e. repeated sublist) is expanded as a single row in the data.frame. Identical names between different sublists are aligned as individual data.frame columns:
<<>>=
## Nested list of Pokemon properties in Pokemon GO
data("pokedex")

str(pokedex, list.len = 3)

## Unnest list as a wide data.frame
pokedex_wide <- rrapply(
  pokedex,
  how = "bind"
)

head(pokedex_wide[, c(1:3, 5:7)])
@
\begin{remark}
Repeated observations (i.e. sublists) are recognized as several sublists on the same nested level that are either unnamed or have the same name for each sublist. Conceptually, each repeated observation is unnested into a flat list and appended as an individual row to the wide data.frame.
\end{remark}

\subsection{Condition function}

Base \code{rapply()} allows to apply \code{f} to leaf elements of certain types or classes via the \code{classes} argument, which might not always provide sufficient control to partition leaf elements. For this purpose, \code{rrapply()} includes an additional \code{condition} argument, which accepts any principal argument function to use as a condition or predicate to select leaf elements to which \code{f} is applied. Conceptually, the \code{f} function is applied to all leaf
elements for which the \code{condition} function exactly evaluates to \code{TRUE} similar to the \code{isTRUE} function. If the \code{condition} function is missing, \code{f} is applied to all leaf elements. In combination with \code{how = "prune"}, the \code{condition} function provides a flexible way to select and filter elements from the nested list.\\[3mm]
Using the \code{condition} argument, we can update the above function call to better reflect our purpose:
<<>>=
na_drop_oceania_list2 <- rrapply(
  renewable_oceania, 
  condition = function(x) !is.na(x), 
  f = function(x) x, 
  how = "prune"
)
str(na_drop_oceania_list2, list.len = 3, give.attr = FALSE)
@
\code{rrapply()} allows the \code{f} argument to be missing, in which case no function is applied to the leaf elements. Using the \code{Negate} function, we can rewrite the above expression somewhat more concisely as, 
<<>>=
na_drop_oceania_list3 <- rrapply(
  renewable_oceania, 
  condition = Negate(is.na), 
  how = "prune"
)
str(na_drop_oceania_list3, list.len = 3, give.attr = FALSE)
@
A more interesting example is to consider a \code{condition} that is not also replicable using the \code{classes} argument. For instance, we can filter all countries with a renewable
energy share above 85 percent, or all countries with a renewable energy share
of 0 percent: 
<<>>=
renewable_energy_above_85 <- rrapply(
  renewable_energy_by_country, 
  condition = function(x) x > 85, 
  how = "prune"
)
str(renewable_energy_above_85, give.attr = FALSE)
## passing arguments to condition via ...
renewable_energy_equal_0 <- rrapply(
  renewable_energy_by_country, 
  condition = `==`, 
  e2 = 0, 
  how = "prune"
)
str(renewable_energy_equal_0, give.attr = FALSE)
@
Note that the \code{NA} elements are not returned, as the \code{condition} does not evaluate to \code{TRUE} for \code{NA} values. \\[3mm]
As the \code{condition} function is a generalization of the \code{classes} argument to have more
flexible control of the predicate, it is also possible to use the \code{deflt} argument
together with \code{how = "list"} or \code{how = "unlist"} to set a default value to all
leaf elements for which the \code{condition} does not evaluate to \code{TRUE}:
<<>>=
na_zero_oceania_list2 <- rrapply(
  renewable_oceania, 
  condition = Negate(is.na), 
  deflt = 0, 
  how = "list"
)
str(na_zero_oceania_list2, list.len = 3, give.attr = FALSE)
@
To be consistent with base \code{rapply()}, the \code{deflt} argument can still only be used
together with \code{how = "list"} or \code{how = "unlist"}. With \code{how = "replace"}, we can replace \code{NA} values by zeros using the \code{f} function in the same way as before, 
<<>>=
na_zero_oceania_replace2 <- rrapply(
  renewable_oceania, 
  condition = is.na, 
  f = function(x) 0, 
  how = "replace"
)
str(na_zero_oceania_replace2, list.len = 3, give.attr = FALSE)
@
\subsubsection{Using the \texttt{...} argument}

In base \code{rapply()}, the first argument to \code{f} always evaluates to the content of
the leaf element to which \code{f} is applied. Any further arguments (besides the special arguments \code{.xname}, \code{.xpos}, \code{.xparents} and \code{.xsiblings} discussed below) that are independent of the node content are supplied via the dots \code{...} argument. Since \code{rrapply()} accepts a function in two of its arguments \code{f} and \code{condition}, any further arguments defined via the \code{dots} also need to be defined as function arguments in \emph{both} the \code{f} and \code{condition} function (if existing), even if they are not used in the function itself.\\[3mm]
To illustrate, consider the following example where we replace all \code{NA} elements by a value defined in a separate argument \code{newvalue}:
<<>>=
## this is not ok!
tryCatch({
  rrapply(
    renewable_oceania, 
    condition = is.na, 
    f = function(x, newvalue) newvalue, 
    newvalue = 0, 
    how = "replace"
  )
}, error = function(error) error$message)
## this is ok
na_zero_oceania_replace3 <- rrapply(
  renewable_oceania, 
  condition = function(x, newvalue) is.na(x), 
  f = function(x, newvalue) newvalue, 
  newvalue = 0, 
  how = "replace"
)
str(na_zero_oceania_replace3, list.len = 3, give.attr = FALSE)
@
\subsection{Special arguments \texttt{.xname}, \texttt{.xpos}, \texttt{.xparents} and \texttt{.xsiblings}}

For illustration purposes, let us return all non-missing values in
\code{renewable\_oceania} as a non-nested flattened list:
<<>>=
renewable_oceania_flat <- rrapply(
  renewable_oceania, 
  condition = Negate(is.na), 
  how = "flatten"
)
head(renewable_oceania_flat)
@
Suppose that we wish to apply a function to each list element that relies on the name of the node. A possible way to achieve this using \code{mapply()} would be:
<<>>=
renewable_oceania_flat_text <- mapply(
  FUN = function(name, value) sprintf("Renewable energy in %s: %.2f%%", name, value),
  name = names(renewable_oceania_flat),
  value = renewable_oceania_flat,
  SIMPLIFY = FALSE
)
str(renewable_oceania_flat_text, list.len = 6)
@
\begin{remark}
Note that the \texttt{purrr}-package also contains the convenience function \code{imap} for exactly this purpose.
\end{remark}
In base \code{rapply()}, the \code{f} function only has access to the content of a leaf
element through its principal argument, but there is no convenient way to access the list element its name or location from
inside the \code{f} function. This makes \code{rapply()} impractical if we want to apply
a function \code{f} that relies on e.g. the name of the leaf element as in the above example.\\[3mm]
To address this and similar issues, \code{rrapply()} allows the use of a number of special arguments 
\code{.xname}, \code{.xpos}, \code{.xparents} and \code{.xsiblings} in addition to the principal argument in the \code{f} and \code{condition} functions. The \code{.xname} argument evaluates to the name of the leaf element. The \code{.xpos} argument evaluates to the position of the leaf element in the nested list structured as an integer vector. For instance, if \code{x = list(list("y", "z"))}, then an \code{.xpos} location of \code{c(1, 2)} corresponds to the leaf element \code{x[[1]][[2]]} or equivalently
\code{x[[c(1, 2)]]}. The \code{.xparents} argument evaluates to a vector of all parent node names in the path to the list element. The names in \code{.xparents} are ordered by increasing depth in the nested list, with the final element always matching to \code{.xname}. The \code{.xsiblings} argument evaluates to the parent list containing the current list element and all of its direct siblings. \\[3mm]
The arguments \code{.xname}, \code{.xpos}, \code{.xparents} and \code{.xsiblings} need to be defined explicitly as function arguments in \code{f} and \code{condition} whenever they are used. Note that the principal function arguments of \code{f} and \code{condition} always evaluate to the content of the list element, for this reason the arguments \code{.xname}, \code{.xpos},  \code{.xparents} and \code{.xsiblings} should always be defined in addition to a principal function argument. It is allowed to use any combination of special arguments in \code{f} and \code{condition}.\\[3mm]
Using the \code{.xname} argument, we can reproduce the \code{mapply()} example above also from a nested list as input:
<<>>=
renewable_oceania_flat_text <- rrapply(
  renewable_oceania,
  f = function(x, .xname) sprintf("Renewable energy in %s: %.2f%%", .xname, x),
  condition = Negate(is.na),
  how = "flatten"
)
head(renewable_oceania_flat_text, n = 6)
@
Since these special arguments can also be used in the \code{condition} 
function, it is now possible to filter elements or apply a function only to a part of the list based on the node names or their positions.\\[3mm]
As an example, let us extract the renewable energy shares of Belgium, the Netherlands and
Luxembourg while preserving the nested structure of the filtered elements:
<<>>=
renewable_benelux <- rrapply(
  renewable_energy_by_country, 
  condition = function(x, .xname) .xname %in% c("Belgium", "Netherlands", "Luxembourg"), 
  how = "prune"
)
str(renewable_benelux, give.attr = FALSE)
@
Knowing that Europe is located under the node \code{renewable\_energy\_by\_country[[c(1, 5)]]}, we can filter all European countries
with a renewable energy share above 50 percent by using the \code{.xpos} argument,
<<>>=
renewable_europe_above_50 <- rrapply(
  renewable_energy_by_country,
  condition = function(x, .xpos) identical(head(.xpos, 2), c(1L, 5L)) & x > 50,
  how = "prune"
)
str(renewable_europe_above_50, give.attr = FALSE)
@
This can be done more conveniently using the \code{.xparents} argument, as this does not require us to look up the location of Europe in the list beforehand,
<<>>=
renewable_europe_above_50 <- rrapply(
  renewable_energy_by_country,
  condition = function(x, .xparents) "Europe" %in% .xparents & x > 50,
  how = "prune"
)
@
Using the \code{.xpos} argument, we could also look up the location of a particular country in the nested
list,
<<>>=
(xpos_sweden <- rrapply(
  renewable_energy_by_country,
  condition = function(x, .xname) identical(.xname, "Sweden"),
  f = function(x, .xpos) .xpos,
  how = "flatten"
))

## sanity check
renewable_energy_by_country[[xpos_sweden$Sweden]]
@
We could even use the \code{.xpos} argument to determine the maximum depth of the list
or the length of the longest sublist,
<<>>=
## maximum depth
depth_all <- rrapply(
  renewable_energy_by_country, 
  f = function(x, .xpos) length(.xpos), 
  how = "unlist"
)
max(depth_all) 
## longest sublist length
sublist_count <- rrapply(
  renewable_energy_by_country, 
  f = function(x, .xpos) max(.xpos), 
  how = "unlist"
)
max(sublist_count)
@
The \code{.xsiblings} argument allows us to evaluate the sibling nodes of the current list element under evaluation. An example usage is to look up all direct neighbors of a particular country in the nested list:
<<>>=
## look up sibling countries of Sweden in list
siblings_sweden <- rrapply(
  renewable_energy_by_country,
  condition = function(x, .xsiblings) "Sweden" %in% names(.xsiblings),
  how = "flatten"
)
head(siblings_sweden, n = 6)
@
When unnesting nested lists containing repeated observations with \code{how = "bind"}, the \code{.xname}, \code{.xpos} or \code{.xparents} arguments can be useful to parse only a particular set of data.frame columns:
<<>>=
## parse only Pokemon number, name and type columns 
pokedex_small <- rrapply(
  pokedex,
  condition = function(x, .xpos, .xname) {
    length(.xpos) < 4 & .xname %in% c("num", "name", "type")
    },
  how = "bind"
)

head(pokedex_small)
@

\subsection{Avoid recursing into list nodes}

By default, if \code{classes = "ANY"} both base \code{rapply()} and \code{rrapply()} recurse into any list-like element. Using \code{classes = "list"} in base \code{rapply()} has no effect as the function descends into any list node before evaluating the \code{classes} argument. In contrast, \code{rrapply()} does detect \code{classes = "list"}, in which case the \code{f} function is applied to list elements that satisfy the \code{condition} function. If the \code{condition} is not satisfied for a list element, \code{rrapply()} will recurse further into the sublist, apply the \code{f} function to the nodes that satisfy \code{condition} and so on. The use of \code{classes = "list"} tells \code{rrapply()} to not descend into list objects by default. For this reason this behavior can only be triggered with the \code{classes} argument and *not* through the use of e.g. \code{condition = is.list}. \\[3mm]
The choice \code{classes = "list"} is useful to calculate summary statistics across nodes or to look up the position of intermediate nodes in a nested list. To illustrate, we can return the mean and standard deviation of the renewable energy share in Europe as follows:
<<>>=
rrapply(
  renewable_energy_by_country,  
  classes = "list",
  condition = function(x, .xname) .xname == "Europe",
  f = function(x) list(
    mean = mean(unlist(x), na.rm = TRUE), 
    sd = sd(unlist(x), na.rm = TRUE)
  ),
  how = "flatten"
)
@
\begin{remark}
Note that the principal \code{x} argument in the \code{f} function now evaluates to a list for the node satisfying \code{condition}. For this reason, we first \code{unlist} the sublist before passing it to \code{mean} and \code{sd}.
\end{remark}
The same result could be obtained by defining a \code{condition} based on the \code{"M49-code"} attribute of the list element. This can be convenient to filter or summarize nodes in nested lists coming from XML (or HTML) files based on their attribute values.
<<>>=
rrapply(
  renewable_energy_by_country,
  classes = "list",
  condition = function(x) attr(x, "M49-code") == "150",
  f = function(x) list(
    mean = mean(unlist(x), na.rm = TRUE), 
    sd = sd(unlist(x), na.rm = TRUE)
  ),
  how = "flatten"
)
@
We can use the \code{.xpos} argument to apply the \code{f} function only at specific locations or depths in the nested list. For instance, we could return the mean renewable energy shares for each continent by observing that the \code{.xpos} vector of each continent has length (i.e. depth) 2:
<<>>=
renewable_continent_summary <- rrapply(
  renewable_energy_by_country,  
  classes = "list",
  condition = function(x, .xpos) length(.xpos) == 2,
  f = function(x) mean(unlist(x), na.rm = TRUE)
)

## Antarctica has a missing value
str(renewable_continent_summary, give.attr = FALSE)
@
If \code{classes = "list"}, the \code{f} function is only applied to the (non-terminal) list nodes. If \code{classes = "ANY"}, the \code{f} function is only applied to any (terminal) non-list node. To apply \code{f} to any terminal \emph{and} non-terminal node of the nested list, we should combine \code{classes = c("list", "ANY")}. This is illustrated by searching both terminal and non-terminal nodes for the country or region with M49-code \code{"155"} (Western Europe):

<<>>=
## Filter country or region by M49-code
rrapply(
  renewable_energy_by_country,
  classes = c("list", "ANY"), 
  condition = function(x) attr(x, "M49-code") == "155",
  f = function(x, .xname) .xname,
  how = "unlist")
@
Another illustration of the use of \code{classes = "list"} is to preprocess selection of list columns before unnesting a nested list with repeated observations into a wide data.frame with \code{how = "bind"}:
<<>>=
## Simplify Pokemon evolution sublists to character vectors 
pokedex_wide2 <- rrapply(
  pokedex,
  classes = c("character", "list"),
  condition = function(x, .xname) .xname %in% c("name", "next_evolution", "prev_evolution"), 
  f = function(x) if(is.list(x)) sapply(x, `[[`, "name") else x,
  how = "bind"
)
    
head(pokedex_wide2, n = 9)
@

\subsection{Recursive list node modification}

If \code{classes = "list"}, \code{rrapply()} applies the \code{f} function to any list element that satisfies the \code{condition} function, but will not recurse further into these list elements. This make it for instance impossible to recursively update the name of each list element in a nested list, as \code{rrapply()} stops recursing after updating the first list layer. For this purpose, set \code{classes = "list"} combined with \code{how = "recurse"}, in which case \code{rrapply()} recurses further into any *updated* list element after application of the \code{f} function (using \code{how = "replace"}). In this context, the \code{condition} argument is interpreted as a passing criterion: as long as the \code{condition} and \code{classes} arguments are satisfied, \code{rrapply()} will try to recurse further into any list-like element. \\[3mm]
Using \code{how = "recurse"}, we can recursively replace all names in the nested list by their M49-codes:
<<>>=
renewable_M49 <- rrapply(
  list(renewable_energy_by_country), 
  classes = "list",
  f = function(x) {
    names(x) <- vapply(x, attr, character(1L), which = "M49-code")
    return(x)
  },
  how = "recurse"
)

str(renewable_M49[[1]], max.level = 3, list.len = 3, give.attr = FALSE)
@
\begin{remark}
Here we passed \code{list(renewable\_energy\_by\_country)} to the call of \code{rrapply()} in order to start application of the \code{f} function at the level of the list \code{renewable\_energy\_by\_country} itself, instead of starting at its list elements. 
\end{remark}

\subsection{Miscellaneous}

\subsubsection{Unmelting data.frames to lists}

The option \code{how = "melt"} returns a melted data.frame similar in format as \code{reshape2::melt()} applied to a nested list. Suppose that we have manipulated the data.frame, by e.g. filtering a number of rows, and wish to convert it back to a nested list in order to write it to e.g. a JSON- or XML-object. For this purpose, \code{rrapply()} includes the option \code{how = "unmelt"}, which performs the inverse operation of \code{how = "melt"}.
No skeleton object is needed in this case (as required in e.g. base R's \code{relist()} function), only a data.frame in the same format as returned by \code{how = "melt"}. To illustrate, we can reconstruct a nested list from the data.frame \code{na\_drop\_melt} produced above as follows,
<<>>=
## melted data.frame
head(na_drop_melt)

na_drop_unmelt <- rrapply(na_drop_melt, how = "unmelt")

str(na_drop_unmelt, list.len = 3, give.attr = FALSE)
@
\begin{remark}
\code{how = "unmelt"} is based on a greedy approach parsing data.frame rows as list elements starting from the top of the data.frame. That is, \code{rrapply()} continues collecting children nodes as long as the parent node name remains unchanged. If, for instance, we wish to create two separate nodes (on the same list layer) with the name \code{"Western Europe"}, these nodes should not be listed after one another in the melted data.frame as \code{rrapply()} will group all children under a single \code{"Western Europe"} list element.
\end{remark}
\begin{remark}
Internally, \code{how = "unmelt"} reconstructs a nested list from the melted data.frame and subsequently applies the same conceptual framework as \code{how = "replace"}. Any other function arguments, such as \code{f} and \code{condition}, should be used in exactly the same way as one would use them with \code{how = "replace"} applied to a nested list object.
\end{remark}
\begin{remark}
\code{how = "unmelt"} does (currently) not restore the attributes of intermediate list nodes and is therefore not an exact inverse of \code{how = "melt"}. Melting an already unmelted nested list will always produce exactly the same results:
<<>>=
na_drop_remelt <- rrapply(na_drop_unmelt, how = "melt")
identical(na_drop_melt, na_drop_remelt)
@
\end{remark}

\subsubsection{Data.frames as lists}

If \code{classes = "ANY"}, \code{rrapply()} recurses into any list-like object equivalent to base \code{rapply()}. Since data.frames are list-like objects, the \code{f} function is applied to the individual columns instead of the data.frame object as a whole. To avoid this behavior, set \code{classes = "data.frame"}, in which case the \code{f} and \code{condition} functions are applied directly to the data.frame object itself and not its columns. Note that with base \code{rapply()} using \code{classes = "data.frame"} has no effect as \code{rapply()} descends into the columns of any data.frame object before evaluating the \code{classes} argument.
<<>>=
## create a list of data.frames
oceania_df <- list(
  Oceania = lapply(
    renewable_oceania[["Oceania"]], 
    FUN = function(x) data.frame(value = unlist(x))
  )
)
## this does not work!
tryCatch({
  rrapply(
    oceania_df,
    f = function(x) subset(x, !is.na(value)), ## filter NA-rows of data.frame
    how = "replace"
  )
}, error = function(error) error$message)
## this does work
rrapply(
  oceania_df,
  classes = "data.frame",
  f = function(x) subset(x, !is.na(value)),
  how = "replace"
)
@

\begin{remark}
Note that the same result can also be obtained using \code{classes = "list"} and checking that the list element under evaluation is a data.frame:
<<eval=FALSE>>=
rrapply(
  oceania_df,
  classes = "list",
  condition = is.data.frame,
  f = function(x) subset(x, !is.na(value)),
  how = "replace"
)
@
\end{remark}

\subsubsection{List attributes}

Base \code{rapply()} may produce different results when using \code{how = "replace"} or \code{how = "list"} when working with list attributes. The former preserves intermediate list attributes whereas the latter does not. To avoid unexpected behavior,
\code{rrapply()} always preserves intermediate list attributes when using \code{how = "replace"}, \code{how = "list"} or \code{how = "prune"}.  Note that if we set \code{how = "flatten"}, \code{how = "melt"} or \code{how = "unlist"} intermediate list attributes cannot be preserved as the result is no longer a nested list.
<<>>=
## how = "list" now preserves all list attributes
na_drop_oceania_list_attr2 <- rrapply(
  renewable_oceania, 
  f = function(x) replace(x, is.na(x), 0), 
  how = "list"
)
str(na_drop_oceania_list_attr2, max.level = 2)
## how = "prune" also preserves list attributes
na_drop_oceania_attr <- rrapply(
  renewable_oceania, 
  condition = Negate(is.na), 
  how = "prune"
)
str(na_drop_oceania_attr, max.level = 2)
@

\subsection{Using \texttt{rrapply()} on data.frames}

The previous sections explained how to avoid recursing into list-like elements using \code{rrapply()}. However, it can also be useful to exploit the property that a data.frame is a list-like object and use base \code{rapply()} to apply a function \code{f} to data.frame columns of certain classes. For instance, it is straightforward to standardize all \code{numeric} columns in the \code{iris}
dataset by their sample mean and standard deviation:
<<>>=
iris_standard <- rapply(iris, f = scale, classes = "numeric", how = "replace")
head(iris_standard)
@
Using the \code{condition} argument in \code{rrapply()}, we obtain more flexible control in selecting the columns to which \code{f} is applied. For instance, it is now straightforward to apply the \code{f} function only to the \code{Sepal} columns using the \code{.xname} argument:
<<>>=
iris_standard_sepal <- rrapply(
  iris,                    
  condition = function(x, .xname) grepl("Sepal", .xname), 
  f = scale
)
head(iris_standard_sepal)
@
Instead of \emph{mutating} columns, we can also \emph{transmute} columns (referencing
to the semantics of the \code{dplyr}-package) keeping only the columns to which
\code{f} is applied by setting \code{how = "prune"}: 
<<>>=
iris_standard_transmute <- rrapply(
  iris, 
  f = scale, 
  classes = "numeric", 
  how = "prune"
)
head(iris_standard_transmute)
@
In order to \emph{summarize} a set of selected columns, use \code{how = "flatten"}
instead of \code{how = "prune"}, as the latter preserves list
attributes --including data.frame dimensions-- which should not be kept. 
<<>>=
## summarize columns with how = "flatten"
iris_standard_summarize <- rrapply(
  iris, 
  f = summary, 
  classes = "numeric", 
  how = "flatten"
)
iris_standard_summarize
@

\section{Using \texttt{rrapply()} on expressions}

Unevaluated R code is parsed or captured as a set of \emph{expressions}, which is a collective term used to refer to any of the following types of objects: scalar constants e.g. \code{TRUE} or \code{1}, symbols e.g. \code{quote(x)}, call objects e.g. \code{quote(x <- 1)}, expression vectors e.g. \code{expression(a <- 1, 2 * b)} or pairlists e.g. \code{formals(seq.default)}. Call objects and expression vectors are hierarchically structured objects, i.e. \emph{abstract syntax trees}, that can be decomposed into symbols and scalar constants as atomic building blocks.\\[3mm]
Call objects and expression vectors generally behave as nested lists, e.g. subsetting a call object works the same as subsetting a nested list. To illustrate, we can retrieve the abstract syntax tree of a call object by recursing through the object in the same way as for a nested list:
<<>>=
## recurse through call as nested list
ast <- function(expr) {
  lapply(expr, function(x) {
    if(is.call(x) || is.expression(x)) {
      ast(x) 
    } else {
      x
    }
  })
}

## decompose call object
str(ast(quote(y <- x <- 1 + TRUE)))
@
Given this information, we might be inclined to assume that base \code{rapply()} also recurses through call objects and expression vectors in the same way as for nested lists, but this is unfortunately not the case. To be precise, \code{rapply()} does accept expression vectors as input for R >= 3.6, but effectively treats them as flat lists of call objects similar to \code{lapply()}. Call objects are not accepted by \code{rapply()} (for any R version) and return an error.
<<>>=
## rapply on an expression vector (ok for R >= 3.6)
tryCatch({
  rapply(expression(y <- x <- 1, f(g(2 * pi))), f = identity)
}, error = function(error) error$message)

## rapply on a call object (not ok!) 
tryCatch({
  rapply(quote(y <- x <- 1), f = identity)
}, error = function(error) error$message)
@
Starting from version 1.2.0 \code{rrapply()} also supports recursion of call objects and expression vectors, which are treated as nested lists based on their internal abstract syntax trees. As such, all functionality described in the previous sections extends directly to call objects and expression vectors. 

\subsection{Structuring the result}

When applying \code{rrapply()} (or base \code{rapply()}) to nested lists the difference between \code{how = "replace"} and \code{how = "list"} is relatively minor. Both choices return a nested list, but only \code{how = "list"} replaces elements not subject to \code{f} by the argument \code{deflt}. For call objects and expression objects, the difference is more important as \code{how = "replace"} always maintains the type of the object after application of \code{rrapply()}, whereas \code{how = "list"} returns the object formatted as a nested list. \\[3mm]
With \code{how = "replace"}, we can for instance directly update the abstract syntax tree of a call object:
<<>>=
call_old <- quote(y <- x <- 1 + TRUE)
str(call_old)

## update call object
call_new <- rrapply(
  call_old, 
  classes = "logical", 
  f = as.numeric, 
  how = "replace"
)
str(call_new)
@
Using \code{how = "list"}, we can update the abstract syntax tree and return it as a nested list:
<<>>=
## update and decompose call object
call_ast <- rrapply(
  call_old, 
  f = function(x) ifelse(is.logical(x), as.numeric(x), x), 
  how = "list"
)
str(call_ast)
@
\begin{remark}
Note that in the second function call we did not use \code{classes = "logical"} to avoid that all list elements that are not of class \code{"logical"} are replaced by the \code{deflt} argument (\code{NULL}).
\end{remark}
The choices \code{how = "prune"}, \code{how = "flatten"} and \code{how = "melt"} return the pruned abstract syntax tree as: a nested list, a flattened list and a melted data.frame respectively. This is identical to application of \code{rrapply()} to the abstract syntax tree formatted as a nested list. To illustrate, let us return all names in the abstract syntax tree of an expression vector that are not part of base R.
<<>>=
## example expression
expr <- expression(y <- x <- 1, f(g(2 * pi)))
## helper function
is_new_name <- function(x) !exists(as.character(x), envir = baseenv())
@
<<>>=
## prune and decompose expression
expr_prune <- rrapply(
  expr, 
  classes = "name", 
  condition = is_new_name, 
  how = "prune"
)
str(expr_prune)

## prune and flatten expression
expr_flatten <- rrapply(
  expr, 
  classes = "name", 
  condition = is_new_name, 
  how = "flatten"
)
str(expr_flatten)

## prune and melt expression
expr_melt <- rrapply(
  expr, 
  classes = "name", 
  condition = is_new_name, 
  f = as.character,
  how = "melt"
)
expr_melt
@

\subsection{Avoid recursing into expression nodes}

If \code{classes = "ANY"} (the default), \code{rrapply()} recurses into any list-like element, which for expression objects are: call objects, expression vectors and pairlists. To avoid recursing into list elements of a nested list, we can use \code{classes = "list"}. Analogously, to avoid recursing into list-like elements of the abstract syntax tree, we should use \code{classes = "language"}, \code{classes = "expression"}, \code{classes = "pairlist"} or any combination thereof. If the \code{condition} and \code{classes} arguments are not satisfied for a given list-like element, \code{rrapply()} will recurse further into the object, apply the \code{f} function to the nodes that satisfy \code{condition} and \code{classes} and so on. Note that this behavior can only be triggered with the \code{classes} argument and \emph{not} through the use of e.g. \code{condition = is.call}. \\[3mm]
To illustrate, we extract all most deeply nested call objects of the example expression above and return these as a flat list:
<<>>=
## extract all terminal call nodes of AST
rrapply(
  expr, 
  classes = "language", 
  condition = function(x) !any(sapply(x, is.call)),
  how = "flatten"
)
@
For additional details and worked out code examples, also visit the \emph{Articles} section at the package website: \href{https://jorischau.github.io/rrapply/}{https://jorischau.github.io/rrapply/}.

\end{document} 

<<echo=false>>=
options(op)
@
