% \VignetteIndexEntry{The Bigmemory Project Overview}
% \VignetteDepends{bigmemory}
% \VignettePackage{bigmemory}
% \VignetteEngine{knitr::knitr}
\documentclass[12pt]{article}

\usepackage{graphics}
\usepackage{graphicx}

\usepackage{accents}

% New from euler:
\usepackage{ae}
\usepackage{color}
\usepackage{url}

\topmargin=-0.85in
\textheight=9.5in
\textwidth=6.5in
\oddsidemargin=0in
%-0.25in

%\usepackage{CJK}
%\usepackage{pinyin}
\def\E{\mathord{I\kern-.35em E}}
\def\R{\mathord{I\kern-.35em R}}
\def\P{\mathord{I\kern-.35em P}}
\def\I{\mathord{1\kern-.35em 1}}
\def\wt{\mathord{\widehat{\theta}}}

\newcommand{\proglang}[1]{\textbf{#1}}
\newcommand{\pkg}[1]{\texttt{\textsl{#1}}}
\newcommand{\code}[1]{\texttt{#1}}
\newcommand{\mg}[1]{{\textcolor {magenta} {#1}}}
\newcommand{\gr}[1]{{\textcolor {green} {#1}}}
\newcommand{\bl}[1]{{\textcolor {blue} {#1}}}

\newtheorem{thm}{Theorem}[section]
\newtheorem{myexplore}[thm]{Explore}
\newtheorem{mybackground}[thm]{Background}
\newtheorem{myquestion}[thm]{Question}
\newtheorem{myexample}[thm]{Example}
\newtheorem{mydefinition}[thm]{Definition}
\newtheorem{mytheorem}[thm]{Theorem}

%\pagestyle{myheadings}    % Go for customized headings
%\markboth{notused left title}{John W. Emerson, Department of Statistics, Yale University \copyright 2009}
%\newcommand{\sekshun}[1]                % In 'article' only the page
%        {                               % number appears in the header.
%        \section{#1}                    % I want the section name AND
%        \markboth{#1 \hfill}{#1 \hfill} % the page, so I need a new kind
%        }                               % of '\sekshun' command. 

\begin{document}

\setkeys{Gin}{width=1.0\textwidth} 

<<include=FALSE>>=
library(knitr)
opts_chunk$set(
fig.path='graphics/stat'
)
@


<<setup,include=FALSE,echo=FALSE>>=
options(keep.source = TRUE, width = 75)
@

\begin{center}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

{\Large\bf The Bigmemory Project}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\vspace*{0.5cm}
{\bf Michael J. Kane and John W. Emerson\\
Yale University\\
April 29, 2010}

\vspace*{0.25cm}

\end{center}

%\begin{raggedright}
\parindent=0.5in

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{quotation}
Multi-gigabyte data sets often challenge and frustrate \proglang{R}
users.  \proglang{C/C++} programming can provide efficiencies,
but is cumbersome for interactive data analysis and
lacks the flexibility and power of \proglang{R}'s rich statistical
programming environment.  The package \pkg{bigmemory} and sister
packages \pkg{biganalytics}, \pkg{synchronicity}, \pkg{bigtabulate},
and \pkg{bigalgebra} bridge this gap, implementing massive matrices
and supporting their manipulation and exploration.
The data structures may be allocated to shared memory, allowing separate 
processes on the same computer to share access to a single copy of the
data set.  The data structures may also be file-backed, allowing users
to easily manage and analyze data sets larger than available RAM and
share them across nodes of a cluster.
These features of the Bigmemory Project open the door for powerful and
memory-efficient parallel analyses and data mining of massive data sets,
even on modest hardware.
\end{quotation} 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\vspace*{0.5cm}
\noindent
{\bf Introductory Example: the 2009 JSM Data Expo}
\vspace*{0.5cm}

Consider the complete airline on-time performance data
from the 2009 JSM Data Expo.  The processed data set, \texttt{airline.csv},
is approximately 11 GB (about 120 million rows and 29 columns)
with factors coded as integers (see \url{http://www.bigmemory.org/} for
processing information).
The \texttt{read.big.matrix()} call creates the binary
file-backing \texttt{airline.bin}
associated with the \texttt{big.matrix} object \texttt{x}.
Subsequent \proglang{R} sessions can attach instantly to \texttt{airline.bin}
without incurring the one-time overhead (about 25 minutes)
associated with creating the backing.  
A summary of the entire data set is easily obtained using the new
\texttt{summary()} method.  Note the surprising
presence of negative arrival and departure delays: exploratory data analysis
in action via \pkg{bigmemory}.  The summary
only takes 3-4 minutes to process the 11 GB of data on a laptop with only
4 GB of RAM.
<<read, eval = FALSE>>=
library(bigmemory)
library(biganalytics)
x <- read.big.matrix("airline.csv", type="integer", header=TRUE, 
                      backingfile="airline.bin",
                      descriptorfile="airline.desc",
                      extraCols="Age")
summary(x)

#                          min        max        mean        NA's
#Year                     1987       2008     1998.62           0
#Month                       1         12        6.55           0
#DayofMonth                  1         31       15.72           0
#DayOfWeek                   1          7        3.94           0
#ArrDelay                -1437       2598        7.05     2587529
#DepDelay                -1410       2601        8.17     2302136
#... (other variables omitted here) ...
@


\noindent
{\bf Overview}
\vspace*{0.5cm}

Data frames and matrices in \proglang{R} are easy to use,
with typical manipulations executing quickly on
data sets much smaller than available RAM.  They suit the needs of many
\proglang{R} users and work seamlessly with existing \proglang{R} functions
and packages.  However, problems arise with larger data sets and when
increased memory requirements of parallel programming strain the system.

The Bigmemory Project offers packages
for two purposes.  First, \pkg{bigmemory}, \pkg{biganalytics}, and
\pkg{bigtabulate} have been designed to provide a minimalist,
elegant framework for users to manage and explore large data sets, even
on modest hardware (expensive workstations or clusters are not required).
The interface is designed to mimic \proglang{R}'s familiar \code{matrix}
syntax.  Matthew Keller, Assistant Professor of
Psychology, University of Colorado at Boulder offered the following
testimonial about \pkg{bigmemory}: ``I love that it's intuitive and
doesn't require a lot of learning new ways to code things.''

Second, the packages of the Bigmemory Project provide a foundation for
memory-efficient parallel programming and can serve as building blocks
for developers of new high-performance computing tools in \proglang{R}.
When used in conjunction with a parallel package (such as \pkg{foreach},
\pkg{snow}, \pkg{Rmpi}, or \pkg{multicore}, for example),
even shared-memory parallel-computing becomes
accessible to non-experts.  
The programming interface is stable, and offers the flexibility to support
the development of
algorithms working seamlessly on both \texttt{big.matrix} and traditional
\texttt{matrix} objects.  For examples of this, look first
at the function \texttt{mwhich()}; it offers flexible \texttt{which()}-like
functionality that is computationally efficient and avoids memory overhead.
In addition, all the functions provided by \pkg{bigtabulate} may be used
with \texttt{matrix} and \texttt{big.matrix} objects alike.

\vspace*{0.5cm}
\noindent
{\bf Underneath the Hood of the Bigmemory Project}
\vspace*{0.5cm}

The packages of the Bigmemory Project use the Boost Interprocess
\proglang{C++} library to provide platform-independent support for
massive matrices that may be shared across \proglang{R} processes.
Innovative use of \proglang{C++} accessors supports matrices of
\texttt{double}, \texttt{integer}, \texttt{short}, and \texttt{char},
as well as the development of algorithms working seamlessly on
\texttt{big.matrix} objects or traditional \proglang{R} matrices.

\vspace*{0.5cm}
\noindent
{\bf Example: Airplane Ages and Parallel Processing}
\vspace*{0.5cm}

We would like to approximate the age of each plane at the time of
each flight.  This first requires calculation of an approximate
``birthmonth'' for each plane: the month of the first
appearance in the data set.  Given a matrix
\texttt{y} containing \texttt{Year} and \code{Month} for all flights
of a given plane, \texttt{birthmonth(y)} returns the
month (in months AD) of the earliest flight:
<<birth, eval = FALSE>>=
birthmonth <- function(y) {
   minYear <- min(y[,'Year'], na.rm=TRUE)
   these <- which(y[,'Year']==minYear)
   minMonth <- min(y[these,'Month'], na.rm=TRUE)
   return(12*minYear + minMonth - 1)
}
@
A traditional approach to calculating all the birthmonths might use a \code{for()} loop:
<<run_birth, eval = FALSE>>=
allplanes <- unique(x[,'TailNum'])
planeStart <- rep(0, length(allplanes))
for (i in allplanes) {
  planeStart[i] <- birthmonth( x[mwhich(x, 'TailNum', i, 'eq'),
                              c('Year', 'Month'), drop=FALSE] )
}
@
With about 13,000 flights this takes about 9 hours, even with the relative
fast and memory-efficient use of \texttt{mwhich()}.

A far more efficient alternative is to first obtain a list of row indices
for each plane:
<<bigsplit, eval = FALSE>>=
library(bigtabulate)
planeindices <- bigsplit(x, 'TailNum')
@
Here, the use of the new function \code{bigsplit()} is equivalent to
<<split, eval = FALSE>>=
planeindices <- split(1:nrow(x), x[,'TailNum'])
@
but is faster (16 versus 29 seconds) and more memory efficient (with
peak memory usage of 2 versus 3 GB).
Either way,
\texttt{planeindices[i]} contains all row indices corresponding to flights
with \texttt{TailNum} equal to \texttt{i}.  This requires several hundred MB,
but is computationally more efficient in this problem.  For example,
\texttt{planeindices} may be used with \code{sapply()} in the obvious way,
completing the task in a mere 30 seconds:
<<sapply, eval = FALSE>>=
planeStart <- sapply(planeindices,
                  function(i) birthmonth(x[i, c('Year','Month'),
                                           drop=FALSE]))
@ 

The looping structure \texttt{foreach()} of package \pkg{foreach}
can be a powerful and flexible alternative to \texttt{for()} or
functions like
\texttt{lapply()} and \texttt{sapply()}.  It can also
take advantage of the shared-memory
capability of \pkg{bigmemory}.  Package \pkg{doMC} provides one of several
available ``parallel backends'' for the function \texttt{foreach()}, allowing
the work to be automatically distributed to available processor cores:
<<parallel, eval = FALSE>>=
library(doMC)
registerDoMC(cores=2)
planeStart <- foreach(i=planeindices, .combine=c) %dopar% {
  return(birthmonth(x[i, c('Year','Month'), drop=FALSE]))
}
@
The syntax of a \code{foreach()} loop is slightly different from the
syntax of  a traditional loop, but its benefits are clear:
in this example, it takes only 14 seconds
to calculate the plane birthmonths using two processor cores.\footnote{We
should note that \pkg{doMC} and \pkg{multicore} are particularly well-suited
for this.  When other parallel backends are used, one additional command is
required in the \code{birthmonth()} function: \code{x <- attach.big.matrix(xdesc)}
where \code{xdesc <- describe(x)} would be required just prior to the
\code{foreach()} loop, providing explicit shared-memory access across processes.
In contrast, \code{multicore} automatically operates on shared memory,
avoiding the need for this extra step.}
Both cores share access to the same master copy
of the airline data (with \texttt{Year} and \texttt{Month} cached in RAM);
individual calls to \texttt{birthmonth()} are relatively small in size.
Without the \texttt{registerDoMC()}
initialization, the \code{foreach()} loop would run on a single processor core, much
like \code{sapply()}, but taking about 24 seconds in this problem
with lower memory overhead than \code{sapply()}.

Finally, the plane ages at the time of all flights may be calculated:
<<ages, eval = FALSE>>=
x[,'Age'] <- x[,'Year']*as.integer(12) +
          x[,'Month'] - as.integer(planeStart[x[,'TailNum']])
@
This arithmetic is conducted on \proglang{R} vectors extracted from
the \code{big.matrix}; use of
\code{as.integer()} helps keep the memory consumption under control.

\vspace*{0.5cm}
\noindent
{\bf Concluding Example: a Big Regression}
\vspace*{0.5cm}

In addition to providing basic functions for exploratory data analysis, the
package \pkg{biganalytics} provides a wrapper for Thomas Lumley's
\pkg{biglm} package, supporting massive
linear and generalized linear models.\footnote{Package \pkg{biganalytics}
also provides \code{bigkmeans()}, and other analytics may be added to the
package in the future.}  The following toy example examines
the airline arrival delays as a linear function of the age of the plane
at the time of the flight and the year of the flight.  About 85 million
flights are used (because of missing airplane tailcodes). 
We estimate that use of \proglang{R}'s \texttt{lm()}
function would require more than 10 GB of RAM of memory overhead, while
this example runs in about 3 minutes with only several hundred MB of memory
overhead.
<<biglm, eval = FALSE>>=
blm <- biglm.big.matrix(ArrDelay ~ Age + Year, data=x)
summary(blm)
@
<<biglm_output, eval = FALSE>>=
#Large data regression model: biglm(formula = formula, data = data, ...)
#Sample size = 84216580 
#               Coef    (95%     CI)     SE p
#(Intercept) 91.6149 87.6509 95.5789 1.9820 0
#Age          0.0144  0.0142  0.0146 0.0001 0
#Year        -0.0424 -0.0444 -0.0404 0.0010 0
@
From this, we might conclude that older planes are associated with increased predicted
delays, and predicted delays in recent years are lower.  However, this
exercise is merely for illustrative purposes; a serious study of airline delays would
quickly reject this oversimplification and discover problems with this particular
regression.

\vspace*{0.5cm}
\noindent
{\bf Additional Information and Supporting Material}
\vspace*{0.5cm}

These examples were tested both in Linux 64-bit and Windows 7 Enterprise 64-bit
environments.
Older versions of Windows operating systems (including Vista 64-bit) seem
to suffer from extremely inefficient caching behavior with filebackings and
are not recommended for use with
\pkg{bigmemory}; 32-bit environments will be limited by approximately 2 GB
of addressable memory.

The packages are available via R-Forge and on CRAN as of
late April, 2010; please see
\url{http://www.bigmemory.org/} for more information.
There is a short vignette available in the Documentation area,
as well as presentation slides introducing \pkg{bigmemory}
and providing some benchmarks and shared-memory parallel programming
examples.  Please do not use the older version of \pkg{bigmemory} 
archived on CRAN (versions <= 3.12).

\newpage

\noindent
{\bf Citations}
\vspace*{0.5cm}

\begin{enumerate}
\item The Bigmemory Project, \url{http://www.bigmemory.org/}, the home of \proglang{R} packages
\pkg{bigmemory}, \pkg{biganalytics}, \pkg{bigtabulate}, \pkg{bigalgebra}, and
\pkg{synchronicity}.  Packages available from CRAN or R-Forge.

\item 2009 JSM Data Expo: Airline on-time performance.  \url {http://stat-computing.org/dataexpo/2009/}.

\item Thomas Lumley (2009). \pkg{biglm}: bounded memory linear and generalized
  linear models. \proglang{R} package version 0.7,
  \url{http://CRAN.R-project.org/package=biglm}.

\item \proglang{R} Development Core Team (2009). \proglang{R}: A language and environment for
  statistical computing. \proglang{R} Foundation for Statistical Computing,
  Vienna, Austria. ISBN 3-900051-07-0, \url{http://www.R-project.org}.

\item Luke Tierney, A. J. Rossini, Na Li and H. Sevcikova (). \pkg{snow}: Simple
  Network of Workstations. \proglang{R} package version 0.3-3,
  \url{http://CRAN.R-project.org/package=snow}.

\item Simon Urbanek (2009). \pkg{multicore}: Parallel processing of \proglang{R} code on
  machines with multiple cores or CPUs. \proglang{R} package version 0.1-3,
  \url{http://www.rforge.net/multicore/}.

\item Stephen Weston and REvolution Computing (2009). \pkg{doMC}: Foreach parallel adaptor for the
  \pkg{multicore} package. \proglang{R} package version 1.2.0,
  \url{http://CRAN.R-project.org/package=doMC}.

\item Stephen Weston and REvolution Computing (2009). \pkg{foreach}: Foreach looping
construct for \proglang{R}. \proglang{R} package version 1.3.0,
\url{http://CRAN.R-project.org/package=foreach}.

\item Hao Yu (2010).  \pkg{Rmpi}: Interface (Wrapper) to MPI (Message-Passing Interface).
  \proglang{R} package version 0.5-8, \url{http://www.stats.uwo.ca/faculty/yu/Rmpi}.
\end{enumerate}


%\end{raggedright}

\end{document}
