% \VignetteIndexEntry{Penalties in grpreg}
% \VignettePackage{grpreg}
% \VignetteEngine{knitr::knitr}

\documentclass{article}

\usepackage{amsmath}
\usepackage{fullpage}
\usepackage[colorlinks=true, urlcolor=blue, pdfborder={0 0 0}]{hyperref}

\providecommand{\lam}{\lambda}
\providecommand{\gam}{\gamma}
\providecommand{\bb}{\boldsymbol{\beta}}
\providecommand{\bh}{\widehat{\beta}}
\providecommand{\bbh}{\widehat{\boldsymbol{\beta}}}
\providecommand{\abs}[1]{\left\lvert#1\right\rvert}
\providecommand{\norm}[1]{\lVert#1\rVert}
\providecommand{\as}[1]{\begin{align*}#1\end{align*}}
\providecommand{\x}{\mathbf{x}}
\providecommand{\X}{\mathbf{X}}
\providecommand{\y}{\mathbf{y}}
\providecommand{\MCP}{\textrm{MCP}}
\providecommand{\SCAD}{\textrm{SCAD}}

\title{Penalties available in grpreg}
\author{Patrick Breheny}
\date{\today}

<<include=FALSE, purl=FALSE>>=
require(knitr)
opts_chunk$set(background = "gray90")
opts_chunk$set(prompt = TRUE)
opts_chunk$set(comment = NA)
opts_template$set(fig = list(fig.height=5, fig.width=5, out.width='.5\\linewidth', fig.align='center'))
@

\begin{document}
\maketitle

\section{Introduction}

This vignette summarizes the penalties available in grpreg and describes how
specifying various options in the code corresponds to different mathematical
forms of the penalty.

The following notation will be used.  The objective function $Q(\bb)$ being minimized can be decomposed into
\as{Q(\bb|\X,\y) = L(\bb|\X,\y) + P(\bb),}
where $L(\bb|\X,\y)$ denotes the loss function (negative log-likelihood) and $P(\bb)$ denotes the penalty.  In what follows,
\begin{itemize}
  \item $\bb$ denotes the entire vector of regression coefficients
  \item $\bb_j$ denotes the vector of regression coefficients corresponding to the $j$th group
  \item $\beta_{jk}$ denotes $k$th regression coefficient in the $j$th group
  \item $\norm{\bb_j}$ denotes the Euclidean ($L_2$) norm of $\bb_j$
  \item $\norm{\bb_j}_1$ denotes the $L_1$ norm of $\beta_j$ (Manhattan distance)
\end{itemize}

\section{Group selection}

\subsection{Group lasso}

<<eval=FALSE>>=
grpreg(X, y, group, penalty="grLasso")
@

\as{P(\beta) = \lam \sum_j \norm{\bb_j}}

\subsection{Group MCP}
\label{Sec:grMCP}

<<eval=FALSE>>=
grpreg(X, y, group, penalty="grMCP")
@

\as{P(\bb) = \sum_j \MCP_{\lam, \gamma}(\norm{\bb_j}}

where $\MCP_{\lam, \gamma}(\cdot)$ denotes the MCP penalty with regularization parameter $\lam$ and tuning parameter $\gamma$.

\subsection{Group SCAD}

<<eval=FALSE>>=
grpreg(X, y, group, penalty="grSCAD")
@

\as{P(\bb) = \sum_j \SCAD_{\lam, \gamma}(\norm{\bb_j}}

where $\SCAD_{\lam, \gamma}(\cdot)$ denotes the SCAD penalty with regularization parameter $\lam$ and tuning parameter $\gamma$.

If you use any of the above penalties, please cite

\begin{quote}
Breheny P and Huang J (2015). Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors. {\em Statistics and Computing}, {\bf 25}: 173-187. [\href{http://myweb.uiowa.edu/pbreheny/publications/group-computing.pdf}{pdf}].
\end{quote}

The article goes into more mathematical details, discusses issues of standardization in the group sense, and provides references.

\section{Bi-level selection}

\subsection{Group exponential lasso (GEL)}
\label{Sec:gel}

<<eval=FALSE>>=
grpreg(X, y, group, penalty="gel")
@

\as{P(\beta) = \sum_j f_{\lam, \tau}(\norm{\bb_j})}

where $f(\cdot)$ denotes the exponential penalty with regularization parameter $\lam$ and tuning parameter $\tau$:

\as{f_{\lam, \tau}(\theta) &= \frac{\lam^2}{\tau}\left\{1-\exp\left(-\frac{\tau\theta}{\lam}\right)\right\}}

If you use the GEL penalty, please cite

\begin{quote}
Breheny P (2015). The group exponential lasso for bi-level variable selection. {\em Biometrics}, {\bf 71}: 731-740. [\href{http://onlinelibrary.wiley.com/doi/10.1111/biom.12300/pdf}{pdf}]
\end{quote}

\subsection{Composite MCP}

<<eval=FALSE>>=
grpreg(X, y, group, penalty="cMCP")
@

\as{P(\bb) = \sum_j \MCP_{\lam, \gam_1} \left( \sum_k \MCP_{\lam, \gam2} (\abs{\beta_{jk}}) \right)}

where $\textrm{MCP}_{\lam, \gamma}(\cdot)$ denotes the MCP penalty with regularization parameter $\lam$ and tuning parameter $\gamma$.

If you use the composite MCP penalty, please cite either of the following papers:

\begin{quote}
Breheny P and Huang J (2009). Penalized methods for bi-level variable selection. {\em Statistics and Its Interface}, {\bf 2}: 369-380. [\href{http://myweb.uiowa.edu/pbreheny/publications/Breheny2009.pdf}{pdf}]\\
Huang J, Breheny P and Ma S (2012). A Selective Review of Group Selection in High-Dimensional Models. {\em Statistical Science}, {\bf 27}: 481-499. [\href{http://myweb.uiowa.edu/pbreheny/publications/Huang2012.pdf}{pdf}]
\end{quote}

Please note that there is some confusion around the name ``group MCP''.  In the first paper above (2009), the composite MCP penalty was referred to as the ``group MCP'' penalty; the second paper (2012), in reviewing the various kinds of group penalties that had been proposed, recommended changing the name to ``composite MCP'' to avoid confusion with the ``group MCP'' of Section~\ref{Sec:grMCP}.

\subsection{Group bridge}

<<eval=FALSE>>=
gBridge(X, y, group)
@

\as{P(\bb) = \lambda \sum_j K_j^\gamma \norm{\bb_j}_1^\gamma}

where $K_j$ denotes the number of elements in group $j$.

Please note that the group bridge penalty uses a very different algorithm from the other penalties.  Due to the nature of the penalty, model fitting is slower and less stable for group bridge models.  This is, in fact, the main motivation of the GEL penalty of Section~\ref{Sec:gel}: to offer a more tractable alternative to group bridge that has similar estimation properties but is much better behaved from a numerical optimization perspective.

If you use the group bridge penalty, please cite either of the following papers:

\begin{quote}
Huang J, Ma S, Xie H and Zhang C (2009). A group bridge approach for variable selection. {\em Biometrika}, {\bf 96}: 339-355.\\
Breheny P and Huang J (2009). Penalized methods for bi-level variable selection. {\em Statistics and Its Interface}, {\bf 2}: 369-380. [\href{http://myweb.uiowa.edu/pbreheny/publications/Breheny2009.pdf}{pdf}]
\end{quote}

The first paper proposed the method; the second paper proposed the algorithm that is used in the {\tt grpreg} package.

\section{Specifying an additional ridge component}

For all of the penalties in the previous section, {\tt grpreg} allows the specification of an additional ridge ($L_2$) component to the penalty.  This will set $\lam_1 = \alpha\lam$ and $\lam_2=(1-\alpha)\lam$, with the penalty given by

\as{P(\bb) = P_1(\bb|\lam_1) + \frac{\lam_2}{2}\norm{\bb}^2,}

where $P_1$ is any of the penalties from the earlier sections.  So, for example

<<eval=FALSE>>=
grpreg(X, y, group, penalty="grLasso", alpha=0.75)
@

will fit a model with penalty

\as{P(\beta) = 0.75\lam \sum_j \norm{\bb_j} + \frac{0.25\lam}{2}\norm{\bb}^2.}

\end{document}
