%\VignetteIndexEntry{The Problem of Spatial Autocorrelation}
%\VignetteDepends{}
%\VignetteKeywords{spatial}
%\VignettePackage{spdep}
\documentclass[a4paper,10pt]{article} 
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
%\usepackage[dvips]{graphicx,color}
\usepackage{times}
\usepackage{hyperref}
\usepackage{natbib}
\usepackage[english]{babel}
\usepackage{xspace}

\usepackage{Sweave}
\usepackage{mathptm}
\usepackage{natbib}

\setkeys{Gin}{width=0.95\textwidth}
\newcommand{\strong}[1]{{\normalfont\fontseries{b}\selectfont #1}}
\let\pkg=\strong
\RequirePackage{alltt}
\newenvironment{example}{\begin{alltt}}{\end{alltt}}
\newenvironment{smallexample}{\begin{alltt}\small}{\end{alltt}}
\newcommand{\code}[1]{\texttt{\small #1}}
\def\RR{\textsf{R}\xspace}
\def\SP{\texttt{S-PLUS}\xspace}
\def\SS{\texttt{S}\xspace}
\SweaveOpts{keep.source=FALSE}

\title{``The Problem of Spatial Autocorrelation:'' forty years on} 
\author{Roger Bivand} 

\begin{document} 

\maketitle 

<<echo=FALSE>>= 
owidth <- getOption("width")
options("width"=90)
ow <- getOption("warn")
options("warn"=-1)
.PngNo <- 0
@

<<label=afig,echo=FALSE,eval=FALSE>>= 
.PngNo <- .PngNo + 1; file <- paste("Fig-bitmap-", .PngNo, ".pdf", sep="")
pdf(file=file, width = 6.5, height = 3.5, pointsize = 12, bg = "white")
opar <- par(mar=c(3,3,1,1)+0.1)
@
<<label=afigl,echo=FALSE,eval=FALSE>>= 
.PngNo <- .PngNo + 1; file <- paste("Fig-bitmap-", .PngNo, ".pdf", sep="")
pdf(file=file, width = 6.5, height = 3.5, pointsize = 12, bg = "white")
@
<<label=bfigl,echo=FALSE,eval=FALSE>>= 
.PngNo <- .PngNo + 1; file <- paste("Fig-bitmap-", .PngNo, ".pdf", sep="")
pdf(file=file, width = 6.5, height = 5, pointsize = 12, bg = "white")
@
<<label=bfig,echo=FALSE,eval=FALSE>>= 
.PngNo <- .PngNo + 1; file <- paste("Fig-bitmap-", .PngNo, ".pdf", sep="")
pdf(file=file, width = 6.5, height = 5, pointsize = 12, bg = "white")
opar <- par(mar=c(3,3,1,1)+0.1)
@

<<label=zfig,echo=FALSE,eval=FALSE>>=
par(opar)
dev.null <- dev.off()
cat("\\includegraphics[width=0.95\\textwidth]{", file, "}\n\n", sep="")
@
<<label=zfigl,echo=FALSE,eval=FALSE>>=
dev.null <- dev.off()
cat("\\includegraphics[width=0.95\\textwidth]{", file, "}\n\n", sep="")
@

\section{Introduction}

\citet{cliff+ord:69}, published forty years ago, marked a turning
point in the treatment of spatial autocorrelation in quantitative
geography. It provided the framework needed by any applied researcher
to attempt an implementation for a different system, possibly using a
different programming language. In this spirit, here we examine how
spatial weights have been represented in implementations and may be
reproduced, how the tabulated results in the paper may be reproduced,
and how they may be extended to cover simulation.

One of the major assertions of \citet{cliff+ord:69} is that their
statistic advances the measurement of spatial autocorrelation with
respect to \citet{moran:50} and \citet{geary:54} because a more general
specification of spatial weights could be used. This more general form
has implications both for the preparation of the weights themselves,
and for the calculation of the measures. We will look at spatial weights
first, before moving on to consider the measures presented in the paper
and some of their subsequent developments. Before doing this, we will
put together a data set matching that used in \citet{cliff+ord:69}. They
provide tabulated data for the counties of the Irish Republic, but omit
Dublin from analyses. A shapefile included in this package, kindly made
available by Michael Tiefelsdorf, is used as a starting point:

\begin{footnotesize}
<<echo=TRUE,eval=TRUE,results=hide>>= 
library(spdep)
require(maptools)
fn <- system.file("etc/shapes/eire.shp", package="spdep")[1]
prj <- CRS("+proj=utm +zone=30 +units=km +ellps=mod_airy")
eire <- readShapeSpatial(fn, ID="names", proj4string=prj)
@
<<echo=TRUE,eval=TRUE>>= 
class(eire)
names(eire)
@
\end{footnotesize}

\noindent
and read into a SpatialPolygonsDataFrame --- classes used for handling 
spatial data in \RR are fully described in \citet{bivandetal:08}. To 
this we need to add the data tabulated in the paper in Table 
2,\footnote{cropped scans of tables are available from 
\url{http://spatial.nhh.no/R/etc/CO69-PNGs.zip}.} p. 40, 
here in the form of a text file with added rainfall values from 
Table 9, p. 49:
\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
fn <- system.file("etc/misc/geary_eire.txt", package="spdep")[1]
ge <- read.table(fn, header=TRUE)
names(ge)
@
\end{footnotesize}

Since we assigned the county names as feature identifiers when 
reading the shapefiles, we do the same with the extra data, and 
combine the objects:
\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
row.names(ge) <- as.character(ge$county)
all.equal(row.names(ge), row.names(eire))
eire_ge <- spCbind(eire, ge)
@
\end{footnotesize}

\noindent
Finally, we need to drop the Dublin county omitted in the analyses 
conducted in \citet{cliff+ord:69}:
\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
eire_ge1 <- eire_ge[!(row.names(eire_ge) %in% "Dublin"),]
length(row.names(eire_ge1))
@
\end{footnotesize}

\noindent
To double-check our data, let us calculate the sample Beta coefficients, using the formulae given in the paper for sample moments:
\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
skewness <- function(z) {z <- scale(z, scale=FALSE); ((sum(z^3)/length(z))^2)/((sum(z^2)/length(z))^3)}
kurtosis <- function(z) {z <- scale(z, scale=FALSE); (sum(z^4)/length(z))/((sum(z^2)/length(z))^2)}
@
\end{footnotesize}

\noindent
These differ somewhat from the ways in which skewness and kurtosis are computed in modern statistical software, see for example \citet{joanes+gill:98}. However, for our purposes, they let us reproduce Table 3, p. 42:
\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
print(sapply(as(eire_ge1, "data.frame")[13:24], skewness), digits=3)
print(sapply(as(eire_ge1, "data.frame")[13:24], kurtosis), digits=4)
print(sapply(as(eire_ge1, "data.frame")[c(13,16,18,19)], function(x) skewness(log(x))), digits=3)
print(sapply(as(eire_ge1, "data.frame")[c(13,16,18,19)], function(x) kurtosis(log(x))), digits=4)
@
\end{footnotesize}

\noindent
Using the tabulated value of $23.6$ for the percentage of agricultural holdings above 50 in 1950 in Waterford, the skewness and kurtosis cannot be reproduced, but by comparison with the \code{irishdata} dataset in \pkg{ade4}, it turns out that the value should rather be $26.6$, which yields the tabulated skewness and kurtosis values.

Before going on, the variables considered are presented in Table \ref{vars}.

\begin{table}[htb]
\begin{center}
\caption{Description of variables in the Geary data set.}
\label{vars}
\begin{footnotesize}
  \addvspace{5pt}
\begin{tabular}{ll}
  \hline
variable &  description \\ 
  \hline
pagval2\_10 & Percentage number agricultural holdings in valuation group £2--£10 (1950) \\ 
pagval10\_50 & Percentage number agricultural holdings in valuation group £10--£50 (1950) \\ 
pagval50p & Percentage number agricultural holdings in valuation group above £50 (1950) \\ 
cowspacre & Milch cows per 1000 acres crops and pasture (1952) \\ 
ocattlepacre & Other cattle per 1000 acres crops and pasture (1952) \\ 
pigspacre & Pigs per 1000 acres crops and pasture (1952) \\ 
sheeppacre & Sheep per 1000 acres crops and pasture (1952) \\ 
townvillp & Town and village population as percentage of total (1951) \\ 
carspcap & Private cars registered per 1000 population (1952) \\ 
radiopcap & Radio licences per 1000 population (1952) \\ 
retailpcap & Retail sales £ per person (1951) \\ 
psinglem30\_34 & Single males as percentage of all males aged 30--34 (1951) \\ 
rainfall & Average of rainfall for stations in Ireland, 1916--1950, mm \\ 
   \hline
\end{tabular}
\end{footnotesize}
\end{center}
\end{table}


\section{Spatial weights}

As a basis for comparison, we will first read the unstandardised 
weighting matrix given in Table A1, p. 54, of the paper, reading a 
file corrected for the misprint giving O rather than D as a neighbour 
of V:
\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
fn <- system.file("etc/misc/unstand_sn.txt", package="spdep")[1]
unstand <- read.table(fn, header=TRUE)
summary(unstand)
@
\end{footnotesize}

\noindent
In the file, the counties are represented by their serial letters, so 
ordering and conversion to integer index representation is required to 
reach a representation similar to that of the \SP SpatialStats module 
for spatial neighbours:
\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
class(unstand) <- c("spatial.neighbour", class(unstand))
of <- ordered(unstand$from)
attr(unstand, "region.id") <- levels(of)
unstand$from <- as.integer(of)
unstand$to <- as.integer(ordered(unstand$to))
attr(unstand, "n") <- length(unique(unstand$from))
@
\end{footnotesize}

\noindent
Having done this, we can change its representation to a \code{listw} 
object, assigning an appropriate style (generalised binary) for 
unstandardised values:
\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
lw_unstand <- sn2listw(unstand)
lw_unstand$style <- "B"
lw_unstand
@
\end{footnotesize}

\noindent
Note that the values of S0, S1, and S2 correspond closely with those 
given on page 42 of the paper, $0.84688672$, $0.01869986$ and 
$0.12267319$. The discrepancies appear to be due to rounding in 
the printed table of weights.

The contiguous neighbours represented in this object ought to match 
those found using \code{poly2nb}. However, we see that the reproduced 
contiguities have a smaller link count:
\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
nb <- poly2nb(eire_ge1)
nb
@
\end{footnotesize}

\noindent
The missing link is between Clare and Kerry, perhaps by the 
Tarbert--Killimer ferry, but the counties are not contiguous, as 
Figure \ref{plot_nb} shows:
\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
xx <- diffnb(nb, lw_unstand$neighbours, verbose=TRUE)
@
<<echo=TRUE,eval=FALSE,results=hide>>= 
plot(eire_ge1, border="grey60")
plot(nb, coordinates(eire_ge1), add=TRUE, pch=".", lwd=2)
plot(xx, coordinates(eire_ge1), add=TRUE, pch=".", lwd=2, col=3)
@
\end{footnotesize}

\begin{figure}[htbp]
\begin{center} 
<<results=tex,eval=TRUE,echo=FALSE>>= 
<<afig>>
par(mfrow=c(1,2))
plot(eire_ge1, border="grey40")
title(xlab="25 Irish counties")
text(coordinates(eire_ge1), labels=as.character(eire_ge1$serlet), cex=0.8)
plot(eire_ge1, border="grey60")
title(xlab="Contiguities")
plot(nb, coordinates(eire_ge1), add=TRUE, pch=".", lwd=2)
plot(xx, coordinates(eire_ge1), add=TRUE, pch=".", lwd=2, col=3)
legend("topleft", legend=c("Contiguous", "Ferry"), lwd=2, lty=1, col=c(1,3), bty="n", cex=0.7)
par(mfrow=c(1,1))
<<zfig>>
@ 
\caption{County boundaries and contiguities}
\label{plot_nb}
\end{center}
\end{figure}

An attempt has also been made to reproduce the generalised
weights for 25 Irish counties reported by \citet{cliff+ord:69}, after
Dublin is omitted. Reproducing the inverse distance 
component $d_{ij}^{-1}$ of the
generalised weights $d_{ij}^{-1} \beta_{i(j)}$ is eased by the
statement in \citet[][p. 55]{cliff+ord:73} that the points chosen
to represent the counties were their ``geographic centres,'' so
not very different from the centroids yielded by applying a chosen
computational geometry function. The distance metric is not given,
and may have been in kilometers or miles --- both were tried, but the
results were not sensitive to the difference as it applies equally
across the weights; miles are used here. Computing the 
proportion of shared distance measure
$\beta_{i(j)}$ is harder, because it requires the availability of the
full topology of the input polygons. \citet[][p. 244]{bivandetal:08} show
how to employ the \code{vect2neigh} function (written by Markus Neteler)
in the \RR \pkg{spgrass6} package when using GRASS GIS vector handling
to create a full topology from spaghetti vector data and to extract border
segment lengths; a similar approach also is mentioned there using ArcGIS
coverages for the same purpose. GRASS was used to create the topology,
and next the proportion of shared distance measure was calculated.

\begin{footnotesize}
<<echo=FALSE,eval=TRUE>>= 
load(system.file("etc/misc/raw_grass_borders.RData", package="spdep")[1])
@
<<echo=TRUE,eval=FALSE,results=hide>>= 
library(maptools)
SG <- Sobj_SpatialGrid(eire_ge1)$SG
library(spgrass6)
grass_home <- "/home/rsb/topics/grass/g64/grass-6.4.0svn"
initGRASS(grass_home, home=tempdir(), SG=SG, override=TRUE)
writeVECT6(eire_ge1, "eire", v.in.ogr_flags=c("o", "overwrite"))
res <- vect2neigh("eire", ID="serlet")
@
<<echo=TRUE,eval=TRUE>>= 
grass_borders <- sn2listw(res)
raw_borders <- grass_borders$weights
int_tot <- attr(res, "total") - attr(res, "external")
prop_borders <- lapply(1:length(int_tot), function(i) raw_borders[[i]]/int_tot[i])
dlist <- nbdists(grass_borders$neighbours, coordinates(eire_ge1))
inv_dlist <- lapply(dlist, function(x) 1/(x/1.609344))
combo_km <- lapply(1:length(inv_dlist), function(i) inv_dlist[[i]]*prop_borders[[i]])
combo_km_lw <- nb2listw(grass_borders$neighbours, glist=combo_km, style="B")
summary(combo_km_lw)
@
\end{footnotesize}

\noindent
To compare, we need to remove the Tarbert--Killimer ferry link manually, 
and view the summary of the original weights, as well as a correlation 
coefficient between these and the reconstructed weights. Naturally, 
unless the boundary coordinates used here are identical with those in 
the original analysis, presumably measured by hand, small differences 
will occur.

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
red_lw_unstand <- lw_unstand
Clare <- which(attr(lw_unstand, "region.id") == "C")
Kerry <- which(attr(lw_unstand, "region.id") == "H")
Kerry_in_Clare <- which(lw_unstand$neighbours[[Clare]] == Kerry)
Clare_in_Kerry <- which(lw_unstand$neighbours[[Kerry]] == Clare)
red_lw_unstand$neighbours[[Clare]] <- red_lw_unstand$neighbours[[Clare]][-Kerry_in_Clare]
red_lw_unstand$neighbours[[Kerry]] <- red_lw_unstand$neighbours[[Kerry]][-Clare_in_Kerry]
red_lw_unstand$weights[[Clare]] <- red_lw_unstand$weights[[Clare]][-Kerry_in_Clare]
red_lw_unstand$weights[[Kerry]] <- red_lw_unstand$weights[[Kerry]][-Clare_in_Kerry]
summary(red_lw_unstand)
cor(unlist(red_lw_unstand$weights), unlist(combo_km_lw$weights))
@
\end{footnotesize}

\noindent
Even though the differences in the general weights, for identical 
contiguities, are so small, the consequences for the measure of 
spatial autocorrelation are substantial, Here we use the fifth variable, 
other cattle per 1000 acres crops and pasture (1952), and see that the
reconstructed weights seem to ``reveal'' more autocorrelation than the
original weights.

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
flatten <- function(x, digits=3, statistic="I") {
  res <- c(format(x$estimate, digits=digits),
    format(x$statistic, digits=digits),
    format.pval(x$p.value, digits=digits))
  res <- matrix(res, ncol=length(res))
  colnames(res) <- paste(c("", "E", "V", "SD_", "P_"), "I", sep="")
  rownames(res) <- deparse(substitute(x))
  res
}
`reconstructed weights` <- moran.test(eire_ge1$ocattlepacre, combo_km_lw)
`original weights` <- moran.test(eire_ge1$ocattlepacre, red_lw_unstand)
print(rbind(flatten(`reconstructed weights`), flatten(`original weights`)), quote=FALSE)
@
\end{footnotesize}

\section{Measures of spatial autocorrelation}

Our targets for reproduction are Tables 4 and 5 in
\citet[][pp. 43--44]{cliff+ord:69}, the first containing standard
deviates under normality and randomisation for the original Moran measure
with binary weights, the original Geary measure with binary weights,
the proposed measure with unstandardised general weights, and the
proposed measure with row-standardised general weights. In addition,
four variables were log-transformed on the basis of the skewness and
kurtosis results presented above. We carry out the transformation of
these variables, and generate additional binary and row-standardised
general spatial weights objects --- note that the weights constants for
the row-standardised general weights agree with those given on p. 42 in
the paper, after allowing for small differences due to rounding in the
weights values displayed in the paper (p. 54):

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
eire_ge1$ln_pagval2_10 <- log(eire_ge1$pagval2_10)
eire_ge1$ln_cowspacre <- log(eire_ge1$cowspacre)
eire_ge1$ln_pigspacre <- log(eire_ge1$pigspacre)
eire_ge1$ln_sheeppacre <- log(eire_ge1$sheeppacre)
vars <- c("pagval2_10", "ln_pagval2_10", "pagval10_50", "pagval50p",
 "cowspacre", "ln_cowspacre", "ocattlepacre", "pigspacre",
 "ln_pigspacre", "sheeppacre", "ln_sheeppacre", "townvillp",
 "carspcap", "radiopcap", "retailpcap", "psinglem30_34")
nb_B <- nb2listw(lw_unstand$neighbours, style="B")
nb_B
lw_std <- nb2listw(lw_unstand$neighbours, glist=lw_unstand$weights, style="W")
lw_std
@
\end{footnotesize}

The standard representation of the measures is:

\[
I = \frac{n}{\sum_{i=1}^{n}\sum_{j=1}^{n}w_{ij}}
\frac{\sum_{i=1}^{n}\sum_{j=1}^{n}w_{ij}(x_i-\bar{x})(x_j-\bar{x})}{\sum_{i=1}^{n}(x_i - \bar{x})^2}
\]

\noindent
for Moran's $I$ --- in the paper termed the proposed statistic, and for Geary's $C$:

\[
C = \frac{(n-1)}{2\sum_{i=1}^{n}\sum_{j=1}^{n}w_{ij}}
\frac{\sum_{i=1}^{n}\sum_{j=1}^{n}w_{ij}(x_i-x_j)^2}{\sum_{i=1}^{n}(x_i - \bar{x})^2}
\]

\noindent
where $x_i, i=1, \ldots, n$ are $n$ observations on the numeric
variable of interest, and $w_{ij}$ are the spatial weights. In order to
reproduce the standard deviates given in the paper, it is sufficient to
apply \code{moran.test} to the variables with three different spatial
weights objects, and two different values of the \code{randomisation=}
argument. In addition, \code{geary.test} is applied to a single spatial
weights objects, and two different values of the \code{randomisation=}
argument.

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
system.time({
MoranN <- lapply(vars, function(x) moran.test(eire_ge1[[x]], listw=nb_B, randomisation=FALSE))
MoranR <- lapply(vars, function(x) moran.test(eire_ge1[[x]], listw=nb_B, randomisation=TRUE))
GearyN <- lapply(vars, function(x) geary.test(eire_ge1[[x]], listw=nb_B, randomisation=FALSE))
GearyR <- lapply(vars, function(x) geary.test(eire_ge1[[x]], listw=nb_B, randomisation=TRUE))
Prop_unstdN  <- lapply(vars, function(x) moran.test(eire_ge1[[x]], listw=lw_unstand, randomisation=FALSE))
Prop_unstdR  <- lapply(vars, function(x) moran.test(eire_ge1[[x]], listw=lw_unstand, randomisation=TRUE))
Prop_stdN  <- lapply(vars, function(x) moran.test(eire_ge1[[x]], listw=lw_std, randomisation=FALSE))
Prop_stdR  <- lapply(vars, function(x) moran.test(eire_ge1[[x]], listw=lw_std, randomisation=TRUE))
})
res <- sapply(c("MoranN", "MoranR", "GearyN", "GearyR", "Prop_unstdN", "Prop_unstdR", "Prop_stdN", "Prop_stdR"), function(x) sapply(get(x), "[[", "statistic"))
rownames(res) <- vars
ores <- res[,c(1,2,5:8)]
@
\end{footnotesize}

\noindent
In order to conduct 8 different tests on 16 variables, we use
\code{lapply} on the list of variables in the specified order, then
\code{sapply} on a list of output objects by name to generate a table
in the same row and column order as the original (we save a copy of six
columns for comparison with bootstrap results below):

\begin{footnotesize}
<<echo=FALSE,eval=TRUE>>= 
options("width"=100)
@
<<echo=TRUE,eval=TRUE>>= 
print(formatC(res, format="f", digits=4), quote=FALSE)
@
<<echo=FALSE,eval=TRUE>>= 
options("width"=90)
@
\end{footnotesize}

The values of the standard deviates agree with those in Table 4 in the
original paper, with the exception of those for the proposed statistic
with standardised weights under normality for all untransformed
variables. We can see what has happened by substituting the weights
constants for the standardised weights with those for unstandardised
weights:

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
wc_unstd <- spweights.constants(lw_unstand)
wrong_N_sqVI <- sqrt((wc_unstd$nn*wc_unstd$S1 - wc_unstd$n*wc_unstd$S2 + 3*wc_unstd$S0*wc_unstd$S0)/((wc_unstd$nn-1)*wc_unstd$S0*wc_unstd$S0)-((-1/(wc_unstd$n-1))^2))
raw_data <- grep("^ln_", vars, invert=TRUE)
I <- sapply(Prop_stdN, function(x) x$estimate[1])[raw_data]
EI <- sapply(Prop_stdN, function(x) x$estimate[2])[raw_data]
res <- (I - EI)/wrong_N_sqVI
names(res) <- vars[raw_data]
print(formatC(res, format="f", digits=4), quote=FALSE)
@
\end{footnotesize}

Next, let us look at Table 5 in the original paper. Here we only tabulate
the values of the measures themselves, and, since the expectation is
constant for each measure, the square root of the variance of the measure
under randomisation --- extracting values calculated above:

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
res <- lapply(c("MoranR", "GearyR", "Prop_unstdR", "Prop_stdR"), function(x) sapply(get(x), function(y) c(y$estimate[1], sqrt(y$estimate[3]))))
res <- t(do.call("rbind", res))
colnames(res) <- c("I", "sigma_I", "C", "sigma_C", "unstd_r", "sigma_r", "std_r", "sigma_r")
rownames(res) <- vars
print(formatC(res, format="f", digits=4), quote=FALSE)
@
\end{footnotesize}

The values are as follows, and match the original with the exception of
those for the initial version of Moran's $I$ in the first two columns. If
we write a function implementing equations 3 and 4:

\[
I = \frac{\sum_{i=1}^{n}\sum_{j=i+1}^{n}w_{ij}(x_i-\bar{x})(x_j-\bar{x})}{\sum_{i=1}^{n}(x_i - \bar{x})^2}
\]

where crucially the inner summation is over $i+1 \ldots n$, not $1 \ldots
n$, we can reproduce the values of the measure shown in the original
Table 5:

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
oMoranf <- function(x, nb) {
  z <- scale(x, scale=FALSE)
  n <- length(z)
  glist <- lapply(1:n, function(i) {ii <- nb[[i]]; ifelse(ii > i, 1, 0)})
  lw <- nb2listw(nb, glist=glist, style="B")
  wz <- lag(lw, z)
  I <- (sum(z*wz)/sum(z*z))
  I
}
res <- sapply(vars, function(x) oMoranf(eire_ge1[[x]], nb=lw_unstand$neighbours))
print(formatC(res, format="f", digits=4), quote=FALSE)
@
\end{footnotesize}

\noindent
The variance term given in equation 7 in the original paper is for the
case of normality, not randomisation; the reference on p. 28 to equation
38 on p. 26 does not permit the reproduction of the values in the second
column of Table 5. The variance equation given as equation 1.35 by
\citet[][p. 9]{cliff+ord:73} does not do so either, so for the time being
it is not possible to say how the tabulated values were computed. Note
that since the standard deviances are reproduced correctly, and can
be reproduced from the second column values using the measure and its
expectance, it is just a matter of establishing which formula was used,
but this has so far not proved possible.


\section{Simulating measures of spatial autocorrelation}

\citet{cliff+ord:69} do not conduct simulation experiments, although their
sequels do, notably \citet{cliff+ord:73}, among many others. Simulation
studies are necessarily more demanding computationally, especially
if spatially autocorrelated variables are to be created, as in
\citet[][pp. 146--153]{cliff+ord:73}. In the same book, they also
report the use of permutation tests, also known as Monte Carlo or Hope
hypothesis testing procedures \citep[][pp. 50--52]{cliff+ord:73}. These
procedures provided a way to examine the distribution of the statistic of
interest by exchanging at random the observed values between observations,
and then comparing the simulated distribution under the null hypothesis
of no spatial patterning with the observed value of the statistic in question.

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
MoranI.boot <- function(var, i, ...) {
  var <- var[i]
  return(moran(x=var, ...)$I)
}
Nsim <- function(d, mle) {
  n <- length(d)
  rnorm(n, mle$mean, mle$sd)
}
f_bperm <- function(x, nsim, listw) {
  boot(x, statistic=MoranI.boot, R=nsim, sim="permutation", listw=listw,
    n=length(x), S0=Szero(listw))
}
f_bpara <- function(x, nsim, listw) {
  boot(x, statistic=MoranI.boot, R=nsim, sim="parametric", ran.gen=Nsim,
    mle=list(mean=mean(x), sd=sd(x)), listw=listw, n=length(x),
    S0=Szero(listw))
}
nsim <- 4999
set.seed(1234)
@
\end{footnotesize}

First let us define a function \code{MoranI.boot} just to return the value
of Moran's $I$ for variable \code{var} and permutation index \code{i},
and a function \code{Nsim} to generate random samples from the variable of
interest assuming Normality. To make it easier to process the variables
in turn, we encapsulate calls to \code{boot} in wrapper functions
\code{f\_bperm} and \code{f\_bpara}. Running \Sexpr{format(nsim)}
simulations for each of \Sexpr{format(length(vars))} for three different
weights specifications and both parametric and permutation bootstrap
takes quite a lot of time.

\begin{footnotesize}
<<echo=TRUE,eval=FALSE>>= 
system.time({
MoranNb <- lapply(vars, function(x) f_bpara(x=eire_ge1[[x]], nsim=nsim, listw=nb_B))
MoranRb <- lapply(vars, function(x) f_bperm(x=eire_ge1[[x]], nsim=nsim, listw=nb_B))
Prop_unstdNb  <- lapply(vars, function(x) f_bpara(x=eire_ge1[[x]], nsim=nsim, listw=lw_unstand))
Prop_unstdRb  <- lapply(vars, function(x) f_bperm(x=eire_ge1[[x]], nsim=nsim, listw=lw_unstand))
Prop_stdNb  <- lapply(vars, function(x) f_bpara(x=eire_ge1[[x]], nsim=nsim, listw=lw_std))
Prop_stdRb  <- lapply(vars, function(x) f_bperm(x=eire_ge1[[x]], nsim=nsim, listw=lw_std))
})
@
<<echo=FALSE,eval=FALSE>>= 
zzz <- system.time({
MoranNb <- lapply(vars, function(x) f_bpara(x=eire_ge1[[x]], nsim=nsim, listw=nb_B))
MoranRb <- lapply(vars, function(x) f_bperm(x=eire_ge1[[x]], nsim=nsim, listw=nb_B))
Prop_unstdNb  <- lapply(vars, function(x) f_bpara(x=eire_ge1[[x]], nsim=nsim, listw=lw_unstand))
Prop_unstdRb  <- lapply(vars, function(x) f_bperm(x=eire_ge1[[x]], nsim=nsim, listw=lw_unstand))
Prop_stdNb  <- lapply(vars, function(x) f_bpara(x=eire_ge1[[x]], nsim=nsim, listw=lw_std))
Prop_stdRb  <- lapply(vars, function(x) f_bperm(x=eire_ge1[[x]], nsim=nsim, listw=lw_std))
})
res <- lapply(c("MoranNb", "MoranRb", "Prop_unstdNb", "Prop_unstdRb", "Prop_stdNb", "Prop_stdRb"), function(x) sapply(get(x), function(y) (y$t0 - mean(y$t))/sd(y$t)))
res <- t(do.call("rbind", res))
colnames(res) <- c("MoranNb", "MoranRb", "Prop_unstdNb", "Prop_unstdRb", "Prop_stdNb", "Prop_stdRb")
rownames(res) <- vars
save(zzz, res, file="backstore/boot_res.RData")
@
<<echo=FALSE,eval=TRUE>>=
bsfn <- system.file("doc/backstore/boot_res.RData", package="spdep")
load(bsfn)
zzz
@
<<echo=TRUE,eval=FALSE>>= 
res <- lapply(c("MoranNb", "MoranRb", "Prop_unstdNb", "Prop_unstdRb", "Prop_stdNb", "Prop_stdRb"), function(x) sapply(get(x), function(y) (y$t0 - mean(y$t))/sd(y$t)))
res <- t(do.call("rbind", res))
colnames(res) <- c("MoranNb", "MoranRb", "Prop_unstdNb", "Prop_unstdRb", "Prop_stdNb", "Prop_stdRb")
rownames(res) <- vars
@
\end{footnotesize}

We collate the results to compare with the analytical standard
deviates under Normality and randomisation, and see that in fact the
differences are not at all large, as expressed by the median absolute
difference between the tables. We can also see that inferences based on
a one-sided $\alpha=0.05$ cut-off are the same for the analytical and
bootstrap approaches. This indicates that we can, in general, rely on
the analytical standard deviates, and that bootstrap methods will not
help if assumptions underlying the measures are not met.

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
print(formatC(res, format="f", digits=4), quote=FALSE)
oores <- ores - res
apply(oores, 2, mad)
alpha_0.05 <- qnorm(0.05, lower.tail=FALSE)
all((res >= alpha_0.05) == (ores >= alpha_0.05))
@
\end{footnotesize}

These assumptions affect the shape of the distribution of the measure in
its tails; one possibility is to use a Saddlepoint approximation to find
an equivalent to the analytical or bootstrap-based standard deviate for
inference \citep{tiefelsdorf:02}. The Saddlepoint approximation requires
the eigenvalues of the weights matrix and iterative root-finding for
global Moran's $I$, while for local Moran's $I_i$, analytical forms are
known. Even with this computational burden, the Saddlepoint approximation
for global Moran's $I$ runs quite quickly. First we need to fit null
linear models (only including an intercept) to the variables, then apply
\code{lm.morantest.sad} to the fitted model objects:

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
lm_objs <- lapply(vars, function(x) lm(formula(paste(x, "~1")), data=eire_ge1))
system.time({
MoranSad <- lapply(lm_objs, function(x) lm.morantest.sad(x, listw=nb_B))
Prop_unstdSad  <- lapply(lm_objs, function(x) lm.morantest.sad(x, listw=lw_unstand))
Prop_stdSad  <- lapply(lm_objs, function(x) lm.morantest.sad(x, listw=lw_std))
})
res <- sapply(c("MoranSad", "Prop_unstdSad", "Prop_stdSad"), function(x) sapply(get(x), "[[", "statistic"))
rownames(res) <- vars
@
\end{footnotesize}

Although the analytical standard deviates (under Normality) are larger
than those reached using the Saddlepoint approximation when measured
by median absolute deviation, the differences do not lead to different
inferences at this chosen cut-off. This reflects the fact that the shape
of the distribution is very sensitive to small $n$, but for moderate
$n$ and global Moran's $I$, the effects are seen only further out in
the tails. The consequences for local Moran's $I_i$ are much stronger,
because the clique of neighbours of each observation is typically very
small. It is perhaps of interest that the differences are much smaller
for the case of general weights than for unstandardised binary weights.

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
print(formatC(res, format="f", digits=4), quote=FALSE)
oores <- res - ores[,c(1,3,5)]
apply(oores, 2, mad)
all((res >= alpha_0.05) == (ores[,c(1,3,5)] >= alpha_0.05))
@
\end{footnotesize}

In addition we could choose to use the exact distribution of Moran's $I$,
as described by \citet{tiefelsdorf:00}; its implementation is covered in
\citet{bivandetal:09}. The global case also needs the eigenvalues of the
weights matrix, and the solution of a numerical integration function,
but for these cases, the timings are quite acceptable.

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
system.time({ 
MoranEx <- lapply(lm_objs, function(x) lm.morantest.exact(x, listw=nb_B))
Prop_unstdEx  <- lapply(lm_objs, function(x) lm.morantest.exact(x, listw=lw_unstand))
Prop_stdEx  <- lapply(lm_objs, function(x) lm.morantest.exact(x, listw=lw_std))
})
res <- sapply(c("MoranEx", "Prop_unstdEx", "Prop_stdEx"), function(x) sapply(get(x), "[[", "statistic"))
rownames(res) <- vars
@
\end{footnotesize}

The output is comparable with that of the Saddlepoint approximation,
and the inferences drawn here are the same for the chosen cut-off as
for the analytical standard deviates calculated under Normality.

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
print(formatC(res, format="f", digits=4), quote=FALSE)
oores <- res - ores[,c(1,3,5)]
apply(oores, 2, mad)
all((res >= alpha_0.05) == (ores[,c(1,3,5)] >= alpha_0.05))
@
\end{footnotesize}

\citet{lietal:07}  take up the challenge in \citet[][p. 31]{cliff+ord:69},
to try to give the statistic a bounded fixed range. Their APLE measure is
intended to approximate the spatial dependence parameter of a simultaneous
autoregressive model better than Moran's $I$, and re-scales the measure
by a function of the eigenvalues of the spatial weights matrix. APLE 
requires the use of row standardised weights.

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
vars_scaled <- lapply(vars, function(x) scale(eire_ge1[[x]], scale=FALSE))
nb_W <- nb2listw(lw_unstand$neighbours, style="W")
pre <- spdep:::preAple(0, listw=nb_W)
MoranAPLE <- sapply(vars_scaled, function(x) spdep:::inAple(x, pre))
pre <- spdep:::preAple(0, listw=lw_std, override_similarity_check=TRUE)
Prop_stdAPLE <- sapply(vars_scaled, function(x) spdep:::inAple(x, pre))
res <- cbind(MoranAPLE, Prop_stdAPLE)
colnames(res) <- c("APLE W", "APLE Gstd")
rownames(res) <- vars
@
\end{footnotesize}

In order to save time, we use the two internal functions
\code{spdep:::preAple} and \code{spdep:::inAple}, since for each
definition of spatial weights, the same eigenvalue calculations need to be
made. The notation using the \code{:::} operator says that the function
with named after the operator is to be found in the namespace of the
package named before the operator. The APLE values repeat the pattern that
we have already seen --- for some variables, the measured autocorrelation
is very similar irrespective of spatial weights definition, while for
others, the change in the definition from binary to general does make
a difference.

\begin{footnotesize}
<<echo=TRUE,eval=TRUE>>= 
print(formatC(res, format="f", digits=4), quote=FALSE)
@
\end{footnotesize}

\begin{figure}[htbp]
\begin{center} 
<<results=tex,eval=TRUE,echo=FALSE>>= 
<<afig>>
pal <- grey.colors(9, 1, 0.5, 2.2)
oopar <- par(mfrow=c(1,3), mar=c(1,1,3,1)+0.1)
z <- t(listw2mat(nb_B))
brks <- c(0,0.1,1)
image(1:25, 1:25, z[,ncol(z):1], breaks=brks, col=pal[c(1,9)],
 main="Binary", axes=FALSE)
box()
z <- t(listw2mat(lw_unstand))
brks <- c(0,quantile(c(z)[c(z) > 0], seq(0,1,1/8)))
image(1:25, 1:25, z[,ncol(z):1], breaks=brks, col=pal, main="General", axes=FALSE)
box()
z <- t(listw2mat(lw_std))
brks <- c(0,quantile(c(z)[c(z) > 0], seq(0,1,1/8)))
image(1:25, 1:25, z[,ncol(z):1], breaks=brks, col=pal,
 main="Std. general", axes=FALSE)
box()
par(oopar)
<<zfig>>
@ 
\caption{Three contrasted spatial weights definitions.}
\label{plot_wts}
\end{center}
\end{figure}

\begin{figure}[htbp]
\begin{center} 
<<results=tex,eval=TRUE,echo=FALSE>>= 
<<afigl>>
eire_ge1$nb_B <- sapply(nb_B$weights, sum)
eire_ge1$lw_unstand <- sapply(lw_unstand$weights, sum)
library(lattice)
trellis.par.set(sp.theme())
p1 <- spplot(eire_ge1, c("nb_B"), main="Binary")
p2 <- spplot(eire_ge1, c("lw_unstand"), main="General")
print(p1, split=c(1,1,2,1), more=TRUE)
print(p2, split=c(2,1,2,1), more=FALSE)
<<zfigl>>
@ 
\caption{Sums of weights by county for two contrasted spatial weights definitions --- for row standardisation, all counties sum to unity.}
\label{plot_map}
\end{center}
\end{figure}

\section{Odds and ends $\ldots$}

The differences found in the case of a few variables in inference using the original binary weights, and the general weights proposed by \citet{cliff+ord:69} are necessarily related to the the weights thenselves. Figures \ref{plot_wts} and \ref{plot_map} show the values of the weights in sparse matrix form, and the sums of weights by county where these sums are not identical by design. In the case of binary weights, the matrix entries are equal, but the sums up-weight counties with many neighbours. 

General weights up-weight counties that are close to each other, have more neighbours, and share larger boundary proportions (an asymmetric relationship). There is a further impact of using boundary proportions, in that the boundary between the county and the exterior is subtracted, thus boosting the weights between edge counties and their neighbours, even if there are few of them. Standardised general weights up-weight further up-weight counties with few neighbours, chiefly those on the edges of the study area.

With a small data set, here with $n=25$, it is very possible that edge and other configuration effects are relatively strong, and may impact inference in different ways. The issue of egde effects has not really been satisfactorily resolved, and should be kept in mind in analyses of data sets of this size and shape.



<<echo=FALSE>>=
options("width"=owidth)
options("warn"=ow)
@

\begin{thebibliography}{}

\bibitem[\protect\citeauthoryear{Bivand, Pebesma, and G{\'o}mez-Rubio}{Bivand, Pebesma, and G{\'o}mez-Rubio}{2008}]{bivandetal:08}
Bivand, R. S., E. J. Pebesma, and V. G{\'o}mez-Rubio. (2008).
\newblock {\emph{Applied Spatial Data Analysis with {R}.}}
\newblock New York: Springer.

\bibitem[\protect\citeauthoryear{Bivand, M\"{u}ller, and Reder}{Bivand, M\"{u}ller, and Reder}{2009}]{bivandetal:09}
Bivand, R. S., W. M\"{u}ller and M. Reder. (2009).
\newblock {``Power calculations for global and local {M}oran's {$I$}.''}
\newblock {\emph{Computational Statistics and Data Analysis}}
\newblock 53, 2859--2872.

\bibitem[\protect\citeauthoryear{Cliff and Ord}{Cliff and Ord}{1969}]{cliff+ord:69}
Cliff, A. D. and J. K. Ord. (1969).
\newblock {``The problem of Spatial autocorrelation.''}
\newblock {In \emph{London Papers in Regional Science 1, Studies in Regional Science}, 25--55, edited by A. J. Scott, London: Pion}.

\bibitem[\protect\citeauthoryear{Cliff and Ord}{Cliff and Ord}{1973}]{cliff+ord:73}
Cliff, A. D. and J. K. Ord. (1973).
\newblock {\emph{Spatial autocorrelation.}}
\newblock London: Pion.

\bibitem[\protect\citeauthoryear{Geary}{Geary}{1954}]{geary:54}
Geary, R. C. (1954).
\newblock {``The contiguity ratio and statistical mapping.''}
\newblock {\emph{The Incorporated Statistician}}
\newblock 5, 115--145.

\bibitem[\protect\citeauthoryear{Joanes and Gill}{Joanes and Gill}{1998}]{joanes+gill:98}
Joanes D. N. and C. A. Gill (1998).
\newblock {``Comparing measures of sample skewness and kurtosis.''}
\newblock {\emph{The Statistician}}
\newblock 47, 183--189.

\bibitem[\protect\citeauthoryear{Li, Calder, and Cressie}{Li, Calder, and Cressie}{2007}]{lietal:07}
Li, H., C. A. Calder and N. Cressie. (2007).
\newblock {``Beyond {M}oran's $I$: Testing for spatial   dependence based on the spatial autoregressive model.''}
\newblock {\emph{Geographical Analysis}}
\newblock 39, 357--375.

\bibitem[\protect\citeauthoryear{Moran}{Moran}{1950}]{moran:50}
Moran, P. A. P. (1950).
\newblock {``Notes on continuous stochastic phenomena.''}
\newblock {\emph{Biometrika}}
\newblock 37, 17--23.

\bibitem[\protect\citeauthoryear{Tiefelsdorf}{Tiefelsdorf}{2000}]{tiefelsdorf:00}
Tiefelsdorf, M. (2000).
\newblock {\emph{Modelling Spatial Processes, The Identification and Analysis of Spatial Relationships in Regression Residuals by Means of Moran's I.}}
\newblock Springer, Berlin.

\bibitem[\protect\citeauthoryear{Tiefelsdorf}{Tiefelsdorf}{2002}]{tiefelsdorf:02}
Tiefelsdorf, M. (2002).
\newblock {``The Saddlepoint approximation of {M}oran's $I$ and local {M}oran's $I_i$ reference distributions and their numerical evaluation.''}
\newblock {\emph{Geographical Analysis}}
\newblock 34, 187--206.


\end{thebibliography}

\end{document}

