R: Dirichlet Prior Bayesian Estimators of Entropy, Mutual...

entropy.Dirichlet {entropy} | R Documentation

## Dirichlet Prior Bayesian Estimators of Entropy, Mutual Information and Other Related Quantities

### Description

`freqs.Dirichlet` computes the Bayesian estimates of the bin frequencies using the Dirichlet-multinomial pseudocount model.

`entropy.Dirichlet` estimates the Shannon entropy H of the random variable Y from the corresponding observed counts `y` by plug-in of Bayesian estimates of the bin frequencies using the Dirichlet-multinomial pseudocount model.

`KL.Dirichlet` computes a Bayesian estimate of the Kullback-Leibler (KL) divergence from counts `y1` and `y2`.

`chi2.Dirichlet` computes a Bayesian version of the chi-squared statistic from counts `y1` and `y2`.

`mi.Dirichlet` computes a Bayesian estimate of mutual information of two random variables.

`chi2indep.Dirichlet` computes a Bayesian version of the chi-squared statistic of independence from a table of counts `y2d`.

### Usage

    freqs.Dirichlet(y, a)
    entropy.Dirichlet(y, a, unit=c("log", "log2", "log10"))
    KL.Dirichlet(y1, y2, a1, a2, unit=c("log", "log2", "log10"))
    chi2.Dirichlet(y1, y2, a1, a2, unit=c("log", "log2", "log10"))
    mi.Dirichlet(y2d, a, unit=c("log", "log2", "log10"))
    chi2indep.Dirichlet(y2d, a, unit=c("log", "log2", "log10"))

### Arguments

`y` | vector of counts.
`y1` | vector of counts.
`y2` | vector of counts.
`y2d` | matrix of counts.
`a` | pseudocount per bin.
`a1` | pseudocount per bin for first random variable.
`a2` | pseudocount per bin for second random variable.
`unit` | the unit in which entropy is measured. The default is "nats" (natural units). For computing entropy in "bits" set `unit="log2"`.

### Details

The Dirichlet-multinomial pseudocount entropy estimator is a Bayesian plug-in estimator: in the definition of the Shannon entropy the bin probabilities are replaced by the respective Bayesian estimates of the frequencies, using a model with a Dirichlet prior and a multinomial likelihood.

The parameter `a` is a parameter of the Dirichlet prior, and in effect specifies the pseudocount per bin. Popular choices of `a` are:

- a=0:maximum likelihood estimator (see `entropy.empirical`)

- a=1/2:Jeffreys' prior; Krichevsky-Trovimov (1991) entropy estimator

- a=1:Laplace's prior

- a=1/length(y):Schurmann-Grassberger (1996) entropy estimator

- a=sqrt(sum(y))/length(y):minimax prior

The pseudocount `a` can also be a vector so that for each bin an individual pseudocount is added.

### Value

`freqs.Dirichlet` returns the Bayesian estimates of the frequencies .

`entropy.Dirichlet` returns the Bayesian estimate of the Shannon entropy.

`KL.Dirichlet` returns the Bayesian estimate of the KL divergence.

`chi2.Dirichlet` returns the Bayesian version of the chi-squared statistic.

`mi.Dirichlet` returns the Bayesian estimate of the mutual information.

`chi2indep.Dirichlet` returns the Bayesian version of the chi-squared statistic of independence.

### Author(s)

Korbinian Strimmer (http://strimmerlab.org).

### References

Agresti, A., and D. B. Hitchcock. 2005. Bayesian inference for categorical data analysis. Stat. Methods. Appl. 14:297–330.

Krichevsky, R. E., and V. K. Trofimov. 1981. The performance of universal encoding. IEEE Trans. Inf. Theory 27: 199-207.

Schurmann, T., and P. Grassberger. 1996. Entropy estimation of symbol sequences. Chaos 6:41-427.

### See Also

`entropy`, `entropy.shrink`, `entropy.empirical`, `entropy.plugin`, `mi.plugin`, `KL.plugin`, `discretize`.

### Examples

    # load entropy library
    library("entropy")

    # a single variable

    # observed counts for each bin
    y = c(4, 2, 3, 0, 2, 4, 0, 0, 2, 1, 1)

    # Dirichlet estimate of frequencies with a=1/2
    freqs.Dirichlet(y, a=1/2)

    # Dirichlet estimate of entropy with a=0
    entropy.Dirichlet(y, a=0)

    # identical to empirical estimate
    entropy.empirical(y)

    # Dirichlet estimate with a=1/2 (Jeffreys' prior)
    entropy.Dirichlet(y, a=1/2)

    # Dirichlet estimate with a=1 (Laplace prior)
    entropy.Dirichlet(y, a=1)

    # Dirichlet estimate with a=1/length(y)
    entropy.Dirichlet(y, a=1/length(y))

    # Dirichlet estimate with a=sqrt(sum(y))/length(y)
    entropy.Dirichlet(y, a=sqrt(sum(y))/length(y))

    # example with two variables

    # observed counts for two random variables
    y1 = c(4, 2, 3, 1, 10, 4)
    y2 = c(2, 3, 7, 1, 4, 3)

    # Bayesian estimate of Kullback-Leibler divergence (a=1/6)
    KL.Dirichlet(y1, y2, a1=1/6, a2=1/6)

    # half of the corresponding chi-squared statistic
    0.5*chi2.Dirichlet(y1, y2, a1=1/6, a2=1/6)

    ## joint distribution example

    # contingency table with counts for two discrete variables
    y2d = rbind( c(1,2,3), c(6,5,4) )

    # Bayesian estimate of mutual information (a=1/6)
    mi.Dirichlet(y2d, a=1/6)

    # half of the Bayesian chi-squared statistic of independence
    0.5*chi2indep.Dirichlet(y2d, a=1/6)

[Package entropy version 1.2.1 Index]