entropy

entropy.matrix

R Documentation

Description

The entropy() function calculates the Entropy of given probability distributions.

Usage

## S3 method for class 'matrix'
entropy(pk, dim = 0L, base = -1, ...)

## S3 method for class 'matrix'
relative.entropy(pk, qk, dim = 0L, base = -1, ...)

## S3 method for class 'matrix'
cross.entropy(pk, qk, dim = 0L, base = -1, ...)

entropy(...)

relative.entropy(...)

cross.entropy(...)

Arguments

`pk`	A \(n \times k\) `<numeric>`-matrix of observed probabilities. The \(i\)-th row should sum to 1 (i.e., a valid probability distribution over the \(k\) classes). The first column corresponds to the first factor level in `actual`, the second column to the second factor level, and so on.
`dim`	An `<integer>` value of length 1 (Default: 0). Defines the dimensions of to calculate the entropy. 0: Total entropy, 1: row-wise, 2: column-wise
`base`	A `<numeric>` value of length 1 (Default: -1). The logarithmic base to use. Default value specifies natural logarithms.
`…`	Arguments passed into other methods
`qk`	A \(n \times k\) `<numeric>`-matrix of predicted probabilities. The \(i\)-th row should sum to 1 (i.e., a valid probability distribution over the \(k\) classes). The first column corresponds to the first factor level in `actual`, the second column to the second factor level, and so on.

Value

A <numeric> value or vector:

A single <numeric> value (length 1) if dim == 0.
A <numeric> vector with length equal to the length of rows if dim == 1.
A <numeric> vector with length equal to the length of columns if dim == 2.

Calculation

Entropy:

\[H(pk) = -\sum_{i} pk_i \log(pk_i)\]

Cross Entropy:

\[H(pk, qk) = -\sum_{i} pk_i \log(qk_i)\]

Relative Entropy

\[D_{KL}(pk \parallel qk) = \sum_{i} pk_i \log\left(\frac{pk_i}{qk_i}\right)\]

Examples

# 1) Define actual
# and observed probabilities

# 1.1) actual probabilies
pk <- matrix(
  cbind(1/2, 1/2),
  ncol = 2
)

# 1.2) observed (estimated) probabilites
qk <- matrix(
  cbind(9/10, 1/10), 
  ncol = 2
)

# 2) calculate
# Entropy
cat(
  "Entropy", entropy(
    pk
  ),
  "Relative Entropy", relative.entropy(
    pk,
    qk
  ),
  "Cross Entropy", cross.entropy(
    pk,
    qk
  ),
  sep = "\n"
)