Entropy

entropy.matrix

R Documentation

Description

The entropy() function calculates the Entropy of given probability distributions.

Usage

## S3 method for class 'matrix'
entropy(pk, dim = 0L, base = -1, ...)

## S3 method for class 'matrix'
relative.entropy(pk, qk, dim = 0L, base = -1, ...)

## S3 method for class 'matrix'
cross.entropy(pk, qk, dim = 0L, base = -1, ...)

## Generic S3 method
entropy(
 pk,
 dim  = 0,
 base = -1,
 ...
)

## Generic S3 method
relative.entropy(
 pk,
 qk,
 dim  = 0,
 base = -1,
 ...
)

## Generic S3 method
cross.entropy(
 pk,
 qk,
 dim  = 0,
 base = -1,
 ...
)

Arguments

pk	A \(n \times k\) <numeric>-matrix of observed probabilities. The \(i\)-th row should sum to 1 (i.e., a valid probability distribution over the \(k\) classes). The first column corresponds to the first factor level in actual, the second column to the second factor level, and so on.
dim	An value of length 1 (Default: 0). Defines the dimension along which to calculate the entropy (0: total, 1: row-wise, 2: column-wise).
base	A <numeric> value of length 1 (Default: -1). The logarithmic base to use. Default value specifies natural logarithms.
…	Arguments passed into other methods
qk	A \(n \times k\) <numeric>-matrix of predicted probabilities. The \(i\)-th row should sum to 1 (i.e., a valid probability distribution over the \(k\) classes). The first column corresponds to the first factor level in actual, the second column to the second factor level, and so on.

Value

A <numeric> value or vector:

• A single <numeric> value (length 1) if dim == 0.

• A <numeric> vector with length equal to the length of rows if dim == 1.

• A <numeric> vector with length equal to the length of columns if dim == 2.

Definition

Entropy:

\[H(pk) = -\sum_{i} pk_i \log(pk_i)\]

Cross Entropy:

\[H(pk, qk) = -\sum_{i} pk_i \log(qk_i)\]

Relative Entropy

\[D_{KL}(pk \parallel qk) = \sum_{i} pk_i \log\left(\frac{pk_i}{qk_i}\right)\]

Examples

# 1) Define actual
# and observed probabilities

# 1.1) actual probabilies
pk <- matrix(
  cbind(1/2, 1/2),
  ncol = 2
)

# 1.2) observed (estimated) probabilites
qk <- matrix(
  cbind(9/10, 1/10), 
  ncol = 2
)

# 2) calculate
# Entropy
cat(
  "Entropy", entropy(
    pk
  ),
  "Relative Entropy", relative.entropy(
    pk,
    qk
  ),
  "Cross Entropy", cross.entropy(
    pk,
    qk
  ),
  sep = "\n"
)