Question

1. What is the minimum value of the entropy H(X) of a random variable X with an alphabet size M. What is the pmf that achieves this minimum value.

Answer 1

Entropy is a measure of uncertainty of a random variable. Let X be a discrete random variable with alphabet M and probability mass function p(x) = Pr{X = x}, x $\epsilon$ M. We denote the probability mass function by p(x) rather than p_x(x) for convenience. Thus, p(x) and p(y) refer to two different random variables and are in fact different probability mass functions (pmf), p_x(x) and p_y(y) respectively.

The entropy H(X) of a discrete random variable X is defined by: H(X) = - $\sum p(x) log p(x)$ where $x\epsilon M$