Mutual information in the context of Expected value

Information theory is the mathematical study of the quantification, storage, and communication of information. The field was established and formalized by Claude Shannon in the 1940s, though early contributions were made in the 1920s through the works of Harry Nyquist and Ralph Hartley. It is at the intersection of electronic engineering, mathematics, statistics, computer science, neurobiology, physics, and electrical engineering.

A key measure in information theory is entropy. Entropy quantifies the amount of uncertainty involved in the value of a random variable or the outcome of a random process. For example, identifying the outcome of a fair coin flip (which has two equally likely outcomes) provides less information (lower entropy, less uncertainty) than identifying the outcome from a roll of a die (which has six equally likely outcomes). Some other important measures in information theory are mutual information, channel capacity, error exponents, and relative entropy. Important sub-fields of information theory include source coding, algorithmic complexity theory, algorithmic information theory and information-theoretic security.

The mathematical theory of information is based on probability theory and statistics, and measures information with several quantities of information. The choice of logarithmic base in the following formulae determines the unit of information entropy that is used. The most common unit of information is the bit, or more correctly the shannon, based on the binary logarithm. Although bit is more frequently used in place of shannon, its name is not distinguished from the bit as used in data processing to refer to a binary value or stream regardless of its entropy (information content). Other units include the nat, based on the natural logarithm, and the hartley, based on the base 10 or common logarithm.

In what follows, an expression of the form ${\displaystyle p\log p\,}$ is considered by convention to be equal to zero whenever ${\displaystyle p}$ is zero. This is justified because ${\displaystyle \lim _{p\rightarrow 0+}p\log p=0}$ for any logarithmic base.

In information theory, joint entropy is a measure of the uncertainty associated with a set of variables.

In information theory, the conditional entropy quantifies the amount of information needed to describe the outcome of a random variable ${\displaystyle Y}$ given that the value of another random variable ${\displaystyle X}$ is known. Here, information is measured in shannons, nats, or hartleys. The entropy of ${\displaystyle Y}$ conditioned on ${\displaystyle X}$ is written as ${\displaystyle \mathrm {H} (Y|X)}$.