Entropy: Information Theory

Entropy in information theory is a mathematical measure of uncertainty associated with a probability distribution. A source whose next outcome is highly predictable has low entropy, while a source with several similarly likely outcomes has higher entropy.

For a discrete random variable X, Shannon entropy is commonly written as:

H(X) = -Σ p(x) log₂(p(x))

where p(x) is the probability of outcome x.

When logarithms use base 2, entropy is measured in bits.

For a fair binary source with outcomes 0 and 1:

p(0) = 0.5

p(1) = 0.5

the entropy is:

H(X) = -[0.5 log₂(0.5) + 0.5 log₂(0.5)]

Since:

log₂(0.5) = -1

we get:

H(X) = -[0.5(-1) + 0.5(-1)]

H(X) = 1 bit

A fair binary outcome therefore has 1 bit of entropy per outcome.

What Is Entropy in Information Theory?

Entropy quantifies uncertainty before an outcome is observed.

Suppose a source always produces the same symbol:

P(A) = 1

There is no uncertainty about what comes next.

Its entropy is:

H = 0

Now suppose a binary source produces:

P(A) = 0.5

P(B) = 0.5

Neither outcome is more predictable than the other.

Its entropy is:

H = 1 bit

So information-theory entropy rises when probability is spread more evenly across possible outcomes.

Entropy Formula

For a discrete random variable X with outcomes x₁, x₂, ..., xₙ:

H(X) = -Σ p(xᵢ) log_b(p(xᵢ))

The base b determines the unit.

If:

b = 2

entropy is measured in:

bits

If:

b = e

entropy is measured in:

nats

If:

b = 10

the unit is sometimes called:

hartleys or decimal digits of information

Base 2 is especially common in computing and digital communication because binary numbers use two symbols.

Why Is There a Negative Sign?

Probabilities satisfy:

0 < p ≤ 1

For probabilities below 1:

log₂(p) < 0

For example:

log₂(0.5) = -1

and:

log₂(0.25) = -2

Without the leading negative sign, the weighted sum would be negative.

The entropy formula uses:

-Σ p(x) log₂(p(x))

so the result is nonnegative.

The mathematical behavior of the logarithm itself is treated more generally under logarithms.

Information Content of a Single Outcome

Before calculating average uncertainty, information theory can assign an information value to one event.

For an event with probability p:

I = -log₂(p)

An unlikely event carries more information when it occurs than a highly predictable event.

For example, if:

p = 1/2

then:

I = -log₂(1/2)

I = 1 bit

If:

p = 1/8

then:

I = -log₂(1/8)

I = 3 bits

The rarer event produces a larger information value.

Entropy as Average Information

Entropy is the probability-weighted average of individual information values.

If outcome x has information:

I(x) = -log₂(p(x))

then:

H(X) = Σ p(x)I(x)

Substituting the information formula gives:

H(X) = -Σ p(x)log₂(p(x))

This interpretation is important: entropy is not simply the information in one event. It represents average uncertainty or average information across the entire distribution.

Example: Fair Coin

Suppose a coin has:

P(H) = 0.5

P(T) = 0.5

Then:

H = -[0.5 log₂(0.5) + 0.5 log₂(0.5)]

Since:

log₂(0.5) = -1

we obtain:

H = -[-0.5 – 0.5]

H = 1 bit

A fair coin has maximum entropy among distributions with two possible outcomes.

Example: Biased Coin

Suppose:

P(H) = 0.9

P(T) = 0.1

Then:

H = -[0.9 log₂(0.9) + 0.1 log₂(0.1)]

Using approximate values:

log₂(0.9) ≈ -0.1520

log₂(0.1) ≈ -3.3219

Therefore:

H ≈ -[0.9(-0.1520) + 0.1(-3.3219)]

H ≈ -[-0.1368 – 0.3322]

H ≈ 0.4690 bits

So:

H ≈ 0.469 bits

The biased coin has less entropy than the fair coin because its next result is more predictable.

Example: Certain Outcome

Suppose:

P(A) = 1

and every other possible outcome has probability zero.

Then:

H = -1 × log₂(1)

Since:

log₂(1) = 0

we get:

H = 0

A deterministic source has no uncertainty about its next outcome.

Handling Zero Probabilities

The entropy expression contains terms such as:

p log(p)

At:

p = 0

the logarithm itself is not finite, but the entropy term is defined by its limiting behavior:

0 log(0) = 0

for entropy calculations.

This means impossible outcomes contribute no entropy.

Only outcomes with positive probability need to contribute numerically to the sum.

Maximum Entropy

For a fixed number n of possible outcomes, entropy is maximized when every outcome is equally likely:

p(xᵢ) = 1/n

Substitute into the entropy formula:

H = -Σ (1/n) log₂(1/n)

There are n equal terms, giving:

Hmax = log₂(n)

For two equally likely outcomes:

Hmax = log₂(2) = 1 bit

For four equally likely outcomes:

Hmax = log₂(4) = 2 bits

For eight equally likely outcomes:

Hmax = log₂(8) = 3 bits

This pattern connects entropy directly with powers and exponents.

Example: Four Equally Likely Symbols

Suppose a source produces:

A, B, C, D

with:

P(A) = P(B) = P(C) = P(D) = 0.25

Each symbol has information:

-log₂(0.25) = 2 bits

Therefore the average entropy is also:

H = 2 bits

This agrees with:

Hmax = log₂(4)

Hmax = 2

Example: Eight Equally Likely Outcomes

If there are eight outcomes, each with probability:

1/8

then:

H = log₂(8)

Since:

2³ = 8

we have:

H = 3 bits

The relationship shows why base-2 entropy naturally describes binary encoding.

Entropy and Probability

Entropy depends on the entire probability distribution, not merely on the number of outcomes.

Two sources may both have four possible outcomes but very different entropy.

Source A:

0.25, 0.25, 0.25, 0.25

has:

H = 2 bits

Source B:

0.97, 0.01, 0.01, 0.01

is much more predictable and therefore has substantially lower entropy.

Concepts such as conditional probability become relevant when the probability distribution changes after other information is known.

Entropy and Expected Value

Entropy is a probability-weighted average, which gives it a structural relationship with expected value.

For expected value:

E[X] = Σ p(x)x

For entropy, the quantity being averaged is the information content:

I(x) = -log₂(p(x))

Therefore:

H(X) = Σ p(x)I(x)

The two concepts measure different things, but both use probability-weighted sums.

Entropy Is Not Variance

Entropy and variance both describe aspects of uncertainty or spread, but they are not interchangeable.

Variance depends on numerical distances from a mean.

Entropy depends on probabilities.

A random variable may have outcomes whose numerical labels change while its entropy remains the same if the probabilities remain unchanged.

For example, distributions:

{0: 0.5, 1: 0.5}

and:

{100: 0.5, 1000: 0.5}

both have entropy:

1 bit

despite having very different numerical variance.

Entropy and Binary Information

A bit represents one binary digit:

0 or 1

A fair binary source contains:

1 bit of entropy per symbol

because two equally likely alternatives require one binary decision to distinguish.

This connects information entropy with the practical representation of data in bits, while a conversion such as bits to bytes answers a different question about storage units.

Eight bits make one byte:

8 bits = 1 byte

but eight bits of storage do not automatically mean a source has exactly eight bits of entropy. Redundancy and unequal probabilities can reduce the actual information content.

Entropy and Data Compression

Entropy provides a theoretical benchmark for lossless compression.

If some symbols occur much more frequently than others, the source has redundancy. A well-designed coding system can assign shorter codewords to common symbols and longer codewords to rare ones.

For example, a source that produces one symbol 99% of the time is highly predictable. Representing every symbol with an equally long code may waste space relative to a probability-aware encoding.

Entropy describes the average information inherent in the source distribution.

It does not by itself specify a particular compression algorithm.

Entropy Does Not Equal File Size

A file may occupy a certain number of bits or bytes, but its entropy depends on the statistical model used for the data.

For example, a long sequence consisting entirely of zeros may occupy many bits in an uncompressed file while containing very little uncertainty under a model that recognizes the repetition.

Therefore:

storage size and information entropy are related concepts but not the same quantity.

Entropy and Predictability

Lower entropy generally corresponds to greater predictability.

Suppose Source A produces:

0 with probability 0.99

1 with probability 0.01

while Source B produces:

0 with probability 0.5

1 with probability 0.5

Before observing the next symbol, Source A is easier to predict.

Source B therefore has higher entropy.

This does not mean every individual prediction for Source A will be correct. Entropy describes the distribution’s average uncertainty.

Entropy and Number of Possible Outcomes

More possible outcomes can permit more entropy, but only if their probabilities are sufficiently distributed.

For n possible outcomes:

H(X) ≤ log₂(n)

Equality occurs when every outcome has probability:

1/n

Therefore, simply adding extremely unlikely outcomes does not necessarily increase entropy to the maximum allowed by the new outcome count.

Entropy of a Three-Outcome Distribution

Suppose:

P(A) = 0.5

P(B) = 0.25

P(C) = 0.25

Then:

H = -[0.5 log₂(0.5) + 0.25 log₂(0.25) + 0.25 log₂(0.25)]

Use:

log₂(0.5) = -1

and:

log₂(0.25) = -2

Therefore:

H = -[0.5(-1) + 0.25(-2) + 0.25(-2)]

H = -[-0.5 – 0.5 – 0.5]

H = 1.5 bits

So:

H = 1.5 bits

The maximum entropy for three outcomes would be:

log₂(3) ≈ 1.585 bits

Thus this distribution is uncertain but not perfectly uniform.

Entropy of a Loaded Four-Symbol Source

Consider:

P(A) = 0.5

P(B) = 0.25

P(C) = 0.125

P(D) = 0.125

Calculate each information value:

-log₂(0.5) = 1

-log₂(0.25) = 2

-log₂(0.125) = 3

Then take the probability-weighted average:

H = 0.5(1) + 0.25(2) + 0.125(3) + 0.125(3)

H = 0.5 + 0.5 + 0.375 + 0.375

H = 1.75 bits

Therefore:

H = 1.75 bits

The maximum for four outcomes is 2 bits, so the unequal distribution has lower entropy.

Why Base 2 Produces Bits

The logarithm base determines the scale of information.

Using:

log₂

asks how many binary decisions correspond to a given probability.

For an event with:

p = 1/8

we have:

-log₂(1/8) = 3

because:

2^-3 = 1/8

The event therefore carries:

3 bits

of self-information.

Changing the Logarithm Base

The same distribution can be measured using a different logarithm base.

For base 2:

H₂ = -Σ p(x)log₂(p(x))

For natural logarithms:

Hₑ = -Σ p(x)ln(p(x))

The numerical values differ by a constant conversion factor, but the underlying ordering of distributions by uncertainty remains consistent.

The choice of base changes the unit, not the fundamental idea.

Conditional Entropy

Sometimes uncertainty is measured after another variable is known.

If X and Y are related, conditional entropy asks how much uncertainty remains in X after observing Y.

Conceptually:

H(X | Y)

can be smaller than:

H(X)

because knowing Y may help predict X.

If Y determines X completely:

H(X | Y) = 0

If knowing Y provides no useful information about X, the remaining uncertainty may be unchanged.

The probability relationships underlying such conditioning connect naturally with the separate conditional probability topic.

Joint Entropy

Joint entropy measures uncertainty in two variables considered together.

It is commonly written:

H(X, Y)

A useful relationship is:

H(X, Y) = H(X) + H(Y | X)

This says the information needed to describe the pair can be viewed as the information in X plus the remaining information needed for Y after X is known.

Entropy and Independent Variables

If X and Y are independent, knowing one does not reduce uncertainty about the other.

Then:

H(Y | X) = H(Y)

and:

H(X, Y) = H(X) + H(Y)

For example, two independent fair coin flips each contribute 1 bit of entropy:

H(X) = 1

H(Y) = 1

Therefore:

H(X, Y) = 2 bits

There are four equally likely pairs:

HH, HT, TH, TT

and:

log₂(4) = 2

Entropy and Information Gain

When new information reduces uncertainty, the reduction can be interpreted as information gained.

If uncertainty begins at:

H(X)

and after observing Y becomes:

H(X | Y)

then the reduction is:

H(X) – H(X | Y)

This idea is important in communication, statistical learning, and decision trees.

The current article remains focused on entropy itself rather than building a separate treatment of every derived information measure.

Entropy and Coding Efficiency

Suppose a source has entropy:

H = 1.5 bits/symbol

This does not necessarily mean every individual symbol can literally be stored using exactly 1.5 physical bits.

Codewords normally have integer lengths.

The entropy value represents a theoretical average information rate. With suitable coding over long sequences, average encoding length can approach the entropy limit under appropriate assumptions.

Entropy Rate

If a source produces a sequence whose symbols are statistically dependent, uncertainty per symbol may differ from the entropy of one isolated observation.

An entropy rate considers uncertainty across a process as the sequence grows.

A memoryless source with independent identically distributed symbols has a simpler relationship: its entropy rate equals the entropy of one symbol.

When dependencies exist, observing earlier symbols may make later symbols more predictable.

Entropy and Division

Probabilities used in entropy calculations are frequently obtained through division.

Suppose a dataset contains 200 symbols, with symbol A appearing 50 times.

Its empirical probability is:

p(A) = 50 / 200

p(A) = 0.25

The entropy calculation then applies a logarithm to that probability.

Division determines the observed proportion; entropy measures uncertainty across the resulting probability distribution.

Entropy and the Euclidean Algorithm

The Euclidean algorithm and information entropy both belong to mathematics, but they solve fundamentally different problems.

The Euclidean algorithm uses repeated integer division and remainders to find a greatest common divisor.

Entropy uses probabilities and logarithms to quantify uncertainty.

Keeping those intents separate matters because a page about entropy should not expand into integer GCD procedures simply because both involve mathematical formulas.

Entropy and Divisibility Rules

Likewise, divisibility rules test whether one integer divides another exactly.

Entropy does not depend on whether a probability numerator or denominator passes a particular digit-based divisibility test.

A probability might first be simplified arithmetically, but the entropy calculation ultimately depends on its numerical probability and logarithm.

Entropy and Determinants

A determinant reduces a square matrix to a scalar value used in questions about invertibility, linear transformations, and systems of equations.

Entropy also produces a scalar number, but the similarity stops there.

A determinant is calculated from matrix entries and their arrangement.

Entropy is calculated from a probability distribution.

The two quantities answer different mathematical questions and should not be confused merely because both summarize more complex structures with one number.

Entropy and Exponential Relationships

Because information uses logarithms, entropy has a natural inverse relationship with powers.

If an equally likely source has:

H = 5 bits

then its number of equally likely outcomes satisfies:

log₂(n) = 5

Using exponents:

n = 2^5

n = 32

Thus 5 bits can distinguish among 32 equally likely possibilities.

Entropy and Binomial Probability

A binomial probability calculation assigns probabilities to counts of successes in repeated Bernoulli trials.

Those resulting probabilities can themselves form a distribution whose entropy may be analyzed.

However, entropy does not calculate the binomial probabilities. It takes an already defined probability distribution and measures its uncertainty.

Entropy Is Not Randomness in a Single Sequence

Entropy describes a probability model or distribution.

A particular observed sequence may appear irregular or highly patterned, but assigning information-theoretic entropy requires specifying how outcomes are generated or estimating an underlying distribution.

For example, seeing:

0101010101

does not by itself determine one universal entropy value without stating the probabilistic model being used.

This distinction prevents entropy from being treated as a simple visual score of how “random” a finite string looks.

Entropy Is Not Thermodynamic Entropy

The word entropy also appears in thermodynamics.

The concepts have mathematical and historical connections, but they are used in different frameworks.

This page concerns information-theory entropy: uncertainty measured from probabilities.

Its core formula is:

H(X) = -Σ p(x)log_b(p(x))

rather than a thermodynamic state equation.

Common Entropy Mistakes

One common mistake is forgetting the negative sign.

Since logarithms of probabilities below 1 are negative, omitting the leading minus sign would produce a negative result for ordinary Shannon entropy.

Another error is using probabilities that do not sum to 1.

A valid discrete probability distribution must satisfy:

Σ p(x) = 1

A third mistake is mixing logarithm bases within one calculation. Doing so mixes units.

If the answer is required in bits, use:

log₂

consistently.

Another common error is assuming that more possible outcomes automatically means greater entropy. Probability balance matters.

Finally, entropy should not be confused with the storage size of one particular file or message.

How to Check an Entropy Calculation

Several checks can catch mistakes.

First, entropy should not be negative:

H(X) ≥ 0

Second, for n possible outcomes:

H(X) ≤ log₂(n)

when entropy is measured in bits.

Third, a deterministic distribution should produce:

H = 0

Fourth, an equal distribution across n outcomes should produce:

H = log₂(n)

These bounds provide useful reasonableness checks even before recalculating every logarithm.

Worked Example: Three Symbols

Suppose:

P(A) = 0.50

P(B) = 0.25

P(C) = 0.25

Calculate:

H = -[0.50 log₂(0.50) + 0.25 log₂(0.25) + 0.25 log₂(0.25)]

Use:

log₂(0.50) = -1

log₂(0.25) = -2

Then:

H = -[0.50(-1) + 0.25(-2) + 0.25(-2)]

H = -[-0.50 – 0.50 – 0.50]

H = 1.50 bits

Check the maximum possible entropy:

log₂(3) ≈ 1.585 bits

Since:

1.50 < 1.585

the answer falls within the valid range.

Worked Example: Highly Predictable Source

Suppose:

P(A) = 0.98

P(B) = 0.02

Then:

H = -[0.98 log₂(0.98) + 0.02 log₂(0.02)]

Using approximate values:

log₂(0.98) ≈ -0.0291

log₂(0.02) ≈ -5.6439

Therefore:

H ≈ -[0.98(-0.0291) + 0.02(-5.6439)]

H ≈ -[-0.0285 – 0.1129]

H ≈ 0.1414 bits

The result is much lower than 1 bit because the next outcome is highly predictable.

Frequently Asked Questions

What is entropy in information theory?

Entropy is a mathematical measure of the average uncertainty or information associated with a probability distribution.

What is the entropy formula?

For a discrete random variable:

H(X) = -Σ p(x)log_b(p(x))

where p(x) is each outcome’s probability and b is the logarithm base.

Why is entropy measured in bits?

When the formula uses base-2 logarithms, the result is measured in bits.

What is the entropy of a fair coin?

A fair coin has:

H = 1 bit

because its two outcomes are equally likely.

What is the entropy of a certain event?

If one outcome has probability 1:

H = 0

There is no uncertainty.

When is entropy highest?

For a fixed number of possible outcomes, entropy is highest when all outcomes are equally likely.

What is the maximum entropy of four outcomes?

For four equally likely outcomes:

Hmax = log₂(4) = 2 bits

Can entropy be negative?

Ordinary Shannon entropy for a discrete probability distribution is nonnegative.

Does a rare event contain more information?

Under self-information:

I = -log₂(p)

a lower-probability event has a larger information value when it occurs.

Does a larger file always have more entropy?

No. File size measures stored data quantity, while entropy depends on a probability model and statistical predictability.

What happens if an outcome has probability zero?

Its contribution to Shannon entropy is conventionally taken as:

0 log(0) = 0

using the appropriate limit.

What is the difference between entropy and expected value?

Expected value averages numerical outcomes. Entropy averages the information content associated with outcome probabilities.

Final Example

Suppose a source produces four symbols with probabilities:

P(A) = 1/2

P(B) = 1/4

P(C) = 1/8

P(D) = 1/8

Their information values are:

A: -log₂(1/2) = 1 bit

B: -log₂(1/4) = 2 bits

C: -log₂(1/8) = 3 bits

D: -log₂(1/8) = 3 bits

Calculate the probability-weighted average:

H = (1/2)(1) + (1/4)(2) + (1/8)(3) + (1/8)(3)

H = 0.5 + 0.5 + 0.375 + 0.375

H = 1.75 bits

Therefore:

Entropy = 1.75 bits per symbol

The essential idea is that entropy measures how uncertain a source is before its next outcome is known. Equal probabilities maximize uncertainty; increasingly predictable distributions reduce it.