Information Theory Information is surprise

In daily life, we think of information as meaning: stories, messages, and ideas. But Claude Shannon approached it differently. He focused on how reliably a message can be transmitted through noise. His theory reinterprets information as uncertainty, unlocking powerful tools for communication, coding, and understanding complex systems.

When we think of information, we often think of meaning, words on a page, a message in a bottle, a story told over coffee. However, in the mid-20th century, engineer and mathematician proposed a radically different approach. He asked not what a message means, but how much information it contains and how reliably it can be transmitted.

This shift gave rise to information theory, a foundational concept in fields ranging from the internet to genetics to artificial intelligence.

The Core Idea

Shannon’s insight was to treat information as a measurable quantity, like weight or distance. His 1948 paper, A Mathematical Theory of Communication, reframed communication as a problem of signal and noise, focusing on encoding and decoding. The goal: to send a message across a noisy channel with as little distortion as possible.

To do this, he introduced a key concept: entropy.

Entropy: Measuring Uncertainty

• If I send you a message that says “the sun will rise tomorrow,” it contains very little new information, because it’s highly predictable.
• But if I tell you “a comet will hit the Pacific at 3 PM,” the message has high information content, because it is unexpected.

In other words, information is surprise. And entropy is a way of measuring that surprise.

Shannon’s entropy formula gives us the average amount of information per symbol in a message, based on the probability of each possible symbol. More uniform probabilities (resulting in greater uncertainty) correspond to higher entropy.

Signal, Noise, and Compression

Shannon also clarified the roles of signal and noise:

• A signal is what you want to communicate.
• Noise is what gets in the way: static on the line, typos, dropped packets.

He demonstrated that, given a noisy channel, perfect transmission can still be achieved as long as the message is encoded efficiently and the channel capacity remains below a critical threshold.

This led to practical breakthroughs, such as error-correcting codes, data compression (e.g., ZIP files, MP3), and the modern architecture of digital communication.

Meaning Left Out—Deliberately

Warm Data
Understanding meaning through context, not just information
We live in overlapping layers of meaning: personal, cultural, emotional, and historical. The challenge is that most data ignores this complexity, offering facts without context. Warm data provides a different approach: attending to the subtle, relational patterns that shape how things work and what truly matters in any situation.

One of the most striking aspects of Shannon’s theory is what it leaves out: meaning. In contrast to Nora Bateson’s warm data, information theory doesn’t care what the message is “about.” It only cares how likely it is.

This made the theory astonishingly general. You can apply it to:

• Morse code and satellite signals
• DNA sequences and neural spikes
• Economic models and ecological networks

Wherever there’s a pattern and uncertainty, Shannon’s tools can be applied.

Why It Matters

Shannon’s work didn’t just solve a technical problem; it reshaped how we think about communication, knowledge, and even life itself. It provided us with a quantitative framework for understanding the abstract and laid the groundwork for fields such as machine learning, network science, and cognitive neuroscience.

As we enter an age where information flows define our societies and systems, it’s worth returning to this foundational insight:

Information is not meaning. Information is choice, variation, and uncertainty.

And yet, almost everything else can be built from that basic principle.

We need to be clear about what information is and what it is not. Shannon’s theory helps us measure and move information, but it doesn’t explain meaning. To work well in complex settings, we must combine this technical view with attention to context, relationships, and the patterns that shape understanding.