Before Shannon, and the 1948 paper
As telegraphy and telephony spread, Harry Nyquist (1924) and Ralph Hartley (1928) at Bell Labs began treating transmission speed and quantity of information mathematically. In 1948 their colleague Claude Shannon published A Mathematical Theory of Communication, which separated information from meaning and defined it as the reduction of uncertainty about which message, among the possible ones, was chosen. An event of probability p carries −log₂p bits, and the average information of a source is its entropy, H = −Σ p log₂ p. The formula has the same form as thermodynamic entropy; by a well-known anecdote, John von Neumann urged Shannon to use the name. Norbert Wiener's Cybernetics, published the same year, likewise brought information and control together.
Limits of compression and transmission
Shannon's source coding theorem showed that entropy is the limit to lossless compression, and David Huffman (1952) devised optimal codes that give shorter codewords to more frequent symbols, a foundation for today's ZIP, JPEG and MP3 formats. The channel coding theorem proved that even over a noisy channel, errors can be made as rare as desired as long as the rate stays below the channel capacity. Beginning with Richard Hamming's error-correcting codes (1950), and later turbo codes (1993), LDPC and polar codes that approach the Shannon limit, the theory underpins the reliability of cellular networks, satellite links and storage devices.
Extensions and connections
Andrey Kolmogorov (1965) and others developed algorithmic information theory, which defines the information in a string as the length of the shortest program that produces it. Rolf Landauer (1961) linked information to physics with the principle that erasing one bit must release a minimum amount of heat, an idea that fed into quantum information and quantum computing. Biologists treat DNA sequences as information, and in statistics and machine learning, mutual information, Kullback–Leibler divergence and cross-entropy loss are everywhere, closely tied to Bayesian inference. Information theory is regarded as the mathematical foundation of the digital technologies behind the information society.
FAQ
- Is information in information theory the same as everyday information?
- No. Shannon's information measures how unpredictable a message is, regardless of its meaning or value; a meaningful sentence and a random string can carry the same amount of information.
- Why is the bit the basic unit?
- With logarithms in base 2, a choice between two equally likely options, such as yes or no, carries exactly one unit. The word bit, coined by John Tukey, first appeared in print in Shannon's 1948 paper.
Sources reviewed: 2026-10-26
References
- Nyquist, H. (1924). Certain factors affecting telegraph speed. Bell System Technical Journal, 3(2), 324–346.
- Hartley, R. V. L. (1928). Transmission of information. Bell System Technical Journal, 7(3), 535–563.
- Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656.
- Wiener, N. (1948). Cybernetics: Or Control and Communication in the Animal and the Machine. Cambridge, MA: MIT Press.
- Shannon, C. E., & Weaver, W. (1949). The Mathematical Theory of Communication. Urbana: University of Illinois Press.
- Hamming, R. W. (1950). Error detecting and error correcting codes. Bell System Technical Journal, 29(2), 147–160.
- Huffman, D. A. (1952). A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40(9), 1098–1101.
- Landauer, R. (1961). Irreversibility and heat generation in the computing process. IBM Journal of Research and Development, 5(3), 183–191.
- Kolmogorov, A. N. (1965). Three approaches to the quantitative definition of information. Problems of Information Transmission, 1(1), 1–7.
- Berrou, C., Glavieux, A., & Thitimajshima, P. (1993). Near Shannon limit error-correcting coding and decoding: Turbo-codes. Proceedings of ICC '93, 1064–1070.
- Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Hoboken, NJ: Wiley.