Shannon's source coding theorem - Revision history

Karl Jones: /* See also */

2016-09-19T19:04:48Z

‎See also

Karl Jones at 19:03, 19 September 2016

2016-09-19T19:03:45Z

Karl Jones at 19:03, 19 September 2016

2016-09-19T19:03:11Z

Karl Jones: Created page with "In information theory, '''Shannon's source coding theorem''' (or '''noiseless coding theorem''') establishes the limits to possible data compression, and the operation..."

2016-09-19T19:02:18Z

Created page with "In information theory, '''Shannon's source coding theorem''' (or '''noiseless coding theorem''') establishes the limits to possible data compression, and the operation..."

New page

In [[information theory]], '''Shannon's source coding theorem''' (or '''noiseless coding theorem''') establishes the limits to possible [[data compression]], and the operational meaning of the [[Entropy (information theory)|Shannon entropy]].

== Description ==

The source coding theorem shows that (in the limit, as the length of a stream of independent and identically-distributed random variable (i.i.d.) data tends to infinity) it is impossible to compress the data such that the code rate (average number of bits per symbol) is less than the [[Entropy (information theory)|Shannon entropy]] of the source, without it being virtually certain that information will be lost.

However it is possible to get the code rate arbitrarily close to the Shannon entropy, with negligible probability of loss.

The source coding theorem for symbol codes places an upper and a lower bound on the minimal possible expected length of codewords as a function of the entropy of the input word (which is viewed as a [[random variable]]) and of the size of the target alphabet.

== See also ==

* [[Asymptotic Equipartition Property]] (AEP)
* [[Channel coding]]
* [[Claude E. Shannon]]
* [[Error exponent]]
* [[Noisy Channel Coding Theorem]]
* [[Random variable]]

== External links ==

* [https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theorem Shannon's source coding theorem] @ Wikipedia

[[Category:Information theory]]

@@ Line 12: / Line 12: @@
 * [[Asymptotic Equipartition Property]] (AEP)
 * [[Claude Shannon]]
 * [[Error exponent]]
 * [[Noisy Channel Coding Theorem]]
 * [[Random variable]]

@@ Line 3: / Line 3: @@
 == Description ==
-The source coding theorem shows that (in the limit, as the length of a stream of independent and identically-distributed random variable (i.i.d.) data tends to infinity) it is impossible to compress the data such that the code rate (average number of bits per symbol) is less than the [[Entropy (information theory)|Shannon entropy]] of the source, without it being virtually certain that information will be lost.
+The source coding theorem shows that (in the limit, as the length of a stream of [[Independent and identically distributed random variables|independent and identically-distributed random variable]] (i.i.d.) data tends to infinity) it is impossible to compress the data such that the code rate (average number of bits per symbol) is less than the [[Entropy (information theory)|Shannon entropy]] of the source, without it being virtually certain that information will be lost.
 However it is possible to get the code rate arbitrarily close to the Shannon entropy, with negligible probability of loss.