Introduction to Information Theory

Information theory was briefly explained in chapter 1. This chapter will take a more formal approach. Information theory grew out of our need to transmits information. Before the telephone, people communicated with telegraphs in morse code. About 5 words per minute can be transmitted in this manner. Not much information. So it was desirable to make sure that the information was being transmitted in the most efficient manner.

With the invention of the telephone, the amount of information that can be transmitted increased dramatically. Telephones are fine for talking, but they do not transmit enough information to be used effectively by today’s fastest computers.

How information is transmitted can and often is the bottleneck. Engineers have worked on this problem for a long time. The field of information theory tackles these difficult issues. The goal of information theory is to describe the information content of various messages and determine how best to transmit such messages. The same mathematical formulas can be applied to the information found in DNA. And this process is very enlightening. But before this can be done, considerations should be given to how information theory is applied in man made communication systems.

In communication systems, it is common for the transmission media to limit how much information that can be sent. The transmission media can be a phone line or any pair of wires. It can also be wireless. Examples of wireless transmission are cell phones and radios. As electronics improve, more information needs to be sent. The figure 6.1 shows how information theory can help. Without information theory, the message would be sent in some format over the transmission media. Parts of the message may be lost, especially if the system is overwhelmed by the information in the message. If this happens the system limits how much information can be sent. To avoid this problem, engineers over designed the system so that it can handle very large messages that are never sent. Or they may just decide the loss of data is a natural consequence of sending information. Information theory solves these problems. Furthermore, if the information is encoded to be sent in a more efficient manner, information theory defines the best possible code. Coding allows more information to be sent per unit time. The same transmission media may be able to send much more information. The message is coded by the encoder before it is transmitted. At the receiver it is decoded, so that people can read it. Information theory is all about optimizing how information is transmitted. The improvement in transmission rate is determined by the information that needs to be sent. Some information can be coded in such a way that a great improvement in transmission speed may result, and some cannot realize any improvement.

Figure 6.1 - Communication Systems

home: Intelligent Design                  

next: definition of information