Theories are constructed with reasonable assumption that one or a few human brains can manage. Real world is full of noisy observations. A beautiful theory considers this noise and handles it. As an example I would like to talk about two most important contributions from two of the best scientists and engineers of the century. The first contribution was by Shanon. In his 1984's paper "A mathematical theory of communication." I would like to quote a few sentences from NASA's web page on Shanon's contribution since I could not say anything better than the article has said about this phenomenal work.
"Many concepts that seem obvious today were first introduced in his paper, "A Mathematical Theory of Communication," published in two parts in the Bell System Technical Journal. The use of the term "bit" as a unit of information first appeared in this paper. In an age in which communication took place with continuous, analog waveforms, it was a startling notion that information traveling across any communications system could be defined mathematically as some quantity of binary symbols. It was entirely new that information of any kind-- whether for use on a telegraph, telephone, radio, or television-- could be decomposed into zeros and ones, encoded, transmitted, and decoded at the other end.
That was groundbreaking, but it was only one component of his theory. He went on to present the concept of the maximum rate of transmission on a channel-- the capacity or "Shannon limit"-- which provides the benchmark against which all codes and modulations are measured. He invented adding redundancy to a transmitted signal in order to enable correcting for transmission errors at the receiving end. He provided the foundation of data compression, years ahead of any widespread implementations of digital communications or storage.
In this one paper, Shannon founded the field of information theory. In a world in which no one knew the concept of a bit, asking the question, "How many bits per second can be transmitted a channel?" and doing nothing more would have been momentous. Shannon did this and proceeded to clearly state the answer, prove it, and begin to show how to design systems that achieve the limit. He presented a clear picture of the whole field of information theory, all at once."
The second person I would like to mention is Jon Von Neumann's classic work on introducing theory of games and economic behavior. Again I would like to quote from an article from Princeton university press on his classic work.
"This is the classic work upon which modern-day game theory is based. What began more than sixty years ago as a modest proposal that a mathematician and an economist write a short paper together blossomed, in 1944, when Princeton University Press published Theory of Games and Economic Behavior. In it, John von Neumann and Oskar Morgenstern conceived a groundbreaking mathematical theory of economic and social organization, based on a theory of games of strategy. Not only would this revolutionize economics, but the entirely new field of scientific inquiry it yielded--game theory--has since been widely used to analyze a host of real-world phenomena from arms races to optimal policy choices of presidential candidates, from vaccination policy to major league baseball salary negotiations. And it is today established throughout both the social sciences and a wide range of other sciences."
The characteristic of both of these great work is keen observation to identify a simplified theory of complex systems that revolutionizes science and engineering and enormously reduces the gap between the abstract world of theory and real world.
Both information theory and game theory have been proven to be applicable in modeling many complex engineering systems till date and will continue to do so.
