On Machine Learning

The discipline of Machine Learning has inherited a major portion of it's theory from classical statistics. Today, the area is quite vast and it consists of many topics including classification, regression, clustering, reinforcement learning etc. We often see interview candidates with machine learning in resume. In academic circle, typically a researcher in machine learning is expected to publish in ICML, NIPS, COLT community.
Application research can be published in various different conferences such as KDD, ICDM, SDM, CVPR, ECCV, ICPR, SIGIR and even in WWW. However, the question arises that what knowledge is really required or valuable for software engineering positions. I feel that someone claiming some practical and academic knowledge in machine learning should be able to answer some of the questions such as follows:

1. What are sample complexity and model complexity?
2. What are the characteristics of feature selection algorithms?
3. How one handle sparsity in features?
4. What are various loss functions used for SVM, clustering, decision trees?
5. What are the desirable characteristics of loss functions and why?
6. What is regularization?
7. Why generalized linear models are called "generalized"?
8. What are exponential family distributions? What is their significance in machine learning?
9. Can one make a classifier using regression model?
10. How do one compare various models with same target and with different targets in regression?
11. Can one apply k-means algorithms for a data set with Zipf distribution? What modification is required?
12. What is Gibb's sampling?
13. What is the difference between HMM and CRF?
14. What is the complexity of inference in Graphical model?

The above questions are a few of the sample questions. I believe a good reading of selected chapters of Duda and Hurt, Bishop and Hastie's book (3 books total) can provide one a basic background in the subject. Boyd's convex optimization book and Feller's volume one in statistics can be used as references for knowledge in convex optimization and statistics.

An analysis of MSFT's search war strategy

Comscore report published that Bing combined with Yahoo has now 41% search market share. This result is doubtful. However, it is possible for Bing to get increased search share from Facebook queries.

The MSFT's search strategy began with an ambition to make another search engine like Goog. I believe that Gates thought about using their PC domination, friend company network in gaining market from Goog exactly the way they killed Netscape. However, Goog rises into a giant business with increasing advertising dollar.

In phase one of the search war, MSFT had their own loyal engineers working in search. But, the leadership soon realized that search has more "algorithms" and "AI" and it is development of "distributed systems" and internet services which is certainly different that PC based software applications. In second phase, MSFT started following Goog's recruitment pattern hiring PhDs and scientists. They also replaced the position of head of search with someone who has more academic background in IR. However, the strategy of making a better Google search remained same. Goog at the same time made better email service, a better translation service and many other new services that MSFT leaders could not think ahead of time. In the third phase, MSFT leaders realizes that Goog has a platform and two sided effect has already made them very powerful and it is almost impossible to gain search market share without innovation. Traditionally innovation is weakest in MSFT's product development. They then hired veterans from falling Yahoo to run their search business and they started making "Bing" as a decision engine. This is a new phase for MSFT's search and also it is a new generation for MSFT's strategy. They still lack an out of the box original something that can make Bing attractive choice for search and unfortunately, this is hard unless MSFT makes it strategically important to make innovation in search and can spread across their org. An example of such innovation by MSFT is "kinnect" and "xbox live".

On Oracle's legal battle with Google on Java!

Oracle's legal battle with Google brings the debate on U.S patenting system itself. It is ridiculous to patent a programming language architecture. Imagine the situation if Tony Hoare wants a share for using quick sort which is almost ubiquitous in databases or any large computing systems or UT Austin wants share for Dijkstra's shortest path algorithm from Cisco or Google or any other company that is using shortest path for any application. Although, I got a few patents and I completely understand that it is just a corporate tool to protect them in face of competition and it has no other purpose. I would always vote for eradicating this ridiculous system. Oracle's motivation is in fetching some free money from Google. Oracle really did nothing with the patents they had purchased from Sun and now they wanted to use those patents.

G+ and Facebook comments

G+ with their circle and hangout features provides more natural and richer expressiveness for social than Facebook. It is a clean UI design and a neat piece of software and has been distributed so far quite cleverly. Facebook will need to innovate and make their product better. It's not hard for G+ to gain market share using their existing products. I believe G+ will stay and will be more competitive soon in future. G leadership is one of the best among Tech corps.

Google's voice based search Impressive

It is impressive that Google is able to obtain high accuracy for frequently used search terms in voice based search. This is an excellent engineering work. This is not an easy task since the speech recognizer needs to be stable under various accents. It must have required a lot of training data. I believe that they probably used a large scale finite state transducer to map to the correct word. It will be interesting to know what are the main tricks they employed in order to get this high accuracy.

More Binary search variants

# Search a sorted array of unknown length
# Degree of difficulty: Easy

Assume: Exception will be thrown for array out of bound.

Descriptive answer:
First find the length.
Then do a normal bin search.

How to find the length?
Assume, length 2^n.
Find out valid k for which no exception is thrown.

Again binsearch between 0 and 2^k+1. Consider exception in the
termination criteria.

# Search a sorted array of distinct integer for A[i]=i

# Degree of difficulty: Easy.



def binsearch_var3(list):

    l=0

    u=len(list)-1

    while l<=u:

        m=(l+u)/2

        print '%d %d' %(m,list[m])

        if list[m]==m:

            return m

        elif list[m]            l=m+1

        else:

            u=m-1



    return -1

On auction

Here's Amy Greenwald's essay on Game Theory and the Design of Electronic Markets.

A news article on IBM

On IBM

What does it mean by MSFT's purchase of Skype?

MSFT is going through stiff competition from GOOG and APPL. It has been barely keeping a presence in Internet search industry, even a purchase of sinking YHOO search did not put them in a better shape yet. The PC industry trend is going downward. MSFT is still stronger there but the growth of the PC market itself has already been penetrated by the "Tablets and other devices". Moreover, MSFT has almost lost its presence in the Mobile market which is one of the highest growth segment. The only good news for the software giant is perhaps its growing influence in the games market. Anyway, lets analyze what could be the reasons for MSFT paying a large sum of money for VoIP service like Skype. Skype has 663 million registered users and very significant gross profitability. Skype contributed more net new minutes of international voice than the rest of the industry put together in 2010, according to Telegeography. Skype has never struggled to achieve growth, but its profitability has often been criticized. MSFT can potentially develop a technology with Skype to disrupt the smart phone mobile industry. However, this is an absolutely daunting task to challenge APPL and GOOG in mobile market. But, only MSFT posses the strength for that. However, they need to find one or multiple people who can make that happen. They have many talented leaders. But they need someone different and it is a hard business+technology problem.

An interesting talk

This talk was given in 2004 on collaborative giftedness (work and development). by Henry Tirri.

Growth of eBay An article from 2003

An article published in Forbes magazine in 2003:


NEW YORK - Why did eBay work where Enron failed? Both set out to create revolutionary new Internet-based marketplaces. One of them is a legend in the making, the other is already the stuff of history.

This is no matter of idle curiosity to economists, who are paying increasing attention to how electronic markets work. In one sense, it is work they thought they wouldn't need to do. The mechanisms of markets are well understood and have been little changed--from an economist's perspective--for hundreds of years.

Save for the goods, a medieval European market is much the same as a 21st-century financial market: It is a place where buyers and sellers can come together to trade information and strike bargains; it vouches for the trustworthiness of participants and their goods, and regulates them through codes of behavior, tacit or otherwise; it lets the appropriate prices for a bargain be discovered; and it provides a clearing house for the financial side of transactions.

That seems to describe eBay (nasdaq: EBAY - news - people ), in an economist's dry sort of way. Yet if that is all there is to markets, the world wouldn't be littered with the remains of moribund business-to-business electronic markets. Many of these were started in the bubble-era years on the if-you-build-it-they-will-come principle. When they didn't come, these B2B Web exchanges failed. Electronic markets are more complex and multifaceted, it seems, than traditional markets.

In multifaceted markets, the value of a product to one group depends on the value of another product to another one. To grasp a sense of that complexity, take this example from the physical world: A real estate agent's value to a house seller is a function of the agent's value as a house finder to home seekers.

Marco Iansiti, a professor at Harvard Business School, believes there are eight such principles that have to be honored if such a market--or market platform, as the jargon now has it--is to succeed.

Pricing. Prices have to be set so that the relevant communities are attracted to the market in appropriate numbers. An effective pricing strategy won't necessarily reflect the incremental costs on either side of a market, and establishing optimal prices that maximize profits are complex calculations. A magazine publisher may be prepared to subsidize readers through discounted subscriptions in order to attract more advertisers, but a change in subscription rates could change the numbers and demographics of the readers, thus affecting the price that can be charged for advertisements.

Operations. To thrive, markets have to generate more value than they cost. Many B2B electronic markets failed because they failed to integrate into their buyers' and sellers' own procurement and account management systems.

Technology. Enron's (otc: ENRNQ - news - people ) experience showed that it is difficult to have core technology expertise across a large number of markets. The risk-management-focused technology a bond market needs is very different from the matching algorithms that dating markets like Yahoo! (nasdaq: YHOO - news - people ) Personals rely on. And the best technology in the world won't matter if market participants can't connect to it. Two years ago eBay introduced third-party APIs, which now drive 30% of the site's revenue.

Liquidity. Achieve it first, then scale up. EBay is a textbook example of becoming pervasive among a small community with minimal investment in technology and operating infrastructure before growing.

Flexibility. EBay, Iansiti says, has gone through five generations of technology in its relatively short life. A market has to be able to evolve as it grows.

Adaption. Enron encouraged employees to target markets and go after them. EBay encourages its employees to follow its customers and build up the areas of its market that are working well. Just such an approach led to the creation of eBay Business, potentially a billion-dollar market in its own right. (See: "EBay's Newest $1 Billion Business.")

Leverage participants. EBay created easy-to-use tools and rules that made it simple for their buyers and sellers to clear their transactions.

Trust. Trustworthiness of the participants is an essential foundation of any successful market. EBay's distributed rating system for sellers lets it exercise a trust management system despite handling millions of transactions a day. Market owners should also not be market participants. Enron was a player in its own markets, eBay is not. Enron, of course, also developed bigger trust issues, but those turned bad into disastrous for its marketplace because it was such an important market participant.

The market-platform view extends the concept of a market to all sorts of business value and supply chains. Companies like Sony (nyse: SNE - news - people ), IBM (nyse: IBM - news - people ), Sun Microsystems (nasdaq: SUNW - news - people ) and Palm (nasdaq: PALM - news - people ) are applying the concepts to software marketing, where success depends on attracting both developers of applications and users of them. They manage their intellectual property as if it were a market place.

This approach is not without commercial risk. As David Evans of NERA Economic Consulting and Peter Passell of the Milken Institute say, the recent antitrust suits against Microsoft (nasdaq: MSFT - news - people ) "were directly related to marketing practices peculiar to two-sided markets."

Market platforms encourage concentration more than traditional ones. Multisided markets create network effects that increase the size at which the market will be most efficient. The more people that use eBay overall, the more it will be valued by individual buyers and sellers.

"This creates a considerable challenge for antitrust regulators to distinguish between actions that would undermine competition in one-sided markets yet serve the long-term interests of consumers in two-sided markets," say Evans and Passell.

85% of Web-based auction revenue goes to eBay, they note--a bigger share than Microsoft has of software sales.

Zynga

Zynga has seen phenomenal growth since its inception. They are now almost a 6 billion dollar brand soon to be IPO company. However, they are under a sustainability risk. The problem is that their games are too simple and it is not hard to make similar games with better graphics and better other attributes. I found at least a few example of such games that are coming to facebook very soon. These games are made by well known gaming companies such as Electronic Arts. Here is a discussion on one such game.

Computer science conference ranking

Link: ualberta's page

Pandora's plight

Pandora has an interesting business in constructing radio channels automatically based on some kind of user preference and/or content analysis. The idea is simple and appealing. Realization is hard and it requires advanced machine learning technology. I have been using this radio service for some time. However, it seems that their collection of music leaves a lot to be desired and is not as satisfying as one might expect. This is a typical problem in the collaborative filtering type of applications. In academic jargon, this can be considered as the problem of incorporating diversity with limited supply. Limited supply in this case is clearly a business limitation related to finance and the current state of music industry. However, Pandora definitely requires to address this problem.

Computer science good conferences in ML/IR/Data mining/Vision and NLP

1. NIPS.
2. COLT (Theory)
3. FOCS (Theory)
4. AIStat (Theory)
5. ICML (Application)
6. KDD (Data mining, applications)
7. SDM (Data Mining)
8. ICDM (Data mining)
9. SIGIR (IR)
10. WWW (IR)
11. CIKM (IR)
12. CVPR (Vision)
13. ECCV (Vision)
14. ICCV (Vision)
15. ACL (NLP)
16. EMNLP (NLP)

Learning to rank.

There have been tons of research papers on learning to rank. However, a critical problem of all these methods is that they are some kind of classification or regression techniques that are dependent on human judgement. While it is possible to obtain human judgement for large scale of data, it is hard to obtain good judgement for sufficient amount of search results. Behavioral signals such as clicks can play a role for using unsupervised data, however there are a lot of documents without any behavioral data. Hence, the real problem is actually harder than the theoretical problem. It is also to note that learning to rank is actually a static solution and in order to take the relevance to a new level simulations (such as what is used in economics) are the only useful tools. Fortunately, search engines can iterate through the experiment and can validate the changes using hypothesis testing kind of frameworks (Ron Kohavi, Roger Longbotham, and Toby Walker, Online Experiments: Practical Lessons, IEEE Computer, Vol 43, issue 9, pp. 82-85, Sept 2010. DOI.). This means although the theoretical work on learning to rank is interesting, the major revenue improvement for engines are coming from intelligent trial and error frameworks which has not been captured in theory yet. So interestingly, we perform many minor miracles through these trial and error, excessive use of brute force, and lots and lots of testing, and--so often- and so far -- it has been found to be quite enough. I would view this as something that requires more attention from the theory community.

Bing vs Google search

1. Bing follows almost exactly same underlying IR technology as Google. Hence, it may have similar results for many high frequency key words but Google will always have the advantage of larger traffic and it is unlikely that Bing's ML ranking can catch up Google in this.

2. Bing actually require more infrastructure work. The search engine results are not optimized for viewing in chrome browser. In general, they seem to be a bit slow and not as clean as Google at this moment. Added social "component" is something that Google does not yet have. But there is zero innovation in terms of technology. It will take 7 days for Google to make a better social "component" if they want with the already existing data. Only advantage of Bing is probably the special collaborations they can get with Facebook.

3. They have a bit new user interface. However, I dont think UI will be the prime component that can beat wonderful service that Google provides today. Typically, UI innovations are trivial. Even if it is not trivial it is easy to copy.

US patent systems!!

I only know about US patent system and hence I can not really comment on any other country's patent system. However, US patent system seems incredibly idiotic. It is typically written in a format that is quite unreadable. People apply many patents that are almost similar. Patents exist to protect big corporations. The infrastructure for patents cost quite a lot of money for any country and I feel that it is a huge waste of money and man hours. Anyway, I don't really care much about patents, but I can not restrain laughing seeing snippets of an Yahoo patents that gave them a right for any web search that ranks the search results!!

See this snippet :
"A search system provides search results to searchers in response to search queries and the search results are ranked. The ranking is determined by an automated ranking process in combination with human editorial input. A search system might comprise a query server for receiving a current query, a corpus of documents to which the current query is applied, ranking data storage for storing information from an editorial session involving a human editor and a reviewed query at least similar to the current query, and a rank adjuster for generating a ranking of documents returned from the corpus responsive to the current query taking into account at least the information from the editorial session."
and ... continuing
"Promotions and demotions might be absolute ("Rank this document first highest."), relative to itself ("Rank this document four positions higher than it would otherwise be."), or relative to another document ("Rank this document higher than this other document."). Other types of promotion/demotion might include "remove this document from consideration no matter what the automatic system suggests", "this set of documents are to be given equal (`tied") rankings", "do not rank this document higher than position P" for some integer P, or the like."

Advice for Computer Science College Students

I came across a blog article (http://www.joelonsoftware.com/articles/CollegeAdvice.html) written by Joel Spolsky on this topic and can not restrain myself saying a few words on his advices. His first advice was learn to write. I did not understand why a moderately good student will not learn writing. This is a skill people generally acquire in high school. His second advice was to learn c programming. I differ on this since I believe the general knowledge of programming languages concept is more useful than one programming language. One language is not enough and I would expect a computer science graduate will know C, an OOP language and one scripting language and a functional language. I would also expect one in his undergrad would write at least one non-trivial large system in any language. Example of non-trivial programs are compilers, interpreters, toy database, toy operating systems, toy distributed storage etc. Microeconomics is an interesting class. I dont know why Joel is particularly interested in that. I believe sociology, game theory classes are also interesting and are important.I do not think that programming should be focus. One should focus on learning the science. Programming is the way of expression and if one understands how things work, expression should not be problem. I felt that the summary of his article will be that his advice to a new graduate is that he/she should be well prepared to write c/x code in his company. Anyway, I dont think a smart student needs advice on what he/she needs to learn but he may be interested in one's view on what could be interesting in future in computer science.

A recent Google spam

Interestingly, www.ourglocal.com broke Google's ranking system. They are able to occupy multiple top ranks for academic queries. This is transient. I think Google will catch it probably in a week algorithmically or they will black list it very soon. However, it shows their ranking may not be hard to break. So, if one can hide IP and can create many random IPs for the attacks from cloud and then generate automated targeted contents, Google will find it absolutely hard to manage unless they do some fundamental change in their algorithm. This fundamental change is hard. I guess this kind of attacks require a lot of money (or talent to hack into clouds). I dont think the current web site is so sophisticated but they do launch a browser redirection script once someone visits their web site. I have not investigated further but this is an interesting ranking attack worth studying.

Mac OS output of ifconfig

It's a BSD copy. Here are a few things I discovered.


* lo0 = loopback
* gif0 = Software Network Interface
* stf0 = 6to4 tunnel interface
* en0 = Ethernet 0
* fw0 = Firewire
* en1 = Ethernet 1
* vmnet8 = Virtual Interface
* vmnet1 = Virtual Interface

How can an experienced engineer improve his/her skills?

Engineering knowledge does not necessarily grow with experience. One of the ways one can improve one's knowledge is working on non-trivial problems and in getting results faster. This is definitely riskier because in any corporate scenario the myth for strategic success is to work on low hanging fruits. However, I think only working on low hanging fruits is detrimental to an Engineer's career. Two things are important. The first one is the scientific knowledge in a domain and the second one is the ability to quickly express algorithmic thoughts in terms of mathematics and programming languages. One of the way to verify if one's programming skill is at per with the best people is to take on a programming olympiad question paper and solve the problems in a given time, send it to online judge and see your score. Also, one can verify the domain knowledge by choosing a moderately good theoretical paper in his/her domain and then attempting to solve that problem addressed in that paper by themselves. Later they can compare their solution with the solution given in the paper.

On QWiki

This is a new start up based on an idea of providing better information experience. They fetch Wikipedia data, extract topics and probably headers (named entities) then extract relevant images from either Wikipedia or from other sources. Then they construct a video with the still photos and then just convert a few Wikipedia sentences to speech. They got about 8.5 million dollar funding. This means that the investors believe that it can have at least 50/60 million dollar business. I like the demo and it is done quite neatly. However, I will need to see some negative cases to have more confidence in the implementation they had. In one glance the technology looks cool. However, it is doubtful how much business value they can have. It may be interesting for automatically constructing audio books for kids. I do not know how much the web site may be useful. May be it will be. It is yet to see.

NetBeans IDE

I was literally happy at Microsoft using visual studio for editing and debugging millions of lines of C/C++ code and was quite unhappy using eclipse so far whenever I have used the CDT. I actually preferred using the gvim or emacs (not for Java) than eclipse. However, recently I used NetBeans for java and found it a bit easier to manage compared to eclipse. Then I also wrote some scala code using NetBeans and the experience was quite good. The last thing I tried was using NetBeans to write a fairly nontrivial c++ code that also uses many external libraries. Surprisingly, the experience was smooth. I found that the NetBeans interfaces are much easier to understand compared to its eclipse counter part. I would recommend it strongly for Java, Scala and for C/C++.

When software quality is not the most important thing?

There are quite a few times when one does not need to adhere to a high software quality standard. For example, if you are realizing a concept and you want to iterate through the design to understand the product better. So you want to optimize the development for fast release. Nothing is more important than speed of pushing the product in production. In this scenario, you build the software as quick as possible avoiding any complex issues that can consume your time. This is exactly what is in practice in the internet industry for feature based or functional development. Of-course this is not always true. For example if Netflix offers you a slow web site with a crappy search engine with it, you will not enjoy the web site. However, still, one can focus on developing high quality softwares where it directly add value to the business. For all search engine companies, performance is a big deal and the code needs to be written carefully to allow fast innovation in the application layer. Code quality is important for that. Increasingly, there is a trend for using better and safer programming languages in the industry that takes care of a lot of concerns related to code quality. For example, if you have an application written in OCaml or Scala, they are definitely better suited for debugging. The point of this writing is that I realized increasingly in the internet industry one can safely ignore the aspects of code quality until and unless spending time in that gives some amount of value. Stroustrup is right in his article that we are in the industry is in perpetual rush and he commented that it is hard to work on code quality in such a rush. I somehow believe the language in future will take care of the quality issues to some extent and currently Industry rarely requires the kind of quality that Stroustrup had in his mind when he gave the interview to Business Insider.

Excerpts from Mystical man month

Found this note from a software engineering lecture notes and thought to post it here.

Mythical Man Month
Why does software fail?

Difficult to estimate time
Optimism
Effort != progress
Schedule not monitored
Slippage induces addition of people
Brooks says programmers are optimists (everything will go right etc...). Incompleteness and inconsistencies become clear only during implementation. He concludes that experimenting, "working out" are essential disciplines.

Each task has a nonzero probability of failure or slippage. Probability that all will go well is near zero.

Cost varies with manpower and resources, but progress does not! Hence, using "man month" (person month) as a measure is misleading and dangerous. They are interchangeable only when there is no interaction whatsoever between tasks.

Draw line downwards top left to lower right months (y) vs people (x) for a perfectly partionable task.
For semi-partionable task, show convex downward curve top left to bottom right
For unpartionable task, show flat line from same months vs people graph.
For partionable tasks that require communication, must add communication to the cost of completion. Communication is:

training (technology, goals, overall strategy, plan of work)
intercommunication; if each pair must communicate, cost grows with "n(n-1)/2". Meetings of >2 people makes it even worse!
For building a system (requires lots of communication), the communication effort quickly dominates the effort. Adding more people lengthens not shortens the schedule

Testing cost underestimated always. Brooks suggests:

1/3 planning (yes this costs a lot)
1/6 coding
1/4 component test, early system test
1/4 system test, all components in hand
TJP: don't forget that writing test harnesses can be almost as much work or sometimes more as writing the actual code.

Delays during final testing are very demoralizing!

Two reasons of failures:
Gutless estimating
Urgency of boss forces programmers to agree to unrealistic schedules.

It is very hard to defend an estimate (good or bad); people use "hunches"

How to check if opencv is properly installed in linux/mac

Replace 2.0 with whatever version you have installed. This is correct till 2.1.

In mac, if you have installed using macport then it should be opencv2.1 and it typically install them in /opt/local/lib.

In /usr/local/lib you should have the following libraries

libcvaux.so -> libcvaux.so.2.0
libcvaux.so.2.0 -> libcvaux.so.2.0.0
libcvaux.so.2.0.0
libcv.so -> libcv.so.2.0
libcv.so.2.0 -> libcv.so.2.0.0
libcv.so.2.0.0
libcxcore.so -> libcxcore.so.2.0
libcxcore.so.2.0 -> libcxcore.so.2.0.0
libcxcore.so.2.0.0
libhighgui.so -> libhighgui.so.2.0
libhighgui.so.2.0 -> libhighgui.so.2.0.0
libhighgui.so.2.0.0
libml.so -> libml.so.2.0
libml.so.2.0 -> libml.so.2.0.0
libml.so.2.0.0

And in /usr/local/include/opencv you should have the following header files.

cvaux.h, cvcompat.h, cv.hpp, cvver.h, cvwimage.h, cxcore.hpp, cxflann.h,
cxmisc.h, cxtypes.h, highgui.hpp, cvaux.hpp, cv.h, cvtypes.h, cvvidsurv.hpp,
cxcore.h, cxerror.h, cxmat.hpp, cxoperations.hpp, highgui.h, ml.h

Uninstalling OpenCV older version needs uninstalling the packages and then manually removing the above before making a clean installation.

Bjarne Stroustrup's interview

I found an interview of Stroustrup very compelling and this is in the same direction of my thoughts on similar topics. Hence I thought to keep his entire interview in my blog.

Technology Review: Why is most software so bad?

Bjarne Stroustrup: Some software is actually pretty good by any standards. Think of the Mars Rovers, Google, and the Human Genome Project. That's quality software! Fifteen years ago, most people, and especially most experts, would have said each of those examples was impossible. Our technological civilization depends on software, so if software had been as bad as its worst reputation, most of us would have been dead by now.

On the other hand, looking at "average" pieces of code can make me cry. The structure is appalling, and the programmers clearly didn't think deeply about correctness, algorithms, data structures, or maintainability. Most people don't actually read code; they just see Internet Explorer "freeze."

I think the real problem is that "we" (that is, we software developers) are in a permanent state of emergency, grasping at straws to get our work done. We perform many minor miracles through trial and error, excessive use of brute force, and lots and lots of testing, but--so often--it's not enough.

Software developers have become adept at the difficult art of building reasonably reliable systems out of unreliable parts. The snag is that often we do not know exactly how we did it: a system just "sort of evolved" into something minimally acceptable. Personally, I prefer to know when a system will work, and why it will.

TR: How can we fix the mess we are in?

BS: In theory, the answer is simple: educate our software developers better, use more-appropriate design methods, and design for flexibility and for the long haul. Reward correct, solid, and safe systems. Punish sloppiness.

In reality, that's impossible. People reward developers who deliver software that is cheap, buggy, and first. That's because people want fancy new gadgets now. They don't want inconvenience, don't want to learn new ways of interacting with their computers, don't want delays in delivery, and don't want to pay extra for quality (unless it's obvious up front--and often not even then). And without real changes in user behavior, software suppliers are unlikely to change.


We can't just stop the world for a decade while we reprogram everything from our coffee machines to our financial systems. On the other hand, just muddling along is expensive, dangerous, and depressing. Significant improvements are needed, and they can only come gradually. They must come on a broad front; no single change is sufficient.

One problem is that "academic smokestacks" get in the way: too many people push some area as a panacea. Better design methods can help, better specification techniques can help, better programming languages can help, better testing technologies can help, better operating systems can help, better middle-ware infrastructures can help, better understanding of application domains can help, better understanding of data structures and algorithms can help--and so on. For example, type theory, model-based development, and formal methods can undoubtedly provide significant help in some areas, but pushed as the solution to the exclusion of other approaches, each guarantees failure in large-scale projects. People push what they know and what they have seen work; how could they do otherwise? But few have the technical maturity to balance the demands and the resources.

BS: C++ wasn't designed specifically for the large switching machines, but for a huge range of applications. Bell Labs was the home of an incredible range of interesting projects spanning every scale and using essentially every kind of computer and operating system. But yes, the average Bell Labs programmer was significantly more able than most people's notion of an "average programmer," and reliability and performance (in that order) were considered significantly more important than in most other places.

Performance is still an issue in many of the applications that I'm interested in: responsiveness of interfaces, start-up and close-down time of applications. Software developers have neutralized the astounding performance of modern computer hardware by adding layer upon layer of overelaborate [software] abstractions. We seem to have hit the limits of linear speedup for hardware, but in many cases, we could win a couple of orders of magnitude back from the software.

That said, C++ has indeed become too "expert friendly" at a time where the degree of effective formal education of the average software developer has declined. However, the solution is not to dumb down the programming languages but to use a variety of programming languages and educate more experts. There has to be languages for those experts to use--and C++ is one of those languages.

TR: In retrospect, in designing C++, wasn't your decision to trade off programmer efficiency, security, and software reliability for run time performance a fundamental mistake?

BS: Well, I don't think I made such a trade-off. I want elegant and efficient code. Sometimes I get it. These dichotomies (between efficiency versus correctness, efficiency versus programmer time, efficiency versus high-level, et cetera.) are bogus.

What I did do was to design C++ as first of all a systems programming language: I wanted to be able to write device drivers, embedded systems, and other code that needed to use hardware directly. Next, I wanted C++ to be a good language for designing tools. That required flexibility and performance, but also the ability to express elegant interfaces. My view was that to do higher-level stuff, to build complete applications, you first needed to buy, build, or borrow libraries providing appropriate abstractions. Often, when people have trouble with C++, the real problem is that they don't have appropriate libraries--or that they can't find the libraries that are available.

Other languages have tried to more directly support high-level applications.

That works, but often that support comes at the cost of specialization. Personally, I wouldn't design a tool that could do only what I wanted--I aim for generality.

TR: How do you account for the fact that C++ is both widely criticized and resented by many programmers but at the same time very broadly used? Why is it so successful?
BS: The glib answer is, There are just two kinds of languages: the ones everybody complains about and the ones nobody uses.

There are more useful systems developed in languages deemed awful than in languages praised for being beautiful--many more. The purpose of a programming language is to help build good systems, where "good" can be defined in many ways. My brief definition is, correct, maintainable, and adequately fast. Aesthetics matter, but first and foremost a language must be useful; it must allow real-world programmers to express real-world ideas succinctly and affordably.

The main reason for C++'s success is simply that it meets its limited design aims: it can express a huge range of ideas directly and efficiently. C++ was not designed to do just one thing really well or to prevent people doing things considered "bad." Instead, I concentrated on generality and performance.

I'm sure that for every programmer that dislikes C++, there is one who likes it. However, a friend of mine went to a conference where the keynote speaker asked the audience to indicate by show of hands, one, how many people disliked C++, and two, how many people had written a C++ program. There were twice as many people in the first group than the second. Expressing dislike of something you don't know is usually known as prejudice. Also, complainers are always louder and more certain than proponents--reasonable people acknowledge flaws. I think I know more about the problems with C++ than just about anyone, but I also know how to avoid them and how to use C++'s strengths.

And then, of course, you don't expect proponents of languages that lost out in competition with C++ to be polite about it. Software development doesn't have that degree of professionalism--though I hope it eventually will. Science is different in this respect: when a new tool, technique, or theory wins out, people see that as progress. In software, contributions by competitors and predecessors are not widely acknowledged, appreciated, or even understood.

TR: In The Design and Evolution of C++, you claim that Kierkegaard was an influence on your conception of the language. Is this a joke?

BS: A bit pretentious, maybe, but not a joke. A lot of thinking about software development is focused on the group, the team, the company. This is often done to the point where the individual is completely submerged in corporate "culture" with no outlet for unique talents and skills. Corporate practices can be directly hostile to individuals with exceptional skills and initiative in technical matters. I consider such management of technical people cruel and wasteful. Kierkegaard was a strong proponent for the individual against "the crowd" and has some serious discussion of the importance of aesthetics and ethical behavior. I couldn't point to a specific language feature and say, "See, there's the influence of the nineteenth-century philosopher," but he is one of the roots of my reluctance to eliminate "expert level" features, to abolish "misuses," and to limit features to support only uses that I know to be useful. I'm not particularly fond of Kierkegaard's religious philosophy, though.

TR: What do you regret the most?


BS: No regrets! Well, of course I dream of what I might have done differently and better, but seriously, who am I to second-guess, say, 1984 vintage Bjarne? He may have been less experienced than I, but he was no less smart, probably smarter, and he had a better understanding of the word of 1984 than I have. C++ has been used to build many systems that enhance our lives, and it has been a significant positive influence on later languages and systems. That's something to be proud of.

Technology Review: Name the coolest and lamest programs ever written in C++, and say what worked and didn't work.



Bjarne Stroustrup: Google! Can you even remember the world before Google? (It was only five years ago, after all.) What I like about Google is its performance under severe resource constraints. It possesses some really neat parallel and distributed algorithms. Also, the first Web browsers. Can you imagine the world without the Web? (It was only about 10 years ago.) Other programs that I find cool are examples of embedded-systems code: the scene-analysis and autonomous driving systems of the Mars Rovers, a fuel-injection control for a huge marine engine. There is also some really cool code in Photoshop's image processing and user interfaces. What I like about these programs is how they are structured to be reliable and responsive under pretty harsh resource constraints. Some of Photoshop's ways of managing internal complexity (for instance, the graphical user interface [GUI] layout and access from image-processing algorithms to the pixel data) are just beautiful.

If you look at code itself, rather than considering the resulting program, we could look at something like Shape [an abstract class that defines an interface by which shapes like circles and rectangles can be manipulated in C++], which I borrowed from Simula. It's still the language-technical base of most GUIs, such as the one on your computer or your iPod, or whatever. A more modern example would be the find or the sort algorithm in the Standard Template Library [STL] in C++. That's the language technical basis for much modern high-performance C++ code, especially of programs that need to do simple operations on lots of data. What is common to these examples of code is that they cleanly separate concerns in a program, allowing separate developments of parts. That simplifies understanding and eases maintenance. These basic language techniques allow separate "things" to be separately represented in code and combined only when needed. However, code is something appreciated by programmers, rather than most people. I have always been a bit envious of graphics people, robotics people, etc. What they do is so concrete and visible; what I do is invariably invisible and incomprehensible to most people. I know many mathematicians feel the same way.

Sorry, I'm not going to shame anyone by naming their work "the lamest C++ program ever." It's oh so tempting . . . but no, it wouldn't be fair.

TR: Following structured programming (of which the best known example is Pascal) and object-oriented programming (your own C++), what will be the next big conceptual shift in the design of programming languages? Some years ago, Technology Review put its money on aspect-oriented programming. Does AO represent a conceptual shift of the same kind that OO did?

BS: I hope you didn't put too much money on it! I don't see aspect-oriented programming escaping the "academic ghetto" any day soon, and if it does, it will be less pervasive than OO. When it works, aspect-oriented programming is elegant, but it's not clear how many applications significantly benefit from its use. Also, with AO, it appears difficult to integrate the necessary tools into an industrial-scale programming environment.

One reason I am cautious is that I saw an earlier language built on similar ideas, called R++, struggle to gain acceptance some 15 years ago. It did well for a major application, but repeatedly, enthusiastic people discovered [that] introducing major changes into their tool chains and processes was complicated and expensive, and that educating new developers was difficult and time-consuming. Naturally, aspect-oriented programming may avoid some of these problems, but to succeed it needs to dodge all, and more. In computer science, a major new idea will succeed only if it is sufficiently capable in every relevant area. To succeed on a large scale, a new language must be good enough in all areas (even some the language designer has never heard of), rather than just superb at one or two things (however important). This is a variant of the simple rule that to function, all essential parts of a machine must work; remove one, and the machine stops. The trick is to know which parts are essential. Please note that I'm not saying, "Aspect-oriented programming doesn't work," but to be "the next big thing," you have to provide major gains in an enormously broad range of application areas.
All that said, I don't know what the next major conceptual shift will be, but I bet that it will somehow be related to the management of concurrency. As programmers, we have been notoriously bad at thinking about lots of things happening simultaneously, and soon our everyday computers will have 32 cores.

You didn't mention generic programming. It is definitely worth thinking about in this context. Over the last decade, it has made a major change to the way C++ libraries are designed and has already led to the addition of language features in Java and C#. I don't think of generic programming as the "next paradigm," because C++ has directly supported it since about 1990, and the C++ standard library crucially depends on it. What if the next big thing has already arrived and nobody (except programmers) noticed?

It's worth noting, perhaps, that I don't actually believe in a popular [use] of the word "paradigm" (that derives from Thomas Kuhn's ideas in The Structure of Scientific Revolutions) when it is applied to programming. I do not believe that a paradigm completely replaces previous paradigms in one revolutionary moment (or "shift"). Instead, each programming paradigm adds to what worked previously, and as a paradigm matures, it is increasingly integrated with previous paradigms. Kristen Nygaard was fond of saying that multiplication didn't completely replace addition, and, by analogy, whatever would come after object-oriented programming would include object-oriented programming as a subset. I tend to agree. The evolution of C++ was guided by this view, and the evolution of Java and C# provides further examples.

TR: Computer languages remain generally difficult to learn. One might argue that for computers to become more than "helper" tools that enable mass computations and widespread communications, they must evolve again--and one key may be in simplifying the process of coding so that more individuals are able to participate in development.

BS: I think that would be misguided. The idea of programming as a semiskilled task, practiced by people with a few months' training, is dangerous. We wouldn't tolerate plumbers or accountants that poorly educated. We don't have as an aim that architecture (of buildings) and engineering (of bridges and trains) should become more accessible to people with progressively less training. Indeed, one serious problem is that currently, too many software developers are undereducated and undertrained.

Obviously, we don't want our tools--including our programming languages--to be more complex than necessary. But one aim should be to make tools that will serve skilled professionals--not to lower the level of expressiveness to serve people who can hardly understand the problems, let alone express solutions. We can and do build tools that make simple tasks simple for more people, but let's not let most people loose on the infrastructure of our technical civilization or force the professionals to use only tools designed for amateurs.
In any case, I don't think it is true that the programming languages are so difficult to learn. For example, every first-year university biology textbook contains more details and deeper theory than even an expert-level programming-language book. Most applications involve standards, operating systems, libraries, and tools that far exceed modern programming languages in complexity. What is difficult is the appreciation of the underlying techniques and their application to real-world problems. Obviously, most current languages have many parts that are unnecessarily complex, but the degree of those complexities compared to some ideal minimum is often exaggerated.

We need relatively complex language to deal with absolutely complex problems. I note that English is arguably the largest and most complex language in the world (measured in number of words and idioms), but also one of the most successful.

TR: Please talk about the pros and cons of maintaining backward compatibility with the existing knowledge base (for example, consider your determination to maintain high compatibility to C when you developed C++). It might seem that a clean break would produce leaps of progress; but is this really a realistic proposition?

BS: Java shows that a (partial) break from the past--supported by massive corporate backing--can produce something new. C++ shows that a deliberately evolutionary approach can produce something new--even without significant corporate support. To give an idea of scale: I don't know what the marketing budget for Java has been so far, but I have seen individual newspaper advertisements that cost more than the total of AT&T's C++ marketing budget for all time. "Leaps" can be extremely costly. Is such money well spent? Maybe from the point of view of corporate budgets and corporate survival, but given a choice (which of course I'll never have), I'd spend that money on research and development of evolutionary changes. Note that almost by definition, research money is used to fund attempted leaps that tend to fail while competing with evolutionary changes.

However, evolution shouldn't be an excuse for doing things the way we've always done them. I would like for evolutionary changes to occur at a faster pace than they do in C++, and I think that's feasible in theory. However, that would require funding of "advanced development," "applied research," and "research into application" on a scale that I don't see today. It would be essential to support the evolution of languages and libraries with tools to ease upgrades of real systems and tools that allowed older applications to survive in environments designed for newer systems. Users must be encouraged to follow the evolutionary path of their languages and systems. Arrogantly damning older code as "legacy" and recommending "just rewrite it" as a strategy simply isn't good enough. "Evolutionary languages" tend to be very conservative in their changes because there is no concept of supporting upgrades. For example, I could imagine accepting radical changes in source code if the change was universally supported by a solid tool for converting from old style to new style. In the absence of such tools, language developers must be conservative with the introduction of new features, and application developers must be conservative with the use of language features.

One problem with an evolutionary approach is that there are few academic outlets for incremental change--especially not when that incremental change directly relates to real-word systems. This is a sign that computer science and software engineering haven't yet emerged as the theoretically and empirically well-founded foundation of real-world software development. Theory and practice seem rarely to meet, and researchers are pushed away from real-world software towards less messy academic topics. Conversely, many developers completely ignore research results.
A major issue in language evolution is control of its direction. For a new language, users have a choice: they can adopt it or not (though this becomes less clear-cut if the new language is heavily pushed by their platform supplier). But once they use a language, it is harder to ignore new features (e.g., one user might not like a feature, but a colleague or a library supplier may think it great and use it). And it is impossible to ignore incompatible changes. The ISO C++ standards process relies on volunteers, who can devote only part of their time to the standards work. This implies that it is slow moving. It also seems to imply that end users are consistently underrepresented compared to suppliers of compilers and tools. Fortunately, the C++ committee has always been able to attract many dozens of active members for its meetings and many more online, so it avoided the parochialism of a small group. Until very recently, academics have been completely absent.

C++ provides a nice, extended case study in the evolutionary approach. C compatibility has been far harder to maintain than I or anyone else expected. Part of the reason is that C has kept evolving, partially guided by people who insist that C++ compatibility is neither necessary nor good for C. Another reason-- probably even more important--is that organizations prefer interfaces that are in the C/C++ subset so that they can support both languages with a single effort. This leads to a constant pressure on users not to use the most powerful C++ features and to myths about why they should be used "carefully," "infrequently," or "by experts only." That, combined with backwards-looking teaching of C++, has led to many failures to reap the potential benefits of C++ as a high-level language with powerful abstraction mechanisms.

Python image display

PIL has a weaker way to display images since it calls xv and there is no interaction possible with the image window.

I found a neat code example with Tkinter and have modified that to get image ground truth.

import os, sys
import Image, ImageTk
import Tkinter

filename = "test.dat"
FILE = open(filename,"a")

def write_to_file(txt):
    FILE.write(txt)

def button_click_exit_mainloop (event):
    write_to_file('\t'+event.char)
    event.widget.quit() # this will cause mainloop to unblock.


root = Tkinter.Tk()
root.bind("", button_click_exit_mainloop)
root.geometry('+%d+%d' % (100,100))


dirlist = os.listdir('.')
old_label_image = None
for f in dirlist:
    try:
        image1 = Image.open(f)
        write_to_file(f)
        root.geometry('%dx%d' % (image1.size[0],image1.size[1]))
        tkpi = ImageTk.PhotoImage(image1)
        label_image = Tkinter.Label(root, image=tkpi)
        label_image.place(x=0,y=0,width=image1.size[0],height=image1.size[1])
        root.title(f)
        if old_label_image is not None:
            old_label_image.destroy()
        old_label_image = label_image
        root.mainloop() # wait until user clicks the window
    except Exception, e:
        pass
FILE.close()

 

Variants of binary search.

It is possible to derive algorithms from the problem by considering the invariants. Lets apply

this technique to derive some variants of binary search.

Question: Find out the first occurrence of a key from a sorted array?

This is a variation of binary search and is not very trivial. To synthesize this algorithm let's see what invariant we may be looking for:

Clearly, if l=lower and u=upper, we are looking for a situation where x[l]=key and l+1=u in the post condition.

Obviously the precondition can be using only the first two constraints.

So we have a proof of correctness from the two conditions.

Now we can write the algorithm (courtesy Benteley)

def binsearch_1st_log(list,k):
    l=-1
    u=len(list)
    while l+1 != u:
        m=(l+u)/2
        if list[m]            l=m
        else:
            u=m
    p=u
    if p >= len(list) or list[p] != k:
          p=-1
    return p

  

This is a neat python implementation. This is also an efficient implementation of binary search.

Question 2: Search a sorted array for the first element larger than k.

The loop invariant for the above problem can be considered as the following:

x[l]<=key and x[u]>key.

The post condition should have an additional constraint l+1 = u.

This gives a very similar code as the first one:

 
def binsearch_var2(list,k):
    l=-1
    u=len(list)
    while l+1 != u:
        m=(l+u)/2
        if list[m]<=k:
            l=m
        else:
            u=m
    if u>0 and list[u-1] != k:
       return -1
    return u

Moderately good problems in computer science.

It seems that there is no lack of important moderately difficult problems in Computer Science that require about 1 or 2 days of thinking and about a months of implementation. Lets talk about one such example problem:

Problem 1: Find near duplicate documents?

Application: Search.

One reasonable problem definition: make hash of random points from the documents such that if x1 is a near duplicate of x2, both x1,x2 will have same hash.

A reasonable solution: Make a 128 bit MD5 of the document (better, do that for paragraphs, sentences, n-grams). Choose part of the hashes from the parts of the documents in an order to increase probability of matching near duplicates.

Broader's paper: This paper is actually a good paper since it not only provides a reasonable solution, it defined the problem nicely and gave a solution with a bound for near duplicate matching. Similar techniques are implemented in Goog and in MSFT and in many other search companies for their production near duplicate detection tasks.

You can read Broader's slides here.

Excerpts for understanding developments in Programming Languages

Last fifty years in computer science research has produced a series of fascinating innovation in programming languages. It started with Church and Turing's work on lamda calculus. In 1954 John Backus assembled a team to define and develop Fortran for the IBM 704 computer. Fortran was the first high level programming language to be put to broad use. Later John McCarthy rediscovered lamda calculus and created Lisp

In the late 1950s and 1960s, ALGOL was developed in order to allow mathematical algorithms to be more easily expressed, and even served as the operating system’s target language for some computers. COBOL (1960) and BASIC (1964) were both attempts to make programming syntax look more like English. In the 1970s, Pascal was developed by Niklaus Wirth, and C was created by Dennis Ritchie while he was working at Bell Laboratories.

The 1980s saw a rapid growth in interest in object-oriented programming. These languages were im- perative in style, but added features to support objects. The last two decades of the 20th century saw the development of a considerable number of such programming languages. Smalltalk-80, originally conceived by Alan Kay in 1969, was released in 1980 by the Xerox Palo Alto Research Center. Drawing from concepts in another object-oriented language – Simula (which is considered to be the world’s first object-oriented programming language, developed in the late 1960s)Bjarne Stroustrup designed C++, an object-oriented language based on C. C++ was first implemented in 1985. In the late 1980s and 1990s, the notable im- perative languages drawing on object-oriented concepts were Perl, released by Larry Wall in 1987; Python, released by Guido van Rossum in 1990; PHP, released by Rasmus Lerdorf in 1994; Java, first released by Sun Microsystems in 1994 and Ruby, released in 1995 by Yukihiro Matsumoto. Functional programming languages, especially purely functional ones, have largely been emphasized in academia rather than in commercial software development. However, prominent functional programming languages such as Scheme, Erlang, OCaml, and Haskell, have been used in industrial and commercial ap- plications by a wide variety of organizations. Functional programming also finds use in industry through domain-specific programming languages like R (statistics), Mathematica (symbolic math), J and K (financial analysis), and XSLT (XML). Widespread declarative domain specific languages like SQL and Lex/Yacc, use some elements of functional programming, especially in eschewing mutable values. Spreadsheets can also be viewed as functional programming languages. John Backus presented the FP programming language in his 1977 Turing Award lecture “Can Program- ming Be Liberated From the von Neumann Style? A Functional Style and its Algebra of Programs”. He defines functional programs as being built up in a hierarchical way by means of “combining forms” that allow an “algebra of programs”; in modern language, this means that functional programs follow the principle of compositionality. Backus’s paper popularized research into functional programming, though it emphasized function-level programming rather than the lambda-calculus style which has come to be associated with functional programming. grew. By 1987, more than a dozen non-strict, purely functional programming languages existed. Of these, Miranda was the most widely used, but was not in the public domain. At the conference on Functional Programming Languages and Computer Architecture (FPCA ’87) in Portland, Oregon, a meeting was held during which participants formed a strong consensus that a committee should be formed to define an open standard for such languages. The committee’s purpose was to consolidate the existing functional languages into a common one that would serve as a basis for future research in functional-language design. The first version of Haskell (”Haskell 1.0”) was defined in 1990. The committee’s efforts resulted in a series of language definitions. In late 1997, the series culminated in Haskell 98, intended to specify a stable, minimal, portable version of the language and an accompanying standard library for teaching, and as a base for future extensions. Following paper has a comparative study on various programming language implementations http://arxiv.org/pdf/1009.0305.

Theory, real world and two remarkable pieces of work

Theories are constructed with reasonable assumption that one or a few human brains can manage. Real world is full of noisy observations. A beautiful theory considers this noise and handles it. As an example I would like to talk about two most important contributions from two of the best scientists and engineers of the century. The first contribution was by Shanon. In his 1984's paper "A mathematical theory of communication." I would like to quote a few sentences from NASA's web page on Shanon's contribution since I could not say anything better than the article has said about this phenomenal work.

"Many concepts that seem obvious today were first introduced in his paper, "A Mathematical Theory of Communication," published in two parts in the Bell System Technical Journal. The use of the term "bit" as a unit of information first appeared in this paper. In an age in which communication took place with continuous, analog waveforms, it was a startling notion that information traveling across any communications system could be defined mathematically as some quantity of binary symbols. It was entirely new that information of any kind-- whether for use on a telegraph, telephone, radio, or television-- could be decomposed into zeros and ones, encoded, transmitted, and decoded at the other end.

That was groundbreaking, but it was only one component of his theory. He went on to present the concept of the maximum rate of transmission on a channel-- the capacity or "Shannon limit"-- which provides the benchmark against which all codes and modulations are measured. He invented adding redundancy to a transmitted signal in order to enable correcting for transmission errors at the receiving end. He provided the foundation of data compression, years ahead of any widespread implementations of digital communications or storage.

In this one paper, Shannon founded the field of information theory. In a world in which no one knew the concept of a bit, asking the question, "How many bits per second can be transmitted a channel?" and doing nothing more would have been momentous. Shannon did this and proceeded to clearly state the answer, prove it, and begin to show how to design systems that achieve the limit. He presented a clear picture of the whole field of information theory, all at once."

The second person I would like to mention is Jon Von Neumann's classic work on introducing theory of games and economic behavior. Again I would like to quote from an article from Princeton university press on his classic work.

"This is the classic work upon which modern-day game theory is based. What began more than sixty years ago as a modest proposal that a mathematician and an economist write a short paper together blossomed, in 1944, when Princeton University Press published Theory of Games and Economic Behavior. In it, John von Neumann and Oskar Morgenstern conceived a groundbreaking mathematical theory of economic and social organization, based on a theory of games of strategy. Not only would this revolutionize economics, but the entirely new field of scientific inquiry it yielded--game theory--has since been widely used to analyze a host of real-world phenomena from arms races to optimal policy choices of presidential candidates, from vaccination policy to major league baseball salary negotiations. And it is today established throughout both the social sciences and a wide range of other sciences."

The characteristic of both of these great work is keen observation to identify a simplified theory of complex systems that revolutionizes science and engineering and enormously reduces the gap between the abstract world of theory and real world.

Both information theory and game theory have been proven to be applicable in modeling many complex engineering systems till date and will continue to do so.

Share holder's value and product quality

Optimizing a development process for quarterly gain can be quite dangerous for long term growth of a company. However, a steady growth is also important. This is not difficult when the scope of the firm is small and focused and it becomes harder when the scope increases and the company started diversifying. In a lot of articles, I found that people often correlate share holder's value with quarterly profit. It is true that finance companies optimize their portfolios that way but a share holder's value should be judged with the promise of long term sustainable growth that the firm is bringing every quarter to its share holders. Often the specific attribute that gets sacrificed for rushing for quarterly gain is the "quality". Quality of a product from a firm is actually the most important aspect that can make the firm stand out in a competitive market. Hence, a development process needs to be optimized for making high quality product. The metrics for quality should be chosen very carefully and should be fully result oriented and independent of the process.

Technology and medical

Technology can be very powerful and it can create amazing result in changing the way people see, the way people live. We are amidst an era of a revolutionary change in consumer devices technology. It is only the beginning and I think we will see the change happening in a few years from now. However, I feel the medical technology today is completely controlled by a bunch of selfish pharma companies and that needs to be changed. Open source movement should happen in drug discovery in order to bring that change.

On important problems

There are two important question. What are the attributes that make a problem important and what are some examples of such problems. First question is a hard one and can be answered when one can try analyzing a few important results published in last few decades. For example, I consider Shannon's development of information theoretical framework, Daubechis's work on wavelets, Valiant's PAC learning framework, Vapnik's work on VC dimension, Friend and Schapire's work on adaboost are pieces of a few of many contributing work developed in last few decades. Often these results have far reaching consequences in Engineering and Science and they were developed based on keen observation on real life problems and then constructing a reasonable mathematical (or I prefer to say algorithmic) framework for addressing the problems.