The Theory Of Emergence–Dumb Individuals, Smart Crowds

In philosophy, systems theory, science, and art, The concept of Emergence describes the way complex systems and patterns arise out of a multiplicity of relatively simple interactions.  Emergence has been around for a long time – at least since Aristotle - and been applied to science, philosophy, systems analysis and sociology at least since Aristotle. The best and most frequently cited example of emergence is that of an ant colony:

An example to consider in detail is an ant colony.  The queen does not give direct orders and does not tell the ants what to do. Instead, each ant reacts to stimuli in the form of chemical scent from larvae, other ants, intruders, food and build up of waste, and leaves behind a chemical trail, which, in turn, provides a stimulus to other ants. Here each ant is an autonomous unit that reacts depending only on its local environment and the genetically encoded rules for its variety of ant.

Despite the lack of centralized decision making, ant colonies exhibit complex behavior and have even been able to demonstrate the ability to solve geometric problems. For example, colonies routinely find the maximum distance from all colony entrances to dispose of dead bodies (Wikipedia)

Emergence has gained in popularity thanks to Big Data.  Although individuals may be dumb, aggregations of them are not. Francis Galton (1822-1911) a British anthropologist once attended a county fair and was drawn to a popular raffle. Visitors were asked to guess the weight of a prize ox on display, and nearly 800 people had offered their estimates.  Galton was not so much interested in the winning guess, but the average of all guesses.  To his surprise, the average was within a pound of the actual weight, far closer than the winning guess or of separate estimates made by cattle experts.

The Jelly Bean Experiment is done in psychology and sociology classes throughout the US.  Students are shown a large glass canister filled with jelly beans, and they are asked to guess the number.  As in the case of the ox, the average of all guesses was far closer to the actual number than any individual guess.  More importantly even when mathematicians provided calculated estimates based on geometrical measurements, they fell far short of the average of all uninformed guesses.

Crowdsourcing builds on this phenomenon. Research labs are now realizing that they do not have to rely on their in-house mathematicians for solutions to complex problems.  If they ask a million random individuals to solve a problem, the chances are greater of success than if they turn only to their ‘experts’.  Google, the master of emergence (no one in Cupertino is deciding on which websites are the most popular. That ranking is based only on numbers of hits) went to crowdsourcing to look for improvements on its search engine and offered a significant prize to anyone who could provide innovative changes.  Credentials, experience, or publications did not matter.  Anyone was eligible.

Google made its crowdsourcing approach perfectly clear when its head of research confronted the well-known cognitive researcher, Noam Chomsky, at a recent Artificial Intelligence conference at MIT.  Chomsky insisted that in order to create language recognition programs, the key to Artificial Intelligence, we must understand the nature of language and how the brain works.  Humbug, said the Google scientist.  All we need to do is to find out how language is used, and from that study build intelligent programs. In other words, the nature of language can be built by tracking the usage of trillions of bits of language-based data.

The theory of emergence is universally accepted.  Aggregations of random ‘dumb’ individuals – such as ants – can indeed create complex societies and carry out sophisticated tasks.  As cited above, ants together can solve difficult problems of geometry.  Currently researchers at Stanford are trying to determine exactly how ants find the shortest distance between two points (colony to food); and how they decide on when to move the colony to another location.

Most research has focused on pheromones – scents that are used to enable ant-to-ant communication.  Such scents may enable individual ants to pass on information about where, how far, and how valuable is a new food supply.  More difficult problems, such as where to carry the bodies of dead ants – in a spot equidistant from all colony entrances – are much more complex, and although pheromones play a role, they do not sufficiently explain the behavior.

Emergence has caught on for two reasons.  First, it is fascinating to think that random, leaderless individuals can produce complex societies, build sophisticated architecture, and behave productively, cooperatively, and efficiently. Could it be possible to envisage human societies without leaders? American utopian societies from Oneida to Sixties hippie communes have all experimented with communalism; and early New England was the home of ‘pure’ democracy embodied in participatory town meetings.

Our current fling with individualism is conservative in the truest sense of the word, for we long to go back to the rugged days of the frontier, the first colonies, and a life free from the overweening influence of government and corporations.  What could be better than a return to that simpler life, one without corrupting influences from above?  Emergence is the ideal Libertarian political philosophy – the people know best; and in the words of Thomas Jefferson, “That government is best that governs least”.

Emergence also appeals to believers in New Age mysticism, of which the best example is perhaps ‘Gaia Theory’, the belief that every bit of the world is connected to every other.  The Earth is a living, breathing, interconnected colony of life. Every action has consequences for the entire system.  More than an ant colony directed by pheromones, animal genetics, and behavior theory, the Earth is a spiritual colony.  God is the Earth, and the Earth is God. 



Finally emergence is gaining in popularity because of the rise of Big Data.  We are now able to collect an almost inconceivable amount of information and create algorithms to make sense out of it.  Why rely on experts, pundits, or crude social surveys and focus groups, when we have access to everyone’s opinion.  In one fell swoop, Big Data can provide a more rational basis for decision-making, and get rid of the pointy-headed intellectuals and Sunday morning blabbermouths. 

Nate Silver is America’s hero because he uses data to predict the outcome of American elections and has been extremely successful in doing so.  Who needs political scientists, historians, and political analysts? We need them no more than Google needs Noam Chomsky or others who insist on answering the question ‘Why’.  We no longer need to know why, only what.

New urbanists champion unplanned emergent cities. Neighborhoods come and go based on the collective decisions of individuals and the laws of supply and demand.  There is no need for city planners, zoning commissions, or officials designers.  A familiar example of this approach is found in footpaths.  When residents of an urban neighborhood find that a planned network of sidewalks and paths does not conform to their preferences, they create their own network – informal tracks that go where they want to go.

Atheists love emergence because it confirms their belief that there is no Divine Intelligence, no super-smart Supreme Being behind life.  Even the most complex systems – the ones at which even the most committed rationalists shake their heads- are nothing more than emergent phenomenon.  There are no leaders in ant colonies or bee hives, and no God in the heavens.

Critics of emergence, Big Data, and the rise of crowdsourcing lament what they see is the passing of individual creativity and genius. Everything from sculpture to movies will soon rely on nothing more than audience preference survey data.  Netflix has already moved in that direction.  The popular House of Cards was created largely on the basis of the reams of data collected from individual streaming viewers who ‘told’ Netflix of their preferences for movie themes, directors, actors, action, and love scenes.

For these humanists the idea that we are all are nothing but ants in a vast colony is frightening to say the least.  It is the worst of Behaviorism, Atheism, and Right Wing Conservatism combined.  The worst, however, is yet to come.  When our brains are soon connected to the computer and a virtual world replaces the real one, human activity will be nothing but information, data, and interconnectivity.  We will all be part of a random, universal electronic network.  On our own, and free from God, Obama, and McDonalds.

I for one am a big fan of emergence, Big Data, and virtuality.  There is nothing special about human beings as a group or any individual among them.  We all follow instructions from our genes, from pheromones, and from the environment.  Accidentally we create complex systems and profit and benefit from them.  Let’s just not take too much credit.