Nick Bostrom, in a wonderful Ted Talk (below), muses about the problem of emerging superintelligence within the human society, so far calibrated to the distribution of biological brains. From the village idiot to Ed Witten.
In “God and Golem”, Wiener evocatively captured that in the story of “Monkey Pawn”, whereby the Golem ruthlessly pursues the assigned target without any human regard.
What binds the two examples together, Wiener’s and Bostrom’s, their common conceptual frame, is the specification of a correct cost function in an optimization problem. Every superintelligence has to be assigned a cost function which is rich enough to incorporate human criteria. The alternative is of course to be swept away.

But humans have already created superintelligent mechanisms, here defined as calculating algorithms that vastly outperforms our species’ computational capabilities. Perhaps the most important (so far) is of course a market economy, whereby a blind optimization engine solves a huge system of (non-linear) equations to compute the price of scarce commodities, thus outperforming any conscious solver, as the Soviet planner painfully experienced and as Enrico Barone had already demonstrated in 1908.
Nobel prize winner soviet mathematician Leontieff had already spoken of the market as a optimizer, based on collective intelligence, in here (jstor). The market is a very powerful approximation engine of a vast system of algebraic equations.
Wiener had already known all this and, similarly to Marx’ 1844 Manuscripts, he famously said the following:

“Perhaps I may clarify the historical background of the present situation if I say that the first
industrial revolution, the revolution of the “dark satanic mills,” was the devaluation of the
human arm by the competition of machinery. . . . The modern industrial revolution [i.e., the
computer revolution] is similarly bound to devalue the human brain. . . . The answer, of course,
is to have a society based on human values other than buying and selling.”
(Wiener, Cybernetics, pp.37-38, bracketed words added)

The issue at stake is then the same: are we able as a species to define an objective function which is rich enough to incorporate human values before unleashing the power of the blind optimization engine?
Stiglitz’ recent proposal of a “minimum corporate tax” is then best framed, for example, as a control problem in the optimization routine which, since at least Adam Smith, has been known as Invisible Hand.

Categories: Technology