Notes on ‘The Black Swan’ by Nicholas Nassim Taleb (2007)

(Ed. This piece was first published at cityofsound.com on 1 January 2009.)

This is a book that I almost didn’t read. Like The Long Tail or Here Comes Everybody, for instance. Both books I own but don’t feel the need to read, feeling that I’ve already having experienced much of what lies inside. This betrays my own arrogance I suppose, and I’ve no doubt I’ve missed a few profound insights this way. But given the choice I prefer to read about things I don’t know, books that don’t promise to back up my existing ideas. Then there are those like Gladwell’s Blink or The Tipping Point, books whose title more or less says it all. A quick rifle through the pages of these books in an airport bookshop—in that peculiar pre-flight mode of having no time and time on your hands—is enough to get the gist, and speculate as to their point.

The Black Swan almost fell into this category, but a recommendation by Paul Schütze and a few others meant that I did pick it up —at Melbourne Airport, natch—and consumed it voraciously.

It’s not so much a popular science book as a popular statistics book, not a genre I would’ve thought probable to emerge, and thus something of a black swan in itself. Black swans are, according to that back cover blurb, “random events that underlie our lives … their impact is huge; they’re nearly impossible to predict; yet after they happen we always try to rationalize them.” That describes the phenomenon that underpins this book—drawn from the well-known example indicating the flaws in inductive reasoning, and suggests the corollary, the validity of falsification—but that doesn’t describe the significance of Nassim Nicholas Taleb’s message here.

As a book, it serves as a bracing rejoinder to many ideas I’d begun to almost instinctively value, particularly the value of expertise and forecasting. And while Taleb doesn’t go after all expertise (although he prefers “sophisticated craft”), nor does he suggest completely ignoring prediction, he systematically takes apart much of the practice of predicting, planning and popular approaches to working with data. Put simply, too much of what is taken to be expert prediction is based on simplistic, flawed induction and subjective, overly-optimistic and simplistic stories, no matter how dressed up in economic theory, and thus horribly exposed to random black swans.

Taleb forcefully asserts that black swans are the only things that meaningfully and powerfully shape our world, whether it’s the stockmarket crash of 1987, 9/11, the spread of the internet, or the assassination of minor Hungarian royalty in Sarajevo in 1914. So when portfolio managers talk about measuring risk without taking into account the possibility of black swans, which they do, Taleb asserts this is no better than astrology. And given that astrology generally gets one page at best in newspapers, one more than it deserves, yet finance often an entire section, it is thus more dangerous than astrology. (Incidentally, Taleb is scathing about the value of daily newspapers in general, recommending a total rejection of them—“additional knowledge of the minutiae of daily business can be useless, even actually toxic”—due to our propensity to be swayed by the sensationalist).

His primary focus is indeed finance and economics, something I have generally had little more than a passing interest in. Yet I find much here that made me pause to consider new ways of thinking my own work through, and equally reinforced that design, in general, should more deeply understand economic behaviour, or at relevant subsets of it. Equally, those working with data in almost any field will find much of value here.

In the field of finance Taleb talks from near unbeatable position of having been a day trader for many years as well as an educator and writer, thus working with financial data from the ‘bottom-up’, as he puts it. His scorn for academics who reject practitioners is consistent throughout, and he much prefers the instinctive approach of the latter, derived from revelling in the complexity of everyday data.

It’s particularly instructive to be reading it amidst the smoking ruins of the global financial collapse of 2008, a scenario largely predicted by Taleb in this book, including a specific footnoted warning as to the precarious position of Fannie Mae, never mind the dangerously interconnected nature of the handful of financial players that dominate the global banking system.

Not that I am truly amidst those smoking ruins as yet; Australia is one of the very few developed nations that the OECD predicts will avoid recession. But you get my point, and I’m now extremely aware of the glaring presence of the phrase “the OECD predicts” in that previous sentence. Indeed the data when I wrote this piece (04.12.08), two weeks after that OECD prediction, indicated the Australian economy was now sailing extremely close to negative growth. Still, better to be sailing at all, and the relatively conservative practices of Australian banks and government, plus trade with China-India, has left it, temporarily at least, as pretty much the only developed ‘western’ member of the G20 not in recession.

Taleb is deeply concerned that much of the analytical framework the modern world is constructed on is not calibrated to deal with black swans, the very things that have indeed tended to shape the world, for better sometimes and yet often for worse.

“That we got here by accident does not mean that we should continue to take the same risks. We are mature enough a race to realise this point, enjoy our blessings, and try to preserve, by becoming more conservative, what we got by luck. We have been playing Russian roulette; now let’s stop and get a real job.”

Had he written this book a year later, his scathing critique of contemporary financial engineering would hardly have been necessary. He could simply have printed the market data and listed verbatim quotes from the likes of Alan Greenspan : “the reason I was shocked (was) because I had been going for 40 years or more with very considerable evidence that it was working very well”.

Or as Greenspan told Congress, “data put into the risk-management models only covered the past two decades of euphoria, excluding previous instances of financial stress” (from ‘Up in smoke’, Australian Literary Review, December 3 2008).

George Soros notes how the crisis has “contradicted the prevailing theory, which holds that financial markets tend towards equilibrium”, a Gaussian distribution-based equilibrium which Taleb proves time and again is an inappropriate concept for capital markets.

Yet fortunately, and despite Taleb’s sometimes merciless evisceration of his foes, this is a far from nihilistic book and Taleb replaces much of the rotten economic superstructure with a scaffolding more suited to taking advantage of significant unforeseen events.

Taleb’s conservatism applies mainly to handling of finance and economics, but he appears to represents a far from conservative kind of person. He does not dismiss expertise from the position of an anti-intellectual, but instead provides a vigorous example of perhaps an earlier kind of intellectual; the broad-minded, highly-cultured and independent intellectual position long since greatly diminished in number and influence by the rise of discipline-based experts.

Railing against the monocultures created by ‘nerds’ who focus on one intellectual pursuit to the eventual exclusion of others, Taleb suggests the importance of remaining open-minded, curious and interdisciplinary, a position I can’t endorse enough. Only in his careless and frequent lashing out at pretty much all social sciences do we find him to be rejecting his own medicine. It’s an attack that may often be warranted, but certainly not as much as Taleb appears to believe.

Experts are systematically taken apart here, through much detailed evidence, ranging from the apparent lack of anyone to usefully predict financial markets ever, to the detailed work of psychologist Philip Tetlock, who conducted psychological research on ‘experts’ with quite astonishing results.

Taleb frequently attacks the egotism of these experts, particularly those who loudly proclaim the virtues of their (inherently flawed) methodologies, accepting the rewards that a gullible world endows upon them. (And not just the “fuck you” money, as Taleb has it; he is particularly scathing about the Nobel prize for economics).

Yet Taleb’s own ego appears to be in a fairly healthy condition throughout, thanks to the book often possessing the air of an autobiography as much as an intellectual treatise. This makes the book a far more interesting read, of course, and The Black Swan is often amusing and always engaging, even when knee-deep in theory. Taleb comes across as complex, flawed, impassioned, and worth sharing a bottle of wine with.

For those were told “there would be no math”, as Chevy Chase’s befuddled Gerald Ford character would’ve said, you’ve been misinformed. There is the occasional dose of mildly mind-bending maths in here, hovering between philosophy and raw statistics. But Taleb easily signposts ways around this, and is a good-natured guide through it for those with the stomach. I confess to having been lost on several occasions, but Taleb ensures that you never lose sight of the central arguments, whether he’s explaining how casinos or capital markets work (at least with casinos, you can calculate the odds, he notes.)

Partly thanks to the breadth of Taleb’s erudition, the more interesting passages here concern our psychological makeup, with advantages and disadvantages endowed by the process of evolution as much as anything. You’ll find much here to aid understanding of how people think, and how people act on instinct, for that matter. Beyond that, our almost subconscious attraction to finding confirmation of one’s notions through active construction of data to support that notion is particularly interesting, and worrying.

Taleb recognises that the open-minded empirical skepticism he recommends takes supreme efforts, but this is his suggested method: to live as a practitioner amidst empirical data, and attempt to build knowledge through falsification rather than confirmation. His advice is rather more complicated and multi-faceted than this, yet he puts much of this so elegantly at times that it often appears to be common sense. However, when you want a detailed summation of his specific advice it is not presented as punchily as his earlier critique. (I’d also like to know more about his simulations, for instance).

This ultimately means you have to construct your own findings, which is no bad thing but does lend a little credence to the notion that Taleb has intellectually taken apart much of the financial system (just a year or so before it self-immolated anyway) yet doesn’t produce an equivalently detailed replacement.

A further criticism of The Black Swan might be that it makes its points early, and well, but then drives deeper and deeper into exploring the issue, rather than devoting equivalent weight to possible solutions. It’s thoroughly fascinating but similar learning can be found in an excellent essay on behavioural economics and optimism, by the consistently excellent Gideon Haigh in the Australian magazine The Monthly, and in a sharp critique of the US economy post-Greenspan by Joseph A. Stieglitz in Vanity Fair. There, the context of magazines ensures a punchy one-two delivery, but one just as likely to land a knockout as Taleb’s lengthy war of attrition. Stieglitz sums things up by relating Greenspan’s now infamous recent performance:

“The truth is most of the individual mistakes boil down to just one: a belief that markets are self-adjusting and that the role of government should be minimal. Looking back at that belief during hearings this fall on Capitol Hill, Alan Greenspan said out loud, “I have found a flaw.” Congressman Henry Waxman pushed him, responding, “In other words, you found that your view of the world, your ideology, was not right; it was not working.” “Absolutely, precisely,” Greenspan said. The embrace by America-and much of the rest of the world-of this flawed economic philosophy made it inevitable that we would eventually arrive at the place we are today.” [‘Capitalist Fools’, Joseph A. Stieglitz, Vanity Fair]

Taleb draws frequently on Mandelbrot and Popper above all, and with Hayek in a supporting role. The latter two might lead some to question the political backdrop to this stance, yet Taleb rarely directly wanders off towards a fully libertarian direction. He rails against large corporations as much as government. He finds flaws in laissez-faire solutions, which would continue to produce the inequalities that his frequently proclaimed humanism rejects.

Those of us who have struggled through nearly 20 years of hearing that the market was the best way to run everything—from schools and hospitals to childcare, housing, energy generation and transport infrastructure—have this recent ‘performance’ of the financial system itself as an additional indication that, well, it patently isn’t that simple. It’ll take some time for those purveyors of that mindless rhetoric to take on board that ever freer markets are not the only solution. Much of this book would quickly help with that understanding, even if it is written from the perspective of someone who lives in, and loves at some level, those markets.

Yet now that that same rhetoric has left government often shrivelled in significance—but not size, oddly—and with its skillset blunted accordingly, we are not left with much. For instance, here in Sydney, you can’t walk down the streets without bumping into legions of fiercely bright financial analysts (mainly because they’re looking at their BlackBerrys).

Yet if they could focus their intellect on something broader than bolstering a system based on financial self-interest—which behavioural economics indicates is far from rational and, yes, doesn’t tend towards equilibrium—we might get somewhere. Their lack of work might even generate a useful ‘cognitive surplus’, to appropriate Clay Shirky’s phrase.

Taleb doesn’t really give us any way out of this, perhaps as this is a deeper cultural problem, rather than economic. That Taleb doesn’t go anywhere near that greatest market failure of all, of global warming, is also interesting, but also worrying.

Still, as a bracing, intellectually stimulating, often funny, clear-sighted and conservatively radical read, The Black Swan does begin to back us out of a cul-de-sac. Where we go next is an interesting dilemma.

Below, a few bullet-points indicating particular aspects drawn from The Black Swan as regards cities, design, data, and so on. It’s particularly sobering reading when one works in a team with ‘planning’ in the title (urban in my case) but I can recommend reading alongside the excellent First Read This: Systems Engineering in Practice by Ed van Hinte and Michel Van Tooren ( 010 Publishers), which makes some similar cautionary remarks to Taleb, but in the context of something akin to planning of complex systems nonetheless. Despite the title, you don’t have to read it first but you do have to read it as well.

A few bonus bullets

• The value of not constructing a narrative from data. During the forum ending our recent course at UTS (“The Street as Platform”), we had a debate as to the value of displaying the data versus constructing narratives around the data. Whilst the act of selecting and displaying data is of course its own narrative, many of the visualisations the students had built were open-ended in nature, in part informed by Mitchell Whitelaw’s interest in this as an artistic strategy, and mine as a design tool-making strategy. Andrew vande Moere indicated that he would’ve liked to have seen more narratives drawn out of the data, in order to illustrate points, or draw the viewer towards the data, a view several others at the forum agreed with. Mitchell and I remained uneasy about the desire to pin the data to a narrative however, and Taleb also takes this position several times, but also indicates how hard it is. In the following excerpt, Taleb is indicating the value of remaining open, but also that interpretation is so instinctive it is rarely even perceptible by those doing it. In terms of our data experiment the other day, I think Andrew may well have been right in most instances there, but that perhaps a more sophisticated version of our visualisations would’ve resisted narrative and interpretation even more.

“Counter to what everyone believes, not theorizing is an act – thus theorizing can correspond ot the absence of willed activity, the ‘default’ option. It takes considerable effort to see facts (and remember them) while withholding judgment and resisting explanations. And this theorizing disease is rarely under our control: it is largely anatomical, part of our biology, so fighting it requires fighting one’s own self. So the ancient skeptics’ precepts to withhold judgment go against our nature.” (Taleb)

• Related to the above:

“The more information you give someone, the more hypotheses they will formulate along the way, and the worse off they will be. They see more random noise and mistake it for information. The problem is that our ideas are sticky: once we produce a theory, we are not likely to change our minds – so those who delay developing their theories are better off.” (Taleb)

• Also related to the above:

“The world, epistemologically, is literally a different place to a bottom-up empiricist. We don’t have the luxury of sitting down to read the equation that governs the universe; we just observe data and make an assumption about what the real process might be, and ‘calibrate’ by adjusting our equation in accordance with additional information. As events present themselves to us, we compare what we see to what we expected to see. It is usually a humbling process, particularly for someone aware of the narrative fallacy, to discover that history runs forward, not backward.” (Taleb)

• On why Benoit Mandelbrot took an interest in finance:

“Mandelbrot’s answer was, ‘Data, a gold mine of data.’ Indeed, everyone forgets that he started in economics before moving on to physics and the geometry of nature. Working with such abundant data humbles us; it provides the the intuition of the following error: travelling the road between representation and reality in the wrong direction.”

• Taleb writes specifically of the hubris of those who survive a process, and how the survivor is ill-equipped to commentate from their position (being largely unaware of the “cemetery”, as he puts it, of non-survivors. The “silent evidence”.) Specifically in terms of cities, and even more specifically, New York City. Or rather the idea of “the resilience of New York City”, as if it were an internal quality of the city.

“Let’s put a collection of cities in a simulator of history: Rome, Athens, Carthage, Byzantium, Tyre, Catal Hyuk (located in modern-day Turkey, it is one of the first known human settlements), Jericho, Peoria, and, of course, New York City. Some cities will survive the harsh conditions of the simulator. As to others, we know that history might not be too kind. I am sure that Carthage, Tyre, and Jericho had their local, no less eloquent (commentator) saying “Our enemies have tried to destroy us many times, but we always come back more resilient than before. We are now invincible.”

This bias causes the survivor to be an unqualified witness of the process. Unsettling? The fact that you survived is a condition that may weaken your interpretation of the properties of the survival, including the shallow notion of ‘cause’.”

• The limits of (some) predictive models:

“Statisticians have concentrated their efforts in building more sophisticated models without regard to the ability of such models to more accurately predict real-life data.”

• Some specific advice, which reinforces the urban advantage:

“Collect as many free nonlottery tickets (those with open-ended payoffs) as you can, and, once they start paying off, do not discard them. Work hard, not in grunt work, but in chasing such opportunities and maximising exposure to them. This makes living in big cities invaluable because you increase the odds of serendipity. The idea of settling in a rural area on grounds that one has good communications “in the age of the internet” tunnels out of such sources of positive uncertainty.”

And Caravaggio’s ‘The Fortune-Teller’ is a great image to use when opening presentations on foresight and innovation work.

Ed. This piece was first published at cityofsound.com on 1 January 2009.