• The theme that emerges from these three chapters can be summarized in one sentence, which will be a key theme of this book: wherever there is judgment, there is noise—and more of it than you think.
• Our conclusion is simple: wherever there is judgment, there is noise, and more of it than you think.
• Our topic is human error. Bias and noise—systematic deviation and random scatter—are different components of error. The targets illustrate the difference.
• Some judgments are biased; they are systematically off target. Other judgments are noisy, as people who are expected to agree end up at very different points around the target. Many organizations, unfortunately, are afflicted by both bias and noise.
• In real-world decisions, the amount of noise is often scandalously high.
• Wherever you look at human judgments, you are likely to find noise.
Whenever you observe noise, you should work to reduce it!
• As we will see throughout this book, the amount of noise observed when an organization takes a serious look almost always comes as a shock.
• Judgment is a form of measurement in which the instrument is a human mind.
• In noisy systems, errors do not cancel out. They add up.
Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein, Noise: A Flaw in Human Judgement
There is no other way to say this: these are uncertain times.
We all know the VUCA designation, which was first taught to Army Generals:
There may be a lot of causes for all the uncertainty and volatility. But, one cause is that we are learning, with more precision, just how wrong we get things.
Daniel Kahneman, who won a Nobel Prize in Economic Sciences, is well-known for his book Thinking, Fast and Slow. As a careful reader of business books, I think I can state with confidence that that book has been the most quoted and referred to book by other authors over the last many years. It seems to pop up everywhere. His key understanding from that book is found in his description of two kinds of thinking:
- System 1 thinking – fast, intuitive; judgment is achieved quickly and effortlessly
- System 2 thinking – slow, deliberative thought.
And, we get in trouble because we engage in so much System 1 thinking to the exclusion of System 2 thinking.
I presented my synopsis of his new book Noise at the July 9 First Friday Book Synopsis.
As always, I ask What is the point? Here it is for this book: We make too many bad judgments. We need to get better at making better judgments. This book points the way.
And I ask Why is this book worth our time? Here are my three reasons for this book:
#1 – This book is thorough; it shows us where we make bad judgments in practically every part of our work life; and all other parts of life.
#2 – This book is filled with understandable examples, even as it teaches us from well-researched findings.
#3 – This book reminds us – me; and you – that we are part of the problem; we have flaws in our own judgment.
In my synopses, I always include a number of Quotes and Excerpts from the book – the “best of” Randy’s highlighted Passages. Here are the best of the best that I included in my synopsis:
• Interjudge disparities increased significantly after 2005. When the guidelines were mandatory, defendants who had been sentenced by a relatively harsh judge were sentenced to 2.8 months longer than if they had been sentenced by an average judge. When the guidelines became merely advisory, the disparity was doubled. …After the guidelines became advisory, judges became more likely to base their sentencing decisions on their personal values.
• First, judgment is difficult because the world is a complicated, uncertain place. Disagreement is unavoidable wherever judgment is involved.
Second, the extent of these disagreements is much greater than we expect.
Third, noise can be reduced.
Fourth, efforts at noise reduction often raise objections and run into serious difficulties.
• For another exercise in counterfactual thinking, consider how different countries and regions responded to the COVID-19 crisis. Even when the virus hit them roughly at the same time and in a similar manner, there were wide differences in responses.
• Whether you make a decision only once or a hundred times, your goal should be to make it in a way that reduces both bias and noise.
• Agreement is especially easy when a judgment is absurd. …Judges at wine competitions differ greatly on which wines should get medals, but are often unanimous in their contempt for the rejects.
• The measurement and reduction of noise should have the same high priority as the measurement and reduction of bias.
• Suppose that doctors are deciding whether to admit people for hospitalization, that companies are deciding whom to hire, that lawyers are deciding which cases to bring, or that Hollywood executives are deciding which television shows to produce. In all these cases, there will be pattern noise, with different judges producing different rankings of the cases.
• “Level noise is when judges show different levels of severity. Pattern noise is when they disagree with one another on which defendants deserve more severe or more lenient treatment. And part of pattern noise is occasion noise—when judges disagree with themselves.”
• Simply put, just like a basketball player who never throws the ball twice in exactly the same way, we do not always produce identical judgments when faced with the same facts on two occasions.
• Or to put it differently, you are not always the same person, and you are less consistent over time than you think. But somewhat reassuringly, you are more similar to yourself yesterday than you are to another person today.
• His pattern resembles what you would see in an experiment in which two groups of investors read business plans that are substantively identical but printed in a different font and on a different paper. If these irrelevant details make a difference in the investors’ judgment, there is psychological bias.
• We don’t know if the investors who were impressed by the sleek font and glossy paper are too positive or if those who read the rougher version are too negative. But we know their judgments are different, although they should not be.
• For example, the perception of the risk of airplane crashes or hurricanes rises briefly after well-publicized instances of such events. In theory, a judgment of risk should be based on a long-term average. In reality, recent incidents are given more weight because they come more easily to mind.
• Because of confirmation bias and desirability bias, we will tend to collect and interpret evidence selectively to favor a judgment that, respectively, we already believe or wish to be true.
• The psychologist Paul Slovic terms this the affect heuristic: people determine what they think by consulting their feelings. …We like most things about politicians we favor, and we dislike even the looks and the voices of politicians we dislike. That is one reason that smart companies work so hard to attach a positive affect to their brand.
• The rule is simple: if there is more than one way to see anything, people will vary in how they see it.
• “The uniqueness of people’s personalities is what makes them capable of innovation and creativity, and simply interesting and exciting to be around. When it comes to judgment, however, that uniqueness is not an asset.”
• Good judges tend to be experienced and smart, but they also tend to be actively open-minded and willing to learn from new information.
• Psychologists and neuroscientists distinguish between crystallized intelligence, the ability to solve problems by relying on a store of knowledge about the world (including arithmetical operations), and fluid intelligence, the ability to solve novel problems.
• As one newspaper headline put it, “Study Finds That Basically Every Single Person Hates Performance Reviews.” Every single person also knows (we think) that performance reviews are subject to both bias and noise.
• The definition of success is a nontrivial problem.
And, here are a number of the key points made by the authors that I included in my synopsis:
- Let’s start here – a lot of bad decisions and mistakes are made…everywhere…
- And, “the problem is with you, not with me…”
- Indeed, we suspect that underwriters who heard about the noise audit and accepted its validity never truly believed that its conclusions applied to them personally.
- We view the occasional disagreements with colleagues as lapses of judgment on their part. (emphasis added).
- What is this book?
- This is a book about decision-making that is done poorly (incorrectly; wrong) because of too much noise, some bias, and too much System 1 thinking with too little System 2 thinking.
- This book is more diagnosis than solution; but it points plenty to potential solutions.
- Noise, noise everywhere…
- medicine, especially psychiatry; hiring; performance evaluation; forecasting; court (judges; and juries)
- noise between people
- noise within each individual person
- Be wary of:
- Moods — People in a good mood are more cooperative and elicit reciprocation. They tend to end up with better results than do unhappy negotiators. On the other hand, a good mood makes us more likely to accept our first impressions as true without challenging them. People who are in a good mood are more likely to let their biases affect their thinking.
- Overconfidence — “The average expert was roughly as accurate as a dart-throwing chimpanzee.” — People who believe themselves capable of an impossibly high level of predictive accuracy are not just overconfident. They don’t merely deny the risk of noise and bias in their judgments. Nor do they simply deem themselves superior to other mortals. They also believe in the predictability of events that are in fact unpredictable, implicitly denying the reality of uncertainty. — We are not all highly confident all the time, but most of the time we are more confident than we should be.
- Substitution — substituting one question for another will lead to an answer that does not give different aspects of the evidence their appropriate weights, and incorrect weighting of the evidence inevitably results in error. — When we substitute an easier question for the one we should be answering, errors are bound to occur.
- the halo effect – (part of the first impression/exposure effect) — When the initial evaluation is erroneous, however, the tendency to stick to it in the face of contradictory evidence is likely to amplify errors.
- Noise — noise is unwanted variability
- Bias – the evaluator’s preferences (known and unknown…)
- Occasion noise is the variability in judgments of the same case by the same person or group on different occasions. A surprising amount of occasion noise arises in group discussion because of seemingly irrelevant factors, such as who speaks first.
- Objective Ignorance – Some of the executives in our audiences tell us proudly that they trust their gut more than any amount of analysis.
- Really?… Judgements, and decisions, are impacted by:
- the time of day
- the fatigue of the decision maker
- Do not forget:
- regression to the mean
- standard deviation
- the desire for coherence – The aim of judgment, as you experienced it, was the achievement of a coherent solution.
- there is a danger in first impressions…leading to the danger of informational cascades…
- confirmation bias
- fundamental attribution error — A well-documented psychological bias called the fundamental attribution error is a strong tendency to assign blame or credit to agents for actions and outcomes that are better explained by luck or by objective circumstances.
- smarter is better – GMA (General Mental Ability)
- The big suggestions:
- a noise audit
- decision hygiene
- evaluations done independently, in sequence…
- second opinions; free of as much other information as possible (especially the evaluations of others)
- make your own second guess/second opinion
- “You can gain about 1/10th as much from asking yourself the same question twice as you can from getting a second opinion from someone else.” This is not a large improvement. But you can make the effect much larger by waiting to make a second guess.
- the wisdom of crowds – especially crowds of trained experts… (but; be a little wary of groups)
- we need more statistical models; and more algorithms…
- maybe pay more attention to the “average” before you give your judgement… – “base rates”
- aim for probabilistic, not for absolute…
- look for actively open-minded people.
- use decision observers
- aim for more structured systems (guidelines; conversations; meetings; deliberations; interviews; rather than less structured)
- follow a sequence of questions/inquiries; independently — Let’s start by defining much more clearly and specifically what we are looking for in candidates, and let’s make sure we evaluate the candidates independently on each of these dimensions.”
- train yourself to be in perpetual Beta mode — “the strongest predictor of rising into the ranks of superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement.” As he puts it, “What makes them so good is less what they are than what they do—the hard work of research, the careful thought and self-criticism, the gathering and synthesizing of other perspectives, the granular judgments and relentless updating.”
- We conclude by offering a system we call the mediating assessments protocol: a general-purpose approach to the evaluation of options that incorporates several key practices of decision hygiene and aims to produce less noisy and more reliable judgments.
And here are my five lessons and takeaways:
#1 – We allow too much noise into our decision-making processes.
#2 – We are oblivious – blind – to our own failures in this regard.
#3 – We can do better; with training, and decision hygiene, among other steps to take.
#4 – This failure has real-world consequences. In business, and in all aspects of life and society.
#5 – We might need help. (A decision observer, for example).
We have all made some bad decisions. (Okay – I have made some bad decisions. And, if I were a betting man, I would bet that you have also made some bad decisions). The more we reduce noise, and bias, the better we will get at making better decisions.
This book can help.
I have presented synopses of business books monthly for over 23 years. Our synopses are available for purchase. Each synopsis comes with my multi-page, comprehensive synopsis handout, plus the audio recording of the synopsis presentation (recorded at our First Friday Book Synopsis events).
You can order them from our web site. Click on the “Buy Synopses” tab above to search by book title. And click here for our newest additions. My synopsis of this book, Noise, will be available soon.