Recently, I've started asking for people's confidence levels (from 0% to 100%) of project-related events happening. And although in its early stages, I've been disappointed by the results thus far, confidence generally being way higher than the reality.
Richard Feynman beautifully exposed the flawed methodology behind risk assessment at Nasa in his role on the commission to investigate the 1986 Challenger disaster. Nasa failed to realise (admit?) that if there are lots of uncorrelated bits of the Shuttle each with a near 100% probability of surviving the mission, each of which is critical to avoid disaster, then the probability of the Shuttle returning safely to Earth can fall unacceptably short of 100%.
The realisation of risks in projects I manage has a lesser impact. But an impact nonetheless. So I intend to keep a log of all the confidence estimates I receive (column B), together with the person whose confidence is being shared (column A) and the binary outcome of the event in which they have confidence (column C, 1 meaning the predicted event happened, 0 meaning it didn't). I figure that if people's confidence levels are true reflections of reality, then the sum of column B will equal the sum of column C. And sumifs based on people's names will identify the optimists, realists and pessimists.
=SUMIF(A:A,"John Smith",B:B)/SUMIF(A:A,"John Smith",C:C) will give me an optimism quotient for John. I can then divide any confidence percentage I receive from him by the quotient to get a more realistic view of whether the event will happen.



