The following figure is from a recent paper I co-authored*:
What implications does it have for making subjective "consensus" probability maps at the start of a search?
David Mandel has just blogged a summary of our recent Decision Analysis paper**. In laboratory conditions on general knowledge questions, we found:
- People are often incoherent: their probabilities don't add to 100%.
- We get an 18% gain in accuracy if we coherentize their estimates.
- But we get a 30% gain in accuracy if we also assign more weight to coherent estimates.
Suppose this applies to making subjective probability maps -- and we don't know that it does. Recall the original "Mattson" consensus asks everyone to put probabilities in each region. People are bad at this. Often they don't add to 100%, so people correct by getting to about 80% and then just dividing the rest. So we have invented decision aids to help them. The Proportional method says use whatever numbers you like, and normalizes. The O'Connor method uses verbal cues (A = "very likely" ... I = "very unlikely"), then gives each letter a number 9..1, and uses the Proportional method. Etc.
I tend to fall in the "use whatever method you like best" camp. Usually that means Proportional or O'Connor. But if our result applies to making subjective probability maps, it would suggest:
- The O'Connor & Proportional will beat Mattson, simply by forcing coherence. No surprise there.
- But coherence-weighted Mattson might do better still. Our "decision aids" might be hiding carelessness, incapacity, or neglect which we would do better to recognize and ignore.
I think it's time for a follow-up study. I might call Ken Hill, whose earlier study formally established that subjective maps are subject to some standard probability biases.
* The heavy lifting was done by postdocs Chris Karvetski & Ken Olson, with excellent design & writing input from David Mandel.
** Karvetski, Christopher W. and Olson, Kenneth C. and Mandel, David R. and Twardy, Charles R., Probabilistic Coherence Weighting for Optimizing Expert Forecasts (July 26, 2013). Decision Analysis, 10(4), 305-326. Available at SSRN: http://ssrn.com/abstract=2411649