My colleagues just published a paper in Euro Journal on Decision Processes, for their special issue on risk management.
Karvetski, C.W, Olson, K.C., Gantz, D.T., Cross, G.A., "Structuring and analyzing competing hypotheses with Bayesian networks for intelligence analysis". EURO Journal on Decision Processes, Special Issue on Risk Management: http://link.springer.com/article/10.1007/s40070-013-0001-x
Alas, it's behind a paywall and the printed edition isn't due until Autumn. Here's an excerpt from the abstract:
Although ACH aims at reducing confirmation bias, as typically implemented, it can fall short in diagramming the relationships between hypotheses and items of evidence, determining where assumptions fit into the modeling framework, and providing a suitable model for ‘‘what-if’’ sensitivity analysis. This paper describes a facilitated process that uses Bayesian networks to (1) provide a clear probabilistic characterization of the uncertainty associated with competing hypotheses, and (2) prioritize information gathering among the remaining unknowns. We illustrate the process using the 1984 Rajneeshee bioterror attack in The Dalles, Oregon, USA.
I've seen some very good demonstrations of ACH, but when all is said and done, the ACH matrix is a rough approximation to Bayes, justified because it is faster or more intuitive. But in fact it requires just as many judgments. Consider this passage from their conclusion:
Although a Bayesian network is a more sophisticated model than ACH, it can be less tedious by eliminating repeated elicitations after partitioning hypotheses into multiple dimensions and focusing on local relationships between variables. With ACH, 121 inputs were needed to define the model in Table 2, whereas 118 conditional probabilities were needed in Tables 3 and 4 to define the Bayesian network.
And when you're done, the Bayes net can perform instantaneous what-if calculations, and update probabilities as evidence becomes available. (And should you happen to have a combinatorial prediction market, you can crowdsource the probabilities in a distributed fashion. But that's our other research project.)
The paper sets out a hypothetical analysis (with a real historical case). Data collection is ongoing. Their facilitated method has been tried on two groups of analysts and recently on a large (>100) group of students, all with good success.
Don Ferguson has sparked recent listserv discussion on scenario analysis and structured analytic techniques. I think ACH is pretty good at what it does, and I think it's usually better than informal analysis. But I think SAR planning would do better to make any structured scenario analysis fully Bayesian. That's been a part of formal search theory since at least ~1970.
