Incoherence & Mattson

People are often incoherent: their probabilities don't add to 100%. We get an 18% gain in accuracy if we coherentize their estimates. But we get a 30% gain in accuracy if we also assign more weight to coherent estimates.

What implications does this have for making subjective probability maps?

The following figure is from a recent paper I co-authored*:

Figure from Karvetski et al. 2014 showing we get more accuracy by ignoring incoherent estimates than by simple unweighted averages.
Figure from Karvetski et al. 2014 showing we get more accuracy by ignoring incoherent estimates than by simple unweighted averages.  (The unfortunately abbreviated 'BS' means 'Brier Score'. Lower is better, with 0 being perfect.)

What implications does it have for making subjective "consensus" probability maps at the start of a search?

Continue reading "Incoherence & Mattson"

UAVs, MH370, Prediction Markets

In which I discuss two-and-a-half approaches to crowdsourcing Search & Rescue, and invite you to try one -- namely, mine: http://bit.ly/1eBBMkq

In which I discuss two-and-a-half approaches to crowdsourcing Search & Rescue, and invite you to try one -- namely, mine.  (SciCast)

Continue reading "UAVs, MH370, Prediction Markets"

Berkshires 2006 Sweep Width Experiment

In the summer of 2006, Rick Toman and Dan O'Connor (NewSAR) organized a sweep width experiment and summit called "Detection in the Berkshires" at Mount Greylock in Massachusetts. The data from that experiment has been used in previous blog posts, but hasn't been published independently.

In the summer of 2006, Rick Toman (Massachusetts State Police) and Dan O'Connor (NewSAR) organized a sweep width experiment and summit called "Detection in the Berkshires" at Mount Greylock in Massachusetts.  In addition to the sweep-width experiment, Perkins & Roberts provided search tactics training for several teams, and the summit provided a chance for us to explore similarities and differences between formal search theory and formalized search tactics.  It was an important the chance to meet many key people, compare notes, and discuss ideas.  I wish I had been more diligent about following up. Many thanks to Rick & Dan for organizing the event, and to many others listed at the end. However, this post is mostly to provide a reference for the sweep width experiment.

Continue reading "Berkshires 2006 Sweep Width Experiment"

MapScore Updates

MapScore has been updated to include Batch Upload, and fix or improve a number of other features.

The MapScore site has been updated! The most exciting new feature is Batch Upload.

  • Batch Upload!  In one fell swoop, upload and re-score all your models and cases (or as many as necessary).  No more clicking around or messing with “active” vs “inactive” cases.  (Unless you want to...)
MapScore batch upload screen. New Nov. 2013.
MapScore batch upload screen. New Nov. 2013.

Continue reading "MapScore Updates"

Search Theory in C4ISR Journal

The C4ISR Journal had a recent search theory article quoting me along with Larry Stone.

The C4ISR Journal had a recent search theory article quoting me along with Larry Stone.  I'm quite honored, like the British company in Dirk Gently's Holistic Detective Agency:

...was the only British software company that could be mentioned in the same sentence as ... Microsoft.... The sentence would probably run along the lines of ‘...unlike ... Microsoft...’ but it was a start.

It's a good article, covering the undeniably exciting historical origins hunting U-boats, and looking at what may be a modern renaissance.  I think the article stretches to connect search theory with Big Data, but the author does note that when the data is visual, and you have humans scanning it for objects, there is a connection.  With planning, it could have been used to prioritize the Amazon Mechanical Turk search for Jim Gray.  (The resolution of the actual images in that search was probably too low regardless, but the core idea was sound.)

Continue reading "Search Theory in C4ISR Journal"

Structured Methods for Intelligence Analysis

My colleagues just published a paper in Euro Journal on Decision Processes, for their special issue on risk management.

Karvetski, C.W, Olson, K.C., Gantz, D.T., Cross, G.A., "Structuring and analyzing competing hypotheses with Bayesian networks for intelligence analysis".

My colleagues just published a paper in Euro Journal on Decision Processes, for their special issue on risk management.

Karvetski, C.W, Olson, K.C., Gantz, D.T., Cross, G.A., "Structuring and analyzing competing hypotheses with Bayesian networks for intelligence analysis". EURO Journal on Decision Processes, Special Issue on Risk Management: http://link.springer.com/article/10.1007/s40070-013-0001-x

Alas, it's behind a paywall and the printed edition isn't due until Autumn. Here's an excerpt from the abstract:

Continue reading "Structured Methods for Intelligence Analysis"