Our MapScore paper is now in press at Transactions in GIS! From the abstract:
The MapScore project described here provides a way to evaluate probability maps using actual historical searches. In this work we generated probability maps based on the statistical Euclidean distance tables from ISRID data (Koester, 2008), and compared them to Doke’s (2012) watershed model. Watershed boundaries follow high terrain and may better reflect actual barriers to travel. We also created a third model using the joint distribution using Euclidean and watershed features. On a metric where random maps score 0 and perfect maps score 1, the ISRID Distance Ring model scored 0.78 (95%CI: 0.74-0.82, on 376 cases). The simple Watershed model by itself was clearly inferior at 0.61, but the Combined model was slightly better at .81 (95%CI: 0.77-0.84).
We compared the familiar distance-ring model from (Koester 2008) with a novel model counting the number of watersheds (ridge lines) crossed by the subject, and a combined model where the statistics are recalculated using both distances and #watersheds crossed.
The familiar distance-ring model did much better than the watershed alone, though we admit it is not a fair fight: as any Incident Commander knows, the distance-ring model has different statistics for each lost-person type. In contrast, the watershed model has only been learned for hikers, though we apply it to all cases.
However, although watersheds by themselves are much worse, the combined model slightly outperforms the distance-ring model. Yes, the effect is statistically significant, but it is not very large: about 3 absolute percentage points, about 13% of the possible gain. On the other hand, four such gains would yield more than 50% of the possible improvement.
But the point of the paper is not to argue for any of these simple models, but to present MapScore as a method for comparing models by testing the probability maps they generate on actual historical cases from ISRID, and to provide baseline numbers to beat.
Credits and Copies
The citation is: