Wisdom of the 2017 JSM Crowd?
Estimating with Statisticians
We had a fun time at JSM with statisticians trying to “guess the money” in the jar. Our inspiration was the 1907 “Guess the Weight of an Ox” contest chronicled by the statistician Sir Francis Galton. Galton was surprised by how the average of all the guesses was spot on, despite considerable variation among them. The experience changed his view of democracy.
At JSM, it was interesting to see the different approaches to estimation. A few people systematically counted up visible bills, then made an effort to estimate what portion of the total was visible. Several guessers made reference to a Bayesian strategy in which you guess above the expected mean, figuring that the higher payoff in the event of a win, and the lower probability of a tie, would offset the lower probability of winning. Others wanted to weigh the bowl. We had a very interesting discussion with a woodworker about the thickness of a dollar bill (0.0043 inches), and how an expert can slice wood to one-quarter that thickness. One resident of a nearby booth said he thought of salting the bowl on the sly with $100 bills that only he would know about.
And the Winners Are…By now, you’ve probably heard about the winners. We had $117.00 in the jar, and two exact winners - Keenan O’Brien (pictured with Janet Dobbins) and Svetlana Levitan.. Another person just missed with $117.50. So close!
This year the jar had only $1 bills, so the contest was a bit more straightforward than in 2016, when we sprinkled in two $5 and one $10 bills and ended up with an extremely right-skewed distribution, apparently the result of the “wild card” element introduced by these higher denomination bills.
Our 2016 experience was very different from Galton’s Ox Contest! You can view 2016’s contest here.
This time we advertised the fact that the bowl had only $1 bills, and we did end up with a less skewed distribution. The mean was 15% above the actual, and the median was only 5% above. Still not at the Ox Contest level of accuracy, but much closer than in 2016. Additionally, we allowed a second guess, in which the contestant was given more information if they came back on Tuesday after noon. The additional information was the guesses of three other contestants (selected at random).
Some right-skew remained in the distribution, resulting from the 0-truncation combined with the relatively large variance of the guesses:
2017 actual: $117
Mean guess: $135
Median guess $123
Sample size N=258
Sample size N=30
mean second guess: $121
We had 21 kids enter the contest in 2017, and their average guess was $104, or an 11% underestimate.
The Data - See the Data for Yourself
These data, including last year’s, are all available and we invite you analyze, use and publish as desired (with credit to Statistics.com).
Galton's story of the ox contest has been a staple of statistical lore over the past century and served as an inspiration in James Surowiecki's popular book, The Wisdom of Crowds.
But perhaps crowds are not so reliable, after all?
Eric Steiner, writing in Wired in 2015, describes a money-bowl (coins) contest just like the ones we conducted. His results were similar to ours, and he titled his article Turns Out the Internet is Bad at Guessing How Many Coins are in a Jar.
Wisdom of the Crowds TV series (airs Oct 2017) on CBS
Special thanks to Shahar Boneh who brought his students from Metropolitan State University of Denver to JSM and one was lucky enough to guess exactly.
What's your take?
Have you seen this distribution in crowd estimation tasks? Let us know - perhaps we can collaborate on a paper.
Want to hear more?
Peter Bruce sends out periodic newsletters about our upcoming courses with interesting insights. You can choose which type(s) of courses you'd like to hear about (Introductory, Bayesian Statistics, Biostatistics, Clinical Trial Statistics, Data/Text Analytics, Engineering, Environment, Predictive Modeling, Social Sciences, Survey Statistics, or Using R, Python, SQL) by clicking the "Yes" button beneath "Want to be notified of future courses?" above.
Want to be notified of future courses?Yes