In most statistical modeling or machine learning prediction tasks, there will be cases that can be easily predicted based on their predictor values (signal), as well as cases where predictions are unclear (noise). Two statistical learning methods, boosting and ProfWeight, use those difficult cases in exactly opposite ways – boosting up-weights them, and ProfWeight down-weightsContinue reading “Handling the Noise – Boost It or Ignore It?”
Category Archives: Analytics
“Defiant” Supervision
How did the phrase “defiantly recommend”, as in “I defiantly recommend this product,” come into common usage on the internet? The answer is a good look inside the workings of supervised learning. Supervision, generally from humans, is instrumental in much of statistical and machine learning. Google’s precise search algorithms are not public, but the generalContinue reading ““Defiant” Supervision”
Alaskan Generosity
People in Alaska are extraordinarily generous – that’s what a predictive model showed, when applied to a charitable organization’s donor list. A closer examination revealed a flaw – while the original data was for all 50 states, the model’s training data for Alaska included donors, but excluded non-donors. The reason? The data was 99% non-donors,Continue reading “Alaskan Generosity”
Industry Spotlight – The Military
Abraham Wald, a persecuted Jewish mathematician who fled Austria just before World War II, led an analysis of allied bombers returning from missions. Hitherto, the Air Force had focused on reinforcing areas that showed the most damage on return. Wald convinced them instead to focus on the areas that consistently showed no damage. He reasonedContinue reading “Industry Spotlight – The Military”
Why Analytics Projects Fail – 5 Reasons
With the news full of so many successes in the fields of analytics, machine learning and artificial intelligence, it is easy to lose sight of the high failure rate of analytics projects. McKinsey just came out with a report that only 8% of big companies (revenue > $ 1 billion) have successfully scaled and integratedContinue reading “Why Analytics Projects Fail – 5 Reasons”
Political Analytics and Microtargeting
The statistics of targeting individual voters with specific messages, as opposed to messaging that went to whole groups, began in the U.S over a decade ago with the Democrats. Political targeting is now an established business, or at least a discipline within the broader realm of political consulting. By 2016, the Republicans had surged wellContinue reading “Political Analytics and Microtargeting”
The Statistics of Persuasion
The Art of Persuasion is the title of more than one book in the self-help genre, books that have spawned blogs, podcasts, speaking gigs and more. But the science of persuasion is actually of more interest, because it produces useful rules that can be studied and deployed. Marketers and politicians have long been enthusiastic usersContinue reading “The Statistics of Persuasion”
Book Review: Thinking Fast and Slow
Daniel Kahneman won a Nobel Prize in Economics for his work in behavioral economics, much of it with his colleague Amos Tversky, who died in 2006. Kahneman’s 2011 classic, Thinking Fast and Slow, is a superbly-written non-technical summary of their fascinating research and its often counter-intuitive findings. The best feature of the book is theContinue reading “Book Review: Thinking Fast and Slow”
Football Analytics
Preparing for the Superbowl Your team is at midfield, you have the ball, it’s 4th down with 2 yards to go. Should you go for it? (Apologies in advance to our many readers, especially those outside the U.S., who are not aficionados of American football, but it’s Superbowl week in the U.S. A quick guideContinue reading “Football Analytics”
Job Spotlight: Digital Marketer
A digital marketer handles a variety of tasks in online marketing – managing online advertising and search engine optimization (SEO), implementing tracking systems (e.g. to identify how a person came to a retailer), web development, preparing creatives, implementing tests, and, of course, analytics. There are typically three types of employers: Marketing agencies that contract outContinue reading “Job Spotlight: Digital Marketer”
Things are Getting Better
In the visualization below, which line do you think represents the UN’s forecast for the number of children in the world in the year 2100? Hans Rosling, in his book Factfulness, presents this chart and notes that in a sample of Norwegian teachers, only 9% correctly identified the correct answer. Rosling, who died two yearsContinue reading “Things are Getting Better”
Work and Heat
If you are working on New Year’s Eve or New Year’s Day, odds are it is from home, where you can (usually) control the temperature in the home. Which, from the standpoint of productivity, is a good thing. According to a study from Cornell, raising the office temperature from 68 degrees to 77 degrees increasedContinue reading “Work and Heat”
The Statistics of Christmas Trees
A researcher shakes a sprig from a Christmas tree, and counts the number of needles that fall. He then repeats the process for countless other sprigs. The sprigs are from a variety of species, and the goal is to determine which species do the best job of retaining their needles. Falling needles are a definiteContinue reading “The Statistics of Christmas Trees”
The False Alarm Conundrum
False alarms are one of the most poorly understood problems in applied statistics and biostatistics. The fundamental problem is the wide application of a statistical or diagnostic test in search of something that is relatively rare. Consider the Apple Watch’s new feature that detects atrial fibrillation (afib). Among people with irregular heartbeats, Apple claims aContinue reading “The False Alarm Conundrum”
Eli Whitney and Google
This weekend (12/8/2018) marked the 253rd anniversary of the birth of Eli Whitney, inventor of the cotton gin. And 20 years ago, Google received its first big infusions of capital from, among others, Jeff Bezos, the founder of Amazon. Both Eli Whitney and the Google founders instigated economic revolutions, but also illustrate polar opposite approachesContinue reading “Eli Whitney and Google”
How Google Determines Which Ads you See
A classic machine learning task is to predict something’s class, usually binary – pictures as dogs or cats, insurance claims as fraud or not, etc. Often the goal is not a final classification, but an estimate of the probability of belonging to a class (propensity), so the cases can be ranked. A good example ofContinue reading “How Google Determines Which Ads you See”
Triage and Artificial Intelligence
Predictim is a service that scans potential babysitters’ social media and other online activity and issues them a score that parents can use to select babysitters. Jeff Chester, the executive director of the Center for Digital Democracy, commented: There’s a mad rush to seize the power of AI to make all kinds of decisions withoutContinue reading “Triage and Artificial Intelligence”
Deming’s Funnel Problem
W. Edwards Deming’s funnel problem is one of statistics’ greatest hits. Deming was a noted statistician who took the statistical process control methods of Shewhart and expanded them into a holistic approach to manufacturing quality. Initially, his ideas were cooly received in the US and he ended up implementing them first in Japan. The successContinue reading “Deming’s Funnel Problem”
Industry Spotlight: the Auto Industry
The auto industry serves as a perfect exemplar of three key eras of statistics and data science in service of industry: Total Quality Management (TQM) First in Japan, and later in the U.S., the auto industry became an enthusiastic adherent to the Total Quality Management philosophy. Fundamentally, TQM is all about using data to improveContinue reading “Industry Spotlight: the Auto Industry”
Analytics Professionals – Must They Be Good Communicators?
Most job ads in the technical arena list communication among the sought-after skills; it consistently outranks many programming and analytical skills. Is it for real, or is it just thrown in there by the HR Department on general principle? The founder of a leading analytics services firm told me that good biostatisticians in the pharmaceuticalContinue reading “Analytics Professionals – Must They Be Good Communicators?”