#### Making Predictions Self-Fulfilling Prophecies

The Three Stooges comedy sketch “Ants in the Pantry,” (1935), features the Stooges as exterminators whose target customers don’t have enough pests.  The Stooges solve the problem by bringing their own pests (ants and mice) on sales calls, and surreptitiously releasing them. Having created the…

Comments Off on Making Predictions Self-Fulfilling Prophecies

#### Gangs and Covid

Dr. Carlos Carcach is Professor & Director of the Center for Public Policy at the Escuela Superior de Economía y Negocios (ESEN) in Santa Tecla, El Salvador, and coordinator of ESEN's post-graduate program in predictive analytics, which offers online instruction in partnership with Statistics.com, using…

Comments Off on Gangs and Covid

#### Detecting a Slots Payout Difference of 2%

Most businesses use statistics and analytics to one degree or another, but there is only one industry that is built solely on this discipline.  This week we look at the casino business - in particular, the odds on slots. Slot machines are a casino’s best…

Comments Off on Detecting a Slots Payout Difference of 2%

#### Data Analytics

Terminology in Data Analytics As data continue to grow at a faster rate than either population or economic activity, so do organizations' efforts to deal with the data deluge, and use it to capture value.  And so do the methods used to analyze data, which…

#### Social Network Analysis (SNA) in Medicine

In hospitals, “sentinel events” are events that carry with them a significant risk of unexpected death or harm.  It is estimated that ⅔ of such sentinel events result from communications failures during the handoff of a patient from one provider to another (e.g. during a…

Comments Off on Social Network Analysis (SNA) in Medicine

#### Problem of the Week: Simpson’s Paradox – baseball

Question: A baseball team is comparing two of its hitters, Hernandez and Dimock. Hernandez hit .250 in 2017 and .275 in 2018. Dimock did worse in both years - .245 in 2017 and .270 in 2018. Overall, though, Dimock hit better across the two years,…

#### Matching Algorithms

Some applications of machine learning and artificial intelligence are recognizably impressive - predicting future hospital readmission of discharged patients, for example, or diagnosing retinopathy. Others - self-driving cars, for example - seem almost magical. The matching problem, though, is one where your first reaction might…

#### Instructor Spotlight: Cliff Ragsdale

Cliff T. Ragsdale teaches several courses for the Institute in the area of operations research, based on his best selling text “Spreadsheet Modeling and Decision Analysis.”  One of Cliff’s special talents is making his subject, which can be quite challenging technically, widely accessible. His courses do…

Comments Off on Instructor Spotlight: Cliff Ragsdale

#### Industry Spotlight: Consulting

When a new technology arrives, consulting companies can quickly add staff and expertise to build institutional capacity centered around the technology in ways companies focused on delivering their own products and services cannot.  Large consulting companies like Booz Allen and McKinsey, as well as smaller…

Comments Off on Industry Spotlight: Consulting

#### Industry Spotlight: Package Delivery Business

Nothing better illustrates the encroachment of data science and analytics on the older “economy of tangible things” than the business of delivering packages. The use of analytics in package delivery is not new. Companies like UPS and Fedex are longtime users of operations research methods…

#### Industry Spotlight: Credit Scoring

In the U.S., credit scoring is dominated by three companies - Experian, TransUnion and Equifax, employing roughly 30,000 people.  An important player in the scoring methodology is FICO, previously Fair Isaac Corporation, and the scores are typically called “FICO scores.”  Credit scoring is the oldest…

Comments Off on Industry Spotlight: Credit Scoring

#### Industry Spotlight: Automotive

The auto industry serves as a perfect exemplar of three key eras of statistics and data science in service of industry: Total Quality Management (TQM) First in Japan, and later in the U.S., the auto industry became an enthusiastic adherent to the Total Quality Management…

Comments Off on Industry Spotlight: Automotive

#### Feature Engineering and Data Prep – Still Needed?

It is a truism of machine learning and predictive analytics that 80% of an analyst's time is consumed in cleaning and preparing the needed data. I saw an estimate by a Google engineer that 25% of the time was spent just looking for the right…

Comments Off on Feature Engineering and Data Prep – Still Needed?

#### Job Spotlight: Risk Analyst

Many jobs are centered around risk management. If you're looking through job postings, of course, you'll see lots of jobs whose purpose is to make sure that nothing bad happens - the equivalent of locking the doors and closing the windows. More interesting from a…

Comments Off on Job Spotlight: Risk Analyst

#### Problem of the Week: The Value of Bedrooms

Question: You work for an internet real-estate company, building statistical models to predict home price on the basis of square footage, number of bedrooms, number of bathrooms, property type (single family home, townhouse, multiplex), and age. Surprisingly, you find the coefficient for bedrooms is negative,…

Comments Off on Problem of the Week: The Value of Bedrooms

#### Statistically Significant – But Not True

If you are looking for the Feature Engineering blog post, you can find it here: https://www.statistics.com/feature-engineering-data-prep-still-needed/ In 2015, at an Alzheimer's conference, Biogen researchers presented dramatic brain scans showing that the antibody aducanumab effectively cleared out plaque in the brain, plaque that was associated with…

Comments Off on Statistically Significant – But Not True

#### Industry Spotlight – Precision Agriculture

The application of analytics to agriculture has given rise to what is called "precision agriculture", a science that seeks to take advantage of and use detailed information that is local in time and place. Tractors and farm equipment are being equipped with sensors and software…

Comments Off on Industry Spotlight – Precision Agriculture

#### Industry Spotlight: Credit Scoring

In the U.S., credit scoring is dominated by three companies - Experian, TransUnion and Equifax, employing roughly 30,000 people. An important player in the scoring methodology is FICO, previously Fair Isaac Corporation, and the scores are typically called "FICO scores." Credit scoring is the oldest…

Comments Off on Industry Spotlight: Credit Scoring

#### Industry Spotlight: The IRS is Watching You

The IRS (U.S. Internal Revenue Service) has been using computers to choose tax returns for audit since 1962. Early on, the selection was rule-based, but the IRS turned to statistical modeling in 1969, using the oldest predictive analytics model in the toolbox - discriminant analysis.…

Comments Off on Industry Spotlight: The IRS is Watching You

#### Industry Spotlight: Package Delivery

Nothing better illustrates the encroachment of data science and analytics on the older "economy of tangible things" than the business of delivering packages. The use of analytics in package delivery is not new. Companies like UPS and Fedex are longtime users of operations research methods…

Comments Off on Industry Spotlight: Package Delivery

#### Ethical Practice in Data Mining

Prior to the advent of internet-connected devices, the largest source of big data was public interaction on the internet. Social media users, as well as shoppers and searchers on the internet, make an implicit deal with the big companies that provide these services: users can…

Comments Off on Ethical Practice in Data Mining

#### Industry Spotlight: Customer Segmentation

Are you "young and rustic?" Or perhaps a "toolbelt traditionalist?" These are nicknames given to customer segments identified by market research firm Claritas, with its statistical clustering tool. Long before the advent of individualized product recommendations, business sought to segment customers into distinct groups on…

Comments Off on Industry Spotlight: Customer Segmentation

#### Handling the Noise – Boost It or Ignore It?

In most statistical modeling or machine learning prediction tasks, there will be cases that can be easily predicted based on their predictor values (signal), as well as cases where predictions are unclear (noise). Two statistical learning methods, boosting and ProfWeight, use those difficult cases in…

Comments Off on Handling the Noise – Boost It or Ignore It?

#### “Defiant” Supervision

How did the phrase "defiantly recommend", as in "I defiantly recommend this product," come into common usage on the internet? The answer is a good look inside the workings of supervised learning. Supervision, generally from humans, is instrumental in much of statistical and machine learning.…

People in Alaska are extraordinarily generous - that's what a predictive model showed, when applied to a charitable organization's donor list. A closer examination revealed a flaw - while the original data was for all 50 states, the model's training data for Alaska included donors,…

#### Industry Spotlight – The Military

Abraham Wald, a persecuted Jewish mathematician who fled Austria just before World War II, led an analysis of allied bombers returning from missions. Hitherto, the Air Force had focused on reinforcing areas that showed the most damage on return. Wald convinced them instead to focus…

Comments Off on Industry Spotlight – The Military

#### Why Analytics Projects Fail – 5 Reasons

With the news full of so many successes in the fields of analytics, machine learning and artificial intelligence, it is easy to lose sight of the high failure rate of analytics projects. McKinsey just came out with a report that only 8% of big companies…

Comments Off on Why Analytics Projects Fail – 5 Reasons

#### Political Analytics and Microtargeting

The statistics of targeting individual voters with specific messages, as opposed to messaging that went to whole groups, began in the U.S over a decade ago with the Democrats. Political targeting is now an established business, or at least a discipline within the broader realm…

Comments Off on Political Analytics and Microtargeting

#### The Statistics of Persuasion

The Art of Persuasion is the title of more than one book in the self-help genre, books that have spawned blogs, podcasts, speaking gigs and more. But the science of persuasion is actually of more interest, because it produces useful rules that can be studied…

Comments Off on The Statistics of Persuasion

#### Book Review: Thinking Fast and Slow

Daniel Kahneman won a Nobel Prize in Economics for his work in behavioral economics, much of it with his colleague Amos Tversky, who died in 2006. Kahneman's 2011 classic, Thinking Fast and Slow, is a superbly-written non-technical summary of their fascinating research and its often…

Comments Off on Book Review: Thinking Fast and Slow

#### Football Analytics

Preparing for the Superbowl Your team is at midfield, you have the ball, it's 4th down with 2 yards to go. Should you go for it? (Apologies in advance to our many readers, especially those outside the U.S., who are not aficionados of American football,…

#### Job Spotlight: Digital Marketer

A digital marketer handles a variety of tasks in online marketing - managing online advertising and search engine optimization (SEO), implementing tracking systems (e.g. to identify how a person came to a retailer), web development, preparing creatives, implementing tests, and, of course, analytics. There are…

Comments Off on Job Spotlight: Digital Marketer

#### Things are Getting Better

In the visualization below, which line do you think represents the UN's forecast for the number of children in the world in the year 2100? Hans Rosling, in his book Factfulness, presents this chart and notes that in a sample of Norwegian teachers, only 9%…

Comments Off on Things are Getting Better

#### Work and Heat

If you are working on New Year's Eve or New Year's Day, odds are it is from home, where you can (usually) control the temperature in the home. Which, from the standpoint of productivity, is a good thing. According to a study from Cornell, raising…

Comments Off on Work and Heat

#### The Statistics of Christmas Trees

A researcher shakes a sprig from a Christmas tree, and counts the number of needles that fall. He then repeats the process for countless other sprigs. The sprigs are from a variety of species, and the goal is to determine which species do the best…

Comments Off on The Statistics of Christmas Trees

#### The False Alarm Conundrum

False alarms are one of the most poorly understood problems in applied statistics and biostatistics. The fundamental problem is the wide application of a statistical or diagnostic test in search of something that is relatively rare. Consider the Apple Watch's new feature that detects atrial…

Comments Off on The False Alarm Conundrum

This weekend (12/8/2018) marked the 253rd anniversary of the birth of Eli Whitney, inventor of the cotton gin. And 20 years ago, Google received its first big infusions of capital from, among others, Jeff Bezos, the founder of Amazon. Both Eli Whitney and the Google…

A classic machine learning task is to predict something's class, usually binary - pictures as dogs or cats, insurance claims as fraud or not, etc. Often the goal is not a final classification, but an estimate of the probability of belonging to a class (propensity),…

#### Triage and Artificial Intelligence

Predictim is a service that scans potential babysitters' social media and other online activity and issues them a score that parents can use to select babysitters. Jeff Chester, the executive director of the Center for Digital Democracy, commented: There's a mad rush to seize the…

Comments Off on Triage and Artificial Intelligence

#### Deming’s Funnel Problem

W. Edwards Deming's funnel problem is one of statistics' greatest hits. Deming was a noted statistician who took the statistical process control methods of Shewhart and expanded them into a holistic approach to manufacturing quality. Initially, his ideas were cooly received in the US and…

Comments Off on Deming’s Funnel Problem

#### Industry Spotlight: the Auto Industry

The auto industry serves as a perfect exemplar of three key eras of statistics and data science in service of industry: Total Quality Management (TQM) First in Japan, and later in the U.S., the auto industry became an enthusiastic adherent to the Total Quality Management…

Comments Off on Industry Spotlight: the Auto Industry

#### Analytics Professionals – Must They Be Good Communicators?

Most job ads in the technical arena list communication among the sought-after skills; it consistently outranks many programming and analytical skills. Is it for real, or is it just thrown in there by the HR Department on general principle? The founder of a leading analytics…

Comments Off on Analytics Professionals – Must They Be Good Communicators?

In a couple of days, theWall Street Journalwill come out with its November survey of economists' forecasts. It's a particularly sensitive time, with elections in a few days and President Trump attacking the Federal Reserve for for raising interest rates. It's a good time to…

#### 100 years of variance

It is 100 years since R A Fischer introduced the concept of "variance" (in his 1918 paper "The Correlation Between Relatives on the Supposition of Mendelian Inheritance"). There is much that statistics has given us in the century that followed. Randomized clinical trials, and the means…

Comments Off on 100 years of variance

#### Course Spotlight: Structural Equation Modelling (SEM)

SEM stands for "structural equation modeling," and we are fortunate to have Prof. Randall Schumacker teaching this subject at Statistics.com. Randy created the Structural Equation Modeling (SEM) journal in 1994 and the Structural Equation Modeling Special Interest Group (SIG) at the American Educational Research Association…

Comments Off on Course Spotlight: Structural Equation Modelling (SEM)

#### Benford’s Law Applies to Online Social Networks

Fake social media accounts and Russian meddling in US elections have been in the news lately, with Mark Zuckerberg (Facebook founder) testifying this week before the US Congress. Dr. Jen Golbeck, who teaches Network Analysis at Statistics.com, published an ingenious way to determine whether a…

Comments Off on Benford’s Law Applies to Online Social Networks

#### Masters Programs versus an Online Certificate in Data Science from Statistics.com

We just attended the analytics conference of INFORMS' (The Institute for Operations Research and the Management Sciences) this week in Baltimore, and they held a special meeting for directors of academic analytics programs to better align what universities are producing with what industry is seeking.…

Comments Off on Masters Programs versus an Online Certificate in Data Science from Statistics.com

#### Course Spotlight: Customer Analytics in R

"The customer is always right" was the motto Selfridge's department store coined in 1909. "We'll tell the customer what they want" was Madison Avenue's mantra starting in the 1950's. Now data scientists like Karolis Urbonas help companies like Amazon (where he works in Europe as…

Comments Off on Course Spotlight: Customer Analytics in R

#### Course Spotlight: Predictive Analytics

Predicting whether an internet user will click on a link or buy a product, whether an insurance claim is fraudulent, whether a home mortgage will be paid on time (or early), how much a house will sell for, what internet ad you should see next,…

Comments Off on Course Spotlight: Predictive Analytics

#### “Money and Brains” and “Furs and Station Wagons”

"Money and Brains" and "Furs and Station Wagons" were evocative customer shorthands that the marketing company Claritas came up with over a half century ago. These names, which facilitated the work of marketers and sales people, were shorthand descriptions of segments of customers identified through…

Comments Off on “Money and Brains” and “Furs and Station Wagons”

#### Kidney Donations – Optimizing Them

Did you know that there is a country in which kidneys can be legally bought and sold? That country is Iran, and there are actually two models for payment. In one part of the country, donors receive a flat fee, and in the other part…

Comments Off on Kidney Donations – Optimizing Them

When Apple CEO Tim Cook finally unveiled his company's new Apple Watch in a widely-publicized rollout earlier this month, most of the press coverage centered on its cost (\$349 to start) and whether it would be as popular among consumers as the iPod or iMac.…

#### Course Spotlight: The Text Analytics Sequence

Text analytics or text mining is the natural extension of predictive analytics, and Statistics.com's text analytics program starts Feb. 6. Text analytics is now ubiquitous and yields insight in: Marketing: Voice of the customer, social media analysis, churn analysis, market research, survey analysis Business: Competitive…

Comments Off on Course Spotlight: The Text Analytics Sequence

#### Course Spotlight: Constrained Optimization

Say you operate a tank farm (to store and sell fuel). How much of each fuel grade should you buy? You have specified flow and storage capacities, constraints on what types of fuels can be stored in which tanks, prior contractual obligations about minimum monthly…

Comments Off on Course Spotlight: Constrained Optimization

#### Big Data and Clinical Trials in Medicine

There was an interesting article a couple of weeks ago in the New York Times magazine section on the role that Big Data can play in treating patients -- discovering things that clinical trials are too slow, too expensive, and too blunt to find. The…

Comments Off on Big Data and Clinical Trials in Medicine

#### Industry Spotlight: The brand premium for Chanel and Harvard

The classic illustration of the power of brand is perfume - expensive perfumes may cost just a few dollars to produce but can be sold for more than \$500 due to the cachet afforded by the brand. David Malan's Computer Science course at Harvard, CSCI…

Comments Off on Industry Spotlight: The brand premium for Chanel and Harvard

#### Convoys

Ever wonder why, in World War II, ships in convoys were safer than ships traveling on their own? Most people assume it was due to the protection afforded by military escort vessels, of which there was a limited supply (insufficient to protect ships traveling on…

#### Dialects

When talking to several people, do you address them as "you guys"? "Y'all"? Just "you"? And is the carbonated soft drink "soda" or "pop?" Maps based on survey responses to questions like this were published in the Harvard Dialect Survey in 2003. Josh Katz took…

#### Needle in a Haystack

What's the probability that the NSA examined the metadata for your phone number in 2013? According to John Inglis, Deputy Director at the NSA, it's about 0.00001, or 1 in 100,000. A surprisingly small number, given what we've all been reading in the media about…

Comments Off on Needle in a Haystack

#### Predictive Modeling and Typhoon Relief

The devastation wrought by Super-Typhoon Haiyan in the Philippines is the biggest test yet for the nascent technology of "artificial intelligence disaster response," a phrase used by Patrick Meier, a pioneer in the field. When disaster strikes, a flood of social media posts and tweets…

Comments Off on Predictive Modeling and Typhoon Relief

#### Terrorist Clusters

The "righteous vengeance gun attack" is just one of 10 types of terrorism identified by Chenoweth and Lowham via statistical clustering techniques. Another cluster is "bombings of a public population where a liberation group takes responsibility." You can read about the 10 clusters, and the…