In an experiment, an arm is a treatment protocol - for example, drug A, or placebo. In medical trials, an arm corresponds to a patient group receiving a specified therapy. The term is also relevant for bandit algorithms for web testing, where an arm consists…
A sparse matrix typically refers to a very large matrix of variables (features) and records (cases) in which most cells are empty or 0-valued. An example might be a binary matrix used to power web searches - columns representing search terms and rows representing searches,…
We continue our effort to shed light on potentially confusing usage of terms in the different data science communities. In statistics, a sample is a collection of observations or records. It is often, but not always, randomly drawn. In matrix form, the rows are records…
With this entry, we inaugurate a new effort to shed light on potentially confusing usage of terms in the different data science communities. In statistics and machine learning, normalization of variables means to subtract the mean and divide by the standard deviation. When there are…
HDFS is the Hadoop Distributed File System. It is designed to accommodate parallel processing on clusters of commodity hardware, and to be fault tolerant.
The Kruskal-Wallis test is a nonparametric test for finding if three or more independent samples come from populations having the same distribution. It is a nonparametric version of ANOVA.
A "discovery" is a hypothesis test that yields a statistically significant result. The false discovery rate is the proportion of discoveries that are, in reality, not significant (a Type-I error). The true false discovery rate is not known, since the true state of nature is not known (if it were, there would be no need for statistical inference).
The 2006 Netflix Contest has come to convey the idea of crowdsourced predictive modeling, in which a dataset and a prediction challenge are made publicly available. Individuals and teams then compete to develop the best performing model.
This week's word is actually a letter. R is a statistical computing and programming language and program, a derivative of the commercial S-PLUS program, which, in turn, was an offshoot of S from Bell Labs.
In time series forecasting, a moving average is a smoothing method in which the forecast for time t is the average value for the w periods ending with time t-1.
In regression models, an interaction term captures the joint effect of two variables that is not captured in the modeling of the two terms individually.
A naive forecast or prediction is one that is extremely simple and does not rely on a statistical model (or can be expressed as a very basic form of a model).
In discrete response models, overdispersion occurs when there is more correlation in the data than is allowed by the assumptions that the model makes.
In a classification model, the confusion matrix shows the counts of correct and erroneous classifications. In a binary classification problem, the matrix consists of 4 cells.
The predictors in a predictive model are sometimes given different terms by different disciplines. Traditional statisticians think in terms of variables.
Structured data is data that is in a form that can be used to develop statistical or machine learning models (typically a matrix where rows are records and columns are variables or features).
Censoring in time-series data occurs when some event causes subjects to cease producing data for reasons beyond the control of the investigator, or for reasons external to the issue being studied.
Predictive modeling is the process of using a statistical or machine learning model to predict the value of a target variable (e.g. default or no-default) on the basis of a series of predictor variables (e.g. income, house value, outstanding debt, etc.).
Goodness-of-fit measures the difference between an observed frequency distribution and a theoretical probability distribution which
An adjacency matrix describes the relationships in a network. Nodes are listed in the top..
In a test of significance (also called a hypothesis test), Type I error is the error of rejecting the null hypothesis when it is true -- of saying an effect or event is statistically significant when it is not.
Data partitioning in data mining is the division of the whole data available into two or three non-overlapping sets: the training set (used to fit the model), the validation set (used to compared models), and the test set (used to predict performance on new data).
Longitudinal data records multiple observations over time for a set of individuals or units. A typical..
Cross-sectional data refer to observations of many different individuals (subjects, objects) at a given time, each observation belonging to a different individual. A simple...
CHAID stands for Chi-squared Automatic Interaction Detector. It is a method for building classification trees and regression trees from a training sample comprising already-classified objects.
Also called the training sample, training set, calibration sample. The context is predictive modeling (also called supervised data mining) - where you have data with multiple predictor variables and a single known outcome or target variable.
The centroid is a measure of center in multi-dimensional space.
Promoting better understanding of statistics throughout the world.
New Editor of Journal of Statistics Education
Read Peter's Letter to the Editor in Saturday's Washington Post.
Last year's popular story out of the Predictive Analytics World conference series was Andrew Pole's presentation of Target's methodology for predicting which customers were pregnant.
Evidence show that there is no significant difference between taking an online introductory statistics course and a traditional in-person class.
Facebook began trading around 11:30 this morning, and I spent 8 minutes
Newly elected American Statistical Association (ASA) Fellow, and recognized for his outstanding professional contributions to and leadership in the field of statistical science.
Arizona's immigration law goes before the Supreme Court this week...
I saw this job posting a while ago, and, in my next life,
February 12 was the 80th anniversary of the birth of Julian Simon, an early pioneer in resampling methods.
Statistics for Future Presidents - Steve Pierson, Director of Science Policy at ASA wrote interesting blog wondering how statistics for future presidents (or policymakers more generally) would compare with the recommended statistical skills/concepts for others. Take a look and let him know!
Teaching Geographic Information Science and Technology in Higher Education, 2012 (Wiley)
The story of the prospective Facebook IPO, and prior IPO's from LinkedIn, Pandora, and Groupon all involve "data scientists". Read an interview with Monica Rogati - Senior Data Scientist at LinkedIn to see the connection.
Dr. Michelle Everson is recognized for her outstanding contributions to and innovation in the teaching of elementary statistics.
John Elder's presentations on common data mining mistakes are a must-see if you have any experience or plans in the data mining arena.
"Any claim coming from an observational study is most likely to be wrong." Thus begins "Deming, data and observational studies," just published in "Significance Magazine" (Sept. 2011).
I was watching a Washington Nationals game on TV a couple of days ago, and the concept of "expected value" ...
A neurosurgeon, pathologist and epidemiologist are each told to examine a can of sardines on a table in a closed room, and present a report.
What do teenagers want? More importantly for the music industry, what music will they buy?
Advertisers shy away from round numbers, believing that $99 appears significantly cheaper than $100...
Did the NCAA get the March Madness rankings right? Check out SportsMeasures.com
What does Matt Asher's article "Attack of the Hair Trigger Bees" have to do with global warming? Matt Asher runs statisticsblog.com ...
The first Gallup Poll was published in October, 1935. In America Speaks,
Thinking about careers that use statistics? The job title "catastrophe modeling assistant" caught my eye recently in a job announcement. ...
