under the null hypothesis of no difference among treatments, chance might account for the observed difference(s) in the samples. The procedure for a two-sample test of means is as follows (sample 1 has N1 observations, sample 2 has N2 observations):
- Put all observations together in a hat,
- Shuffle the hat
- Draw a sample of size N1 from the hat, find the mean
- Draw the remaining N2 values, find the mean
- Record the difference in means
- Repeat steps 2-5 many times (say 1000)
- Find out how often the resampled difference in means is as extreme as the observed difference in means. This is the p-value.