statistics.com

[Leos1981]

Hi everybody,

I have the following problem/question:

I researched the results of 10 independent surveys conducted in 3 different countries. All surveys analysed 3 identical decsision criteria and their importance for 10 different sample groups. The importance was measured by a 5-point-likert scale (1=very unimportant, 5=very important). Each survey reports the means and standard deviations for each of the three criteria. So for one criteria I have following data available:

Criteria 1 Country 1 Survey 1 Mean x1 Std. dev. y1
Criteria 1 Country 1 Survey 2 Mean x2 Std. dev. y2
Criteria 1 Country 1 Survey 3 Mean x3 Std. dev. y3
Criteria 1 Country 1 Survey 4 Mean x4 Std. dev. y4

Criteria 1 Country 2 Survey 5 Mean x5 Std. dev. y5
Criteria 1 Country 2 Survey 6 Mean x6 Std. dev. y6
Criteria 1 Country 2 Survey 7 Mean x7 Std. dev. y7
Criteria 1 Country 2 Survey 8 Mean x8 Std. dev. y8

Criteria 1 Country 3 Survey 9 Mean x9 Std. dev. y9
Criteria 1 Country 3 Survey 10 Mean x10 Std. dev. y10

I want to figure out if the item "country" functions as a moderator. How can I do this? I conducted an ANOVA a while ago, and if I remember correctly I cannot use this method here as there are different numbers of surveys per country and the means are not normally distributed. Can I use a non-parametric test for this problem? If so, which one?

I also had a look at meta-analytical methods. However, a homogeneity analysis (with chi square as reference value) doesn't work because unrealistic & very high Q values result no matter which criteria I use. Obviously, some requirements for this method are violated.

Thanks so much for any help in advance.

Best

Leo

[alethephant]

Q1. How many subjects are in your groups?

Q2. How do you know the means "are not normally distributed"? (Depends on the answer to Q1.)

Nonparametric tests require assumptions too, typically symmetry and only differences in means, not dispersion. However, an obvious test that comes to mind is a bootstrap or a randomization test on your original data. For the randomization test, you could use the ANOVA fixed effect F-test statistic as the figure of merit.

But, unless your group sizes are very small, the means should be acceptably normally distributed, even if the individual Likert responses are not.

You can also do ordinal logistic regression as a means to your answer.

Have you looked at a tornado or forest plot of your confidence intervals on the individual means?

If you have any sample data for your problem, this would be helpful.

[Leos1981]

Hi Alethephant,

thanks for your answer. Regarding your questions:

Q1: per survey, there are sample sizes ranging from minimum of 12 and max. of 120.

Q2: you are right, I do not know for sure that there is no normal distribution for the means. However, there is no indication in the survey that they are normally distributed.
Moreover, I thought that I cannot use an ANOVA if there are different numbers of items (surveys) per group (in my case: per country).

One actual data set is (5-point-likert scale):

Criteria 1 USA mean = 4.32 std.dev.= 0.54 sample size: 56
Criteria 1 USA mean = 3.94 std.dev.= 0.65 sample size: 12
Criteria 1 USA mean = 4.5 std.dev.= 0.45 sample size: 37
Criteria 1 USA mean = 3.73 std.dev.= 0.87 sample size: 46

Criteria 1 Germany mean = 3.62 std.dev.= 0.33 sample size: 62
Criteria 1 Germany mean = 3.2 std.dev.= 0.76 sample size: 120
Criteria 1 Germany mean = 3.84 std.dev.= 0.5 sample size: 100
Criteria 1 Germany mean = 3.52 std.dev.= 0.64 sample size: 39

Criteria 1 Canada mean = 4.6 std.dev.= 0.32 sample size: 80
Criteria 1 Canada mean = 4.55 std.dev.= 0.39 sample size: 57

I just want to figure out if "country" moderates the means. Unfortunately, I do not have the original data per survey (only means, sample size and std.dev.)

I will follow your suggestions and take a look at tornado or forest plots.

Thanks again for your help.
Best
Leo

[alethephant]

Your sample sizes do appear large enough to presume normality on the mean scores (except the one at 12). So you can start with that assumption.

The within-country standard deviations appear homogeneous enough to consider the surveys as commensurate with respect to variance. However, you may wish to do a Bartlett or Levene test to reassure yourself. (Usually the result doesn't do you much good to know, though.)

Given all this, you can calculate the standard errors of the mean scores by dividing the standard deviations by sqrt(n(i)) in each case. Now you've got means with standard errors.

Do a weighted average for each country individually, and compute the standard deviation of that weighted average.

Once you've done this for all countries, you can compare countries to each other using pairwise normal z tests with Bonferroni multiple comparison corrections, if you wish.

This is really just a study in how to use large-sample normal approximations. There has been so much use of small-sample methods in the last 60 years that people have forgotten how easy and general large-sample methods are.

[Leos1981]

Thanks a lot Alethephant
I will follow your instructions and see what results I get.

Best

Leo

[Leos1981]

Hi again,

I have one more question regarding the Levene test described above. As I have only aggregated data from different studies (mean, std. dev., sample size), I think a normal Levene test is not possible. My total sample size might be very large (see example above) however the single data available constists of 10 mean values only. So, SPSS or any other statistics software will only calculate as if my data consists of a population of 10 samples. This leads almost always to variance heterogeneity. Is there any way around this problem? I thought about a simulation of values?!

Thanks again for your help.

Best Leo

[alethephant]

1. You can presume your means are normally distributed, if they are based on a considerable (e.g., > 30) number of data points.

2. Any test of normality would be applied to the data behind the mean, and then only if the data is scant.

3. Once a mean is identified as normally distributed, any linear combination of means will also be normally distributed.

4. It does not matter if the means have different standard errors, so long as you take this into account when you do testing or analysis.

5. Again, I suggest you analyze your collection of data based solely on normal methods, taking into account the individual standard errors in the proper way.