statistics.com

[henthreery]

Hello,

First time posting here, and am excited to have discovered this community. I'm hoping to get some advice on a modeling problem I'm working on (not homework).

The goal is to measure the difficulty of different environments (a categorical variable) on the probability of successfully performing a task.

I have few thousand subjects who were initially assigned to one of about 20 environments where they each performed a task a different number of times resulting in some number of successes. The subjects are then assigned to a different environment, and repeat the task. We want to know what effect the environment has on the probability of success.

However, the number of task repetitions for each subject in an environment is not controlled, and ranges from 1 to 600 trials. Furthermore, the same subject performs a different number of trials in the second environment than in the first.

So I have data that looks like (N = trials, X = successes, Rate = X/N):

[alethephant]

With respect to your dataset:

1. You have ~ 20 "environments" labeled "A", "B", etc.

2. You have many subjects, labeled by number.

3. Each subject, which is assigned to a 1st environment repeats an unspecified task a number of times with a success rate. The number of trials is denoted N, the number of successes as X and the "Rate" as X/N.

4. The subject is then assigned to a second environment, and repeats the same task again. You are using "1" and "2" to denote the two environments, with the order "1" first and "2" second in a crossover trial for that subject.

Is all of this correct?

If so, how did you assign environments to subjects and the crossovers?

So far your problem appears to be a standard two-arm crossover trial experiment with a binary outcome variable measured in replicate.

A key question is the carryover effect. Is there a learning curve for the activity you are measuring? I.e., do you expect better performance after completing the first set on the first environment, so that a better score will be obtained on the second environment, everything else being equal?

If you measured a subset of subjects on the same environment twice, you would measure the carryover effect.

[henthreery]

Hi, and thanks for the quick reply.

You are correct in your description of the data.

This was not a controlled experiment, and the data were simply collected "in the wild" from historical observations. Thus, the assignment of the 2nd environment was chosen by factors beyond my control, rather than with an experimental design in mind.

To a first approximation could be considered non-uniformly random. I say non-uniformly because there are "clusters" of assignment pairs in the data -- a subject in environment B might get moved to environment C 80% of the time, environment D 20% of the time, and environment A or E 0% of the time. I think this affects the balance of the data, hence my concerns about how to weight subjects.

We can assume the carryover effect & learning curve are washed out between assignments.

Thanks again for any advice or pointers on how to proceed.

Henthreery

[alethephant]

What you have is something similar to an incomplete blocks experiment. Subject is similar to "blocks", and is a random nuisance factor.

The optimal way to analyze the data is a generalized linear mixed model, i.e., logistic regression with a random Subjects factor.

You could use a fixed effect model, but

fit1<- glm(cbind(X, N-X) ~ Subj + Env, data=yourdata, family=binomial(link='logit'))

will result in "Subj" with thousands of levels to be fitted. "Env" is okay as a fixed factor with 20+ levels. Note that the grouped data will be properly weighted in the fit, so you don't have to worry about it. You don't need to worry about X = 0 either.

Modeling "Subj" as a random factor is much, much better in this circumstance, but requires a GLMM program. In R,

require('MASS')
require('nlme')
fit2<- glmmPQL(cbind(X, N-X) ~ Env, random = ~ 1 | Subj, data=yourdata, family = binomial(link='logit')))

You should be aware that mixed modeling for large datasets is a current research area, and generalized linear mixed models also. So problems with the software may occur when you do your fits. In particular, expect it may take a very long time to converge on the optimum solution.

The information you are interested in is the coefficient set of "Env", which will be differences from the Env = A effect. The "Subj" effect will be reported as a standard deviation of the effects of a population of subjects.

If you need more detailed help with the actual dataset,

1. Contact a local statistician.
2. Enroll in the Mixed and Hierarchical Models course at statistics.com and ask you questions of the instructor.
3. Buy some consulting at statistics.com.

Good luck!

[henthreery]

This is great, and just what I needed! Thank you for a fantastic, detailed reply.

[kevin84johnson]

Thank you so much for the very informative information.. Nice post.. Thanks guys..
_________________
modern furniture online
modern sofas