With the growing use of virtual learning environments (VLE), innovative methods to evaluate their performance are increasingly needed. A key difficulty in evaluating VLE using system logs is the large heterogeneity of usage patterns. The current study demonstrates an approach to classify complex patterns of student-level and classroom-level usage with latent class analysis, then estimate average treatment effects (ATEs) of membership in student or classroom classes, as well as joint effects. The approach accounts for uncertainty of latent classes with a three-step method, and nonrandom selection
into classes using inverse probability weighting. We demonstrate the approach with an analysis of usage of an Algebra VLE and estimate causal effects of latent class membership on a high-stakes Algebra
I standardized assessment. Challenges of using system logs for evaluation of VLE are discussed with respect to measurement error, construct validity, latent classes enumeration, and comparison of classes with respect to distal outcomes.
Acknowledgements. The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305C160004 to University of Florida. The opinions expressed are those of the authors and do not represent the views of the Institute or the U.S. Department of Education.