Sensitivity of fi ve information criteria to discriminate covariance structures with missing data in repeated measures designs

  1. Pablo Livacic-Rojas 1
  2. Paula Fernández 2
  3. Guillermo Vallejo 2
  4. Ellián Tuero-Herrero 2
  5. Feliciano Ordóñez 2
  1. 1 Universidad de Santiago de Chile
    info

    Universidad de Santiago de Chile

    Santiago de Chile, Chile

    ROR https://ror.org/02ma57s91

  2. 2 Universidad de Oviedo
    info

    Universidad de Oviedo

    Oviedo, España

    ROR https://ror.org/006gksa02

Journal:
Psicothema

ISSN: 0214-9915,  1886-144X

Year of publication: 2020

Volume: 32

Issue: 3

Pages: 399-409

Type: Article

DOI: 10.7334/PSICOTHEMA2020.63 DIALNET GOOGLE SCHOLAR lock_openOpen access editor

More publications in: Psicothema

Sustainable development goals

Abstract

Backgrounds: This study analyzes the effectiveness of different information criteria for the selection of covariance structures, extending it to different missing data mechanisms, the maintenance and adjustment of the mean structures, and matrices. Method: The Monte Carlo method was used with 1,000 simulations, SAS 9.4 statistical software and a partially repeated measures design (p=2; q=5). The following variables were manipulated: a) the complexity of the model; b) sample size; c) matching of covariance matrices and sample size; d) dispersion matrices; e) the type of distribution of the variable; f) the non-response mechanism. Results: The results show that all information criteria worked well in Scenario 1 for normal and non-normal distributions with heterogeneity of variance. However, in Scenarios 2 and 3, all were accurate with the ARH matrix, whereas AIC, AICCR and HQICR worked better with TOEP and UN. When the distribution was not normal, AIC and AICCR were only accurate in Scenario 3, more heterogeneous and unstructured matrices, with complete cases, MAR and MCAR. Conclusions: In order to correctly select the matrix it is advisible to analyze the heterogeneity, sample size and distribution of the data.

€ View funding

Funding information

Acknowledgements The authors would like express acknowledge the suggestions of revisors to improve this work has been funded by the Chilean National Fund for Scientific and Technological Development (FONDECYT. Ref.: 1170642) and the Spanish Ministry of Science, Innovation and Universities (Ref: PGC2018-101574-B-I00).

Funders

Bibliographic References

  • Aberson, C.L. (2019). Applied Power Analysis for the Behavioral Sciences. Routledge.
  • Blanca, M. J., Arnau, J., López-Montiel, D., Bono, R., & Bendayan, R. (2013). Skewness and kurtosis in real data samples. Methodology:
  • European Journal of Research Methods for the Behavioral and Social Sciences, 9(2), 78-84. https://doi.org/10.1027/1614-2241/a000057.
  • Bono, R., Blanca, M. J., Arnau, J., & Gómez-Benito, J. (2017). Non-normal distributions commonly used in health, education, and social sciences: A systematic review. Frontiers in Psychology-Quantitative Psychology and Measurement, 8, Article 1602, 1-6. https://doi.org/10.3389/ fpsyg.2017.01602
  • Bono, R., Arnau, J., Alarcón, R., & Blanca, M. J. (2020). Bias, precision and accuracy of skewness and kurtosis estimators for frequently used continuous distributions. Symmetry, 12(1), 19. https://doi.org/10.3390/ sym12010019
  • Dong & Peng (2013). Principled missing data methods for researchers. SpringerPlus, 2, 222. https://doi.org/10.1186/2193-1801-2-222
  • Fernández, P., Vallejo G., & Livacic-Rojas P. (2010). Robustez de cinco estadísticos univariados para analizar diseños Split Plot en condiciones adversas [Robustness of fi ve univariate statistics to analyze Split Plot designs in adverse conditions]. Revista Latinoamericana de Psicología, 42(2), 289-309.
  • Fernández, P., Vallejo, G., Livacic-Rojas, P., & Tuero-Herrero, E. (2018). The (Ir)Responsibility of (Under)Estimating Missing Data. Frontiers in Psychology. 9, 556-568. https://doi.org/10.3389/fpsyg.2018.00556
  • Fleishman, A.I. (1978). A method for simulating non-normal distributions. Psychometrika, 43, 521-532. https://doi.org/10.1007/BF02293811
  • Funatogawa, I., & Funatogawa, T. (2019). Longitudinal Data Analysis. Autoregresive Linear Mixed Effects Models. Springer.
  • Garson, G.D. (2020). Multilevel Modeling: Applications in STATA®, IBM® SPSS®, SAS®, R, & HLM™. Sage Publications.
  • Grissom, R., & Kim, J. (2012). Effect size for research: Univariate and multivariate applications. 2nd edition. Routledge.
  • Grund, S., Lüdtke, O., & Robitzsch, A. (2019). Missing data in Multilevel Research. In S.E. Humphrey & J.M. LeBreton (Editors-in-Chief), The Handbook of Multilevel Theory, Measurement and Analysis (pp. 365- 386). American Psychological Association.
  • Heck, R., & Thomas, S.L. (2020). An Introduction to Multilevel Modeling Techniques. MLM and SEM Approaches Using Mplus. Routledge.
  • Leke, C. A., & Marwala, T. (2019). Deep Learning and Missing Data in Engineering Systems (Studies in Big Data Book 48). Springer.
  • Little, R.J.A., & Rubin D.B. (2020). Statistical Analysis with Missing Data (3rd edition). Wiley-Blackwell.
  • Livacic-Rojas P., Vallejo G., & Fernández, P. (2013). Covariance structures selection and Type I Error rates in Split Plot designs. Methodology. European Journal of Research Methods for the Behavioural and Social Sciences, 9(4), 129-136. https://doi.org/10.1027/1614-2241/a000058.
  • Livacic-Rojas P., Vallejo G., & Fernández, P. (2017). Power of modifi ed Brown-Forsythe and Mixed-Model Approaches in Split-Plot Designs. Methodology. European Journal of Research Methods for the Behavioural and Social Sciences, 13(1), 9-22. https://doi.org/10.1027/1614-2241/ a000124
  • Liu, X. (2016). Methods and Applications of Longitudinal Data Analysis. Academic Press.
  • Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166. https://doi.org/10.1037/ 0033-2909.105.1.156
  • Molenberghs, G., Fitzmaurice, G., Kenward, M., Tsiatis, A., & Verbeke, G. (2015). Handbook of Missing Data Methodology. CRC Press.
  • Raghunathan, T. (2016). Missing Data Analysis in Practice. CRC Press.
  • Raghunathan, T., Berglund, P., & Solenberger, P. (2018). Multiple Imputation in Practice. CRC Press. Ripley, B. D. (1987). Statistical Inference for Spatial Processes. Cambrigde University Press.
  • Sullivan, T.R., Yelland L.N., Lee K.L., Ryan, P., & Salter, A.B. (2017). Treatment of missing data follow-up studies of randomized controlled trials: A systematic review of the literature. Clinical Trials, 14(4), 387- 395. https://doi.org/10.1177/1740774517703319
  • Vale, C. D., & Maurelli, V. A. (1983). Simulating multivariate non normal distributions. Psychometrika, 48, 465-471. https://doi.org/10.1007/ BF02293687.
  • Vallejo, G., Fernández, M. P., Livacic-Rojas, P. E., & Tuero-Herrero, E. (2011a). Selecting the best unbalanced repeated measures model. Behavior Research Methods, 43, 18-36. https://doi.org/10.3758/ s13428-010-0040-1
  • Vallejo, G., Fernández, P., Livacic-Rojas, P., & Tuero-Herrero, E.(2011b). Comparison of modern methods for analyzing repeated measures data with missing values. Multivariate Behavioral Research, 46, 900-937. https://doi.org/10.1080/00273171.2011.625320
  • Vallejo G., Tuero-Herrero E., Núñez J.C., & Rosário P. (2014). Performance evaluation of recent information criteria for selecting multilevel models in Behavioral and Social Sciences. International Journal of Clinical and Health Psychology, 14, 48-57. https://doi.org/10.1016/S1697- 2600(14)70036-5
  • Vallejo, G., Ato, M., Fernández, M.P., Livacic-Rojas, P.E., & Tuero-Herrero, E. (2016). Power analysis to detect treatment effect in longitudinal studies with heterogeneous errors and incomplete data. Psicothema, 28(3), 330-339. https://doi.org/10.7334/psicothema2016.92
  • Vallejo, G., Fernández, M. P., & Livacic-Rojas, P. E. (2018). Data analysis of incomplete repeated measures using a multivariate extension of the Brown-Forsythe procedure. Psicothema, 30(4), 434-441. https://doi. org/10.7334/psicothema2018
  • Vallejo, G., Fernández, M. P., & Livacic-Rojas, P. E. (2019). Sample size estimation for heterogeneous growth curve models with attrition. Behavior Research Methods, 51(3), 1216-1243. https://doi.org/10.3758/ s13428-018-1059-y
  • Wilcox, R. (2017). Introduction to Robust Estimation & Hypothesis Testing (4rd ed.). Elsevier.
  • Yoo, J.E. (2013). Multiple Imputation with Structural Equation Modeling. Using auxiliary variables when data are missing. Academic Publishing.
  • Zhang, Z., & Yuan, K. (2018). Practical Statistical Power Analysis. Using Web Power and R. Isdsa Press.
  • Zhou, X.H., Zhou, Ch., Liu, D., & Ding, X. (2014). Applied Missing Data Analysis in the Health Sciences. Wiley & Sons.
Data source: Dialnet