Hypothesis tests for population heterogeneity in meta-analysis
Wolfgang Viechtbauer,
Corresponding Author
Wolfgang Viechtbauer
University of Illinois at Urbana-Champaign, USA and University of Maastricht, The Netherlands
Correspondence should be addressed to Wolfgang Viechtbauer, Department of Methodology and Statistics, University of Maastricht, P.O. Box 616, 6200 MD Maastricht, The Netherlands (e-mail: [email protected]).Search for more papers by this authorWolfgang Viechtbauer,
Corresponding Author
Wolfgang Viechtbauer
University of Illinois at Urbana-Champaign, USA and University of Maastricht, The Netherlands
Correspondence should be addressed to Wolfgang Viechtbauer, Department of Methodology and Statistics, University of Maastricht, P.O. Box 616, 6200 MD Maastricht, The Netherlands (e-mail: [email protected]).Search for more papers by this authorAbstract
Choice of the appropriate model in meta-analysis is often treated as an empirical question which is answered by examining the amount of variability in the effect sizes. When all of the observed variability in the effect sizes can be accounted for based on sampling error alone, a set of effect sizes is said to be homogeneous and a fixed-effects model is typically adopted. Whether a set of effect sizes is homogeneous or not is usually tested with the so-called Q test. In this paper, a variety of alternative homogeneity tests – the likelihood ratio, Wald and score tests – are compared with the Q test in terms of their Type I error rate and power for four different effect size measures. Monte Carlo simulations show that the Q test kept the tightest control of the Type I error rate, although the results emphasize the importance of large sample sizes within the set of studies. The results also suggest under what conditions the power of the tests can be considered adequate.
References
- Alexander, R. A., Scozzaro, M. J., Borodkin, L. J. Statistical and empirical examination of the chi-square test for homogeneity of correlations in meta-analysis Psychological Bulletin 106 329–331 1989.
- Berkey, C. S., Hoaglin, D. C., Mosteller, F., Colditz, G. A. A random-effects regression model for meta-analysis Statistics in Medicine 14 395–411 1995.
- Bond, C. F., Jr, Wiitala, W. L., Richard, F. D. Meta-analysis of raw mean differences Psychological Methods 8 406–418 2003.
- Brockwell, S. E., Gordon, I. R. A comparison of statistical methods for meta-analysis Statistics in Medicine 20 825–840 2001.
- Callender, J. C., Osburn, H. E. Unbiased estimation of sampling variance of correlations Journal of Applied Psychology 73 312–315 1988.
-
Cochran, W. G.
Problems arising in the analysis of a series of similar experiments
Supplement to the Journal of the Royal Statistical Society
4 102–118 1937.
10.2307/2984123 Google Scholar
- Cochran, W. G. The combination of estimates from different experiments Biometrics 10 101–129 1954.
-
Cohen, J., Statistical power analysis for the behavioral sciences, 2nd ed., Hillsdale, NJ: Erlbaum, 1988.
10.1046/j.1526-4610.2001.111006343.x Google Scholar
- Corbeil, R. R., Searle, S. R. A comparison of variance component estimation Biometrics 32 779–791 1976a.
- Corbeil, R. R., Searle, S. R. Restricted maximum likelihood (REML) estimation of variance components in the mixed model Technometrics 18 31–38 1976b.
- DerSimonian, R., Laird, N. Meta-analysis in clinical trials Controlled Clinical Trials 7 177–188 1986.
- Erez, A., Bloom, M. C., Wells, M. T. Using random rather than fixed effects models in meta-analysis: Implications for situational specificity and validity generalization Personnel Psychology 49 275–306 1996.
- Field, A. P. Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods Psychological Methods 6 161–180 2001.
- Fisher, R. A. Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population Biometrika 10 507–521 1915.
- Fleiss, J. L. Measures of effect size for categorical data, H. M. Cooper, L. V. Hedges, The handbook of research synthesis, 245–260 New York: Russell Sage Foundation, 1994.
- Friedman, L. Estimators of random effects variance components in meta-analysis Journal of Educational and Behavioral Statistics 25 1–12 2000.
-
Hardy, R. J.,
Thompson, S. G.
A likelihood approach to meta-analysis with random effects
Statistics in Medicine
15 619–629 1996.
10.1002/(SICI)1097-0258(19960330)15:6<619::AID-SIM188>3.0.CO;2-A CASPubMedWeb of Science®Google Scholar
-
Hardy, R. J.,
Thompson, S. G.
Detecting and describing heterogeneity in meta-analysis
Statistics in Medicine
17 841–856 1998.
10.1002/(SICI)1097-0258(19980430)17:8<841::AID-SIM781>3.0.CO;2-D CASPubMedWeb of Science®Google Scholar
- Hartley, H. O., Rao, J. N. K. Maximum-likelihood estimation for the mixed analysis of variance model Biometrika 54 93–108 1967.
- Harville, D. A. Maximum likelihood approaches to variance component estimation and to related problems Journal of the American Statistical Association 72 320–338 1977.
- Harwell, M. An empirical study of Hedges' homogeneity test Psychological Methods 2 219–231 1997.
-
Hedges, L. V.
Distribution theory for Glass's estimator of effect size and related estimators
Journal of Educational Statistics
6 107–128 1981.
10.3102/10769986006002107 Google Scholar
- Hedges, L. V. Fitting categorical models to effect size from a series of experiments Journal of Educational Statistics 7 119–137 1982a.
- Hedges, L. V. Estimation of effect size from a series of independent experiments Psychological Bulletin 92 490–499 1982b.
- Hedges, L. V. A random effects model for effect sizes Psychological Bulletin 93 388–395 1983.
- Hedges, L. V. An unbiased correction for sampling error in validity generalization studies Journal of Applied Psychology 74 469–477 1989.
- Hedges, L. V., Olkin, I., Statistical methods for meta-analysis San Diego, CA: Academic Press, 1985.
- Hedges, L. V., Pigott, T. D. The power of statistical tests in meta-analysis Psychological Methods 6 203–217 2001.
- Hedges, L. V., Vevea, J. L. Fixed-versus random-effects models in meta-analysis Psychological Methods 3 486–504 1998.
- Hotelling, H. New light on the correlation coefficient and its transforms Journal of the Royal Statistical Society, Series B 15 193–232 1953.
- Hunter, J. E., Schmidt, F. L., Methods of meta-analysis: Correcting error and bias in research findings Newbury Park, CA: Sage, 1990.
- Hunter, J. E., Schmidt, F. L. Estimation of sampling error variance in the meta-analysis of correlations: Use of average correlation in the homogeneous case Journal of Applied Psychology 79 171–177 1994.
- Hunter, J. E., Schmidt, F. L. Fixed effects vs. random effects meta-analysis models: Implications for cumulative research knowledge International Journal of Selection and Assessment 8 275–292 2000.
- Johnson, B. T., Mullen, B., Salas, E. Comparison of three major meta-analytic approaches Journal of Applied Psychology 80 94–106 1995.
- Koslowsky, M., Sagie, A. On the efficacy of credibility intervals as indicators of moderator effects in meta-analytic research Journal of Organizational Behavior 14 695–699 1993.
-
Lehmann, E. L., Elements of large-sample theory New York: Springer-Verlag, 1999.
10.1007/b98855 Google Scholar
- Lipsey, M. W., Wilson, D. B., Practical meta-analysis Thousand Oaks, CA: Sage, 2001.
- Morrell, C. H. Likelihood ratio testing of variance components in the linear mixed-effects model using restricted maximum likelihood Biometrics 54 1560–1568 1998.
- Morris, C. N. Parametric empirical Bayes inference: Theory and practice (with discussion) Journal of the American Statistical Association 78 47–65 1983.
- Morris, S. B. Distribution of the standardized mean change effect size for meta-analysis on repeated measures British Journal of Mathematical and Statistical Psychology 53 17–29 2000.
- National Research Council Combining information: Statistical issues and opportunities for research Washington, DC: National Academic Press, 1992.
- Neyman, J., Scott, E. L. Consistent estimates based on partially consistent observations Econometrica 16 1–32 1948.
-
Normand, S. T.
Meta-analysis: Formulating, evaluating, combining, and reporting
Statistics in Medicine
18 321–359 1999.
10.1002/(SICI)1097-0258(19990215)18:3<321::AID-SIM28>3.0.CO;2-P CASPubMedWeb of Science®Google Scholar
- Olkin, I., Pratt, J. W. Unbiased estimation of certain correlation coefficients Annals of Mathematical Statistics 29 201–211 1958.
- Patterson, H. D., Thompson, R. Recovery of inter-block information when block sizes are unequal Biometrika 58 545–554 1971.
- Patterson, H. D., Thompson, R. Maximum likelihood estimation of components of variance Proceedings of the 8th International Biometrics Conference 197–207 1974.
-
Rao, C. R., Linear statistical inference and its application New York: McGraw-Hill, 1973.
10.1002/9780470316436 Google Scholar
- Rasmussen, J. L., Loher, B. T. Appropriate critical percentages for the Schmidt and Hunter meta-analysis procedure: Comparative evaluation of Type I error rate and power Journal of Applied Psychology 73 683–687 1988.
-
Rosenthal, R., Meta-analytic procedures for social research Newbury Park, CA: Sage, 1991.
10.4135/9781412984997 Google Scholar
- Rosenthal, R. Parametric measures of effect size, H. M. Cooper, L. V. Hedges, The handbook of research synthesis, 231–244 New York: Russell Sage Foundation, 1994.
- Rosenthal, R., DiMatteo, M. R. Meta-analysis: Recent developments in quantitative methods for literature reviews Annual Review in Psychology 52 59–82 2001.
- Rosenthal, R., Rubin, D. B. Comparing effect sizes of independent studies Psychological Bulletin 92 500–504 1982.
- Sackett, P. R., Harris, M. M., Orr, J. M. On seeking moderator variables in the meta-analysis of correlational data: A Monte Carlo investigation of statistical power and resistance to Type I error Journal of Applied Psychology 71 302–310 1986.
- Sagie, A., Koslowsky, M. Detecting moderators with meta-analysis: An evaluation and comparison of techniques Personnel Psychology 46 629–640 1993.
- Sánchez-Meca, J., Marín-Martínez, F. Homogeneity tests in meta-analysis: A Monte Carlo comparison of statistical power and Type I error Quality & Quantity 31 385–399 1997.
- Schmidt, F. L., Hunter, J. E. Comparison of three meta-analysis methods revisited: An analysis of Johnson, Mullen, and Salas (1995) Journal of Applied Psychology 84 144–148 1999.
-
Searle, S. R.,
Casella, G.,
McCulloch, C. E., Variance components New York: Wiley, 1992.
10.1002/9780470316856 Google Scholar
- Self, S. G., Liang, K. Y. Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions Journal of the American Statistical Association 82 605–610 1987.
- Spector, P. E., Levine, E. L. Meta-analysis for integrating study outcomes: A Monte Carlo study of its susceptibility to Type I and Type II errors Journal of Applied Psychology 72 3–9 1987.
- Stram, D. O., Lee, J. W. Variance components testing in the longitudinal mixed effects model Biometrics 50 1171–1177 1994.
- Takkouche, B., Cadarso-Suárez, C., Spiegelman, D. Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis American Journal of Epidemiology 150 206–215 1999.
-
Thompson, S. G.,
Sharp, S. J.
Explaining heterogeneity in meta-analysis: A comparison of methods
Statistics in Medicine
18 2693–2708 1999.
10.1002/(SICI)1097-0258(19991030)18:20<2693::AID-SIM235>3.0.CO;2-V CASPubMedWeb of Science®Google Scholar
- van Houwelingen, H. C., Arends, L. R., Stijnen, T. Advanced methods in meta-analysis: Multivariate approach and meta-regression Statistics in Medicine 21 589–624 2002.
-
Verbeke, G.,
Molenberghs, G., Linear mixed models in practice: A SAS-oriented approach New York: Springer-Verlag, 1997.
10.1007/978-1-4612-2294-1 Google Scholar
- Verbeke, G., Molenberghs, G. The use of score tests for inference on variance components Biometrics 59 254–262 2003.
-
Viana, M. A. G.
Statistical methods for summarizing independent correlational results
Journal of Educational Statistics
5 83–104 1980.
10.2307/1165001 Google Scholar
- Viechtbauer, W. (2004). Choosing between the fixed-, random-, and mixed-effects model in meta-analysis: An analysis of existing and new model selection methods. Unpublished doctoral dissertation, University of Illinois, Urbana-Champaign.
- Viechtbauer, W. (2005). Bias and efficiency of meta-analytic variance estimators in the random-effects model Journal of Educational and Behavioral Statistics, 30, 261–293.
- Wang, M. C., Bushman, B. J., Integrating results through meta-analytic review using SAS software Cary, NC: SAS Institute, 1999.
