Volume 61, Issue 2

A general diagnostic model applied to language testing data

Matthias von Davier

Corresponding Author

Research & Development, Educational Testing Service, Princeton, NJ, USA

Correspondence should be addressed to Dr Matthias von Davier, Research & Development, Educational Testing Service, Rosedale Road, MS 02‐T, Princeton, NJ 08541, USA (e‐mail: mvondavier@ets.org).Search for more papers by this author
First published: 24 December 2010
Citations: 182

Abstract

Probabilistic models with one or more latent variables are designed to report on a corresponding number of skills or cognitive attributes. Multidimensional skill profiles offer additional information beyond what a single test score can provide, if the reported skills can be identified and distinguished reliably. Many recent approaches to skill profile models are limited to dichotomous data and have made use of computationally intensive estimation methods such as Markov chain Monte Carlo, since standard maximum likelihood (ML) estimation techniques were deemed infeasible. This paper presents a general diagnostic model (GDM) that can be estimated with standard ML techniques and applies to polytomous response variables as well as to skills with two or more proficiency levels. The paper uses one member of a larger class of diagnostic models, a compensatory diagnostic model for dichotomous and partial credit data. Many well‐known models, such as univariate and multivariate versions of the Rasch model and the two‐parameter logistic item response theory model, the generalized partial credit model, as well as a variety of skill profile models, are special cases of this GDM. In addition to an introduction to this model, the paper presents a parameter recovery study using simulated data and an application to real data from the field test for TOEFL® Internet‐based testing.

Number of times cited according to CrossRef: 182

  • A Note on the Relationship of the Shannon Entropy Procedure and the Jensen–Shannon Divergence in Cognitive Diagnostic Computerized Adaptive Testing, SAGE Open, 10.1177/2158244019899046, 10, 1, (215824401989904), (2020).
  • Psychometrics, Encyclopedia of Personality and Individual Differences, 10.1007/978-3-319-24612-3, (4157-4161), (2020).
  • Choosing between CDM and Unidimensional IRT: The Proportional Reasoning Test Case, Measurement: Interdisciplinary Research and Perspectives, 10.1080/15366367.2019.1697122, 18, 2, (87-96), (2020).
  • A Semi-supervised Learning Method for Q-Matrix Specification Under the DINA and DINO Model With Independent Structure, Frontiers in Psychology, 10.3389/fpsyg.2020.02120, 11, (2020).
  • undefined, 2020 International Joint Conference on Neural Networks (IJCNN), 10.1109/IJCNN48605.2020.9207587, (1-8), (2020).
  • Examining Parameter Invariance in a General Diagnostic Classification Model, Frontiers in Psychology, 10.3389/fpsyg.2019.02930, 10, (2020).
  • Variational Bayesian inference for the multiple-choice DINA model, Behaviormetrika, 10.1007/s41237-020-00104-w, (2020).
  • Bayesian Estimation of the DINA Model With Pólya-Gamma Gibbs Sampling, Frontiers in Psychology, 10.3389/fpsyg.2020.00384, 11, (2020).
  • Spectral Clustering Algorithm for Cognitive Diagnostic Assessment, Frontiers in Psychology, 10.3389/fpsyg.2020.00944, 11, (2020).
  • Learning Attribute Hierarchies From Data: Two Exploratory Approaches, Journal of Educational and Behavioral Statistics, 10.3102/1076998620931094, (107699862093109), (2020).
  • Hybrid cognitive diagnostic model, Behaviormetrika, 10.1007/s41237-020-00111-x, (2020).
  • Students’ Complex Problem Solving Profiles, Psychometrika, 10.1007/s11336-020-09709-2, (2020).
  • A Constrained Metropolis–Hastings Robbins–Monro Algorithm for Q Matrix Estimation in DINA Models, Psychometrika, 10.1007/s11336-020-09707-4, (2020).
  • Regularized Latent Class Analysis for Polytomous Item Responses: An Application to SPM-LS Data, Journal of Intelligence, 10.3390/jintelligence8030030, 8, 3, (30), (2020).
  • Using Response Times and Response Accuracy to Measure Fluency Within Cognitive Diagnosis Models, Psychometrika, 10.1007/s11336-020-09717-2, (2020).
  • Measuring Skill Growth and Evaluating Change: Unconditional and Conditional Approaches to Latent Growth Cognitive Diagnostic Models, Frontiers in Psychology, 10.3389/fpsyg.2020.02205, 11, (2020).
  • A Class of Cognitive Diagnosis Models for Polytomous Data, Journal of Educational and Behavioral Statistics, 10.3102/1076998620951986, (107699862095198), (2020).
  • A Multivariate Probit Model for Learning Trajectories: A Fine-Grained Evaluation of an Educational Intervention, Applied Psychological Measurement, 10.1177/0146621620920928, (014662162092092), (2020).
  • Extending the Basic Local Independence Model to Polytomous Data, Psychometrika, 10.1007/s11336-020-09722-5, (2020).
  • Estimating CDMs Using the Slice-Within-Gibbs Sampler, Frontiers in Psychology, 10.3389/fpsyg.2020.02260, 11, (2020).
  • On the Sequential Hierarchical Cognitive Diagnostic Model, Frontiers in Psychology, 10.3389/fpsyg.2020.579018, 11, (2020).
  • An empirical Q‐matrix validation method for the sequential generalized DINA model, British Journal of Mathematical and Statistical Psychology, 10.1111/bmsp.12156, 73, 1, (142-163), (2019).
  • A diagnostic classification version of Schizotypal Personality Questionnaire using diagnostic classification models, International Journal of Methods in Psychiatric Research, 10.1002/mpr.1807, 29, 1, (2019).
  • Estimating CDMs Using MCMC, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_31, (629-646), (2019).
  • Explanatory Cognitive Diagnostic Models, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_10, (207-222), (2019).
  • Differential Item Functioning in Diagnostic Classification Models, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_18, (379-393), (2019).
  • Measures of Agreement: Reliability, Classification Accuracy, and Classification Consistency, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_17, (359-377), (2019).
  • Identifiability and Cognitive Diagnosis Models, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_16, (333-357), (2019).
  • Exploratory Data Analysis for Cognitive Diagnosis: Stochastic Co-blockmodel and Spectral Co-clustering, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_14, (287-306), (2019).
  • Cognitive Diagnosis Modeling Using the GDINA R Package, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_29, (593-601), (2019).
  • Nonparametric Methods in Cognitively Diagnostic Assessment, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_5, (107-132), (2019).
  • A Copula Model for Residual Dependency in DINA Model, Quantitative Psychology, 10.1007/978-3-030-01310-3_14, (145-156), (2019).
  • Psychometrics, Encyclopedia of Personality and Individual Differences, 10.1007/978-3-319-28099-8, (1-5), (2019).
  • Global- and Item-Level Model Fit Indices, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_13, (265-285), (2019).
  • Q-Matrix Learning via Latent Variable Selection and Identifiability, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_12, (247-263), (2019).
  • Insights from Reparameterized DINA and Beyond, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_11, (223-243), (2019).
  • Diagnostic Modeling of Skill Hierarchies and Cognitive Processes with MLTM-D, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_9, (187-205), (2019).
  • Loglinear Cognitive Diagnostic Model (LCDM), Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_8, (171-185), (2019).
  • The G-DINA Model Framework, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_7, (155-169), (2019).
  • The R Package CDM for Diagnostic Modeling, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_26, (549-572), (2019).
  • CDMs in Vocational Education: Assessment and Usage of Diagnostic Problem-Solving Strategies in Car Mechatronics, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_22, (461-488), (2019).
  • Nonparametric Item Response Theory and Mokken Scale Analysis, with Relations to Latent Class Models and Cognitive Diagnostic Models, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_2, (21-45), (2019).
  • Cognitive diagnosis models for multiple strategies, British Journal of Mathematical and Statistical Psychology, 10.1111/bmsp.12155, 72, 2, (370-392), (2019).
  • The General Diagnostic Model, Handbook of Diagnostic Classification Models, 10.1007/978-3-030-05584-4_6, (133-153), (2019).
  • Maximum Marginal Likelihood Estimation With an Expectation–Maximization Algorithm for Multigroup/Mixture Multidimensional Item Response Theory Models, ETS Research Report Series, 10.1002/ets2.12272, 2019, 1, (1-16), (2019).
  • Diagnosing EFL learners’ writing ability: a diagnostic classification modeling analysis, Language Testing in Asia, 10.1186/s40468-019-0090-y, 9, 1, (2019).
  • A Comparison of Differential Item Functioning Detection Methods in Cognitive Diagnostic Models, Frontiers in Psychology, 10.3389/fpsyg.2019.01137, 10, (2019).
  • A Longitudinal Higher-Order Diagnostic Classification Model, Journal of Educational and Behavioral Statistics, 10.3102/1076998619827593, (107699861982759), (2019).
  • Using JAGS for Bayesian Cognitive Diagnosis Modeling: A Tutorial, Journal of Educational and Behavioral Statistics, 10.3102/1076998619826040, (107699861982604), (2019).
  • A Blocked-CAT Procedure for CD-CAT, Applied Psychological Measurement, 10.1177/0146621619835500, (014662161983550), (2019).
  • Dynamic Bayesian Network Modeling of Game-Based Diagnostic Assessments, Multivariate Behavioral Research, 10.1080/00273171.2019.1590794, (1-24), (2019).
  • Cognitive Diagnostic Computerized Adaptive Testing for Polytomously Scored Items, Journal of Classification, 10.1007/s00357-019-09357-x, (2019).
  • The Reliability of the Posterior Probability of Skill Attainment in Diagnostic Classification Models, Journal of Educational and Behavioral Statistics, 10.3102/1076998619864550, (107699861986455), (2019).
  • Evaluating the Fit of Sequential G-DINA Model Using Limited-Information Measures, Applied Psychological Measurement, 10.1177/0146621619843829, (014662161984382), (2019).
  • Using Diagnostic Classification Models to Validate Attribute Hierarchies and Evaluate Model Fit in Bayesian Networks, Multivariate Behavioral Research, 10.1080/00273171.2019.1632165, (1-12), (2019).
  • A multilevel logistic hidden Markov model for learning under cognitive diagnosis, Behavior Research Methods, 10.3758/s13428-019-01238-w, (2019).
  • Consistency Theory for the General Nonparametric Classification Method, Psychometrika, 10.1007/s11336-019-09660-x, (2019).
  • Stratified Item Selection Methods in Cognitive Diagnosis Computerized Adaptive Testing, Applied Psychological Measurement, 10.1177/0146621619893783, (014662161989378), (2019).
  • A Partial Mastery, Higher-Order Latent Structural Model for Polytomous Attributes in Cognitive Diagnostic Assessments, Journal of Classification, 10.1007/s00357-019-09323-7, (2019).
  • Adjusting Person Fit Index for Skewness in Cognitive Diagnosis Modeling, Journal of Classification, 10.1007/s00357-019-09325-5, (2019).
  • An Exploratory Diagnostic Model for Ordinal Responses with Binary Attributes: Identifiability and Estimation, Psychometrika, 10.1007/s11336-019-09683-4, (2019).
  • Examining problem-solving skills in technology-rich environments as related to numeracy and literacy, International Journal of Lifelong Education, 10.1080/02601370.2019.1598507, (1-12), (2019).
  • Beliefs and needs of academic teachers: a latent class analysis, Statistical Methods & Applications, 10.1007/s10260-019-00495-5, (2019).
  • Cognitive Diagnostic Models With Attribute Hierarchies: Model Estimation With a Restricted Q-Matrix Design, Applied Psychological Measurement, 10.1177/0146621618765721, 43, 4, (255-271), (2018).
  • Bayesian DINA Modeling Incorporating Within-Item Characteristic Dependency, Applied Psychological Measurement, 10.1177/0146621618781594, 43, 2, (143-158), (2018).
  • Development and Application of an Exploratory Reduced Reparameterized Unified Model, Journal of Educational and Behavioral Statistics, 10.3102/1076998618791306, 44, 1, (3-24), (2018).
  • Information matrix estimation procedures for cognitive diagnostic models, British Journal of Mathematical and Statistical Psychology, 10.1111/bmsp.12134, 72, 1, (18-37), (2018).
  • A diagnostic tree model for polytomous responses with multiple strategies, British Journal of Mathematical and Statistical Psychology, 10.1111/bmsp.12137, 72, 1, (61-82), (2018).
  • A reinforcement learning approach to personalized learning recommendation systems, British Journal of Mathematical and Statistical Psychology, 10.1111/bmsp.12144, 72, 1, (108-135), (2018).
  • A General Cognitive Diagnosis Model for Continuous-Response Data, Measurement: Interdisciplinary Research and Perspectives, 10.1080/15366367.2018.1436817, 16, 1, (30-44), (2018).
  • Mutual Information Reliability for Latent Class Analysis, Applied Psychological Measurement, 10.1177/0146621617748324, 42, 6, (460-477), (2018).
  • How to Build a Complete Q-Matrix for a Cognitively Diagnostic Test, Journal of Classification, 10.1007/s00357-018-9255-0, 35, 2, (273-299), (2018).
  • General mixture item response models with different item response structures: Exposition with an application to Likert scales, Behavior Research Methods, 10.3758/s13428-017-0997-0, 50, 6, (2325-2344), (2018).
  • A Generalized Approach to Defining Item Discrimination for DCMs, Measurement: Interdisciplinary Research and Perspectives, 10.1080/15366367.2018.1436855, 16, 1, (18-29), (2018).
  • Identifiability of the Latent Attribute Space and Conditions of Q-Matrix Completeness for Attribute Hierarchy Models, Quantitative Psychology, 10.1007/978-3-319-77249-3_30, (363-375), (2018).
  • Diagnosing Diagnostic Models: From Von Neumann’s Elephant to Model Equivalencies and Network Psychometrics, Measurement: Interdisciplinary Research and Perspectives, 10.1080/15366367.2018.1436827, 16, 1, (59-70), (2018).
  • Using Response Times to Assess Learning Progress: A Joint Model for Responses and Response Times, Measurement: Interdisciplinary Research and Perspectives, 10.1080/15366367.2018.1435105, 16, 1, (45-58), (2018).
  • Parameter Invariance and Skill Attribute Continuity in the DINA Model, Journal of Educational Measurement, 10.1111/jedm.12175, 55, 2, (264-280), (2018).
  • Measures of Agreement to Assess Attribute‐Level Classification Accuracy and Consistency for Cognitive Diagnostic Assessments, Journal of Educational Measurement, 10.1111/jedm.12196, 55, 4, (635-664), (2018).
  • Hypothesis Testing of the Q-matrix, Psychometrika, 10.1007/s11336-018-9629-6, 83, 3, (515-537), (2018).
  • A Comparison of Score Aggregation Methods for Unidimensional Tests on Different Dimensions, ETS Research Report Series, 10.1002/ets2.12194, 2018, 1, (1-16), (2018).
  • A Review of Subscore Estimation Methods, ETS Research Report Series, 10.1002/ets2.12203, 2018, 1, (1-15), (2018).
  • Identifying Latent Structures in Restricted Latent Class Models, Journal of the American Statistical Association, 10.1080/01621459.2017.1340889, 113, 523, (1284-1295), (2018).
  • Improved Wald Statistics for Item-Level Model Comparison in Diagnostic Classification Models, Applied Psychological Measurement, 10.1177/0146621618798664, (014662161879866), (2018).
  • A Constrained Confirmatory Mixture IRT Model: Extensions and Estimation of the Saltus model using Mplus, The Quantitative Methods for Psychology, 10.20982/tqmp.14.2.p120, 14, 2, (120-136), (2018).
  • Diagnostic Classification Models for Ordinal Item Responses, Frontiers in Psychology, 10.3389/fpsyg.2018.02512, 9, (2018).
  • Nonparametric CAT for CD in Educational Settings With Small Samples, Applied Psychological Measurement, 10.1177/0146621618813113, (014662161881311), (2018).
  • Modeling Learner Heterogeneity: A Mixture Learning Model With Responses and Response Times, Frontiers in Psychology, 10.3389/fpsyg.2018.02339, 9, (2018).
  • Category-Level Model Selection for the Sequential G-DINA Model, Journal of Educational and Behavioral Statistics, 10.3102/1076998618792484, (107699861879248), (2018).
  • ‘Not so globalised’: contrasting media discourses on education and competitiveness in four countries., Journal of Asia Business Studies, 10.1108/JABS-08-2016-0108, (00-00), (2018).
  • Expectation-Maximization-Maximization: A Feasible MLE Algorithm for the Three-Parameter Logistic Model Based on a Mixture Modeling Reformulation, Frontiers in Psychology, 10.3389/fpsyg.2017.02302, 8, (2018).
  • Joint Testlet Cognitive Diagnosis Modeling for Paired Local Item Dependence in Response Times and Response Accuracy, Frontiers in Psychology, 10.3389/fpsyg.2018.00607, 9, (2018).
  • Examining DIF in the Context of CDMs When the Q-Matrix Is Misspecified, Frontiers in Psychology, 10.3389/fpsyg.2018.00696, 9, (2018).
  • Introducing the General Polytomous Diagnosis Modeling Framework, Frontiers in Psychology, 10.3389/fpsyg.2018.01474, 9, (2018).
  • Theorems and Methods of a Complete Q Matrix With Attribute Hierarchies Under Restricted Q-Matrix Design, Frontiers in Psychology, 10.3389/fpsyg.2018.01413, 9, (2018).
  • Attribute Hierarchy Models in Cognitive Diagnosis: Identifiability of the Latent Attribute Space and Conditions for Completeness of the Q-Matrix, Journal of Classification, 10.1007/s00357-018-9278-6, (2018).
  • Estimating the Cognitive Diagnosis $$\varvec{Q}$$Q Matrix with Expert Knowledge: Application to the Fraction-Subtraction Dataset, Psychometrika, 10.1007/s11336-018-9643-8, (2018).
  • A Semiparametric Approach for Modeling Not-Reached Items, Educational and Psychological Measurement, 10.1177/0013164417749679, 79, 1, (170-199), (2017).
  • Comparing Attitudes Across Groups: An IRT-Based Item-Fit Statistic for the Analysis of Measurement Invariance, Applied Psychological Measurement, 10.1177/0146621617748323, 43, 3, (241-250), (2017).
  • Tracking Skill Acquisition With Cognitive Diagnosis Models: A Higher-Order, Hidden Markov Model With Covariates, Journal of Educational and Behavioral Statistics, 10.3102/1076998617719727, 43, 1, (57-87), (2017).
  • See more