Date of Award
Spring 6-13-2025
Document Type
Dissertation
Degree Name
Ph.D. in Quantitative Research Methods
Organizational Unit
Morgridge College of Education, Research Methods and Information Science, Research Methods and Statistics
First Advisor
Yixiao Dong
Second Advisor
Duan Zhang
Third Advisor
Keiko McCullough
Fourth Advisor
Garrett Roberts
Copyright Statement / License for Reuse
All Rights Reserved.
Keywords
Consequential validity, Consequential validity ratio, Construct-irrelevant demographics, Focal and criterion measures, R Shiny, Test fairness
Abstract
Ensuring that test scores support fair and equitable decision-making across diverse demographic groups is an important task as well as a critical challenge in psychological assessment. This issue, represented by the concept of consequential validity, has inspired continuing debates regarding the extent of psychometricians' responsibility for test fairness. Recent advancements have introduced Consequential Validity Ratio (CVR), a metric that quantifies the proportion of variance or influence in a criterion attributable to the focal measure (test scores) relative to the variance or influence explained by construct-irrelevant demographic factors to operationalize fairness in test validation with continuous and binary criterion measures. However, the lack of accessible and user-friendly statistical tools to compute CVRs could limit their practical application. This current study bridges this gap by developing an R Shiny application that automates the calculation of CVRs, as well as integrates several useful functions in the test validation process (e.g., calculating multiple CVRs simultaneously; generating interaction variables; checking reliability of both focal and criterion measures). The side-by-side comparisons with manual analyses in SPSS and STATA yielded identical CVR values, confirming the app’s accuracy, while stress-testing a 100,000-row bootstrapped dataset completed in under half a second (0.38 seconds), demonstrating a strong computational efficiency. The primary utility of this application was demonstrated through empirical analyses of two distinct datasets: the educational dataset Early Childhood Longitudinal Study-Kindergarten (ECLS-K) and the psychological dataset collected as a part of a larger study on vicarious exposure to violent racism through media among AAPI and Black American adults. The empirical results further illustrated the tool’s value: in educational data the app returns a 𝐶𝑉𝑅𝐶 of 0.86 when 3rd grade science assessment score predicts the 5th-grade science scores, showing that 86% of explained variance stems from the focal measure as opposed to demographic factors; whereas, in the psychological dataset collected from a larger study, it outputted a 𝐶𝑉𝑅𝐵 of 0.04 for PHQ-8 predicting a depression diagnosis, indicating minimal influence of the focal measure. By showing these analyses and a user tutorial, the study validates both CVR formulations and demonstrates the application’s capacity to promote fairness in testing.
Copyright Date
6-2025
Publication Statement
Copyright is held by the author. User is responsible for all copyright compliance.
Rights Holder
Kushmakar Baral
Provenance
Received from ProQuest
File Format
application/pdf
Language
English (eng)
Extent
94 pgs
File Size
1.8 MB
Recommended Citation
Baral, Kushmakar, "An R Shiny App and Tutorial for Calculating Consequential Validity Ratio (CVR) with Continuous and Binary Criterion" (2025). Electronic Theses and Dissertations. 2548.
https://digitalcommons.du.edu/etd/2548
Included in
Educational Assessment, Evaluation, and Research Commons, Educational Psychology Commons, Statistical Models Commons