Speaker: Jessica Flake, York University, Quantitative Methods Section, Department of Psychology

Title: Toward Practical Recommendations for Psychometric Best Practice in Applied Research

Abstract: The Standards for Educational and Psychological Testing offer a myriad of best practices for developing and using psychological tests and measures. These are complemented by many reference texts as well as a body of methodological research.  One recommendation that experts tend to agree on is that scale validation is an on-going process: the validity of a scale needs to be re-evaluated and replicated when it is used with a new population, for a different purpose, or in a different context. Despite the large body of work related to psychometrics, there is little guidance for how the applied researcher should conduct on-going validation.  What analyses should be conducted, and how often? And how are they to proceed if the results are not as expected (e.g., partial-measurement invariance or lack of simple structure)? In this talk I will share the results of two studies I have conducted over the past year that are working toward developing recommendations that are practical for applied researchers.

First I will share the results of a systematic review of current practices for on-going validation of studies published in the Journal of Personality and Social Psychology. In the second part of my talk I will share the results of a methodological study about partial measurement invariance, in which we set out to uncover how it influences the downstream statistical analyses central to applied researchers’ hypotheses. I will close with a discussion of how these studies fit into the larger dialog that is currently taking place in the social sciences about replication and how measurement plays a fundamental a role in building a more cumulative science.