To Measure You Must First Define: Excerpts from Psychology

In an interesting and relatively recent article entitled Measurement is Fundamentally a Matter of Definition, Not Correlation, Krause (2012) claimed that psychologists typically pay little attention to defining the things they intend to measure. Instead, they primarily look to establish that putative measures of the concepts they are interested in have correlations with measures of other psychological variables that conform to prediction. This practice is ultimately a problem, Krause contends, because:

“It is only the conformity of a measure to the normative conceptual analysis and so definition of a dimension that can make the measurements it produces valid measurements on that dimension. This makes it essential for each basic science to achieve normative conceptual analyses and definitions for each of the dimensions in terms of which it describes and causally explains its phenomena.” (p. 1)

Given the overwhelming emphasis on statistics in introductory textbooks on scale construction, measurement validity, and psychometrics, I have a feeling that some will consider the idea that measurement is fundamentally a matter of definition rather revolutionary. It is likely to seem even more revolutionary given that many of articles referenced by Krause to support his viewpoint were fairly modern (e.g., Borsboom, 2005; Krause, 2005; Maruan, 1998*).

The point of this (admittedly, quite dry) blog post is to simply point out that, over the last century, a number of psychometricians argued that defining the variable was vital—the first step on the road to successful measurement. The collection of following quotations—all except Clark and Watson (1995) from texts not referenced by Krause—demonstrate this point.

“Measurement, of course, is only a final specialization of description. Measurement...can come in to its own only when qualitative description has truly ripened... no science has reached adult stage without passing through a well-developed descriptive stage” (Cattell, 1946, p. 4)

“Before we can validate the test we have to define the trait which it is designed to measure. Accurate qualitative description therefore has to precede measurement. In other words, psychology has to explore the characters of the unitary traits of man before mental testing can begin” (Cattell, 1948, p. 1)

“The definition of a variable provides the basis for the development of a series of operations that are to be performed in order to obtain descriptions of individuals in terms of the ways in which they manifest the particular property. The characteristics of the variable dictate the nature of the operations. With one variable there will be one series of operations, and with another variable a different series” (Ghiselli, 1964, p. 20)

“If a scale is to be developed to measure a common trait, the scale must contain items. And before the items can be written, the trait or construct that the items will presumably measure must be defined. It is necessary, in other words, to make explicit the nature of the trait or construct that we hope to measure.” (Edwards, 1970, p. 29)

“A measure should spring from a hypothesis regarding the existence and nature of an attribute” (Nunnally, 1970, p. 213)

“We cannot measure well when we cannot specify clearly what we are trying to measure, where it occurs, and when” (Fiske, 1971, p. 30)

“We cannot measure a variable well if we cannot describe it” (Fiske, 1971, p. 117)

“To advance the science of personology, intensive effort must be devoted to each major construct, to delineating it explicitly and systematically, and to create measuring procedures conforming to the blueprints derived from such a conceptualization” (Fiske, 1971, p. 144)

“Let us try to learn to be free from other a priori mathematical and statistical considerations and prescriptions – especially codes of permission. Instead, let us try to think substantively during the initial stages of measurement, and focus directly on the specific universe of observations with which we wish to do business” (Guttman, 1971, p. 346)

“Lazarsfield and Barton (1951) have described the process of measurement in the social sciences in terms of four progressive stages. The first stage they describe is one in which the investigator forms an initial image of the nature of the concept he wishes to measure” (Lemon, 1973, p. 29)

“...the construct definition sets the boundaries for potential measurement techniques. It operates like a test plan for the development of an instrument” (Shavelson, Hubner, & Stanton, 1976, p. 415)

“...the first problem that the psychologist or educator faces as he tries to measure the attributes that he is interested in is that of arriving at a clear, precise, and generally accepted definition of the attribute he proposes to measure” (Thorndike & Hagen, 1977, p. 10)

“A less obvious, but equally important, characteristic shared by enumeration and measurement is that they both require prior definition. We cannot reliably quantify without first defining the objects of properties to be quantified” (Gordon, 1977, p. 2)

“a meaningful and essential question to raise about a measure is whether it is consistent with the definition of the construct it is meant to be tapping” (Pedhazur & Pedhazur-Schmelkin, 1991, p. )

“A critical first step is to develop a precise and detailed conception of the target construct and it theoretical context. We have found that writing out a brief, formal description of the construct is very useful in crystallizing one’s conceptual model” (Clark & Watson, 1995, p. 310)

“Carefully define the domain and facets of the construct and subject them to content validation before developing other elements of the assessment instrument... a construct that is poorly defined, undifferentiated, and imprecisely partitioned will limit the contend validity of the assessment instrument” (Haynes, Richard, & Kubany, 1995, p. 244)

“To summarize, I offer a five-step model for construct validity research (depicted in Figure 1 and heavily influenced by Meehl, 1978, 1990a). The steps are (1) careful specification of the theoretical constructs in question...” (Smith, 2005, p. 399)

“Every new instrument (or even the redevelopment or adaptation of an old instrument) must start with an idea – the kernel of the instrument, the “what” of the “what does it measure?” and the “how” of “how will the measure be used?” (Wilson, 2005, p. 19)

“The measurement of concepts like creativity and intelligence is limited by the clarity with which we are able to define the meaning of these constructs...” (Rust & Golombuk, 2009, p. 31)

“Concepts are the starting point in measurement. Concepts refer to ideas that have some unity or something in common. The meaning of a concept is spelled out in a theoretical definition.” (Bollen, 2011, p. 360)

“Theory enters measurement throughout the process. We need theory to define a concept and to pick out its dimensions. We need theory to develop or to select indicators that match the theoretical definition and its dimensions. Theory also enters in determining whether the indicator influences the latent variable or vice versa.” (Bollen, 2011, p. 361)

“The first, and perhaps most deceptively-simple, facet of scale construction is articulating the construct(s) to be measured. Whether the construct (one or more) is viewed as an attitude, a perception, an attribution, a trait, an emotional response, a behavioural response, a cognitive response, or a physiological response, or-more generally, a psychological response, tendency, or disposition of any kind—it must be carefully articulated and differentiated from other constructs” (Furr, 2011, p. 12)

"The first stage of the scale development and validation process involves defining the conceptual domain of the construct" (MacKenzie, Podsakoff, & Podsakoff, 2011, p. 298).“The topic of conceptualization of measurement variables is probably the least quantitative of the fields of study. Psychometricians have typically shied away from this area of our work, possibly for this exact reason. Yet, it is the most important, for without adequate conceptualization, all else is empty-in particular the whole concept of “validity evidence” becomes moot, as there is no substance to validate to” (Wilson, 2013, p. 222)

* See Krause's references for these references. Sorry!

References

Bollen, K. A. (2011). Evaluating effect, composite, and causal indicators in structural equation models. MIS Quarterly, 35(2), 359-372.

Edwards, A. L. (1970). The measurement of personality traits by scales an inventories. New York: Holt, Rinehart, and Winston.

Cattell, R. B. (1946). Description and measurement of personality. London: George G. Harrap & Co.

Fiske, D. W. (1971). Measuring the concepts of personality. Oxford, England: Aldine.

Furr, R. M. (2011). Scale construction and psychometrics. London: Sage.

Ghiselli, E. E. (1964). Theory of psychological measurement. New York: McGraw-Hill.

Gordon, R. L. (1977). Unidimensional scaling of social variables: Concepts and procedures. New York: The Free Press.

Guttman, L. (1971). Measurement as structural theory. Psychometrika, 36(4), 329-347.

Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7(3), 238-247.

Krause, M. S. (2012). Measurement validity is fundamentally a matter of definition, not correlation. Review of General Psychology, 16(4), 391-400. http://dx.doi.org/10.1037/a0027701

Lemon, N. (1973). Attitudes and their measurement. London: C. Tinling & Co

Nunnally, J. C. (1970). Introduction to psychological measurement. New York: McGraw-Hill.

Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. New York, NY: Psychology Press.

MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques. MIS quarterly, 35(2), 293-334.

Rust,  J., & Golombok, S. (2009). Modern psychometrics: The science of psychological assessment. London: Routledge.

Shavelson, R. J., Hubner, J. J., & Stanton, G. C. (1976). Self-concept: Validation of construct interpretations. Review of Education Research, 407-441.

Smith, G. T. (2005). On construct validity: Issues of method and measurement. Psychological Assessment, 17(4), 396-408.

Thorndike, R. L., & Hagen, E. P. (1977). Measurement and evaluation in psychology and education. New York: John Wiley and Sons. 4^th edition

Wilson, M. (2005). Constructing measures: An item response modelling approach. London: Lawrence Erlbaum Associates.

Wilson, M. (2013). Seeking a balance between the statistical and scientific elements in psychometrics. Psychometrika, 78(2), 211-236.