In an interesting and relatively recent article entitled Measurement is Fundamentally a Matter of Definition, Not Correlation, Krause (2012) claimed
that psychologists typically pay little attention to defining the things they
intend to measure. Instead, they primarily look to establish that
putative measures of the concepts they are interested in have correlations with measures of other psychological variables that conform to prediction. This practice is ultimately a
problem, Krause contends, because:
“It is only the conformity of a measure to the
normative conceptual analysis and so definition of a dimension that can make
the measurements it produces valid measurements on that dimension. This makes
it essential for each basic science to achieve normative conceptual analyses
and definitions for each of the dimensions in terms of which it describes and
causally explains its phenomena.” (p. 1)
Given the overwhelming emphasis on statistics in
introductory textbooks on scale construction, measurement validity, and
psychometrics, I have a feeling that some will consider the idea that measurement
is fundamentally a matter of definition rather revolutionary. It is likely to
seem even more revolutionary given that many of articles referenced by Krause to
support his viewpoint were fairly modern (e.g., Borsboom, 2005; Krause, 2005;
Maruan, 1998*).
The point of this (admittedly, quite dry) blog post is to simply point out that, over the last century, a number of psychometricians argued that defining the variable was vital—the
first step on the road to successful measurement. The collection of following
quotations—all except Clark and Watson (1995) from texts not referenced by
Krause—demonstrate this point.
“Measurement, of course, is
only a final specialization of description. Measurement...can come in to its
own only when qualitative description has truly ripened... no science has
reached adult stage without passing through a well-developed descriptive stage”
(Cattell, 1946, p. 4)
“Before we can validate the test we have to define the trait which
it is designed to measure. Accurate qualitative description therefore has to
precede measurement. In other words, psychology has to explore the characters
of the unitary traits of man before mental testing can begin” (Cattell, 1948,
p. 1)
“The definition of a variable provides the basis for the
development of a series of operations that are to be performed in order to
obtain descriptions of individuals in terms of the ways in which they manifest
the particular property. The characteristics of the variable dictate the nature
of the operations. With one variable there will be one series of operations,
and with another variable a different series” (Ghiselli, 1964, p. 20)
“If a scale is to be developed to measure a common trait, the scale
must contain items. And before the items can be written, the trait or construct
that the items will presumably measure must be defined. It is necessary, in
other words, to make explicit the nature of the trait or construct that we hope
to measure.” (Edwards, 1970, p. 29)
“A measure should spring from a hypothesis regarding the existence
and nature of an attribute” (Nunnally, 1970, p. 213)
“We cannot measure well when we cannot specify clearly what we are
trying to measure, where it occurs, and when” (Fiske, 1971, p. 30)
“We cannot measure a variable well if we cannot describe it”
(Fiske, 1971, p. 117)
“To advance the science of personology, intensive effort must be
devoted to each major construct, to delineating it explicitly and
systematically, and to create measuring procedures conforming to the blueprints
derived from such a conceptualization” (Fiske, 1971, p. 144)
“Let us try to learn to be free from other a priori mathematical
and statistical considerations and prescriptions – especially codes of
permission. Instead, let us try to think substantively during the initial
stages of measurement, and focus directly on the specific universe of
observations with which we wish to do business” (Guttman, 1971, p. 346)
“Lazarsfield and Barton (1951) have described the process of
measurement in the social sciences in terms of four progressive stages. The
first stage they describe is one in which the investigator forms an initial
image of the nature of the concept he wishes to measure” (Lemon, 1973, p. 29)
“...the construct definition sets the boundaries for potential
measurement techniques. It operates like a test plan for the development of an
instrument” (Shavelson, Hubner, & Stanton, 1976, p. 415)
“...the first problem that the psychologist or educator faces as he
tries to measure the attributes that he is interested in is that of arriving at
a clear, precise, and generally accepted definition of the attribute he
proposes to measure” (Thorndike & Hagen, 1977, p. 10)
“A less obvious, but equally important, characteristic shared by
enumeration and measurement is that they both require prior definition. We
cannot reliably quantify without first defining the objects of properties to be
quantified” (Gordon, 1977, p. 2)
“a meaningful and essential question to raise about a measure is
whether it is consistent with the definition of the construct it is meant to be
tapping” (Pedhazur & Pedhazur-Schmelkin, 1991, p. )
“A critical first step is to develop a precise and detailed
conception of the target construct and it theoretical context. We have found
that writing out a brief, formal description of the construct is very useful in
crystallizing one’s conceptual model” (Clark & Watson, 1995, p. 310)
“Carefully define the domain and facets of the construct and
subject them to content validation before developing other elements of the
assessment instrument... a construct that is poorly defined, undifferentiated,
and imprecisely partitioned will limit the contend validity of the assessment
instrument” (Haynes, Richard, & Kubany, 1995, p. 244)
“To summarize, I offer a five-step model for construct validity
research (depicted in Figure 1 and heavily influenced by Meehl, 1978, 1990a).
The steps are (1) careful specification of the theoretical constructs in
question...” (Smith, 2005, p. 399)
“Every new instrument (or even the redevelopment or adaptation of
an old instrument) must start with an idea – the kernel of the instrument, the
“what” of the “what does it measure?” and the “how” of “how will the measure be
used?” (Wilson, 2005, p. 19)
“The measurement of concepts like creativity and intelligence is
limited by the clarity with which we are able to define the meaning of these
constructs...” (Rust & Golombuk, 2009, p. 31)
“Concepts are the starting point in measurement. Concepts
refer to ideas that have some unity or something in common. The meaning of a
concept is spelled out in a theoretical definition.” (Bollen, 2011, p.
360)
“Theory enters measurement throughout the process. We need theory
to define a concept and to pick out its dimensions. We need theory to develop
or to select indicators that match the theoretical definition and its
dimensions. Theory also enters in determining whether the indicator influences
the latent variable or vice versa.” (Bollen, 2011, p. 361)
“The first, and perhaps most deceptively-simple, facet of scale
construction is articulating the construct(s) to be measured. Whether the
construct (one or more) is viewed as an attitude, a perception, an attribution,
a trait, an emotional response, a behavioural response, a cognitive response,
or a physiological response, or-more generally, a psychological response,
tendency, or disposition of any kind—it must be carefully articulated and
differentiated from other constructs” (Furr, 2011, p. 12)
"The first stage of the scale
development and validation process involves defining the conceptual domain of
the construct" (MacKenzie, Podsakoff, & Podsakoff, 2011, p. 298).“The topic of conceptualization of measurement variables is
probably the least quantitative of the fields of study. Psychometricians have
typically shied away from this area of our work, possibly for this exact
reason. Yet, it is the most important, for without adequate conceptualization,
all else is empty-in particular the whole concept of “validity evidence”
becomes moot, as there is no substance to validate to” (Wilson, 2013, p. 222)
* See Krause's references for these references. Sorry!
References
Bollen, K. A. (2011). Evaluating effect,
composite, and causal indicators in structural equation models. MIS
Quarterly, 35(2), 359-372.
Edwards, A. L. (1970). The measurement of
personality traits by scales an inventories. New York: Holt, Rinehart, and
Winston.
Cattell, R. B. (1946). Description and
measurement of personality. London: George G. Harrap & Co.
Fiske, D. W. (1971). Measuring the concepts
of personality. Oxford, England: Aldine.
Furr, R. M. (2011). Scale construction and psychometrics. London: Sage.
Ghiselli, E. E. (1964). Theory of
psychological measurement. New York: McGraw-Hill.
Gordon, R. L. (1977). Unidimensional scaling
of social variables: Concepts and procedures. New York: The Free Press.
Guttman, L. (1971). Measurement as structural
theory. Psychometrika, 36(4), 329-347.
Haynes, S. N., Richard, D.
C. S., & Kubany, E. S. (1995). Content
validity in psychological assessment: A functional approach to concepts and
methods. Psychological Assessment, 7(3), 238-247.
Krause, M. S. (2012). Measurement validity is fundamentally a matter of definition, not correlation.
Review of General Psychology, 16(4), 391-400.
http://dx.doi.org/10.1037/a0027701
Lemon, N. (1973). Attitudes and their
measurement. London: C. Tinling & Co
Nunnally, J. C. (1970). Introduction to psychological
measurement. New York: McGraw-Hill.
Pedhazur, E. J., & Schmelkin, L. P. (1991).
Measurement, design, and analysis: An integrated approach. New York, NY:
Psychology Press.
MacKenzie,
S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement
and validation procedures in MIS and behavioral research: Integrating new and
existing techniques. MIS quarterly, 35(2), 293-334.
Rust,
J., & Golombok, S. (2009). Modern psychometrics: The science of
psychological assessment. London: Routledge.
Shavelson, R. J., Hubner, J. J., & Stanton,
G. C. (1976). Self-concept: Validation of construct interpretations. Review
of Education Research, 407-441.
Smith, G. T. (2005). On construct validity:
Issues of method and measurement. Psychological Assessment, 17(4),
396-408.
Thorndike, R. L., & Hagen, E. P. (1977). Measurement
and evaluation in psychology and education. New York: John Wiley and Sons.
4th edition
Wilson, M. (2005). Constructing measures: An
item response modelling approach. London: Lawrence Erlbaum Associates.
Wilson, M. (2013). Seeking a balance between
the statistical and scientific elements in psychometrics. Psychometrika, 78(2),
211-236.
