Assessing Cognitive Abilities: Intelligence and More

Abstract

In modern cognitive science, rationality and intelligence are measured using different tasks and operations. Furthermore, in several contemporary dual process theories of cognition, rationality is a more encompassing construct than intelligence. Researchers need to continue to develop measures of rational thought without regard to empirical correlations with intelligence. The measurement of individual differences in rationality should not be subsumed by the intelligence concept.

Both the target essay by Hunt and Jaeggi [1] and the target essay by Johnson [2] deal with definitional issues in the cognitive domain, as does the provocative editorial by the Editor De Boeck [3]. Hunt and Jaeggi challenge the field by calling definitional issues the unnoticed gorilla in the study of intelligence—referring to a famous experiment known to all readers of this journal. De Boeck asks whether we should broaden intelligence definitions beyond variables correlated with g, and asks the interesting question: “Let’s assume that the positive manifold is a continent. Are there still unknown territories on the continent?”

Using this continent metaphor, I would say that there are unknown cognitive territories on the continent, but also cognitive territories not on the continent—or at least peninsulas and isthmuses. My position, though, is a little unorthodox in that I do not recommend that these cognitive territories be folded into the concept of intelligence.

Hunt and Jaeggi mention in their essay the heuristics and biases work of Kahneman and Tversky, as well as the dual process theories that explain the operation of many of these biases [4,5]. They also mention the types of processes that underlie human rationality—that humans “can analyze evidence, evaluate arguments, and consider the consequences of choices between possible actions.” They are right to link the heuristics and biases tradition to these issues of rationality, because the voluminous work in that tradition has comprehensively mapped an operational definition of rational thought.

To think rationally means taking the appropriate action given one’s goals and beliefs (instrumental rationality), and holding beliefs that are commensurate with available evidence (epistemic rationality). Collectively, the many tasks of the heuristics and biases program—and the even wider literature in decision science [6]—comprise the operational definition of rationality in modern cognitive science. Psychologists have extensively studied aspects of instrumental rationality and irrationality, such as: the ability to display disjunctive reasoning in decision making; the tendency to show inconsistent preferences because of framing effects; the tendency to show a default bias; the tendency to substitute affect for difficult evaluations; the tendency to over-weight short-term rewards at the expense of long-term well-being; the tendency to have choices affected by vivid stimuli; and the tendency for decisions to be affected by irrelevant context. Likewise, they have studied aspects of epistemic rationality and irrationality, such as: the tendency to show incoherent probability assessments; the tendency toward overconfidence in knowledge judgments; the tendency to ignore base-rates; the tendency not to seek to falsify hypotheses; the tendency to try to explain chance events; the tendency toward self-serving personal judgments; the tendency to evaluate evidence with a myside bias; and the tendency to ignore the alternative hypothesis.

In short, we have an extensive and rich set of operationalizations for the concept of rationality in modern cognitive science. None of these operational measures are assessed on common IQ tests. Yet people (including scientists) often talk as if they were. For example, many conceptions of intelligence define it as involving adaptive decision making. Adaptive decision making is the quintessence of rationality, but the items used to assess intelligence on widely accepted tests bear no resemblance to measures of decision making.

My research group has shown that there are systematic individual differences in the judgment and decision making tasks that operationally define rationality in modern cognitive science. With the assistance of a three-year grant from the John Templeton Foundation, my research group and I are constructing the first prototype of a comprehensive test of rational thinking that encompasses most of the heuristics and biases tasks that have been studied in cognitive science. We have shown that most of these rational thinking abilities are indeed on De Boeck’s continent, but some are peninsulas, if not islands (myside bias; [7]). Nevertheless, we do not view our project as an attempt to improve existing intelligence tests, but rather as an attempt to look at what a comprehensive assessment of an allied construct might look like. Which subcomponents correlate with intelligence and which do not will be revealed after the fact. That is, we are not using extant correlations with intelligence (or the lack of such correlations) as serving in any way as a criterion for our test. We are instead using the theoretical foundations of epistemic and instrumental rationality in the literature of philosophy and decision science, as well as the empirical work in the heuristics’ and biases’ tradition, to structure our test. How it maps onto De Boeck’s continent will be revealed after the fact.

In short, perhaps unlike most readers of this journal, I do not see why everything in human nature, cognitively speaking, has to have the label intelligence—particularly when there are readily existing concepts (both scientific concepts and folk concepts) for some of those things (rationality, creativity, wisdom, critical thinking, open-minded thinking, reflectivity, sensitivity to evidence). Calling everything intelligence has been the strategy of broad theorists, like Gardner and Sternberg, for example. Broad theorists inflate the concept of intelligence. By inflation I mean putting into the term more than what the IQ tests measure. One very strong tendency among broad theorists is to use adjectives to differentiate the more encompassing parts of their intelligence concept from the “IQ test part”—what in 2009 [8] I termed MAMBIT (the mental abilities measured by intelligence tests). So, for instance, when Sternberg discusses high practical intelligence, it can be translated to mean “optimal behavior in the domain of practical affairs”, or when Gardner talks about high bodily-kinesthetic intelligence, he means little more than high functioning in the bodily-kinesthetic domain. The word intelligence is actually superfluous. It is there merely to add status to the domain in question (to put it on equal footing with MAMBIT). The strategy seems to be something like the following: Because intelligence is a valued term and we want bodily-kinesthetic talent to be valued too, we’ll fuse the term intelligence onto it in order to transfer some of the value from intelligence to bodily-kinesthetic talent. Indeed, this is why educators have been so enthusiastic about the “multiple intelligences” idea. Its scientific status is irrelevant to them. They use it as a motivational tool—to show that “everyone is intelligent in some way.” The same is true for the coinages of social intelligence or emotional intelligence.

However, there are unintended consequences—some of them quite ironic—of this strategy, consequences that have been insufficiently appreciated. Labeling different mental entities with the same name will encourage just the assumption that many broad theorists want to attack—it will inflate the esteem given to MAMBIT. In a sense, broad theorists seek to break a rule of construct validity—and of common sense: things that are named the same should go together. If these things really are separate mental faculties, and we wish to emphasize their separateness (as Gardner does), then we should not suggest just the opposite by calling them all “intelligences.” However, by their profligate use of the term intelligence, the broad theorists subvert their very purpose of isolating “the IQ test part of intelligence” (MAMBIT) as only one aspect of many cognitive virtues that we may wish to value. By inflating the word intelligence, by associating it with more and more valued mental activities and behaviors, broad theorists will succeed in doing just the opposite of what many of them intend—cutting “the IQ test part of intelligence” down to size. If you inflate the conceptual term intelligence you will inflate all its close associates as well—and 100 years of mental testing makes it a simple historical fact that the closest associate of the term intelligence is “the IQ test part of intelligence.”

My strategy is different than that of the broad theorists. The tests have the label intelligence and thus MAMBIT will always be dominant in the folk psychology of intelligence. I would argue that it is a mistake to ignore this fact. Instead, my strategy is to open up some space for rationality in the lexicon of mental terms and, in doing so, tame the intelligence concept. We have coherent and well operationalized concepts of rational action and belief formation. We have a coherent and well operationalized concept of MAMBIT. No scientific purpose is served by fusing these concepts, because they are very different. To the contrary, scientific progress is made by differentiating concepts.

The strategy of the broad theorists ends up giving us the worst of all worlds. Short shrift is given to the concept of rationality because it is not separately named (but instead conflated with and lost within the intelligence concept). There is no imperative to actually assess rationality, because its semantic space has been gobbled up by the broadened view of intelligence. It will be even harder than it already is to stress that MAMBIT does not measure rational thinking. Although most people recognize that IQ tests do not encompass all of the important mental faculties, we often act (and talk) as if we have forgotten this fact. Where else does our surprise at smart people doing foolish things come from if not from the implicit assumption that rationality and intelligence should go together?

In our society, what gets measured gets valued. With our comprehensive assessment of rational thinking we aim to draw more attention to the skills of rational thought by measuring them systematically and by examining the correlates of individual differences in these cognitive skills. It was in part a historical contingency that the operational measurement of the concept of intelligence is older than the operational measurement of rationality. Our goal has always been to give the concept of rationality a fair shot—almost as if it had been the first one to be measured. If we say that, theoretically, operationalized intelligence dates from Spearman or Binet, then we could say that operationalized rationality dates from the axiomatic approach to utility theory of von Neumann and Morgenstern and Savage—and here we are talking about the 1940s and 1950s. Empirical progress on the rationality concept has been even more delayed. One way to look at our research program is to say that we are looking at how one would measure rationality if one did not know about the historical contingency that intelligence “got there first”.