Measuring Happiness Is Harder (But Maybe Also Easier) Than You Think

ESM Versus DRM

In addition to comparing ESM and DRM to one another, it is also useful to see how strongly they correlate with additional criteria. Unfortunately, deciding what measure to use as an external criterion is not an easy task, as there is no clear gold-standard measures of well-being that one can use to validate a new technique. In a series of papers, my colleagues and I have tried to examine the links between experiential measures and wide range of alternative forms of assessment and external criteria (not all of which will be covered in this post).

For instance, the table below shows the correlation between ESM and DRM measures of affect and four self-report measures of well-being (a single-item measure of life satisfaction; the Satisfaction With Life Scale; and global, retrospective measures of experienced affect), along with two informant-report measures of life satisfaction. A few patterns are worth pointing out. First, at least for experiential positive affect, correlations between the global self-reports and DRM measures tend to be stronger than the correlations with the ESM measures. For negative affect, on the other hand, the differences are very small (and nonsignificant); and for the informant-rated criteria, correlations with the two types of measures are virtually identical. So far, at least for these limited criteria, there may be a slight advantage for ESM versus DRM, but the differences are not large.¹

ESM and DRM Versus Global Reports

Once we start thinking about comparing the validity of the DRM to that of the ESM, it is also possible to think about how both compare to the validity of global self-reports. Specifically, if we focus on informant reports as a useful criterion, we can test whether correlations are stronger with the DRM versus the frequently criticized global self-report measures. And in our studies, we consistently find that correlations with global self-report measures are at least as strong, if not higher than correlations with the DRM. For instance, the correlations with the two informant-reported life satisfaction measures are r = .37 and .35 for global positive affect and -.27 and -.25 for negative affect, values that are consistently (though not substantially) larger than those from the table above. Our study is not especially well-powered to detect whether these are significantly larger than the correlations with the DRM and ESM; but the same general pattern emerges in a different paper we recently published. The table below shows correlations from two studies that included global, informant, and DRM measures of positive and negative affect (Hudson et al. 2019).²

Again, although the differences are not large, for each construct, global self-reports did a bit better in predicting informant reports than did DRM-based measures. I don’t think informant reports provide the perfect criterion, and it is important to note that all of these correlations are pretty far from an r of 1.00, but the fact that global measures correlate more strongly with informant reports (i.e., have greater evidence of validity using this criterion) than do experiential measures is worth noting.

Some People Report More Within-Person Variability Than Others

The problem is that variability in self-reports of one type of construct correlates too strongly with variability in ratings of completely unrelated phenomena.

The first piece of evidence comes from a seemingly unrelated set of studies: Studies on within-person variability in personality. The idea that some people are more variable—in their affect, their levels of energy, or even their personality—seems pretty intuitive. Many studies have examined within-person variability, assuming that it is a meaningful individual difference in its own right. However, my former student, Brendan Baird, showed that we should be concerned about what we get when we assess within-person variability using repeated self-report measures.

The problem is that variability in self-reports of one type of construct correlates too strongly with variability in ratings of completely unrelated phenomena. For instance, in our study on within-person variability in personality, we asked people about their personality in different roles (e.g., what are you like as a friend? What are you like as a worker?). We found, however, that cross-role variability in ratings of one’s own personality correlated .64 with cross-role variability in ratings of one’s friend’s personality, even though people didn’t see themselves to be that similar to their friends in terms of their average personality (Baird, Lucas, and Donnellan 2017). That seems too strong to me. Even worse, cross-role variability in one’s own personality correlated moderately (around r = .45) with variability in ratings of the personality of various Simpsons characters and even with variability in the desirability of various “neutral objects” like the local public transportation system, local speed limits, and 8 1/2 by 11 inch paper (for an open-access version of this paper, see here).³

This could be a problem if we are looking at the effects of within person changes in situations on variables like personality and affect, as those individuals with more variability may appear to be more strongly affected. In addition, depending on the nature of the situational factors that are being examined, within-person variability could affect overall mean levels that people report. Importantly, the associations were also moderate in size (though weaker) when actual experiential measures were used (daily-diary methods, in this case), and even ESM-based measures of variability correlate pretty strongly with the cross-role measures that were the primary focus (Baird, Le, and Lucas 2006). It does seem to be the case, though, that the effects are exacerbated when the situations or roles across which variability occurs are assessed in a single sitting (as is true with the DRM).

Simply Asking About Variability Causes More Variability

A second concern, and one that is probably more relevant to our assessment of the validity of experiential measures, comes from a different study that Brendan conducted, one in which he experimentally manipulated whether people were asked about their personality in multiple roles versus just one (Baird and Lucas 2011). The trick was figuring out how to assess the effect of this manipulation on reports of variability, because there is no variability when only a single role is assessed. So before participants came to the lab for the study, we administered a short personality questionnaire asking them what their personality was like “in general.” Then, once they got to the lab, half were asked to report on their personality in a single role (e.g., as a friend) and half were asked to complete the same questionnaire multiple times for multiple roles. We then compared the role-specific ratings to the general ratings that had been provided a few days before.

The results were pretty striking. Those people who reported on multiple roles showed greater discrepancies and weaker correlations with their ratings of what they were like in general. In addition, the effect of specific roles was inflated in the multiple role condition in a direction that, in our interpretation, was consistent with stereotypes about how people are supposed to behave. For example, the effect of the friend role on extraversion was more positive in the multiple-role condition than the single-role condition. It seemed as though participants in the single-role condition interpreted our questions about their role-specific personality as just another question about what they were like in general, whereas those in the multiple-role condition interpreted the same question as one about how they differ from what they’re typically like when in that role. This is consistent with a broader literature on how the questions we ask shape the answers that respondents provide (Schwarz 1999).

Back to the DRM

Why does this matter for experiential measures of subjective well-being? Well let’s look at a typical set of findings that emerge when these measures are used. In general, research like that reported in Kahneman et al. (2004) usually shows that DRM measures correlate less strongly with stable, person-level factors (like income, health, etc.) and more strongly with other situational variables that are measured at the same time that experienced well-being is assessed. This pattern is often interpreted to mean that people rely on their beliefs about what factors should matter for happiness (factors like income and health) when answering global measures, and that the experiential measures reveal that what really matters is how we spend our time. The problem is that the DRM looks a lot like our experimentally manipulated multiple-role condition—a condition that pulled for reports that differed from what one is like in general and as well as reports that exaggerated the impact of the specific roles that were made salient.

In fact, when we compared the DRM to the ESM in our new paper, the pattern looked pretty consistent with the effects of the multiple-role condition in Baird and Lucas (2011). I won’t go into detail here (this post is already long enough), but as I noted above, the DRM was often less strongly correlated with global self-reports (i.e., what one thinks they are like in general) than was the ESM. In addition, the DRM was more strongly affected by situational factors and more closely linked with global stereotypes about how people typically feel in various situations than was the ESM. By repeatedly asking “what were you doing and how were you feeling”—especially in a single sitting—researchers may be subtly communicating to participants that they are less interested in what the respondents are like in general and are more focused on participants’ opinions about the situations in which they find themselves. And although I think that this type of effect is likely to be most pronounced when using methods like the DRM (which are completed in a single sitting), it is possible that it occurs any time respondents are required to answer the same questions over and over again (including ESM).

Conclusion

Years ago, when I started doing research on subjective well-being, the story that you used to hear was that life circumstances don’t matter for well-being: How happy you are, it was said, is almost completely unrelated to the income you have, the health problems from which you suffer, or the number of positive and negative life events you’ve experienced. The very fact that these associations were so low was used as evidence that the measures simply could not be right (Schwarz and Strack 1999). This idea played into the counterintuitive narrative that people simple didn’t know, or at the very least couldn’t report, how happy they really were.

Over the years, the field started to recognize that this narrative resulted from weak studies with small sample sizes, combined with problems regarding the interpretation of effect sizes. Now we know that life circumstances do matter, and that global self-reports correlate in pretty intuitive ways with many life circumstance variables (e.g., Lucas 2007).

But around this time, another counterintuitive narrative began to take hold. As the number of studies that included the DRM (or other similar measures) grew, people began to find that these new measures tended to correlate very weakly with the stable life circumstance variables that had been the focus of so much well-being research. So now the narrative was that yes, life circumstances correlate with global well-being measures, but only because people believe that they should. When we assess actual well-being with experiential measures, we find the real association, which is pretty close to zero. Now the same evidence that was once used against global self-reports (their weak associations with life circumstance variables that simply had to matter) didn’t seem relevant when applied to the new class of measures. Moreover, the opposite set of results (the fact that global measures do correlate with life circumstance variables) was now being used as evidence against their validity.

via GIPHY

I want to be clear that I do not think that global self-reports of subjective well-being are flawless. Indeed, I think there is still a lot to be concerned about and a lot more research we need to do. The point of my two posts is this: Although experiential measures seem to be able to fix a few obvious threats to the validity from which global self-reports might suffer, there may be additional threats to the validity of these alternative measures that are rarely considered. And because we don’t know the relative importance of these various threats to validity, it is still quite possible that the often-maligned global self-report measures of well-being—measures that are inexpensive, that are simple to understand, and that can easily be incorporated into just about any type of study—are our best bet for assessing people’s subjective sense of how well their lives are going.