Missing Workers

I went on a podcast binge during some recent holiday travel. In an episode of The Indicator, I heard a pretty striking number: employment levels may be up to 5.5 points higher than reported by the Bureau of Labor Statistics. That’s about 7 million people, which I imagine is an amount economists care about. So what went wrong? I had to find out.

The figure I heard came from a paper published by the Federal Reserve Bank of Boston. The findings are based on a survey fielded over the last eight years that includes both questions used by the BLS in its monthly tracking and a battery about informal work. Informal work here is everything from babysitting to driving for Uber to selling collectibles on eBay.

The survey finds that informal work is a big deal. On average, 1 in 4 Americans participate in informal work of some kind. Many of these informal workers are already classified as employed by the BLS. According to the report, however, some are classified as unemployed or not in the labor force at all. In 2017, the size of this misclassified group may have been enough to increase the employment-population ratio from 61.25 to 66.75 (the 5.5 points quoted in the podcast).

On average, 18 hours are spent on informal work each month. Even among those already classified as employed, these additional labor hours could shift our understanding of American labor and impact policy decisions.

How was such a sizable amount of labor missed? The report offers some answers and the likely culprit is poor survey design. The survey used by the BLS relies on self-reporting of employment, work, and jobs. Many Americans, however, seemingly don’t consider the informal work they do when they see these terms. This may be due to their formal employment, their retirement or disability status, or a drive to do gig work for fun rather than primarily for pay. Whatever the specific reasons, it's a miscommunication potentially as harmful as it was difficult to detect.

What can we do to limit the chances of miscommunication like this in surveys? The right balance between brevity and clarity can be tricky. Even when something seems well-established and simple to define—actually, especially when that’s the case—consider how clear it really is to all respondents.

Here are five tips to make sure what you think you’re asking is what people think they’re answering:

1. Provide definitions and examples. These must still be quick and easy to read (e.g., for work, think of anything you do to provide a good or service in exchange for money).
   
   

2. Include more context. If you’re asking how many hours are spent working each month, you could collect a single number and hope respondents include what aligns with your expectations. Or you could collect a few categories, each helping to clarify the others and offering some flexibility in analysis.

3. User test your surveys. Whenever you can, have people take your survey then interview them. Ask about the experience, points of confusion or frustration, and feedback they wanted to provide but couldn’t.
   
   

4. Don’t force it. Consider whether an answer must be required or whether it’d be beneficial to learn when someone isn’t sure. I don’t know or I’m not sure responses may be followed up by encouraging respondents to submit their best guesses. This approach offers complete responses while also monitoring uncertainty to inform special consideration in analysis and/or potential survey revisions.
   
   

5. Revisit repeated surveys. Circumstances change and so does language. “Work” is very different than it was just a decade ago. The risks of misunderstood questions overshadow the potential value of perfectly static questionnaires.