What is this thing? ( Soerfm/Wikipedia)
What is this thing? ( Soerfm/Wikipedia)

Do You Really Understand Why Water Boils? New Survey Says, Probably Not.

Good news, America: Roughly 75 percent of us know who developed the polio vaccine, that ocean tides are caused by the tugging of the moon’s gravity, and that astrology and astronomy are, in fact, different things. Phew.

Great news! The vast majority of us (86 percent) know the Earth’s core is hotter than its surface, and that if you want to make nuclear weapons, you’re going to need some uranium (82 percent of you know that, which could be somewhat concerning).

The not-so-good news: Two-thirds of you don’t understand why water boils, and figuring out what makes sounds loud is hard.

These are just some of the results released today from a science knowledge survey conducted by the Pew Research Center. Over three weeks in 2014, 3,278 people answered 12 science questions. Overall, 6 percent of respondents got a perfect score, and half answered more than eight questions correctly. Not surprisingly, the number of correct answers lined up closely with education level.

I guess that’s not so bad, especially given that some of the questions were a bit off the beaten track, like asking what a light-year measures (Time? Distance? Brightness? Weight?) or how to interpret a graph with tooth decay on one axis and sugar consumption on the other (take a guess).

Or were they? Based on these results, it might be tempting to think we can relax and stop worrying about Americans performing somewhat dismally in the sciences; but before we do that, we need to take a good look at what the Pew survey — and others like it — are really testing.

The key with such surveys, says the University of Michigan’s Jon Miller, who’s been studying science literacy for nearly four decades, is to ask questions about core concepts. Things like what molecules are, what DNA is, and how the universe is organized. Show people an image of a spiral galaxy, like the Milky Way, and ask if they know what it is and why it’s important.

Conversely, asking about names, dates, or places can tap into someone’s biographical rather than foundational knowledge. “It’s not really getting at whether they have a skill level,” Miller says.

In other words, surveys should not be testing whether people understand the headlines in today’s science stories, but whether they have enough basic knowledge to understand the headlines 20 years from now.

“What you’re not trying to do is measure, do they know what the SARS virus is, or what the West Nile virus is — but do they understand what a virus is?” Miller says. “What we should be doing is thinking about measuring the constructs that people need to know.”

Some of the questions on the Pew survey might do that. Miller points to the question about what a light-year measures (“it offers a basic structural sense of the Universe”), and the scatterplot, which tests whether people can interpret the relationship shown on a graph.

But others might be a bit more like trivia, instead offering a snapshot of this particular moment in time rather than true science comprehension. It’s a pitfall that many surveys have run into, starting with the very first one, conducted by the National Association of Science Writers in 1957. One of the survey questions asked about radioactive fallout, and which type of element commonly rained down from the sky. Would you know the answer now? Maybe not. But in the mid-20th century, Miller notes, the Nuclear Test Ban Treaty hadn’t been signed and nations were still exploding bombs overhead. So, newspapers regularly covered strontium-90 fallout.

Another example, perhaps, is a question on this year’s Pew survey. It shows a photo of a comet and asks people whether the icy object with “a tail of gas and dust that extends millions of miles” is a comet, star, asteroid, or moon. Seventy-eight percent of people answered correctly. Coincidentally, the European Space Agency’s Rosetta mission — in which a spacecraft pulled into orbit around a comet and then sent a lander to the dirtball’s surface – was heavily featured in the popular press around the time of testing (Aug. 11-Sept. 3, 2014).

One more result that might speak to this point comes from a Pew question about Jonas Salk, who developed the polio vaccine in the early 1950s. Among older adults who were alive at the time (65 years and up), 86 percent got the question right. Conversely, only 68 percent of young adults, between the ages of 18-29, got the question right.

In these cases, is it possible to disentangle actual science comprehension from biographical memories or simply reading the news? It’s not clear. And, might it be more important to understand how vaccines work and how Earth’s solar neighborhood is put together? Pew notes in its report that “how much Americans appear to know about science depends on the kinds of questions asked,” and that the survey only represents “a small slice of science knowledge.”

It would be interesting to see how the answers to these questions fluctuate over the next five years, or 10 years, and if popular science knowledge waxes and wanes like the phases of the moon…you do know how that works, right?