Society of Professional Journalists
Improving and protecting journalism since 1909

Advertise with SPJ

News and More
Click to Expand Instantly

SPJ News
Events and Deadlines
SPJ Blogs
Quill Online
SPJ on Tumblr
Journalist's Toolbox

Stay in Touch
Twitter Tumblr Facebook Google Plus
RSS Pinterest Pinterest Storify

Current Issue
Browse Archive
About Quill
Media Kit [PDF, 785 KB]
Advertising Info
Back Issue Request
Reprint Permission Form
Pulliam/Kilgore Internship Info

Search Quill

SPJ Blogs
SPJ Leads
The EIJ News
Press Notes
SPJ News
Open Doors
Geneva Conventions
Annual FOI Reports

Home > Publications > Quill > Diversity Toolbox

Current Issue | Browse Archive | About Quill | Advertising Info
Media Kit [PDF, 785 KB] | Back Issues | Reprint Permission Form | Pulliam/Kilgore Internship Info

Search Quill

Tuesday, September 02, 2014
Diversity Toolbox

Avoid a (biased) data dump

By Sally Lehrman

“Good enough” isn’t good enough if you are a data journalist. Even “accurate” isn’t good enough. When you’re working with data, it’s important to run it through a bias filter at every stage.

That’s what three highly regarded investigative journalists insisted as we prepared for a panel on the ethics of working with data at the Investigative Reporters and Editors conference in June. They were joined by civil rights attorney Eva Jefferson Paterson, president of the Equal Justice Society in Oakland, Calif., in highlighting the dangers of skewed data that could undermine investigative work. Reporters are vulnerable at each step of gathering, interpreting and presenting data, they said.

Journalists commonly guard against data weighted by sources to make a point. But most assume that raw data itself is impartial. That’s not the case, Paterson said. Experimental psychologists are finding that implicit bias can creep into the collection and interpretation of what appear to be straightforward numbers. Our eyes can trick us, especially when we are working with data that trigger our unintentional attitudes about social groups.

“Objective information can be skewed because of our implicit bias about people of color,” Paterson said, urging journalists to consider the implications for their reporting. As an example, Paterson described studies of fingerprint matches by forensic scientists. In one, individual evaluators were asked to re-examine prints that they didn’t know they had matched before. When they were offered potentially biasing “case information,” most made completely different judgments. “Confirmation bias” causes us to choose according to our expectations.

Ricardo Sandoval -Palos, senior editor of NPR’s “Morning Edition,” suggested that journalists avoid thinking about data as proof. It should be a setup for further questions, he said. And those questions should not necessarily be the obvious ones. Think about the population the data represent and develop a familiarity with that group.

“Demographically, am I understanding the right questions to ask?” Sandoval Palos said. “Be skeptical.”

Tips from Ricardo Sandoval-Palos:

• Test your data.

• Test the questions you intend to ask of your data.

• Challenge your assumptions and findings.

• Have a second pair of educated eyes look at your data and conclusions.

• Think through, thoroughly, how you will present data.

Tips from Chrys Wu on data visualization:

• Take courses to train yourself how to think about data visualization.

• Find examples in which data is presented in an informative way.

• If you’re going to try something beyond a pie or bar chart, include annotation to explain what you are doing and why.

• Make the underlying data available, but always with context to assist analysis.

Sandoval -Palos pointed to data-based stories that invite misinterpretation. An Associated Press story about the high proportion of Latinos in federal detention was nuanced, offering vital information about immigration laws that drive the phenomenon. But other reports using the same data made it sound as if Hispanics were unusually prone to crime.

It comes down to making sure your data is fair and accurate, agreed Jennifer LaFleur, senior editor for data journalism at the Center for Investigative Reporting. First, the original data sets should be checked for missing data, duplicates and internal problems. Then use your reporting to check out what you think you found.

LaFleur sends her summary data to the person or organization she is reporting on, she said, and often ends up with a sharpened analysis and a better story. Invite feedback and corrections when you publish, too.

Inaccurate, misleading data can do a lot of damage, said Chrys Wu, developer advocate at The New York Times, who is both journalist and coder. She called attention to data presentation at publication. It’s easy to get carried away with the bells and whistles of presenting visual information, she said, forgetting the central purpose: effective communication. Clever graphics can create confusion.

“If we accept that our job is to convey information, to convey truth, we need to make sure the people who see it understand exactly what we’re trying to present,” she said.

With the advent of great new tools to present data, consider the audience. How do we present information that allows people to think about complex problems in a time- and attention-short environment? The long life of material on the Internet makes accuracy all the more important.

Do a gut check when you’re done, LaFleur concluded at the panel. A key question, she said, is “What else could explain (these) findings?”

For a good resource on using and visualizing data, and things you should consider before publication, check out

Sally Lehrman is the Senior Fellow for Journalism Ethics at the Markkula Center. She is an independent journalist who reports on science and social issues.

Copyright © 1996-2014 Society of Professional Journalists. All Rights Reserved. Legal

Society of Professional Journalists
Eugene S. Pulliam National Journalism Center
3909 N. Meridian St.
Indianapolis, IN 46208
317/927-8000 | Fax: 317/920-4789

Contact SPJ Headquarters
Employment Opportunities
Advertise with SPJ