How The New York Times moderates 12,000 comments a day
Last week, The New York Times announced it would expand the use of automation to open up more articles to reader comments. Using a system called Moderator developed with Jigsaw, the tech incubator from Google’s parent company Alphabet, the Times hopes to expand the number of stories open to comments from 10 percent today to 80 percent by the end of the year. But the tech wouldn’t work without the foundation the Times community desk laid.
That’s because Moderator has been trained by being fed 16 million Times comments that have been hand-moderated over the past 10 years. It uses those judgments to build an algorithm that will prioritize comments for moderation.
The editor of the community desk is Bassey Etim, who leads a team of 14 part- and full-time moderators who review about 12,000 comments a day. It’s a complicated process, so the Times can only open up just so many articles to comments and only for a limited time. Moderators aren’t just screening comments for curses and threats; they’re seeking to create a place where readers can discuss stories, exchange ideas at a high level and see all sides of a debate reflected. Etim calls it “content curation at a very high level.”
The job of the desk has gotten more complicated in the Trump era. There are just more comments, and in this heated political environment, it’s become harder to characterize what’s inflammatory. People have gotten more creative about name-calling those they disagree with, whether it’s “liberal snowflakes” or “followers of the Great Orange,” leaving the Times to figure out how to reflect the culture without letting too much meanness get through.
A lot of how the Times handles these depends on context. It’s OK to say snowflake if someone is discussing liberals generally, but not to attack a specific commenter. “Whereas things like ‘t-Rump’ and ‘Orange Menace’ are never allowed, just like we do not allow ‘DemoRats’ or ‘President O-Bozo,'” Bassey said.
Policing comments can be particularly hard on a day like last week’s congressional baseball practice shooting, which brings up a lot of different hot-button issues. With emotions running high, the risk is that a comment that goes too far will slip by the filters. Despite the high-pressure environment, Etim said the community desk has a low turnover rate, with several staffers who joined the desk before he did in 2008.
The desk is made up of people with a mix of editorial backgrounds. Etim looks for people who have strong knowledge of the news, understand ideas across the political spectrum and are able to put their own biases aside to foster a good discussion because moderators have to be able to handle comments on stories on any topic. He asks candidates to evaluate comments for inclusion and to explain their reasoning.
Still, it takes a full year for new employees to get up to speed, but they eventually develop a shorthand, which is how the Times is able to handle the volume it does by hand. “When you read that many comments, your brain starts to code how people talk; you start to know the commenters. Our moderators are familiar with hundreds, maybe a thousand commenters,” Etim said. The automation will help them get to more comments by filtering them. At the same time, the Times is aware of the possibility of bias creeping into algorithms that can change the tenor of comments.
Many publishers from Popular Science to Quartz have eschewed or given up on comments sections as they’ve become polluted by trolls and more people are sharing their opinions on social media. The Times polices comments on its social accounts, too, where it can’t control the process but it can respond to questions and comments. That job falls under Cynthia Collins, Times social media editor, who has nine social media editors working around the globe. “One of our most talked about stories on Facebook could have up to 5,000 comments,” she said.
But comments can build engagement, and engagement can lead people to becoming paying subscribers, which the Times is increasingly leaning on as ad revenue becomes harder to grow. The Financial Times says readers who comment are seven times more engaged than those who don’t, reading more, spending more time on-site and returning more often.
At such a politically polarized time, Etim sees the comments section as a way for people to be exposed to opposing points of view and challenge them without offending the people holding them. “There’s a lot of mind-changing in the comments. It’s really gratifying because the work we do allows people to have those conversations and understand a little bit of the common humanity that is easy to miss.”
One recent Times article on how climate change science divided a classroom in Ohio generated 857 comments from teachers, students, parents and other citizens. “We were basically hearing from every possible person who could be reading this story,” Etim said. “There was a respectful dialogue. That’s what makes the job really fun.”
Member Exclusive‘Math doesn’t add up’: Publishers still face tough choices
“Just salary cuts will at most bring the costs down by 10%, at most, I can guarantee,” one exec messaged me.
Complex Networks plans to diversify its way through the pandemic
Complex Networks bills itself as one of the most diversified digital media companies in the business, so it’s counting on diversification to protect its business.
‘Rats out of the sewers’: Ad fraudsters are leaping on the coronavirus crisis
For ad fraudsters, the coronavirus pandemic is a crisis too tempting to go to waste. Website traffic is surging. But with advertisers adding coronavirus-related keywords to their block lists and others pausing spend altogether, ad prices on news sites are low. With less competition in the auction, low quality ads — and even publishers’ own […]
SponsoredRegulations are prompting publishers to develop new strategies around user log-ins
In a post-GDPR and post-cookie world, more publishers are making concerted efforts to explain the value of their content to users and increase the volume of consumer authentication.
WTF are post-auction discounts?
Post-auction discounts let advertisers compete in the auction as if it bid $6 or $7 or more, but then benefit from a discount after winning the auction.
Highsnobiety closes commerce, cuts 25% of staff
Highsnobiety was one of a few publishers who invested in product creation for its commerce business, rather than just peppering its site with affiliate links.