Huffpost Technology
Gerry Smith Headshot

How Police Are Scanning All Of Twitter To Detect Terrorist Threats

Posted: Updated:

When Boston officials decided to monitor Twitter during this year's marathon, they didn't scan the site's 500 million daily posts for signs of trouble.

Dataminr did that for them.

The company's software sorts through millions of tweets for clues about major events or emerging threats, flagging mentions of everything from fires to suspicious packages and sending real-time alerts to customers.

Dataminr has been quietly working with public safety officials in Boston and three other cities with the aim of detecting potential criminal or terrorist activity bubbling up on Twitter before it happens.

Boston's use of Dataminr was part of the city's broader effort to tighten security at the 2014 marathon after last year's bombings. At a time when eyewitnesses may tweet about an emergency before -- or even instead of -- calling 911, Dataminr's co-founder Ted Bailey said his company offers a valuable tool to first responders.

"How can you afford to have these blind spots in your area?" Bailey said in an interview Monday at the company's New York City office, where employees with Ph.D.'s in mathematics and linguistics sat in front of computers and tested Dataminr's complex algorithm.

Dataminr is one of several companies marketing such products to police departments. A company called BrightPlanet is selling a tool called Blue Jay that allows law enforcement officers to listen to what gang members say on Twitter and track their movements. The FBI is also building its own application to monitor social media posts for words like "bomb," "suspicious package" and "white powder."

dataminr boston map

Vernon Keenan, director of the Georgia Bureau of Investigation, said that tools to monitor social media posts can provide useful tips to law enforcement.

"If someone is talking [on Twitter] about planting a bomb or they see explosives or they have a weapon and want to disrupt an event, it alerts you that there's a problem," Keenan told The Huffington Post.

But he said police surveillance of social media can raise privacy and civil liberties concerns if used for reasons other than criminal investigations.

"The problem is if you don't have a specific law enforcement purpose for using the monitoring tools," said Keenan. "Why are you monitoring tweets? What type of information are you going to be collecting? How long are you going to retain it? That has to be addressed before you employ the technology."

Two years ago, privacy advocates found the Department of Homeland Security was monitoring social media sites to track public reaction to negative news about the U.S. government.

Ginger McCall, associate director at the Electronic Privacy Information Center, said the episode showed how law enforcement could use surveillance of social media to restrict free speech or government dissent.

"The application needs to be very narrowly focused in such a way that it doesn't sweep in First Amendment activity," McCall told HuffPost.

Bailey said Dataminr customers can only use the software to track major events on Twitter and can't use it to single out individuals or anti-government tweets. He added that Dataminr customers can't store tweets permanently.

Dataminr's alerts can be limited by location, and focused according to topics like "conflicts and violence" that consist of hundreds of related words and phrases. During this year's Boston Marathon, for example, Dataminr sent an alert to city officials that included the text of a tweet that warned about a "possible suspicious package" at an intersection in Cambridge.

It's unclear how Boston officials responded. In a statement, Lindsay Crudele, the city's community and social technology strategist, said that Dataminir's alerts led to "further on-site investigation" about crowding and security during the marathon, but she did not elaborate on what action was taken.

Boston officials plan to use Dataminr's software again next month to monitor Twitter during the city's Fourth of July festivities, Bailey said.

Founded in 2009, Dataminr has about 75 employees and has raised around $50 million in funding from investors. Wall Street traders have used Dataminr to get real-time information on stocks and several media outlets, including The Huffington Post, are testing its software to track breaking news. Bailey declined to provide the company's revenue, how much its product costs, or which other cities use its software.

Law enforcement's growing effort to monitor Twitter also raises another question: How do you distinguish between real tweets and fake ones? Twitter, of course, can be an unreliable source of information.

Last year, hackers sent a fake tweet on the Associated Press' Twitter account about an explosion at the White House that supposedly injured President Barack Obama. The post, which was retweeted thousands of times in mere minutes, caused the Dow Jones Industrial Average to drop sharply before it quickly recovered.

Bailey said Dataminr doesn't have employees verifying tweets, and he emphasized that first responders should confirm alerts with other sources before going to the scene. But he said the company's software can separate fake tweets from real ones by flagging clusters of similar posts from the same location.

For example, after two apartment buildings collapsed in New York City in March, killing eight people, Dataminr spotted tweets from eyewitnesses who posted about an explosion. The company sent alerts to customers three minutes later, but only after numerous posts appeared to confirm what had happened.

"It wasn't just one tweet," Bailey said. "It was a number of sources on the scene corroborating that this happened. It's the corroborative nature of Twitter after an event happens that is an incredibly essential piece of the puzzle."

Around the Web

Dataminr, Twitter unveil early news detection tool - USA Today

New Twitter Tool Finds Hot Topics Before They Trend | Gadget Lab ...

CNN Doubles Down on Twitter-Based Reporting With Dataminr Deal

Dataminr has a product Twitter might have built

We're Wall Street's secret weapon: Dataminr CEO

Mining for tweets of gold

An algorithm acting as the world's managing editor

 
From Our Partners