Nine years and billions of dollars after September 11th, we have a database that can't do something as basic as connect a denied visa and a CIA report on the same person.
The level of database indexing and management that would have identified Umar Farouk Abdulmutallab as a potential threat is fundamental stuff that is practiced every day - by Google, and by any reasonably sophisticated database marketer.
• The National Counterterrorism Master List is made up of 550,000 people who might pose a risk. A subset of them, about 14,000 people, are subject to additional screening; another subset about 4,000 people, are on the no-fly list.
When the list was set up, some famous people ended up on it because of misspellings and other confusion. The press mocked the government's efforts, and public confidence was eroded. This was easily preventable with some basic computer code, which would have cross-checked across multiple databases. The mistakes clearly highlighted a deeper problem.
• I don't know if the Master List is a state-of-the-art database, or a sloppy grab bag of alphabetized names with little or no sophistication behind it. But I have my suspicions.
To be useful, the database needs to be organized in such a way so that it is searchable by various words and concepts, and it needs to be continuously updated based on a) information fed into by other government agencies, and b) information that is gathered by constantly crawling the Internet and looking for new and relevant data - about the individual, and about the individual's nexus of contacts.
The database also needs to be active, not just passive - continually presenting new information to investigators based on instructions they have given it.
• So when the British government - our closest ally, and partner in fighting terrorism - refused to give Abdulmutallab a visa, he should have immediately been populated into our Master List. And the system should have sent out a series of instructions that would have quickly gathered and organized more information about him, and fed it into pre-set data fields.
• When the CIA officers in Lagos were altered to Abdulmutallab's possible radicalization by his father, his name should have already been in the database. With this additional layer of threat warning, all sorts of automated flags should have been generated - a computer cascade of alerts.
His name should at the very least have instantly and electronically been moved to the list of 14,000 people who require additional screening. Just like Amazon treats customers who order 100 books a year differently than those who order just ten, it's not hard to write some computer code that moves someone from Category A to Category B based on an established set of criteria and contingencies.
Similarly, the Master List requires a set of criteria to determine when a case needs to be manually reviewed -- perhaps resulting in a case officer being assigned to a potential terrorist.
This is where human intelligence needs to work seamlessly with database engineers; CIA officials should set a series of automated criteria that automatically trigger the need for a personal review of a case. Those benchmarks would be tied to an individual's behavior as well as the behavior of those in their network.
This should have happened even without any information about a potential Nigerian terrorist being trained in Yemen - which we now know was available.
• Beyond Abdulmutallab, for every individual on the Master List - and working within the appropriate privacy requirements, which is a separate and controversial issue itself - we should have an incredibly detailed, and continuously updated, dataset based on all the breadcrumbs they leave online.
And when someone stops doing something it can be more important than what they are doing. If someone who was an active emailer suddenly goes dark, for example - particularly if that's linked to other behaviors - the system needs to capture and distribute that information.
• All the information that has come out about Abdulmutallab since the incident - the posts that he made to the Islamic Forum Website, the events he ran as president of the Islamic Students Association - should have been crawled by Homeland Security and made part of his digital file. And again, protocols need to be written so that all this happen on an automated basis the moment someone is added to the Master List.
Google sends out its spiders to crawl the web and bring back data that allow it to rank billions of pages and organize search in a highly effective fashion. That's harder than the task Homeland Security has.
Databases can be incredibly useful tools for variety of practices - just ask eHarmony - when they are constructed with the right internal linkages and algorithms. If there is a link between the imam involved in the Fort Hood homicides - and the Northwest incident - that should be algorithmically available to the CIA.
President Obama spoke of a systemic failure and human error as if they are two separate vectors. But the "system" is a merely a long chain of human decisions. If we continue to regard them as separate, we will never use technology to its full potential and we will continually be in more jeopardy than is necessary.
Google's mission is to "organize all the world's information and make it universally accessible and useful." With the exception of the "universally" part, everything else speaks directly to the database challenges involved with stopping terrorism.
I'd feel much safer if the engineers at Google were running the database that identifies and tracks potential terrorists. Wouldn't you?
Follow Adam Hanft on Twitter: www.twitter.com/hanft
The single biggest issue seems to be uniquely identifying the potential terrorists, and unlike in your commercial examples, they won't be signing up and maintaining their own personal data like at Facebook or dating sites. Therefore we don't have natural unique userid's to connect all the reports of them. So it's left up to multiple agencies, and when Joe from the FBI might think Ali from Nigeria is a different person from Ali from London, you now have 2 potential terrorists instead of one (or worse multiple persons are conflated into one).
Even if you have all the right data, to automate an action i.e. "all sorts of automated flags", you need to be able to automatically quantify the threat (again, beforehand), otherwise it's left to human decision-making. Computers are a good tool, but your overheated imaginings end up equating them with magic, and don't really help the advance any positive solutions.
It should be noted that he didn't succeed. Because since 9-11 plane passengers are no longer passive. Real people saw what he was up to, kicked his butt and stopped him. Sounds like it worked out to me....
How long can we rely on terrorists to fail?
It's frightening to think that after all these years, Homeland Security and its system designers cannot imagine and devise better systems. The crossovers, duplications, and other problems exist in most databases--and are solved far better than these were. This was a shocking failure of government.
It's not the quiet ones you have to watch. To paraphrase the late George Carlin, "If you are in a bar, are you going to watch the quiet guy in the corner minding his own business. Or are you gonna pay attention to the guy jumping up and down, waving his machete, and shouting how he's gonna gut the next M.F. that looks at him crosswise?"
Ok, so now we've defined who it should track. Now what specific actions should it alert us to? Buildings bombed? VISA's denied? Politicians Assassinated? Political Speeches? Radical emailings? Angry Phone calls? Lover quarrels?
You seem to say he had a change in behavior. Ok, let's track changes in behavior of the 550000000 people on the list. When they post to their blogs, when they email their friends. When they "like" something on Facebook. We obviously have to have a base line behavior to notice a change. That's just assuming that they all have internet and aren't from third world countries. Does somalia even have a regognized government? I know they must have high speed Comcast internet.
Adam... it sounds pretty the way you say it, but the manpower hours alone would cost billions of dollars to set up. Not to mention all of the other logistical problems.
Good try though.
"The National Counterterrorism Master List is made up of 550,000 people who might pose a risk. A subset of them, about 14,000 people, are subject to additional screening; another subset about 4,000 people, are on the no-fly list." ... So we are talking about over HALF A MILLION people who's crimes "might pose a threat" to America.
So, Adam, what kind of computer search program would be necessary to actually do periodic searches on ALL of those people? What would be the memory requirements? How large a server room would that computer system require?
How do we set it up to search all the terrorists we nab in Iraq and Afganistan? How do we sync it to work over multiple classified networks? You did realize TIDE was a Top Secret database right? Not to mention you want it to use info from foreign nations.
How would you like to bypass the Posse Comitatus Act? Or is it just to collect on foreign terrorists and to ignore homegrown ones like Timothy Mcveigh?
So, now we have a Database of Half a million people and supposedly a computer program to update it. Now what exactly is it looking for? Extremism right? OK, then you give us your definition of extremism. What makes a radical? What's the difference between "Abdulmutallab's possible radicalization by his father" and the KKK right here domestically?
Look folks, magnificent and breathtaking multi-hundred million dollar IT failures happen because people confuse technology with solutions to a problem. A successful solution requires a lot more than whiz-bang technology. It first and foremost requires a very tight definition of the problem to be solved, a description of exactly how the chosen system solves the problem, a careful analysis of the human factors, a description of failure modes, and the impact of those failures. And that is just to start. Read about some of the truly magnificent IT failures that have occurred. Air Traffic control and IRS systems are good starting points but there are many more.
This is our achilles heel. We will pay hundreds of billions for whiz bang technology but we will not set up a program for training and properly paying for decent Arabic/Dari/Pashtu/etc. linguists and SW Asia/ME area experts qualified to do the real hard work. Why not? What if we set up a program that would train anyone who could pass a background check and an aptitude test to become a linguist in the strategic language of their choice? What if we allowed these linguists to serve in their local communities at secure facilities located far from Washington but securely linked to NSA and CIA facilities in the DC area. What if we made those jobs very desirable by pay and benefits so that we would get the best of the best in large numbers?