Security: Airport: June 2010 Archives


June 6, 2010

Sharon Weinberger has a fairly damning article in Nature on DHS's behavioral screening program, SPOT.
"No scientific evidence exists to support the detection or inference of future behaviour, including intent," declares a 2008 report prepared by the JASON defence advisory group. And the TSA had no business deploying SPOT across the nation's airports "without first validating the scientific basis for identifying suspicious passengers in an airport environment", stated a two-year review of the programme released on 20 May by the Government Accountability Office (GAO), the investigative arm of the US Congress.
[GAO report here]. Apparently, the program is based heavily on Paul Ekman's research on microexpressions (see the TV show "Lie to Me"). There's a bunch of unpersuasive stuff here, for instance:
Ekman's work has brought him cultural acclaim, ranging from a profile in bestselling book Blink -- by Malcolm Gladwell, a staff writer for The New Yorker magazine -- to a fictionalized TV show based on his work, called Lie to Me. But scientists have generally given him a chillier reception. His critics argue that most of his peer-reviewed studies on microexpressions were published decades ago, and much of his more recent writing on the subject has not been peer reviewed. Ekman maintains that this publishing strategy is deliberate -- that he no longer publishes all of the details of his work in the peer-reviewed literature because, he says, those papers are closely followed by scientists in countries such as Syria, Iran and China, which the United States views as a potential threat.

The data that Ekman has made available have not persuaded Charles Honts, a psychologist at Boise State University in Idaho who is an expert in the polygraph or 'lie detector'. Although he was trained on Ekman's coding system in the 1980s, Honts says, he has been unable to replicate Ekman's results on facial coding. David Raskin, a professor emeritus of psychology at the University of Utah in Salt Lake City, says he has had similar problems replicating Ekman's findings. "I have yet to see a comprehensive evaluation" of Ekman's work, he says.


A confounding problem is that the methodology used in SPOT, which is only partially based on Ekman's work, has never been subjected to controlled scientific tests. Nor is there much agreement as to what a fair test should entail. Controlled tests of deception detection typically involve people posing as would-be terrorists and attempting to make it through airport security. Yet Ekman calls this approach "totally bogus", because those playing the parts of 'terrorists' don't face the same stakes as a real terrorist -- and so are unlikely to show the same emotions. "I'm on the record opposed to that sort of testing," he says.

These seem like red flags to me: If we're going to base our defenses on a specific scientific theory about what it takes to detect deception, then it would be nice to have some concrete empirical evidence that the relevant techniques work. If we can't even agree on the terms of the test, then it's hard to see how to have confidence in the system.

We do have some data, though:

The TSA does track statistics. From the SPOT programme's first phase, from January 2006 through to November 2009, according to the agency, behaviour-detection officers referred more than 232,000 people for secondary screening, which involves closer inspection of bags and testing for explosives. The agency notes that the vast majority of those subjected to that extra inspection continued on their travels with no further delays. But 1,710 were arrested, which the TSA cites as evidence for the programme's effectiveness. Critics, however, note that these statistics mean that fewer than 1% of the referrals actually lead to an arrest, and those arrests are overwhelmingly for criminal activities, such as outstanding warrants, completely unrelated to terrorism.

According to the GAO, TSA officials are unsure whether "the SPOT program has ever resulted in the arrest of anyone who is a terrorist, or who was planning to engage in terrorist-related activity". The TSA has hired an independent contractor to assess SPOT. Ekman says he has been apprised of the initial findings, and that they look promising. But the results aren't expected until next year. "It'll be monumental either way," says Maccario.

This seems like something it would be easy to do controlled trials on: say you pick 200,000 random passengers and give them secondary screening (apparently also including a check for outstanding warrants), what fraction would you end up arresting? Even so, if TSA officials are "unsure" I think it's safe to assume that practically none of these arrests have been for anything terrorist-related. After all, if GAO comes asking about the success of your program, wouldn't you deliver the most convincing data you had? So, we're looking at a success rate of somewhere between 0 and (say) 1/20,000. That's not really very impressive.