Testimony of Mr. Larry Willis, Program Manager for the Science and Technology Directorate, U.S. House of Representatives, Committee on Science and Technology Subcommittee on Investigations and Oversight, on "Behavioral Science and Security: Evaluating TSA's SPOT (Screening of Passengers by Observational Techniques) Program"

Rayburn House Office Building

Introduction and Study Objective:

Good afternoon, Chairman Broun, Ranking Member Edwards and distinguished Members of the Subcommittee.  I am honored to appear before you today on behalf of the Department of Homeland Security (DHS) Science and Technology Directorate (S&T) to discuss our evaluation of the Transportation Security Administration’s (TSA) Screening of Passengers by Observation Techniques (SPOT) program. SPOT is a behavior observation and analysis program in which personnel are trained to identify behaviors that deviate from an established baseline that could be possible indicators for terrorism or criminal activity.  The SPOT program is based on scientific research and represents the best practices from defense, intelligence, and law enforcement organizations. Today, I will describe S&T’s research assessing the validity of the SPOT Referral Report, which is a checklist of predefined observable indicators used by TSA to identify potentially high risk travelers.  For the purpose of S&T’s study, high risk travelers are defined as those passengers in possession of serious prohibited and/or illegal items or individuals engaging in conduct leading to an arrest.  Specifically, our study offers an assessment of the extent to which the SPOT Referral Report of observable indicators leads to correct screening decisions at the security checkpoint.  

Research Requirements and Background:

Approximately 1.2 million people fly within the United States daily.  The SPOT program trains TSA personnel to serve as an additional layer of security in airports by providing a non-intrusive means of identifying individuals who may pose a risk of terrorism or criminal activity.  In behavior-based screening, trained personnel attempt to identify anomalous behaviors by observing passengers and comparing what they see to an established behavioral baseline of other passengers developed in the same general location and within the same timeframe.  It is important to note that behavioral screening isn’t limited to aviation security and is conducted formally or informally by other DHS agencies, the Department of Defense, the Intelligence Community, and law enforcement worldwide.  The SPOT validation effort appears to be the most rigorous evaluation of behavioral-based screening. 

The SPOT validation effort began in 2007 as a result of the component-led, S&T-managed People Screening Capstone Integrated Product Team (IPT) process that identified and prioritized capability gaps of DHS operational components.   

The “People Screening” Capstone IPT established the research requirement to identify and validate observable behavior indicators of threats and suspicious behaviors in a screening environment.  As an active participant in this IPT, TSA identified the SPOT Referral Report and its associated indicators as a candidate for the validation study.  Through a series of interactions with TSA, S&T determined that the SPOT screening process and the effectiveness of the observable indicators list was testable.  The SPOT Referral Report contains a discrete list of observable indicators which have been designated by TSA as Sensitive Security Information (SSI).  TSA’s Behavior Detection Officers (BDOs) are trained to identify these indicators and use them to make screening decisions, such as referral for additional screening at the TSA checkpoint.  Furthermore, TSA records each behavior-based screening event, as well as its corresponding indicators, screening results, and outcomes to help inform future screening decisions.  The SPOT process leads to three possible actions: the traveler proceeds through the TSA checkpoint and to their flight as normal; the traveler is identified as possibly carrying serious prohibited/illegal items and receives additional screening at the TSA checkpoint; or the traveler is identified to a Law Enforcement Officer (LEO) for appropriate intervention.

Research Approach:

S&T, in cooperation with the American Institutes for Research (AIR), designed the Base Rate Study to compare TSA’s SPOT Referral Report process with a random screening process and to estimate the population base rate of high-risk travelers.  AIR is one of the largest non-profit behavioral science research organizations in North America and has performed numerous validation studies.  Two databases were used for this study.  The first was designed to include case information from randomly selected travelers who were subjected to the SPOT referral process during the Base Rate Study from December 1, 2009 through October 31, 2010, including a total of 71,589 referrals from 43 airports.  To make direct comparisons between the Base Rate database and the Operational SPOT Referrals, a second dataset (SPOT comparison dataset) was extracted from TSA’s SPOT Referral database to contain the 23,265 Operational SPOT referrals collected during the same time period and from locations covered by the Base Rate Study.  Together, these two datasets allowed AIR to assess the extent to which the SPOT Referral Report of observable indicators leads to correct screening decisions at the security checkpoint.

Research Results:

A number of key findings emerged from the analysis of the SPOT Referral Report, including four that I would like to share with you:

1. Operational SPOT identifies high-risk travelers at a significantly higher rate than random screening. The study data indicate that a high risk traveler is nine times more likely to be identified using Operational SPOT versus random screening. (Operational SPOT refers to the standard operating procedure of the BDOs executing the referral reporting process at the checkpoint as opposed to the program as a whole.) Moreover, to achieve these outcomes, BDOs were able to engage with 50,000 fewer travelers using Operational SPOT than they did when using random selection methods.
2. SPOT indicators appear to be observed and utilized consistently across varying airport characteristics. When we examined the consistency in implementation overall, we found that observable indicators within the SPOT Referral Report are used at relatively the same rate regardless of the year, time of year, or size of airport. Moreover, indicators tended to be consistently related to outcomes in the same ways across these characteristics, providing further evidence that the indicators are reliable. These results also serve as initial support for reliability in the use of the SPOT Referral Report, with little to no evidence of major coding variations or random fluctuations.
3. The population base rate for high-risk travelers is extremely low. In other words, the large majority of travelers pose no security risks. Results of the Base Rate Study confirm that the measurable outcomes that represent high-risk travelers are rare events. These data indicate that the estimated population parameter for:
   1. Arrested by Law Enforcement Officer is 1 in 10,000 travelers (or 0.01 percent).
   2. Possession of Fraudulent Documents is 1 in 2,000 travelers (or 0.05 percent).
   3.  Possession of Serious Prohibited/Illegal Items is 1 in 750 travelers (or 0.13 percent).
   4. Combined Outcome, or presence of any outcome (of the above), is 1 in 750 travelers (or 0.13 percent).
4. The population base rate for SPOT indicators is low.  Among those selected for random screening in the Base Rate Study, very few travelers (approximately 8 percent) exhibited any SPOT indicators. The most frequently observed indicator (again, SPOT indicators are designated SSI) was displayed in only 2.8 percent of the randomly selected travelers. In contrast, this indicator is exhibited in more than half of SPOT-referred travelers. All of the other indicators were observed in fewer than 2 percent of the travelers selected by the Base Rate Study.

Conclusion:

In conclusion, these results indicate that the SPOT program is significantly more effective than random screening: a high-risk traveler is nine times more likely to be identified using Operational SPOT versus random screening.  Our validation process, which included an independent and comprehensive review of SPOT, is a key example of how S&T works to enhance the effectiveness of the Department’s operational activities.  Expanding on these initial findings, we would like to conduct further research to assess the screening accuracy of these observable indicators in similar operational screening environments, in aviation and beyond.  Additionally, we would like to work to identify other indicators that could further increase accuracy in operational screening.

Chairman Broun, Ranking Member Edwards, I thank you again for this opportunity to discuss the Screening of Passengers by Observation Techniques program.   I am happy to answer any questions the Subcommittee may have.