.

Human Factors In Forensic Evidence

Marc Green


An FBI publication says, "Of all methods of identification, fingerprinting alone has proved to be both infallible and feasible." This might come as a surprise to Brandon Mayfield and Shirley McKie, among others. The FBI incorrectly matched Mr. Mayfield's fingerprint to one of a terrorist bomber of the Madrid subway. Scottish police incorrectly matched Ms. McKie's fingerprint to one found at a crime scene. This article explains reasons why such errors occur. They usually originate from innate human limitations and predispositions in perceptual judgment, rather than from intentional deception or falsification. The mental processes causing the errors are normal and usually rational, but they are not infallible. Examining the causes of these "normal" errors has two benefits. First, it can suggest methods for their prevention. Second, it underlines the reality that changing human nature is difficult, so the ultimate solution lies more in changing the system, not the examiner.

Expert forensic opinion decides many court cases. The opinion often concerns individualization, where the expert examiner has visually compared objects taken from a crime scene (fingerprint, handwriting, bite marks, hair sample, etc.) with an exemplar taken from a suspect or a database. The expert first judges whether the evidence is usable, and if so, then classifies the comparison as a match, an exclusion or as inconclusive. Individualization judgment is so powerful that juries may convict defendants solely on the basis of the expert's opinion.

Courts have generally accepted examiner judgment with little question. Although there has long been some skepticism about the infallibility of forensic experts1 both research studies and high-profile forensic individualization errors have recently moved this skepticism to the forefront.

The research studies have demonstrated two important facts about forensic examiners. First, they make individualization errors2. This is not surprising in domains such as handwriting, where there is obviously great uncertainty. However, expert examiners also make errors in domains such as fingerprints, which authorities have long held to be virtually 100% objective and infallible. Second, examiners can be biased by "contextual factors," sources of information that are independent of the evidence itself4

The highest profile case is the Mayfield fingerprint error. The FBI ran a partial print left by a Madrid subway bomber through AFIS (Automated Fingerprint Identification System, a database of 40 million sets of fingerprints. One of the candidate matches was Brandon Mayfield, an Oregon attorney who had converted to Islam and defended Muslim suspects. A first FBI expert declared that Mayfield's fingerprints matched the partial. A second and then third FBI expert verified the match. Next, an independent, defense fingerprint expert also verified the FBI's match. However, the match proved to be an error when Spanish police subsequently identified the real bomber.

Neither such examiner errors nor the biasing of perception and judgment by context is surprising to those of us who scientifically study vision and cognition. Visual matching tasks like those performed by individuating examiners have been the subject of intense scientific research for many years. Perceptual scientists have performed many thousands of research studies on this topic and understand the sources of errors and biases reasonably well. Until recently, however, the forensic world's assumption of infallibility and objectivity has largely prevented application of this understanding to the evaluation and improvement of expert judgment.
Below, I briefly outline some of the scientific knowledge needed to understand how examiners make perceptual judgments and where the sources of bias and error lie. The discussion frequently refers to the Mayfield case as an example of these factors in operation. Lastly, I also discuss the need for blind testing to reduce the amount of bias.

The discussion also includes comments on eyewitness identification, since the same scientific principles apply.  The eyewitness and the forensic examiner perform very similar tasks: the examiner matches one physically present image to another while the eyewitness matches one physically present image to a remembered image. In fact, the same sensory and nonsensory factors that bias forensic examiner judgment apply across a wide range of law enforcement domains, such as police judgment in the use of force. For example, a police officer patrolling at night sees a suspect holding an indistinct object in his hand. Is it a gun? Should he fire?5

Biasing Forensic Judgments

The fundamental scientific principle of perception is highly unintuitive: all conscious perception is an interpretation. We are not passive recipients of light from the world who experience an objective reality. Rather, we actively construct our perceptions and interpretations using two distinct classes of information, sensory and nonsensory6. Moreover, we reason with this information using heuristics, rules-of-thumb, that help simplify reasoning in complex situations7.

Viewers derive sensory information directly from the senses. It is the physical properties of images as measured objectively by a light meter. For example fingerprint images contain friction ridge patterns, which are light and dark shapes of differing contrast and brightness, perhaps on a cluttered background. Theoretically, sensory information is objective and unbiased in the sense of being independent of the particular viewer's cognitive processes, knowledge, beliefs, goals and expectations, etc. It is presumably an objective reality.

The other set of information is called nonsensory because it does not arise from the image or the senses. The most common nonsensory information concerns probabilities, and payoffs. Knowledge about probabilities affects viewer expectation while knowledge of payoffs for various judgments biases response in one direction over another.

The moment the examiner views the sensory information, his cognitive processes spring into action using nonsensory information to organize, to interpret and to judge the image's meaning. Sometimes these nonsensory cognitive processes bias judgment. Scientifically, the term "bias" is the tendency to favor one response category over another. It does not imply any errant behavior or misdeed. In fact, I will explain that biased judgment is normal and rational and is often intelligent behavior for improving performance.

The probability information suggests the a priori likelihood that one response category is the correct choice. All perceptual scientists know that seeing is believing but that it is just as true that believing is seeing. A viewer is more likely to give a response that is consistent with pre-existing expectations and with other factors affecting probability that the suspect is guilty. For example, one study8 found that examiners were more likely to find a fingerprint match after learning that the suspect had confessed. Conversely, the examiners viewing the same fingerprints were less likely to find a match when told that the suspect was in police custody at the time of the crime.

Another common example occurs when the forensic laboratory requires that a second examiner verify positive matches from the first. The second examiner knows that someone else has already made a positive match, so there is a very strong a priori likelihood that it really is an individualization. The second examiner will then be biased toward judging the comparison a match. The effect will be stronger if the original examiner is senior and has high credibility or if multiple examiners have already confirmed the match. The Mayfield case is an extraordinary clear example because the independent defense expert confirmed the errant match made by three prosecution experts. An FBI review9 of the error concluded, "Once the mind-set occurred with the initial examiner, the subsequent examinations were tainted"

The expectancy effect in the Mayfield case was a classic case of the "representative heuristic," which says that two objects sharing one attribute likely share another. The reasoning likely was 1) Muslims have the attribute terrorist, 2) Mayfield had the attribute Muslim, and 3) so Mayfield most likely shares the attribute terrorist. (Try substituting "black" for "Muslim" and "drug dealer" for "terrorist.")

Use of nonsensory information is often perfectly rational. Fingerprints from a subject who confessed are more likely to match. Objects sharing one attribute often share another. The problem is that such reasoning is not infallible.
Humans are inherently "satisficers", meaning we seek "good enough" solutions within reasonable bounds of information and effort. Humans have evolved reasoning based on biases and heuristics to aid satisficing, but they do not ensure 100% correctness. However, courts assume and expect forensic experts to be correct 100% of the time. Obviously, there is a mismatch between the court's demands and normal human judgment. This is the crux of the problem, and I'll suggest some solutions later.

Once judgment occurs, however, viewers tend toward "confirmation bias,"10 the strong tendency to perceive and to make decisions consistent with prior held beliefs and to ignore evidence which contradicts the prior beliefs. It is perhaps the most power heuristic in human thinking and has been frequently cited as a problem in the examination of forensic evidence11 and as well as a source of error and accident12. Confirmation bias is why FBI agents, having found a good candidate would then tend to seek out evidence to confirm their conclusion and to dismiss contradictory evidence. It is also why law enforcement personnel and courts in general frequently ignore subsequent, contradictory evidence once having decided on a suspect's guilt.

Lastly, payoffs are the other class of nonsensory information. Viewers automatically consider the consequences of their decisions on their goals. The legal system has a general built-in payoff bias because it is presumably "better to let 100 guilty men go free than to convict one innocent man." While this presumably biases the examiner away from matches, there are also personal goals and payoffs to consider.

An examiner might want to win the approval of the local police by helping them solve a high profile or particularly heinous crime13. Some examiners have large egos and want to prove their skill by finding matches in particularly uncertain evidence and where other examiners have "failed." Finding a match in a high-profile case can mean earn recognition and praise. A junior examiner is unlikely to disconfirm a match by a more senior examiner, especially one who is his boss. (Apparently at the FBI, "To disagree was not an expected response."14 Continued failure to find matches may result in loss of status or even employment. If the suspect is a known "bad guy," the public would be better served by putting him away. It isn't hard to spot the biasing payoffs for the FBI agents in the Mayfield case.

Unbiasing Forensic Judgments

Before considering methods to reduce error and eliminate bias, I summarize the key take-home points to remember about bias in perception and judgment:

1. All human perception and judgment is based on a combination of sensory and nonsensory factors and is inherently subjective;

2. When an image is clear and unambiguous, the viewer is likely to judge primarily on the sensory information and approach the court's ideal. The biasing effects of nonsensory information increase as the image becomes more ambiguous and uncertain. Bias further increases under time pressure and stress causing viewers to prefer simpler solutions based on less information;

3. Reasoning on nonsensory information is largely unconscious. Certainly, there may be deliberate attempts by a few examiners to falsify forensic evidence, but normally the nonsensory factors operate without awareness or volition;

4. The use of nonsensory information is normal. It is the typical human solution to making decisions with uncertain information;

5. The use of nonsensory information is generally rational. Matches made by a senior examiner are highly likely to be correct. In many situations, completely ignoring the nonsensory information would depart from normal reasoning and would be considered irrational.

In sum, bias occurs unconsciously and is perfectly normal and rational. It is human nature to be unconsciously biased by nonsensory information because it is usually helpful. Merely giving viewers instructions to do so is futile. Once learned, nonsensory information cannot be completely ignored or forgotten, even if later proved to be incorrect. An examiner who learns that the suspect confessed, for example, will still be affected by that information even if he later learns that the suspect had recanted or even that the confession never existed.

The courts assume that the examiners should use only sensory information because it is objective. This is naïve thinking because all perception and judgment is subjective. The only issue is the degree to which this subjectivity is biased one way or the other. Obviously, there is a conflict between the court's idealized view and real human nature15. As we all know, however, you can't change human nature.

So how can bias be reduced from individualization without changing human nature? The answer is to change the circumstances under which the examiner works. Two obvious solutions suggest themselves.

First, improve the sensory information. Bias effects are smaller when the judgment is less ambiguous and less uncertain. Other domains such as radiology, where viewers must also make categorical judgments from uncertain images, have done extensive research on finding image processing techniques which clarify the critical features. There is little similar research in the field of forensics.

It may also be possible to choose examiners who have greater innate skill at viewing the sensory information and who would then be less influenced by nonsensory factors. Many domains, such as aviation, have developed tests of candidate aptitude. Simple acuity or other visual tests might help weed out unsuitable candidates. It might also be possible to teach examiners how to better extract sensory information from the image. Current training is highly variable from place to place and not based on any scientific understand of how examiners perform their job.

Second, use blind procedures which isolate the examiner from most nonsensory information. The examiner should know nothing about the circumstances of the crime, the opinion of the police about the suspect's identity, the origin of the fingerprints, the nature of the crime, the opinion of other examiners, etc.

It is especially critical to isolate examiners from police and others who may have knowledge of the case. Humans are extraordinarily keen at detecting social cues such as subtle voice inflections, changes in posture, facial expressions, etc. In eyewitness identification procedures, for example, those conducting eyewitness identifications should not know the suspect's identity. They can provide subtle cues which both direct the eyewitness and reinforce another human bias, "overconfidence."16

Blind judgment is also critical in verification. Given what is known about confirmation bias, it is no surprise that studies17 have found blind verification far better at catching errors than non-blind verification.

Moreover, some,18 have gone so far as to claim that the lack of the blind procedures, among other reasons, renders the current ACE-V fingerprint individualization procedures as scientifically unproven. One experienced Ontario Provincial Police fingerprint examiner19 concluded that ACE-V is inherently subjective and noted that "Reaching these [fingerprint classification] decisions, however, requires something not found in science. In essence they require us to use our training, our judgment and our knowledge, to arrive at an opinion," That is, the examiner must use his cognitive processes, which are susceptible to biasing by nonsensory factors.

Conclusion

The first step in understanding and correcting examiner error and bias is a realistic assessment of perception and judgment. Courts frequently have an idealized of human abilities which assumes, expects and desires infallibility or at least perfect "objectivity." The reason that courts would take this view is easy to understand. It greatly simplifies their task in weighting evidence. Moreover, humans typically prefer black-and-white solutions that minimize complexity, especially in a system under severe time and resource constraints.

Unfortunately, simple is not always better. The vast scientific and applied research literature on human perception and judgment shows that the idealized view is untenable. This knowledge on the sources of error and bias is ready and waiting to help evaluate and to correct the defects in the current system. It cannot be applied, however, until courts adopt a more realistic view of human nature and do not expect the impossible. The first step to a solution is to admit that there is a problem.

Endnotes

1 Miller, L. (1987). Procedural bias in forensic science examination of human hair. Law And Human Behavior, 11, 157-163.

2 E. g., Haber, L. & Haber, R. (2003).Error rates for human latent fingerprint examiners. In Ratha, N. & Bolle, R. (Eds.), Advances in Automatic Fingerprint Recognition. New York: Springer-Verlag.

3 E. g., Federal Bureau of Investigation, (1985). The Science of Fingerprints: Classification and Uses. "Of all methods of identification, fingerprinting alone has proved to be both infallible and feasible."

4 Dror, I.E., Peron, A., Hind, S., & Charlton, D. (2005). When emotions get the better of us: The effect of contextual top-down processing on matching fingerprints. Applied Cognitive Psychology, 19(6), 799-809.

5 Green, M. (2005). "Is It A Gun? Or Is It A Wallet?" Perceptual Factors In Police Shootings of Unarmed Suspects, Police Marksman, July/August, 52-54. http://www.visualexpert.com/Resources/policeshooting.html

6 This distinction also goes by other names: "psychophysical" vs. "cognitive" and "bottom-up" vs. "top-down." The "sensory" vs. "nonsensory" terminology come from Signal Detection Theory, a perceptual approach to separating bias from accuracy. Lastly, some refer to nonsensory effects as "context effects," or "observer effects."

7 Green, et al. (2008). Forensic Vision: With Applications To Highway Safety. Tucson: Lawyers & Judges Publishing.

8 Dror, I., Charlton, D., & Peron A. (2006). Contextual information renders experts vulnerable to making erroneous identifications Forensic Science International, 156 (1), 74-78.

9 Stacey, R. (2004). A report on the erroneous fingerprint individualization in the Madrid train bombing case. Journal of Forensic Identification, 54, 706-718.

10 In one classic example of confirmation bias, police arrested a suspect who had been identified in a photo array as a rapist. The man protested that at the time of the rape he was sitting next to an assistant police commissioner on a live TV show in front of millions of viewers. The police dismissed his alibi saying, "Yes, I suppose you've got Jesus Christ, and the Queen of England, too."

11 E. g., Risinger, D. Saks, M., Thompson, W., & Rosenthal, R. (2002). The Daubert/Kumho Implications of Observer Effects in Forensic Science: Hidden Problems of Expectation and Suggestion, California Law Review, 90, 1-55.

12E. g., Green, M. (2004). Nursing error and human nature. Journal of Nursing Law, 9, 37-44.

13 Dror, I., Peron, A., Hind, S., & Charlton, D. (2005). When emotions get the better of us: The effect of contextual top-down processing on matching fingerprints. Applied Cognitive Psychology, 19(6), 799-809.

14See endnote 9.

15 This is very common in law. For example, statutes often say that a viewer should see whatever is available to be seen. This is unrealistic and ignores all the normal cognitive process of attention, expectation and other cognitive processes discussed here plus much more. See footnote and 5 and Marshall, J. (1966). Law And Psychology In Conflict. New York. Bobs-Merrill

16 Green, M. (2003). Proper Eyewitness Identification Procedures," Law and Order, Oct., 195- 198.

17 E .g., Arvizu, J. (2002) Testimony on Mr. Plaza's motion to exclude the government's latent fingerprint identification evidence, hearing before Judge Louis Pollak, USA v. Plaza, et al., U.S. District Court for the Eastern District of Pennsylvania, February 24,2002.

18 Haber, L. & Haber, R, (2008).Scientific Validation of Fingerprint Evidence Under Daubert, Law, Probability & Risk, 7, Issue 2, pp. 87-109, 2008.

19Ashton, W. Fingerprint Identifications (Science or Not). Retrieved from http://www.4intel.com/images/fingerprints.pdf 2 Feb 09.

Footnotes

*A version of this article was published under the title, "Human Fallibility Woven Into Forensic Science" in For The Defense, 2009.