Rational Bias in Forensics

Thursday, April 8th, 2010

Reason contributors Glen Whitman and Roger Koppl have an article in the journal Law, Probability & Risk on rational bias in forensic analysis. The outright frauds and charlatans in the forensics field are bad enough. But Koppl and Whitman focus on the structural biases inherent in the way forensic analysis is often performed, biases that likely afflict even well-intentioned analysts. From the abstract:

Specifically, forensic examiners’ conclusions are affected not just by objective test results but also by two subjective factors: their prior beliefs about a suspect’s likely guilt or innocence and the relative importance they attach to convicting the guilty rather than the innocent. The authorities—police and prosecutors—implicitly convey information to forensic examiners by their very decision to submit samples for testing. This information induces the examiners to update their prior beliefs in a manner that results in a greater tendency to provide testimony that incriminates the defendant. Forensic results are in a sense ‘contaminated’ by the prosecution and thus do not provide jurors with an independent source of information. Structural reforms to address such problems of rational bias include independence from law enforcement, blind proficiency testing and separation of test from interpretation.

Koppl and I co-wrote a piece for Slate a couple of years ago laying out some of those structural reforms.

The problem goes back to the fact that forensics isn’t really science. With the notable exception of DNA testing, most fields in forensics were invented and developed by police agencies, not scientists. So they have evolved over time not to better seek out the truth, but to help law enforcement personnel prove a hunch or theory. Hence the general absence in most forensics of critical components to the scientific process like peer review and blind testing. The forensic evidence is then still presented to juries with all the gloss, polish, and impressive-sounding vernacular of hard science.

The other problem is that while science might be described as a perpetual journey toward truth, once a jury hands down its verdict, the criminal justice system tends to put a premium on finality and closure. So even though science may later show us that the forensic evidence used to convict someone 10 or 20 years ago is flawed, overstated, or simply false, the courts have been reluctant to reopen those cases.

Digg it |  reddit |  del.icio.us |  Fark

11 Responses to “Rational Bias in Forensics”

  1. #1 |  JP Uno | 

    Fundamentally, people will always interpret scientific data in a light that is most compatible with their pre-conceived notions. This becomes apparent to most people when non-scientists step into the ring to fight for political causes by way of scientific discussion; but my years of University-level chemical research have me thoroughly convinced that it is true of scientists in the laboratory environment as well. I can’t tell you how many times I have had the same argument over whether or not a theoretical model can be said to “fit well” with experimental data; in fact, I had one just today over a calculated light absorption spectrum that kinda-maybe-sorta matched the experimental data, depending on how you define the word “matched.”

    You can never remove the human component from scientifc inquiry; even if you use a computer to scan, say, DNA “matches,” and sort them into a “definite” pile and a “dubious” pile, some person or team has to write the programs and set up the instrument to sort the probable matches. However, Radley makes an extremely wise point about blind checking. It’s almost never the case in the scientific world that a person with a vested interest in the data saying X will argue vehemently that the data say Y, unless it’s absolutely unambiguous. On the other hand, scientists with no vested career interests in X are much more likely to say “Hey, this clearly shows Y; come on.” So while human biases can never be removed from the human endeavor of scientific modeling, they can be greatly mitigated by separation of the police hypothesis from the forensic determination. This is similar to the notion that line-ups should be done in a fashion where there is minimal opportunity for accidental or purposeful police interference.

  2. #2 |  William Anderson | 

    Gee, Radley. Are you saying that maybe Public Choice Theory, as well as “Capture Theory” just might be relevant here? I’m shocked, SHOCKED!

  3. #3 |  Steve Verdon | 

    Prior beliefs…updating….wow, that sounds like they are using Bayesian terminology and a Bayesian approach. Actually I think that is a damn good idea because then the analyst would have to make explicit those prior beliefs in terms of probabilities, and you’d have a point where you could point and highlight the degree of bias. If analysts wanted to be unbiased they’d set the various probabilities accordingly.

    Of course this kind of thing is so far over the head of most lawyers it is rather pathetic. You’d think that people who have to weigh the merits of evidence would have at laest a rudimentary understanding of how to do that via probability theory, but math is hard for most people and when you have to protect that GPA to get into law school…..

    Structural reforms to address such problems of rational bias include independence from law enforcement, blind proficiency testing and separation of test from interpretation.

    Stating one’s prior probability that the defendant is guilty would also work like a freaking charm. It is explicit and obvious and no way to dodge it.

  4. #4 |  johnl | 

    Wow that’s a great journal. I want to read from the previous issue
    William C. Thompson
    Painting the target around the matching profile: the Texas sharpshooter fallacy in forensic DNA interpretation

  5. #5 |  Michael Chaney | 

    As I’ve said before:

    1. Money for forensics should be equally split between prosecution and defense.
    2. Forensics lab should have absolutely no direct contact with lawyers, and ultimately the technician who performs the test should not know what crime it’s connected with, who the client is (prosecution or defense), or anything else beyond the absolute minimum requirements (e.g. Here’s a fingerprint lifted from the scene and here’s a suspect’s fingerprint, see if you think they match).
    3. All tests should be scientifically conducted using double-blind studies where possible.
    4. Any evidence not disclosed in advance to the defense should be disallowed in court.
    5. All studies should be filmed with the film shown to the jury. Studies that can be conducted in the courtroom should be. Example: “this money should be seized – my dog hits on it. Fine. Bailiff collects and tags random money samples from around the court room, numbers each, dog and handler are then brought in to find which of the 10 piles of money are the one. Bailiff leaves after setting up the experiment but before dog and handler come in. After dog and handler are gone, bailiff returns and reveals which pile was the one. The jury can then see if the dog is reliable. Dog hits on more than one pile of cash means he’s not reliable and should be decertified.” Honest prosecutors would be happy with such an arrangement.

    As Radley said, though, they’re not looking for legitimate science – they’re looking to pin the crime on somebody.

  6. #6 |  j a higginbotham | 

    Wow, Chaney said it all.

  7. #7 |  JP Uno | 

    “Here’s a fingerprint lifted from the scene and here’s a suspect’s fingerprint, see if you think they match” is actually too much information for a really good test. I would more likely say, “Here’s a bunch of human subject samples and a bunch of crime scene samples. Some of them might match; tell us if any do.” Most of the time, I would include a suspected match; occasionally, I would not.

  8. #8 |  Aresen | 

    It wasn’t until I started reading Radley’s blog that I found out that most “forensic science” has never been subjected to a peer-reviewed double-blind study, including a study of the likelihood of a ‘false positive’.

    Until that is done, I will regard all ‘forensic science’ as little more than 21st century phrenology.

  9. #9 |  Timothy | 

    As a general rule, fields that put “science” in their names really aren’t. Forensic science? Not science. Political science? Definitely not science. Management science? Only science in a parallel universe where MBAs are worth the paper they’re printed on.

  10. #10 |  Dave Krueger | 

    I think those in the justice system forensic “science” business would argue that the fact that forensic evidence is always open to challenge in court makes it credible.

    Maybe the biggest problem with forensic evidence is that defense attorneys don’t challenge it in court, because they themselves tend to believe it’s real science (as perhaps they are taught in law school).

    Having said that, I have to agree with Michael Chaney. The justice system and the public are always resistant to anything that would impede their ability to get convictions.

  11. #11 |  Buddy Hinton | 

    If spending parity as between criminal defendant and the state was considered as part and parcel of the right to counsel (as it very much should be), then forensic science would be good science.

    This problem is bad, and it is, at its heart, a money issue.

Leave a Reply