Forthcoming NAS Study Expected to Shake Up Forensics World

Friday, February 6th, 2009

A forthcoming study from the National Academy of Sciences is expected to send shockwaves through the criminal justice system.

People who have seen it say it is a sweeping critique of many forensic methods that the police and prosecutors rely on, including fingerprinting, firearms identification and analysis of bite marks, blood spatter, hair and handwriting. The report says such analyses are often handled by poorly trained technicians who then exaggerate the accuracy of their methods in court…

Legal experts expect that the report will give ammunition to defense lawyers seeking to discredit forensic procedures and expert witnesses in court. Lawyers could also use the findings in their attempts to overturn convictions based on spurious evidence. Judges are likely to use the findings to raise the bar for admissibility of certain types of forensic evidence and to rein in exaggerated expert testimony.

Law enforcement organizations have tried to derail the report nearly every step of the way. The report’s critique of forensic evidence is much needed, but the proposed solution doesn’t sound promising:

It concludes that Congress should create a federal agency to guarantee the independence of the field, which has been dominated by law enforcement agencies, say forensic professionals, scholars and scientists who have seen review copies of the study.

I wouldn’t mind seeing an agency within the Department of Justice devoted to investigating and prosecuting cases of forensic fraud. If prosecutors are conspiring with or pressuring experts to deny criminal defendants a fair trial, that would be a due process violation and under the Fourteenth Amendment, the federal government would be permitted, or even obligated, to step in. Certainly Mississippi, for example, has neglected its duty to ensure that its citizens accused of violent crimes are given a fair trial.

But setting aside wilful and criminal misconduct by forensic experts, the problems with the forensics system aren’t going to be resolved by creating a new federal bureaucracy. Lack of federal oversight isn’t the problem. According to the New York Times article linked above, for example, the NAS report is particularly critical of the FBI crime lab, long considered the gold standard in forensics, and whose technicians often advised state crime labs on best practices.

The problem with criminal forensics is the government monopoly on courtroom science in criminal trials. In too many states, forensic evidence is sent only to state-owned or state-operated crime labs. There’s no competition, no peer review, and in some cases, crime lab workers either report to or can be pressured by prosecutors when test results don’t confirm preexisting theories about how a crime may have occurred. This sort of bias can creep in unintentionally, or it can be more overt.  But studies show it’s always there. The only way to diminish is to bring competitors into the game, other labs who gain by revealing another lab’s mistakes.

Every other area of science is steered by the peer review process. It’s really unconscionable that criminal forensics—where there’s so much at stake—has existed and evolved so long without it. The fact that so many people have been convicted solely based on pseudo-sciences like bite mark and hair and fiber analysis, for example, ought to scare the hell out of us.

Roger Koppl wrote an excellent paper for the Reason Foundation (pdf) that outlines the problems with criminal forensics, and offers a series of reforms for ensuring that the science jurors hear in the courtroom is actually science. Koppl and I also co-wrote a shorter piece in Slate making essentially the same points. Koppl’s proposals sound radical, but only because we’re so used to the current system. What he’s proposing is really little more than applying basic scientific principles like peer review, blind testing, and repetition to the evidence and opinions currently presented in criminal cases as actual science.

And we need to start treating criminal forensic science like actual science, with all the skepticism and repetitive testing that comes with it. What we don’t need is another layer of government bureaucracy that imposes a series of negotiated, compromised-for standards and practices, then fails to properly enforce them.

Digg it |  reddit | |  Fark

31 Responses to “Forthcoming NAS Study Expected to Shake Up Forensics World”

  1. #1 |  Michael Chaney | 

    As I’ve been saying, we simply need actual scientific method applied, along with real consequences for misconduct, and the problem goes away. It’s just far more involved (and hence, costly) than the simplistic methodologies employed now. It’s also much more difficult to lie or tamper.

  2. #2 |  Dave Krueger | 

    What you suggest makes sense to anyone who wants to fix the problem. But fixing problems isn’t what government does. Government only strives for the appearance of fixing problems. They go through the motions and proclaim success regardless of the results. Kind of like the DARE program philosophy applied to every single problem that passes before them.

    I am such a depressing pessimist that I believe the world will actually brighten by a measurable amount when I die. Of course, that will just add to global warming.

  3. #3 |  William | 

    I think a potentially bigger problem than the state monopoly on forensic science or forensic science avoiding peer review is the value we place on applied science in our society. We like the idea of science, we like facts, we like imagining that human knowledge has advanced to the point where we can make definitive statements about the Truth, but really we aren’t there in the majority of areas. Science is only rarely about facts and laws (especially when you get to application), instead theories dominate. The entire point of a theory isn’t to express knowledge but to ask “what if this were the case?” Any decent scientist knows that their theory is going to be disproved eventually, but they put it out there anyway so that a new theory can be developed when the flaws in their theory are discovered. That fluidity is wonderful in science, but it doesn’t belong in a courtroom.

  4. #4 |  Michael Pack | 

    I’d include B.A.C. machines also.iIt seems their not very accurate either,plus or minus 20%! Why not flip a coin? After seeing so many D.A ‘s claim they couldn’t prove D.U.I. with out a test in most cases,I’m led to belive they did nothing wrong and were never a danger.

  5. #5 |  Jim Collins | 

    Why not a seperate agency? DOJ should have nothing to do with this. DOJ is too biased in favor of the prosecution. I’d include B.A.C. machines and I would add patrol car camera systems. As a matter of fact any device or technique that is used to gather evidence sould come under the purview of this agency.

  6. #6 |  Michael Chaney | 

    William, you’re using the Discovery Institute definition of “theory”.

    Read more here.

  7. #7 |  Mike T | 

    I wouldn’t mind seeing an agency within the Department of Justice devoted to investigating and prosecuting cases of forensic fraud.

    The FBI is supposed to already have this in its civil rights division. Perhaps the NAS would do well to consider advocating a law that would turn the civil rights division into a separate federal agency that operates as a de facto Office of the Inspector General of Local and State Law Enforcement.

  8. #8 |  ktc2 | 

    Of course they’re going to object to it!

    If they can’t grab the easiest target with no means to defend themself and then rely on their bogus “experts” to paint them guilty . . . my god they’d have to . . . actually do real detective work!

  9. #9 |  EdinTally | 

    My only concern would be the cost involved in privatization. If both sides HAD to use private companies, who would be better able to afford this service? And how would pricing be implemented?

    Therein lies the problem.

  10. #10 |  Jim Collins | 

    Operative words are “supposed to”, MikeT.

  11. #11 |  Dave Krueger | 

    #5 Jim Collins

    Why not a seperate agency? DOJ should have nothing to do with this. DOJ is too biased in favor of the prosecution. I’d include B.A.C. machines and I would add patrol car camera systems. As a matter of fact any device or technique that is used to gather evidence sould come under the purview of this agency.

    That’s the best approach I’ve heard on this. Ever.

    In fact, it could even be placed under a different branch of government. Maybe make it a subsidiary of the GAO which loves to report stuff the government screws up.

  12. #12 |  EdinTally | 

    piggy backing off #10

    The issue is a lack of accountability. You can shuffle the deck an infinite amount of times, but if the dealer is coming off the bottom of the deck, what is the point of shuffling?

  13. #13 |  Ben (the other one) | 

    I’m a former federal prosecutor with a lot of faith in the folks at DOJ, but I agree that placing this function outside of DOJ makes sense– there’s just no other way to ensure independence. It might conceivably be placed within the Administrative Office of the U.S. Courts (basically the core agency of the federal judiciary), or it could be an independent agency entirely. (A similar argument can be made for the Office of the Pardon Attorney, but that’s way off-thread here.)

    Criminal justice is too important to entrust to superstition and pseudo-science.

  14. #14 |  ShelbyC | 

    Privatization, Compitition? Are you kidding? That helps consumers get what they want. And who is the consumer of forensics? The government.

  15. #15 |  David | 

    Law enforcement organizations have tried to derail the report nearly every step of the way.

    Of course, they step in and puff their chests out about any measure that might help people prove their innocence as “anti-cop”.

  16. #16 |  Cynical In CA | 

    This one’s so easy to call, even I can see it.


    On the off-chance I am wrong:


  17. #17 |  MacK | 


    “My only concern would be the cost involved in privatization. If both sides HAD to use private companies, who would be better able to afford this service? And how would pricing be implemented?

    Therein lies the problem.”

    I can see how this may look like a problem, but you may need to look into what it is presently costing first.

    I do not know any actual $$ amounts, but you can bet that a government ran agency is probably wasting bucket loads of money.

    Private organizations on the other hand can’t waste money, because that removes profits, and results have to be accurate, or another company will jump in to replace incompetence.

    As it stands right now only one side uses this service, and that is the government, defendants are barely able to afford an attorney, let alone specialized labs for rebuttals against the prosecution witnesses.

  18. #18 |  John Jenkins | 

    The report says such analyses are often handled by poorly trained technicians who then exaggerate the accuracy of their methods in court…

    I didn’t realize that this was seriously in question. When DNA techs say that the probability that a person other than the defendant left a certain DNA sample is greater than 1 in some number of trillion, based on abstract math that does not ACTUALLY calculate the probability (because some number of combinations simply don’t exist), you know they’re full of shit.

    Someone already mentioned BAC, which is also a joke, and anyone who doesn’t think that forensic experts will outright lie to placate the police is not thinking straight.

    @ #13 | Ben (the other one): I think that placing any agency like that in the Administrative Office of the Courts would present non-delegation problems. I think a cabinet level Office of the Inspector General would be the best way to handle this, at least at the federal level.

    They could consolidate the IG assets from the various departments, and have one outside watchdog that handles IG duties for the federal executive. One division, headed by a Deputy IG, would be solely responsible for USA/AUSA misconduct and trial audits, with investigative powers, administrative adjudicative authority (if the infraction merits only dismissal), and the authority to seek indictments and try cases (if an infraction merits prosecution). A USA would have to be referred for impeachment, I believe, but the AUSA’s could be removed administratively.

    Now, in theory, this is Congress’s oversight role, but given that Congress uses its investigative and oversight roles for grandstanding and making speeches, rather than actually looking for wrongdoing, it seems unlikely that Congress can be trusted to handle the heavy lifting.

    On the policy side, if Congress or the Courts were to extend the rule in Gideon to providing indigent defendants with a positive right to their own forensic experts, that would go a long way to correcting abuses.

  19. #19 |  EdinTally | 


    You fail to address my questions.

    1) Who would be better able to afford such a service? Government, of course.

    2) How would pricing be implemented? As much as the market could bare.

    3) See 1. Rinse and repeat.

    (of course private companies waste money and resources, but lets stick to this issue)

    I haven’t taken an Evidence class yet, but I believe both sides are entitled to evidence through discovery. How this actually works in practice, I’m not sure.

    A small problem I see with the free market argument is how it wants to fix everything as if it is capable of fixing everything. It is not. My initial point was that when something is broken, the best answer isn’t always to take it apart and allow the free market to fix it. Instead, why not just hold the institutions accountable (I can feel the free market come back as I write this). If accountability is off the table, neither government institutions or private companies work effectively.

    So just fix the root problem: Accountability

  20. #20 |  Lou Stone | 

    A client of mine:

  21. #21 |  chance | 

    I’m sorry, I just don’t understand how this is likely to shake up anything. Even an honest, open minded prosecutor would be reluctant to stop using these tools, or use them in ways that make conviction more difficult. And since I believe most prosecutors probably aren’t that open minded, I just don’t see positive change happening.

    Your coverage of the whole Hayes debacle is evidence of that. How many people were put away based on his testimony are still rotting in jail, even though he was discredited?

  22. #22 |  Ben (the other one) | 

    John (#18): Creating the scientific evidence entity in the judiciary strikes me as entirely consistent with the Constitution’s separation of powers. Compare, for example, Fed. R. Evid. 706(a), permitting courts to appoint their own experts.

    Also, creating an über-Inspector General office has its own problems. The IGs have a dual role, both investigating the quality of their own agencies’ work, and supporting prosecutions and civil cases against people who have ripped it off or otherwise impeded its function. Setting up an outside IG agency would tend to frustrate the latter role. Bush has also proved that there’s no agency that a corrupt president cannot bend to his political purposes, even an IG.

  23. #23 |  Helmut O' Hooligan | 

    Great post, Radley. I eagerly await this study. Forensic science, despite its problems, is one area of criminal justice that I feel quite hopeful about. I have continuing ed. in crime scene, so I have seen first hand the potential that forensic science has to move us toward a more enlightened system of justice. It can take us beyond hype, beyond blind guessing, and beyond the thirst for vengance. It can do all these things if it is properly applied and held to the standards of other physical sciences.

    I think a federal agency may indeed be necessary to ensure compliance. And, as Mike T (#7) suggested, we may need a federal entity (whether it’s civil rights or a separate agency) to act as an IG for law enforcement in general. Radley’s example of the State of Mississippi’s failures is a textbook example of a state violating it’s 14th Amendment obligations. While I am much less likely to call for federal intervention than I used to be, I think such cases scream out for federal action.

    On the state level, I believe that governments should consider removing forensic labs from police agencies and setting up separate forensic services agencies. These agencies could offer lab and field investigative services to state and local police, fire departments, etc.. Everyone from forensic biologists, to CSI’s to the Medical Examiner could work out of these agencies. Large local governments might be able to do the same thing (ie. Indianapolis/Marion County has a separate forensic science agency, last I checked). Forensic scientists and investigators need access to crime scenes and police records, and must work well with the police, but it doesn’t necessarily follow that everyone has to work for the same agency. Police departments have clearly become to prosectution-oriented, so the time for independence is now.

  24. #24 |  EdinTally | 

    Ben (the other) “…Bush has also proved that there’s no agency that a corrupt president cannot bend to his political purposes, even an IG.”

    Agreed. If there is no accountability at the top, accountability down the chain is arbitrary and meaningless.

  25. #25 |  Remfin | 

    Uhm, I see a huge hole in your argument. While government monopolies on forensics may be a big deal in some states, we have plenty of states where the government does not have a forensics monopoly, and they were disasters.

    Does Dr. Steven Hayne not ring a bell for some reason? Everything he did to screw up Mississippi was as a private “lab”, not as the state’s law-designated destination for all forensics. Prosecutors were free to use other (better) people, defendants could contract their own (or him, if they beat prosecutors to the punch), and other expert witnesses could be used to challenge him. It didn’t seem to help much, did it?

    Unless you know something else, it also seems your complaints about the pseudo-science of certain evidence is based off the ridiculous lengths people like Hayne took it to, instead of just being the weak circumstantial evidence it is. That would once again be an issue with private forensics, not government forensics.

    I certainly think there are things that need to be changed. The government certainly can’t do it alone, but it also seems quite clear to me the free-market has failed to solve the problem on its own either in spite of having so many years to try.

  26. #26 |  Highway | 

    Remfin, I doubt that Hayne can be used as an argument that there wasn’t a state monopoly on his work. There was plenty of monopoly pressure: from under- and defunding the state Medical Examiner’s office, to resisting all efforts of reform by that office when it was around, to shady practices that either drove out of business, or highly discouraged businesses starting up to compete with Hayne.

    When the government will only pay a single person to do something, it doesn’t matter if they’re technically a government employee or not, they’ve got a government monopoly.

  27. #27 |  AR | 

    Why can’t this problem be handled by defense attorneys? Can they call their own expert witnesses to undermine the state’s evidence (especially if it has such poor scientific footing)?

  28. #28 |  EdinTally | 

    But his point stands;

    The free market could allow the government to “shop” for evidence. Who wouldn’t want the government as a client?

    Accountability: preach it from on high

  29. #29 |  Scott | 

    In a culture that cannot get enough television courtroom dramas on the programming schedule, I’m not sure that increased scientific rigor applied to evidence will have much of an affect on jurors convinced of their keen insight due to repeated viewings of “Law and Order” and CSI.

    A wily prosecutor only has to blow sufficient smoke up a jury’s ass. Let’s hope that this study has some sort of affect on judges and gives them reason to apply much higher standards to evidence admitted at trial.

  30. #30 |  T. Reed | 

    I just want a system that gets tested. People do what you INspect, not what you EXpect.

    If you are a “certified” lab, part of your membership dues go to an independent office–the only job of the independent office is to send known samples to certified labs for blind testing. Does hair A match hair B? Did fiber A come from shirt B?

  31. #31 |  William | 

    @ Michael Pack/#4: Actually, I’m not using the discovery institute definition of theory. Go back and reread. Nowhere did I say that all theories were equal or that any assertion ought to have the same weight. Theories are a way of explaining observed data under a current scientific paradigm. Over time more evidence is observed and theories are either updated or thrown away. The end result isn’t that all theories have equal merit but that all theory is open to challenge and ultimately temporary because something that better explains the observed data WILL come along. Even Newton’s theories eventually fell as we began to be able to observe and understand both very large and very small phenomena. The great thing about science is that there will always be more data, there will always be more observations, and while old theories will continue to be updated or discarded, the process will never actually end. There is always something new to learn. Thomas Kuhn wrote a great book about this very idea called “The Structure of Scientific Revolution.”

    The problem with science in the justice system is that we aren’t generally dealing with scientists but with people who are selling something. Maybe they’re selling a BAC machine, maybe they’re selling their analysis of bite marks, but at the end of the day they aren’t going into a courtroom to add to the body of human knowledge but to get a paycheck. As a result they have an incentive to fudge the data, to push theories that aren’t really robust enough to be turned into technology or technique, and to avoid actually updating and improving theory because invalidating a theory would mean invalidating their past work. They are able to get away with it because, as a society, our eyes tend to glaze over when people start talking about science. We treat science with the same reverence and awe that our ancestors gave to faith and superstition. We let people like Dr. Hayne get away with incredible lies because they dress them up with a couple of poorly designed experiments performed while wearing a white coat in something they labeled a lab. As a result we let phlogiston and phrenology into our courtrooms.