Comments on: Odds n’ DNA Databases http://www.theagitator.com/2008/05/04/odds-n-dna-databases/ It rankles me when somebody tries to tell somebody what to do. Sat, 17 May 2008 01:09:07 +0000 http://wordpress.org/?v=2.5.1 By: Lloyd Flack http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-107028 Lloyd Flack Fri, 09 May 2008 11:10:24 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-107028 Xrlq, In principle both your ideas are good. It would however be more difficult to calculate these probabilities than you might think. The problem is mostly inhomogeneities in the data. In the first approach different groups of criminals might have different probabilities of being in the data base. Still the group of interest might be homogeneous enough for this to work. However we have no way of knowing the size of the group that never gets caught. In the second case the probability of a false positive will vary from case to case. This could make the calculations messy. Still it's worth trying. Xrlq,
In principle both your ideas are good. It would however be more difficult to calculate these probabilities than you might think. The problem is mostly inhomogeneities in the data.

In the first approach different groups of criminals might have different probabilities of being in the data base. Still the group of interest might be homogeneous enough for this to work. However we have no way of knowing the size of the group that never gets caught.

In the second case the probability of a false positive will vary from case to case. This could make the calculations messy.

Still it’s worth trying.

]]> By: Xrlq http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-106832 Xrlq Fri, 09 May 2008 05:27:57 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-106832 I just had a long conversation with Patterico on the topic, and it occurred to me that we should be able to compute a reasonable estimate of the odds of the killer being in the database, in one of two ways: 1. Compute the percentage of rapist-murderers who have prior convictions for sex offenses that would be likely to land them in the sex offender database. 2. Compute the percentage of cold database hits that result in 2 or more hits. Assuming a 1 in 3 chance of one false positive, a 1 in 9 chance of two, and so on, we should be able to compare the expected number of false hits to the number of total hits, and extrapolate that the difference represents the number of true hits. Once we know the relative odds that a true hit is in the database, we should be able to say with a good deal of confidence how reliable a single cold hit is. I just had a long conversation with Patterico on the topic, and it occurred to me that we should be able to compute a reasonable estimate of the odds of the killer being in the database, in one of two ways:

1. Compute the percentage of rapist-murderers who have prior convictions for sex offenses that would be likely to land them in the sex offender database.
2. Compute the percentage of cold database hits that result in 2 or more hits. Assuming a 1 in 3 chance of one false positive, a 1 in 9 chance of two, and so on, we should be able to compare the expected number of false hits to the number of total hits, and extrapolate that the difference represents the number of true hits.

Once we know the relative odds that a true hit is in the database, we should be able to say with a good deal of confidence how reliable a single cold hit is.

]]> By: Pablo http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-106783 Pablo Fri, 09 May 2008 03:45:59 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-106783 Yes, we have no way to compute the probability that the true assailant is in the offender database. So we have to make one of two mutually exclusive assumptions: (1) the true assailant is NOT in the offender database, which then predicts that there is approximately a 1 in 3 chance the evidence DNA profile will match someone who is innocent. Alternatively, (2) the true assailant is in the offender database. Under this second assumption there is a 100% certainty that his profile will be found to match the evidence. Since the database represents a relatively small sample of the general population it seems to me that assumption (1) is more likely than assumption (2). The reason for the 1 in 3 chance is the result of the limited partial evidence DNA profile, there is simply not enough genetic information available to achieve maximum discrimination. If there was a full evidence profile, then the chance could be 1 in trillions that the evidence profile will match someone who is innocent. That probability would be sufficient to believe the defendant is guilty. Whereas a probability of 1/3 that the evidence profile could originate from an innocent falsely accused leaves too much doubt that the defendant is guilty. Without any further evidence I would vote not guilty. Yes, we have no way to compute the probability that the true assailant is in the offender database. So we have to make one of two mutually exclusive assumptions: (1) the true assailant is NOT in the offender database, which then predicts that there is approximately a 1 in 3 chance the evidence DNA profile will match someone who is innocent. Alternatively, (2) the true assailant is in the offender database. Under this second assumption there is a 100% certainty that his profile will be found to match the evidence.
Since the database represents a relatively small sample of the general population it seems to me that assumption (1) is more likely than assumption (2). The reason for the 1 in 3 chance is the result of the limited partial evidence DNA profile, there is simply not enough genetic information available to achieve maximum discrimination. If there was a full evidence profile, then the chance could be 1 in trillions that the evidence profile will match someone who is innocent. That probability would be sufficient to believe the defendant is guilty. Whereas a probability of 1/3 that the evidence profile could originate from an innocent falsely accused leaves too much doubt that the defendant is guilty. Without any further evidence I would vote not guilty.

]]> By: Xrlq http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-106374 Xrlq Thu, 08 May 2008 16:06:36 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-106374 <blockquote>It’s also unfortunate that the judge in the case profiled in the L.A. Times would only allow the prosecution’s miscalculated 1 in 1.1 million chance of a false match into evidence, and not the more statistically sound 1 in 3. </blockquote> I don't think it's an either-or. One measures the odds that <i>someone</i> will be falsely identified, while the other measures the odds that any particular individual will. Which one is relevant depends on other factors. If the only reason to suspect Puckett is the partial DNA match, and we can only assume that the odds are roughly 50-50 that the true killer was in the database to begin with, then there's about a 1 in 3 chance that they fingered the wrong guy. But if there's enough other evidence out there to make him the chief suspect, as the story indicates, then it's fair to say that the odds of the partial DNA match falsely pointing to the <i>same</i> individual are a million to one.

It’s also unfortunate that the judge in the case profiled in the L.A. Times would only allow the prosecution’s miscalculated 1 in 1.1 million chance of a false match into evidence, and not the more statistically sound 1 in 3.

I don’t think it’s an either-or. One measures the odds that someone will be falsely identified, while the other measures the odds that any particular individual will. Which one is relevant depends on other factors. If the only reason to suspect Puckett is the partial DNA match, and we can only assume that the odds are roughly 50-50 that the true killer was in the database to begin with, then there’s about a 1 in 3 chance that they fingered the wrong guy. But if there’s enough other evidence out there to make him the chief suspect, as the story indicates, then it’s fair to say that the odds of the partial DNA match falsely pointing to the same individual are a million to one.

]]> By: Lloyd Flack http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104808 Lloyd Flack Wed, 07 May 2008 03:22:00 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104808 You cannot give meaningful estimates of the probability that the match is the culprit without giving an estimate of the probability of the culprit being in the data base. You can reliably place a floor under this probability, however this floor could and usually will be fairly low. At least a floor that you could defend in court is likely to be. You cannot give meaningful estimates of the probability that the match is the culprit without giving an estimate of the probability of the culprit being in the data base. You can reliably place a floor under this probability, however this floor could and usually will be fairly low. At least a floor that you could defend in court is likely to be.

]]> By: Lloyd Flack http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104723 Lloyd Flack Wed, 07 May 2008 01:59:23 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104723 You would have to use the cautious estimates in this case. You would have to use n/N. The problem with an old case is that N , the population that potentially could be in the database will be large because people could have moved anywhere in the country. You would have to rely on the other evidence that turned up in the investigation. These cold data base hits might only give probable cause. You have to be aware that this might be a false lead. You would have to use the cautious estimates in this case. You would have to use n/N. The problem with an old case is that N , the population that potentially could be in the database will be large because people could have moved anywhere in the country. You would have to rely on the other evidence that turned up in the investigation. These cold data base hits might only give probable cause. You have to be aware that this might be a false lead.

]]> By: Patterico http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104708 Patterico Wed, 07 May 2008 01:36:36 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104708 I don't know. I'm not sure it's possible to tell juries anything meaningful about the "probability" that the true donor is really in the database. I don’t know. I’m not sure it’s possible to tell juries anything meaningful about the “probability” that the true donor is really in the database.

]]> By: Lloyd Flack http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104690 Lloyd Flack Wed, 07 May 2008 01:15:14 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104690 If you have only one match in the database what is the probability that it is the guilty person? Let pf be the the probability of there being one false match in data base (in the case above this will be about 22%). Let pg be the probability that the guilty person is in the data base. This is the hard one to estimate. If people were randomly placed in the data base then pg=n/N where n is the data base size and N is the population size. Since the data base is largely composed of previous offenders pg will actually be somewhat larger than this. But how much larger? The probability that the match is the guilty person is pg x (1-pf) / (pg x (1-pf) + (1 - pg) x pf). Now let us assume pf = 22%. If pg is 50% then the probability that we have the guilty person is 78%. If pg is 10% the the probability of guilt is 28%. If pg is 1% then the probability that we have the guilty person is 3%. Now you are not supposed to use the criminal proclivities of a person as evidence so by the rules of evidence you should use pg =n/N. If you disagree with this so do I. However the point is that even with realistically high probilities of the data base including the culprit the DNA evidence might still be far from conclusive. If you have only one match in the database what is the probability that it is the guilty person? Let pf be the the probability of there being one false match in data base (in the case above this will be about 22%). Let pg be the probability that the guilty person is in the data base. This is the hard one to estimate. If people were randomly placed in the data base then pg=n/N where n is the data base size and N is the population size. Since the data base is largely composed of previous offenders pg will actually be somewhat larger than this. But how much larger?

The probability that the match is the guilty person is pg x (1-pf) / (pg x (1-pf) + (1 - pg) x pf). Now let us assume pf = 22%. If pg is 50% then the probability that we have the guilty person is 78%. If pg is 10% the the probability of guilt is 28%. If pg is 1% then the probability that we have the guilty person is 3%.

Now you are not supposed to use the criminal proclivities of a person as evidence so by the rules of evidence you should use pg =n/N. If you disagree with this so do I. However the point is that even with realistically high probilities of the data base including the culprit the DNA evidence might still be far from conclusive.

]]> By: Patterico http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104685 Patterico Wed, 07 May 2008 01:06:41 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104685 In which case it's 100% that he's not the donor. In which case it’s 100% that he’s not the donor.

]]> By: Patterico http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104684 Patterico Wed, 07 May 2008 01:06:06 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104684 Although certainly, not knowing whether the guilty person is there or not, a jury that heard there had been a database hit could be told this: "even in a situation where we assume that everyone in the database is innocent, there's a 1/3 chance of a hit, which would be a false positive given the assumption." But in a case with only one hit, you can't tell the jury there's a 1/3 chance of a *false* positive because you're thereby telling them there's a 1/3 chance the guy sitting in front of them is not the donor -- and you can't know that unless you assume up front that the donor wasn't in the database. Although certainly, not knowing whether the guilty person is there or not, a jury that heard there had been a database hit could be told this: “even in a situation where we assume that everyone in the database is innocent, there’s a 1/3 chance of a hit, which would be a false positive given the assumption.”

But in a case with only one hit, you can’t tell the jury there’s a 1/3 chance of a *false* positive because you’re thereby telling them there’s a 1/3 chance the guy sitting in front of them is not the donor — and you can’t know that unless you assume up front that the donor wasn’t in the database.

]]> By: Patterico http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104664 Patterico Wed, 07 May 2008 00:43:31 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104664 OK, I get what you are saying. As you folks have worded your comments, I believe you are correct. But the issue relevant to the LAT article is: what is the significance of a *single* hit in a database? And the answer is: you can't talk about the probability of a false positive unless you *assume* that the guilty person is *not* in the database. Are we agreed on that? OK, I get what you are saying. As you folks have worded your comments, I believe you are correct. But the issue relevant to the LAT article is: what is the significance of a *single* hit in a database? And the answer is: you can’t talk about the probability of a false positive unless you *assume* that the guilty person is *not* in the database.

Are we agreed on that?

]]> By: Lloyd Flack http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104532 Lloyd Flack Tue, 06 May 2008 22:00:51 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104532 No we are not. Aaron and I are both saying the same thing. If you add the guilty person to the data base it does not affect the chance of getting a false positive as well. In the the example above the expected number of matches is 1.3 if the guilty person is in the data base and it is 0.3 if the guilty person is not in the data base. No we are not. Aaron and I are both saying the same thing. If you add the guilty person to the data base it does not affect the chance of getting a false positive as well. In the the example above the expected number of matches is 1.3 if the guilty person is in the data base and it is 0.3 if the guilty person is not in the data base.

]]> By: Aaron http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104506 Aaron Tue, 06 May 2008 21:51:21 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104506 My analysis shows that whether the guilty person is in the database or not, the chances of an innocent person being in the database is roughly the same, by analyzing each of these cases separately. My analysis shows that whether the guilty person is in the database or not, the chances of an innocent person being in the database is roughly the same, by analyzing each of these cases separately.

]]> By: Patterico http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104423 Patterico Tue, 06 May 2008 20:36:12 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104423 You're still assuming you know who the guilty person is, and the whole point is that in real life we don't know that. So when we get a result we don't know the chances that *that* result is a false positive -- and *that's* the question juries are interested in. You’re still assuming you know who the guilty person is, and the whole point is that in real life we don’t know that. So when we get a result we don’t know the chances that *that* result is a false positive — and *that’s* the question juries are interested in.

]]> By: Patterico http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104417 Patterico Tue, 06 May 2008 20:33:05 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104417 "If you know the guilty person is in the database, you can exclude him from it, and calculate the probability that an innocent person is in the other (N-1)." How is that different from me saying that you have to start with a database of known innocent people to know the chances of a *false* positive? Your method gets you to the same place, by removing the guilty person. “If you know the guilty person is in the database, you can exclude him from it, and calculate the probability that an innocent person is in the other (N-1).”

How is that different from me saying that you have to start with a database of known innocent people to know the chances of a *false* positive? Your method gets you to the same place, by removing the guilty person.

]]> By: Aaron http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104416 Aaron Tue, 06 May 2008 20:31:55 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104416 Yes, and I can quantify how close: to within one part in 330,000. "He's six feet tall" "Really" "Well, actually, six feet and 0.0002 inches". The difference is smaller than the margin of error on either calculation from not knowing the exact number of people in the database. Yes, and I can quantify how close: to within one part in 330,000.

“He’s six feet tall”

“Really”

“Well, actually, six feet and 0.0002 inches”.

The difference is smaller than the margin of error on either calculation from not knowing the exact number of people in the database.

]]> By: Patterico http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104368 Patterico Tue, 06 May 2008 19:17:25 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104368 I take it that when you say it's a simplification, you actually mean "*not* really, but close." I take it that when you say it’s a simplification, you actually mean “*not* really, but close.”

]]> By: Aaron http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104098 Aaron Tue, 06 May 2008 16:12:20 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-104098 <blockquote>Having the guilty person in the data base does not influence the chance of having a false positive there as well.</blockquote> It's a slight simplification, but really. If you know the guilty person is in the database, you can exclude him from it, and calculate the probability that an innocent person is in the other (N-1). For large N (say, greater than 100), this is virtually the same as the original chance.

Having the guilty person in the data base does not influence the chance of having a false positive there as well.

It’s a slight simplification, but really. If you know the guilty person is in the database, you can exclude him from it, and calculate the probability that an innocent person is in the other (N-1). For large N (say, greater than 100), this is virtually the same as the original chance.

]]> By: Patterico http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-103890 Patterico Tue, 06 May 2008 14:07:51 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-103890 <i>Having the guilty person in the data base does not influence the chance of having a false positive there as well.</i> Really? Having the guilty person in the data base does not influence the chance of having a false positive there as well.

Really?

]]> By: Lloyd Flack http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-103171 Lloyd Flack Tue, 06 May 2008 06:33:28 +0000 http://www.theagitator.com/2008/05/04/odds-n-dna-databases/#comment-103171 But whether the guilty person is in the data base is precisely what you do not know. Having the guilty person in the data base does not influence the chance of having a false positive there as well. You have to consider the population outside the data base as well. If there is one hit in the data base and an estimated 9 matches in the population outside the data base then the chance that the match repesents the guilty person is 1 in 10. But whether the guilty person is in the data base is precisely what you do not know.

Having the guilty person in the data base does not influence the chance of having a false positive there as well.

You have to consider the population outside the data base as well. If there is one hit in the data base and an estimated 9 matches in the population outside the data base then the chance that the match repesents the guilty person is 1 in 10.

]]>