How Accurate are Probabilistic Odds Claims in Criminal Trials? A “Warranted Skepticism” Approach

Abstract

Probabilistic odds claims used in criminal trials are often inaccurate due to subjectivity within the methods of forensic science. The potential sources of subjectivity are wide since forensic science is an adversarial process and a full disclosure of assumptions is not required under Brady. The prior literature has focused on the limits of Bayesian methods and the potential uniqueness of DNA fingerprints to each person. This paper is unique because it focuses on other sources of subjectivity, such as a lack of disclosure of test results, repeated trials, a presumption of laboratory accuracy, absence of tests, and so on. The infamous Gilyard case is helpful as illustration of one common source of subjectivity. The standard approach for DNA analysis was applied to first determine a “match” to Gilyard’s DNA to a test sample, and then to estimate the relative frequency of Gilyard’s DNA in a reference population at odds of 1-in-18-quadrillion. The odds figure is roughly a million times the number of persons now living or that will ever live and amounts to a strong claim that Gilyard's DNA profile is unique. The "reference population" used to derive the odds claim is hypothetical (or, largely non-existential), just as Karl Popper warned. However, in every criminal case with DNA evidence, there are at least 2 DNA samples that appear to match (in Gilyard, there were 7) not included in the reference population. A significant question is whether the reference population should include the samples at issue in the case. Where the composition of the reference population is merely hypothetical it presumably should be updated to reflect any new evidence as a matter of Bayesian science. Of course, the population is extrapolated from a small dataset, so the existence of 2 or more matching DNA profiles might reduce the odds to a figure perhaps in the thousands, rather than quadrillions. This raises the severe problem that the probabilistic odds calculation depends on the subjective determination of a “match” in the first step. One scholar has suggested that "match" claims are objective because the process has been partly mechanized, even though the interpretation of results has not been standardized. In modern science, however, objectivity refers to replicability by experiment including the interpretation of results. This paper develops many other additional sources of subjectivity in forensic science and suggests: If subjectivity exists to degree x, then any related or resulting probabilistic odds claim may not exceed x. As example, if laboratory error is possible at a rate of 1/10,000, then given probabilistic odds should not exceed 1/10,000. The inherent subjectivity of all probabilistic odds claims, as identified by Frank Ramsey, is also explored. The conclusion is that a “warranted skepticism” approach to the use of remote odds claims in criminal cases, such as in Daubert, remains appropriate.