EFTA00100193 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 Co-Chairs David Patton Executive Director Federal Defenders of New York Jon Sands Federal Defender District of Arizona September 13, 2019 David B. Muhlhausen, Ph.D. Director National Institute of Justice Office of Justice Programs Department of Justice 810 7th Street NW Washington, DC 20531 Re: DOJ First Step Act Listening Session on PATTERN ## Dear Dr. Muhlhausen: Thank you for inviting comment from the Federal Public and Community Defenders regarding the Department of Justice's (DOJ) development of the Prisoner Assessment Tool Targeting Estimated Risk and Needs (PATTERN) as part of its obligations under the First Step Act (FSA). The Federal Public and Community Defenders represent the vast majority of defendants in 91 of the 94 federal judicial districts nationwide, and we welcome the opportunity to provide our views. PATTERN will directly affect how much time many of our clients spend in prison. This makes it a high-stakes tool, and means testing for accuracy and bias is crucial. Indeed, Congress understood the stakes and called for transparency throughout the FSA, including a mandate that the risk and needs assessment system be “developed and released publicly.” $ ^{1} $ Congress also repeatedly required that the system be monitored for bias. $ ^{2} $ The limited information released by the DOJ in its July 19, 2019 1 First Step Act of 2018 (FSA), Pub. L. 115-391, Title I, § 101(a) (Dec. 21, 2018) (codified at 18 U.S.C. § 3632(a)). $ ^{2} $ See, e.g., FSA at Title I, §103 (requiring the Comptroller General to conduct an audit of the use of the risk and needs assessment system every two years, which must include an analysis of “[t]he rates of recidivism among similarly classified prisoners to identify any unwarranted disparities, including disparities among similarly classified prisoners of different demographic groups, in such rates.”); FSA at Title I, §107(g) (requiring the Independent Review Committee to submit to Congress a report addressing the demographic percentages of inmates ineligible to receive and apply time credits, including by age, race, and sex); FSA at Title VI, §610(a)(26) (requiring the Director of the Bureau of Justice Statistics to annually submit to Congress statistics on “[t]he breakdown of EFTA00100194 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 report (DOJ Report) confirms the need to assess PATTERN for accuracy and bias. For example, reported data indicates PATTTERN will have a racially disparate impact, particularly on black males. As illustrated in the charts below, based on the DOJ Report, white males are far more likely than black males to fall in the minimum and low risk categories. $ ^{3} $ Racial Disparities in Eligibility for Full Earned Release Incentives White Males Black Males Minimum/Low Medium/High < Minimum/Low * Medium/High* This matters because these are the categories that are eligible for higher rates of earned time credits and eligibility for supervised release and prerelease custody. $ ^{4} $ The DOJ Report fails to provide the level of transparency required for meaningful evaluation of PATTERN. Below, we detail much of the additional information needed to fully assess PATTERN for accuracy and bias. We look forward to providing additional thoughts after the DOJ has released this information and hope our comment here is only the beginning of an ongoing dialogue with the DOJ regarding PATTERN. ## I. RISK ASSESSMENT PATTERN is a risk assessment tool “designed to predict the likelihood of general and violent recidivism for all BOP inmates.”⁵ It places “individuals into four categories: high, medium, low or prisoners classified at each risk level by demographic characteristics, including age, sex, race, and the length of the sentence imposed."). 3 See U.S. Dep't of Just., The First Step Act of 2018: Risk and Needs Assessment System 62, tbl. 8 (2019) (DOJ Report) (reporting 57% of white males in the developmental sample fall in the minimum and low risk categories while only 27% of black males fall in those same categories). 4 See FSA at Title I $ \§101(a) $ (codified at 18 U.S.C. $ \§3632(d)(4)(A) $ , providing more earned time credits for some individuals in the lowest two risk categories); Title I $ \§102(b)(1)(B) $ (codified at 18 U.S.C. $ \§3624(g)(1) $ , restricting eligibility to transfer to supervised release or prerelease custody to individuals in the minimum or low risk categories, absent warden approval under specified circumstances). $$\textcircled{5}$$ DOJ Report at 43. 2 EFTA00100195 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 minimum." $ ^{6} $ These risk categories determine the number of credits an individual may earn by participating in programs and productive activities, and also eligibility to attribute those credits toward supervised release or prerelease custody. $ ^{7} $ In other words, the risk categories will directly affect how much time many individuals spend in prison. The development of PATTERN, as with all risk assessment tools, necessarily relies on both empirical research and moral choices. $ ^{8} $ Based on the DOJ Report, we have concerns, but even more questions, in both areas. Additional information is needed to assess many important issues including: PATTERN's accuracy; its scoring mechanisms; its fairness across age, gender, race and ethnicity; how much it will exacerbate racial disparity in the federal prison population; its impact on privacy interests; and whether it is consistent with the congressional mandate to “ensure” that “all prisoners at each risk level have a meaningful opportunity to reduce their classification during the period of incarceration." A. Transparency & Accountability: Development, Validation and Bias Testing Transparency in the methods for developing, validating and bias testing PATTERN is vital. Full transparency is a primary way (along with accountability and auditability) to create and justify confidence by stakeholders and the public. Indeed, across risk assessments in criminal justice, the secrecy that permeates black box instruments causes significant concerns about how reasonable they are in practice. ## 1. Dataset Full transparency requires DOJ to release the same dataset used by Grant Duwe, Ph.D., and Zachary Hamilton, Ph.D., to create PATTERN. $ ^{10} $ This is consistent not only with the transparency directives in the FSA, $ ^{11} $ but also with the advice of leading organizations such as the National Center for State Courts which recommends that independent evaluators determine whether their independent “research findings support or contradict conclusions drawn by the instrument developers.” $ ^{12} $ 6 DOJ Report at 50. 7 See supra note 4. 8 See Michael Tonry, Legal and Ethical Issues in the Prediction of Recidivism, 26 FED. SENT’G REP. 167, 167 (2014). $$\textsuperscript{9} \text{FSA at Title I } \§ 101(\mathrm{a}) \text{ (codified at 18 U.S.C. } \§ 3632(\mathrm{a})(5)(\mathrm{A})).$$ 10 See DOJ Report at 42-43. 11 See supra notes 1 & 2. 12 Pamela M. Casey et al., National Center for State Courts, Offender Risk & Needs Assessment Instruments: A Primer for Courts 19 (2014) (stressing that third party audits are valued because “it is always helpful to know whether existing research descriptions about the reliability, validity, and fairness of a tool have been replicated by others.” Any “decisions based on a [risk and needs] tool which grossly 3 EFTA00100196 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 - Access to the full dataset would permit independent researchers to assess validity and algorithmic fairness using a variety of measures and calculations. $ ^{13} $ - Despite recognizing the existence of multiple measures and calculations concerning validity, $ ^{14} $ the DOJ Report focused mostly on the Area Under the Curve (AUC). The AUC, however, has limited utility as a measure of relative risk. $ ^{15} $ Further, when tools are assessed using multiple measures of predictive validity (e.g., correlations, calibration metrics, Somers' D), results for the same tools vary. $ ^{16} $ - Access to the dataset would allow interested parties to complete 2 x 2 contingency tables (number of false negatives, false positives, true negatives, true positives) for general and violent recidivism at each cutoff (minimum to low; low to medium; medium to high) by age, gender and race/ethnicity groupings. These contingency tables would provide important information on the degree to which the categorizations created by the cut-points capture true positives and true negatives (in addition to the associated recidivism rates that the DOJ Report included). $ ^{17} $ - The dataset would allow independent researchers to compute the algorithmic fairness measures called balance for the positive and negative classes by calculating average scores by recidivists versus non-recidivists across each age, gender, and racial/ethnic groupings. misclassifies the risk levels of offenders may not simply fail to improve outcomes; they may actually do harm to the offender." As a result, " [i]nstrument validation is not only important to ensure that decision making is informed by data, but to establish stakeholder confidence."); see also Nathan James, CONG. RESEARCH SERV., Risk and Needs Assessment in the Federal Prison System 11 (July 10, 2018) (Congressional Research Service report concerning risk assessment in the federal prison system positively citing the recommendation of the Council of State Governments that independent third parties should be permitted to validate the tool to assess accuracy by race and gender). 13 For example, release of the full dataset would allow independent researchers to calculate relevant measures such as false positive rates, false negative rates, positive predictive value, negative predictive value, equal calibration, balance for the positive class, balance for the negative class, diagnostic odds ratios, correlations, treatment equality, and demographic parity. The importance of these various measures are discussed and calculated regarding other risk tools in sources cited in the DOJ Report. See DOJ Report at 38-39 nn.20-24. 14 See DOJ Report at 28 (discussing multiple algorithmic measures of racial bias). 15 See Melissa Hamilton, Debating Algorithmic Fairness, 52 UC DAVIS L. REV. ONLINE 261 (2019); Jay P. Singh, Predictive Validity Performance Indicators in Violent Risk Assessment, 31 BEHAV. SCI. & L. 8, 16-18 (2013). 16 See generally Sarah L. Desmarais et al., Performance of Recidivism Risk Assessment Instruments in U.S. Correctional Settings, 13 PSYCHOL. SCI. 206 (2016). 17 See Richard Berk et al., Fairness in Criminal Justice Settings: The State of the Art, SOC. METHODS & RES. (forthcoming 2019). 4 EFTA00100197 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 - Access to the dataset would allow interested parties to complete the bivariate correlations between predictors and risk outcomes which the DOJ Report indicates were completed by the developers, but are not reported. $ ^{18} $ - Access to the dataset would permit independent researchers to test for bias, including comparing each racial/ethnic grouping. As discussed above, the DOJ Report indicates the need for additional inquiry regarding racial disparity and other biases. $ ^{19} $ First, DOJ data show that black males are far less likely than white males to fall into the two lower risk categories that receive the full benefits of earned time credit and eligibility to use those credits for supervised release or prerelease custody. $ ^{20} $ In addition, the relative rate index (RRI) of 1.54 reported in Table 8, but not discussed in the text, comparing white to non-white males, also shows PATTERN has a racially disparate impact. $ ^{21} $ More information is needed, including data on Native-Americans and Asians, which is not included in the DOJ Report. $ ^{22} $ Access to the data would allow independent researchers to isolate individual factors and determine which contributed to any disparate impact. For example, research on the Post-Conviction Risk Assessment (PCRA) found that “Black offenders tend to obtain higher scores on the PCRA than do White offenders” and that “most (66 percent) of the racial difference in the PCRA scores is attributable to criminal history.” $ ^{23} $ Because PATTERN plays a role in determining how much time a person spends in prison, a similar finding of racial difference with PATTERN could "exacerbate racial disparities in prison." $ ^{24} $ Identifying 18 See DOJ Report at 65 n.17. 19 See supra note 3 and accompanying text. $$^{20} \text{ See id.}$$ 21 See DOJ Report at 62, tbl. 8 22 See William Feyerherm et al., Identification and Monitoring in Dept. of Just. Office of Juvenile Justice and Delinquency Prevention, Disproportionate Minority Contact Technical Assistance Manual, 1-1, 1-2, 3 (4th ed. 2009) (recommending the RRI be calculated separately for each minority group that comprises at least 1% of the total population scored); BOP Statistics: Inmate Race, Federal Bureau of Prisons, https://www.bop.gov/about/statistics/statistics_inmate_race.jsp. 23 Jennifer L. Skeem & Christopher T. Lowenkamp, Risk, Race, and Recidivism: Predictive Bias and Disparate Impact, 54 CRIMINOLOGY 680, 700 (2016). 24 Id. at 705; see also id. at 703, 705 (explaining that as assessment of whether a tool produces “inequitable consequences” depends on “what decision they inform” and that “some applications of instruments might exacerbate racial disparities in incarceration”). 5 EFTA00100198 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 which factors generate the disparate impact would open an opportunity to brainstorm with people across disciplines about how to ameliorate such impact. $ ^{25} $ - Access to the dataset would allow independent researchers to evaluate test bias employing the hierarchical modeling method considered best practice in the educational testing literature as referred to, but not reported in, the DOJ Report. $ ^{26} $ - Access to the dataset would allow interested parties to determine whether there are mistakes in the DOJ Report regarding the recidivism rates by ordinal ranking. Table 5 reports general recidivism rates of 9% (minimum), 31% (low), 51% (medium), and 73% (high). Table 9 reports identical recidivism rates in each of these categories for white males, $ ^{27} $ which might either be coincidental or a mistake in reporting. - Similarly, access to the dataset would allow independent researchers to determine the correct AUC for violent recidivism as defined by the developers. The DOJ Report is inconsistent, reporting in one table the AUCs for violent recidivism as .78 for males and .77 for females. $ ^{28} $ In another table, they are reversed, indicating AUCs of .77 for males and .78 for females. $ ^{29} $ These differences are not significant in terms of numbers, but flaws such as these (reasonable considering the tight time frame which the PATTERN team faced) call for independent audits to check for other potential errors. ## 2. Eligibility Additional information is needed regarding the assumptions behind the assertion that “99% of offenders have the ability to become eligible for early release through the accumulation of earned time credits even though they may not be eligible immediately upon admission to prison. That is . . . nearly all have the ability to reduce their risk score to the low category.” $ ^{30} $ Without more information it is impossible to test this assertion, but it appears suspect in light of: the percentage of the developmental sample that fell in the medium and high categories (52% of all and 58% of men); $ ^{31} $ that high scores are likely driven by static factors such as age of first conviction and criminal history 25 See Richard Berk, Accuracy and Fairness for Juvenile Justice Risks Assessments, 16 J. EMPIRICAL LEG. STUD. 175, 184 (2019). 26 See DOJ Report at 29 (referring implicitly to what is known as the Cleary method). 27 See DOJ Report at 59, tbl. 5 & 62, tbl. 9. 28 See DOJ Report at 57, tbl. 3. 29 See DOJ Report at 60, tbl. 7. 30 DOJ Report at 57-58. 31 See DOJ Report at 59, tbls. 5 & 6. 6 EFTA00100199 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 score; and the limited number of programs/productive activities currently available (with correspondingly far fewer points allocated by the tool) $ ^{32} $ ## 3. Developmental Sample Additional information is needed regarding the developmental sample. - Additional information is needed regarding the attributes of the developmental sample. The DOJ Report includes apparently contradictory, or at least confusing information, about the composition of the developmental sample. o The DOJ Report indicates the BOP provided its contractors, Duwe and Hamilton, with a dataset used to “develop and validate” $ ^{33} $ PATTERN containing 278,940 BOP inmates released from BOP facilities between 2009 and 2015,” which included “only those inmates released to the community,” and excluded “released inmates who died” and those “scheduled for deportation.” $ ^{34} $ DOJ also reports that developers relied on a smaller “eligible sample size” of 222,970, described as “those who were released from a BOP facility to a location in the United States and had received a BRAVO assessment,” which may mean that 55,970 individuals from the original dataset (20%) were excluded from what became the developmental sample because they had not been scored on BRAVO. $ ^{35} $ More information is needed regarding the excluded individuals, including demographic characteristics, and reasons they may have been released but not scored on BRAVO. Such a reduction in the sample size could introduce sample bias. O It appears that the training sample contained individuals who were released in 2009-2013, and the test (or validation) sample contained individuals who were released in 2014-2015. $ ^{36} $ More information is needed about why the training and test samples were drawn from different years. Information is also needed regarding what consideration was given to the possibility that there were risk-relevant differences between the groups. For example, policy changes, such as the retroactive 2014 amendment to the drug guidelines, may have resulted in a different composition of 32 See Emily Tiry, Julie Samuels, How Can the First Step Act's Risk Assessment Tool Lead to Early Release from Federal Prison?, Urban Wire, Crime and Justice (Sept. 5, 2019), https://www.urban.org/urbanwire/how-can-first-step-acts-risk-assessment-tool-lead-early-release-federal-prison. $$ ^{33} \text{ DOJ Report at 43}. $$ $$ ^{34} \text{ DOJ Report at 42-43}. $$ 35 DOJ Report at 46. 36 See DOJ Report at 49 & 50. 7 EFTA00100200 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 individuals released in 2015 than in prior years. $ ^{37} $ It is important for stakeholders to understand whether the differentials in samples here also embed bias into the tool. More information is needed regarding why the size of the developmental sample used in the DOJ Report is significantly lower than the number of federal prisoners released in those years, as indicated from another official database. An online tool for calculating the number of released prisoners offered by the Bureau of Justice Statistics indicates that 385,405 individuals were released from federal correctional institutions from 2009-2015. $ ^{38} $ Yet, the DOJ Report specifies that its developmental sample includes only 278,940 released prisoners. $ ^{39} $ Specifically, it is important to know whether the reported exclusions for death and deportation $ ^{40} $ account for the entire differential or whether there are additional explanations. Similarly, more information is needed about the size of the training and test groups. The DOJ Report indicated the training group as 66% of the total developmental sample, with the test group as 33% of the sample, but also described the training group as including 5 years of releases, with the test sample including only 2 years of releases. Information is needed to explain this apparent discrepancy. $ ^{41} $ - Additional information is needed regarding the sample descriptive statistics (including recidivism rates). Table 1 provides data on the entire eligible developmental sample, but is also needed separately for each of the (a) training sample and (b) test sample. $ ^{42} $ - Additional information is needed regarding the sample descriptive statistic on “BRAVO-R Initial: History of Escapes.” The total reported percentage is 86%, but no information is provided regarding whether this means there is 14% missing data on this factor, and if so, how missing data cases were scored. $ ^{43} $ - Information is needed regarding the inter-rater reliability scores for the evaluators concerning the development sample, both training and then test data. These statistics will provide information relevant to whether PATTERN can be scored consistently, as 37 See Remarks for Public Meeting of the U.S. Sentencing Comm'n, Washington, D.C., at 2 (Jan. 8, 2016) (Honorable Patti B. Saris, Chair) (recognizing that approximately 6,000 offenders were released on or about November 1, 2015 as a result of the 2014 amendment to the drug guidelines). $$ ^{38} $$ These were calculated using an online tool and narrowing to federal prisoners. See Bureau of Justice Statistics, Corrections Statistical Analysis Tool-Prisoners, https://www.bjs.gov/index.cfm?ty=nps. 39 See DOJ Report at 42. 40 See DOJ Report at 42-43. 41 See DOJ Report at 49-50. 42 See DOJ Report at 46-48, tbl. 1. 43 See DOJ Report at 48, tbl. 1. 8 EFTA00100201 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 recognized by the DOJ Report, but for some reason not reported. $ ^{44} $ Low inter-rater reliability outcomes decrease the utility of a tool. ## 4. Weighting The DOJ Report indicates that PATTERN involves “analytically weighting assessment items,” $ ^{45} $ but more information is needed on whether the weights are assigned solely through the points identified for each of the factors included in Table 2, $ ^{46} $ or are somehow reweighted in an algorithm not discussed in the report. The DOJ Report provides so few details on weighting, it is unclear what type(s) of models were used (such as regressions) and/or whether any type of machine learning (supervised or unsupervised) was employed. If the former, more information is needed regarding whether and how step-wise procedures were used, data on intercorrelations, and if multicollinearity exists. If the algorithm was developed with any form of machine learning, this more “black box” method has different and profound implications on transparency of the developmental procedures. ## 5. Overrides The DOJ Report does not mention overrides. Information is needed regarding whether PATTERN allows for policy overrides and/or discretionary (also referred to as professional) overrides, and if so, whether there will be a supervisory approval process for discretionary overrides. Information is also needed as to whether any of the final scores in the development sample (training and/or testing) involved overrides of original scores and the reasons for such overrides. ## 6. Relevant Research Copies of two governmental papers cited in the DOJ Report, but not readily available to the public, must be made available. Specifically, documents detailing the BRAVO-R, from which "PATTERN builds," $ ^{47} $ and relevant RRI computations are cited as important to understanding PATTERN $ ^{48} $ but are not readily available to the public. ## 7. Definitions & Scoring More information is needed regarding the definitions of key terms and rules for scoring. - Recidivism. It appears that for purposes of developing and testing PATTERN, “general recidivism” is broadly defined to include “any arrest or return to BOP custody following release.” $ ^{49} $ More information is needed to determine whether this is as (unduly) broad as it appears, and includes revocations for minor technical violations such as failure to timely 44 See DOJ Report at 27. 45 DOJ Report at 50. 46 See DOJ Report at 53-56. 47 DOJ Report at 44; 64 nn.8 & 9. 48 See DOJ Report at 66 n.25. 49 DOJ Report at 50. 9 EFTA00100202 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 report a change of residence, purportedly lying in response to queries from a probation officer, or failing to timely notify the probation officer of being questioned by police. $ ^{50} $ Similarly, it appears that for purposes of developing and testing PATTERN, “violent recidivism” is defined as “violent arrests following release.” $ ^{51} $ More information is needed here, as well, regarding what kinds of arrests are considered “violent.” A separate discussion in the DOJ Report regarding whether the instant offense was violent, appears to cite a definition of “violent recidivism.” $ ^{52} $ More information is needed regarding whether this is also the intended definition of violent recidivism. If so, more information is needed about what is included in “other violent.” $ ^{53} $ Defenders are concerned that revocations, arrests, and misdemeanor convictions are poor and biased proxies for the kind of serious re-offenses targeted by the recidivism-reduction programming at the core of the FSA. In addition, more information is needed regarding whether any mechanism was used to exclude pseudo-recidivism (prior offenses that were not detected and pursued—subject to arrest or return to prison as a result—until after the instant offense). - Age of First Arrest/Conviction. More information is needed regarding whether the first risk factor for purposes of developing, testing and implementing PATTERN is age of first arrest or age of first conviction. The DOJ Report contains contradictory information, referring to both arrest and conviction without explanation for the inconsistency. $ ^{54} $ If looking to conviction, is the relevant age determined by the individual’s age on the date of the alleged conduct, date of arrest, or date of conviction? More information is also needed about what is being counted in the “under 18” category. It is unclear whether this factor sweeps in all juvenile adjudications (including status offenses), or is limited to convictions in adult court. Among our many concerns with this factor is the relative unreliability of juvenile 50 See, e.g., USSG §5D1.3(c)(4), (c)(5), (c)(9). 51 DOJ Report at 50. 52 DOJ Report at 46 n.16; 65 n.15. $$ ^{53} \text{ DOJ Report at 65 n.15}. $$ 24 Compare DOJ Report at 46, tbl.1 (age of first arrest) with DOJ Report at 45; 53, tbl.2; 65, n.14 (age of first conviction). 10 EFTA00100203 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 adjudications$^{55}$ and that “youth of color—and especially black youth—experience disproportionate court involvement.”$^{56}$ - Infractions. More information is needed regarding the infraction factors. First, what is meant by an “infraction,” a “conviction” for an infraction, and a “guilty finding” for purposes of these factors? It is unclear whether the infraction factors will count any and all disciplinary misconduct. Second, how are infractions scored? Would multiple acts during a single course of conduct be counted as one or more? Would multiple acts processed at the same time (whether a single course of conduct or not) be considered one or more? Third, what is the empirical basis for treating all 100 and 200 level offenses the same, such that refusing a Breathalyzer and possessing pot are scored the same as killing and taking hostages? $ ^{57} $ Fourth, is there any limitation on the reach of this factor? For example, does it look only to infractions in the past year, all infractions while in prison for the instant offense (and whether serving the original sentence or a revocation sentence), all infractions while serving any federal sentence, or for any offense ever, regardless of jurisdiction? We have numerous concerns about counting infractions in any form, and particularly minor infractions, for the purposes of determining eligibility for earned time credits and release under the Act. First, there is minimal due process structure over BOP disciplinary actions. Second, likely varied and divergent infraction cultures and practices from one BOP facility to another would mean the likelihood of attracting an infraction may be due to luck of the draw on institutional assignment. In addition, we are concerned about ex post facto use of infractions to negatively score defendants on PATTERN when individuals had no notice such infractions would count against them for these purposes, particularly in light of the FSA provisions indicating past participation in programs will not be counted to positively score individuals. $ ^{58} $ - Programs & Technical/Vocational Courses. More information is needed on the types and descriptions of the programs and technical or vocational courses for which points were given for these two variables. For example, information is needed on the name of the programs/courses, the providers, the personnel involved, the number of hours required, the length of the programs/courses, the program/course goals, the definition of completion, 55 For example, the vast majority of states do not provide jury trials for juveniles, and “children routinely waive their right to counsel without first consulting with an attorney.” Nat'l Juvenile Defender Ctr. (NJDC), *Defend Children: A Blueprint for Effective Juvenile Defender Services* 10 (Nov. 2016); NJDC, Juvenile Right to Jury Trial Chart (last rev. July 17, 2014), http://njdc.info/wp-content/uploads/2014/01/Right-to-Jury-Trial-Chart-7-18-14-Final.pdf. 56 Katherine Hunt Federle, The Right to Redemption: Juvenile Dispositions and Sentences, 77 LA. L. REV. 47, 52 (Fall 2016). 57 See Dep't of Justice, Bureau of Prisons, Inmate Discipline Program, Program Statement 5270.09, tbl.1, (July 8, 2011). 58 See FSA at Title I, § 101(a) (codified at 18 U.S.C. § 3632(d)(4)(B)). 11 EFTA00100204 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 and the locations where the programs/courses were made available. Information is needed about why the direction of the points for the number of technical/vocational courses is the reverse of what might be expected. Specifically, information is needed on why the tool penalizes an individual for taking a technical/vocational course. $ ^{59} $ In addition, information is needed on whether there is an error in the description of the technical/vocational factor when it references the number of courses “created” rather than “completed,” and if not, what is meant by courses “created.” - Drug Treatment and Drug Education. More information is needed regarding the difference between drug treatment and drug education for purposes of scoring the PATTERN. More information is also needed regarding how drug treatment “need” is determined and scored, including whether it is based on self-report. The DOJ Report suggests it is tied to the BRAVO drug/alcohol abuse indicator, but it is not clear what data informs this factor, particularly without access to the BRAVO-R document requested above. - - Sex Offender. Additional information is needed on how this factor is scored, including whether it is limited to convictions for sex offenses, or is broader and informed by arrests, self-report, hearsay, and whether it includes exonerated charges. As with other factors, additional information is also needed on whether there are any time limits on how recent the 59 See DOJ Report at 54, tbl. 2. 60 See DOJ Report at 46, n.16 & 65, n.15. The numbering of the Chapter Three endnotes is off, such that the content of the notes does not always match the text. It appears that the mismatch begins with endnote 14, which according to the text should have provided information on “noncompliance with fiscal responsibility” but instead discusses “Age at first conviction.” 61 See DOJ Report at 46, n.16 & 65, n.15. 62 See DOJ Report at 46, n.16 & 65, n.15. 63 See Mariel Alper & Matthew R. Durose, 2018 Update on Prisoner Recidivism: A 9-Year Follow-up Period (2006-2014) (2019) (Special Report, U.S. Dep’t of Just.). 12 EFTA00100205 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 conduct must be for it to count. In addition, information is needed regarding the empirical basis for including this factor. It appears contrary to DOJ studies of national samples that show lower risk of recidivism for individuals convicted of sex offenses than other types of offenses. $ ^{64} $ Is this factor essentially operating as a policy override for other purposes? - Criminal History Score. Information is needed on whether this is a static figure based strictly on the U.S. Sentencing Commission guidelines' criminal history score at the time of sentencing or whether it can increase at reassessment because of events between sentencing and reassessment. Further, can the criminal history score be reduced at reassessment pursuant to a time decay mechanism? - History of Violence. Information is needed regarding the definition of violence, and whether it requires a conviction for a violent crime. Specifically, which crimes are considered "violent" for purposes of this factor? If not limited to convictions for violent offenses, more information is needed regarding the sources of information that may be considered when assessing this factor, and whether it permits consideration of arrests, prison disciplinary records, hearsay, and/or self-reports. In addition, information is needed on whether there is any time limit for this factor, or some time decay mechanism, as would be supported by available research on desistence. - History of Escapes. Information is needed regarding the definition of escape, including, for example, whether it would include failure to appear in a pre-trial context, or walking away from a halfway house. Information is also needed regarding whether there is a time limit for inclusion of old escapes, or a time decay mechanism. - Education Score. Information is needed regarding the ordinal rankings for the education score for the violent recidivism tool. - Databases. Several factors rely on past criminal conduct. More information is needed regarding the databases that will be accessed to determine recidivism, and the known gaps and biases in such databases. - Missing Data. Information is needed regarding what adjustments were made for missing data, and the rate of missing data for each predictor. In addition, information is needed regarding the policy going forward when there is missing data in one of more factor in an individual case. For example, will information about missing data be communicated with the risk score and classification? ## 8. Double Counting More information is needed to determine the scope of double counting under PATTERN, and whether any consideration has been given about ways to ameliorate it. 64 See Matthew R. Durose et al., Recidivism of Prisoners Released in 30 States in 2005: Patterns from 2005 to 2010 (2014) (Special Report, U.S. Dep’t of Just.). 13 EFTA00100206 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 - Age. Young age will be counted twice for young first offenders, who will be young at time of first arrest/conviction, $ ^{65} $ as well as at time of assessment. - Infractions. Information is needed on whether a single “infraction conviction” that is deemed “serious and violent” would count as both “any” and then again as “serious and violent” infraction. In addition, would an “infraction conviction” that resulted in a criminal conviction also count toward a criminal history score if criminal history is not static? And could an “infraction conviction” also result in points under the history of violence and/or “sex offender” factors? - History of Violence. Information is needed on whether a person with multiple violent priors receives multiple point scores in this single variable. For example, in the male general recidivism tool, if an individual had a minor violent offense < 5 years and a serious violent offense > 15 years, would the individual receive 5 points or 7? - Violence. Information is needed on whether the same violent offense can be counted multiple times, such as in the criminal history score, infraction convictions, instant offense violent, history of violence and/or sex offender. - Sex Offense. Information is needed on whether the same sex offense can be counted multiple times, such as in the criminal history score, infraction convictions, instant offense violent, history of violence and/or sex offender. - Criminal History. Information is needed on whether consideration was given to ameliorating the repeated counting of criminal history, first in the imposition of the sentence based on a guideline calculation that relies heavily on criminal history and then throughout PATTERN, including age of first arrest/conviction, sex offender, criminal history score and history of violence. We are concerned about the inclusion and weight (repeatedly) given to this factor for a number of reasons. Some concerns arise from the unique way in which the guidelines count criminal history, such as including all juvenile adjudications on par with adult convictions (with some difference in decay periods), and using sentence imposed rather than time served as a proxy for seriousness of the offense (affecting the number of points received). $ ^{66} $ In addition, as mentioned above, research on other risk tools has shown racial differences in scores with black individuals obtaining higher scores than white individuals, where most of the difference “is attributable to criminal history.” $ ^{67} $ Criminal history correlates with race because it reflects prior instances of racial disparity in the criminal justice system or disadvantage earlier in life. Criminal history is not just the product of participation in crime, but of biased practices throughout the criminal justice system. Blacks do not sell 65 See supra note 54 and related text regarding issue of whether the first predictor looks to age of first conviction or arrest. 66 See USSG §4A1.2(d), (e). 67 Jennifer L. Skeem & Christopher T. Lowenkamp, Risk, Race, and Recidivism: Predictive Bias and Disparate Impact, 54 CRIMINOLOGY 680, 700 (2016). 14 EFTA00100207 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 drugs or possess guns at a greater rate than Whites. $ ^{68} $ Studies show that Blacks are stopped and frisked or searched at higher rates than Whites, but that Whites who are frisked or searched are found with contraband at higher rates than Blacks who are frisked or searched. $ ^{69} $ And Blacks are arrested more than twice as often as Whites. $ ^{70} $ Charging decisions and bail determinations further compound these racial disparities as individuals move through the criminal justice system. $ ^{71} $ We urge the DOJ to open discussion to a multidisciplinary team on methods to ameliorate the overreliance upon, and negative impacts of, criminal history. ## 9. Protective & Promotive Factors Additional information is needed on whether there are any plans to incorporate additional protective and promotive factors in PATTERN. Currently, program/course participation and educational attainment appear to be the only proxies for protective factors included in PATTERN. Similarly, additional information is needed on whether there are plans to incorporate a desistance factor into PATTERN that would significantly adjust the risk rating according to the literature on the age-crime curve and the literature on cessation of offending. $ ^{72} $ We urge DOJ to engage with a multidisciplinary team to consider incorporating more protective and promotive factors to better meet the goals of the FSA. ## 10. Policy Decisions Risk assessments are not simply math. Every risk assessment involves moral choices and tradeoffs. Some of our questions in this area are incorporated above, such as whether consideration has been given to ameliorating the effects of certain factors that are unacceptable regardless of predictive value. In addition, information is needed generally regarding the mechanisms in place to ensure that issues which have distinct policy implications will be resolved by appropriate personnel—ideally a 68 See Amy Baron-Evans & David Patton, A Response to Judge Pryor's Proposal to “Fix” the Guidelines: A Cure Worse than the Disease, 29 FED. SENT’G. REP. 104, 112 (Dec. 1, 2016-Feb. 1, 2017). 69 See id. at 112-13 (collecting studies); see also Radley Balko, Op-Ed., There’s Overwhelming Evidence that the Criminal-Justice System is Racist. Here’s the Proof, WASH. POST, Updated Apr. 10, 2019 (collecting studies). 70 See Bureau of Justice Statistics, Arrest Data Analysis Tool, 2014 (most recent data available), https://www.bjs.gov/index.cfm?ty=datool&surl=/arrests/index.cfm#. 71 See supra note 69; see also USSC, Application and Impact of 21 U.S.C. § 851: Enhanced Penalties for Federal Drug Trafficking Offenders 7, 33-36, figs. 13-14 (2018). 72 See Cecelia Klingele, *Measuring Change: From Rates of Recidivism to Markers of Desistance*, 109 J. CRIM. L. & CRIMINOLOGY (forthcoming 2019), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3142405; Ralph C. Serin & Caleb D. Lloyd, Integration of the Risk Need, Responsivity (RNR) Model and Crime Desistance Perspective: Implications for Community Correctional Practice, 7 ADVANCING CORRECTIONS 37, 38 (2019). 15 EFTA00100208 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 multidisciplinary team that includes policymakers and stakeholders $ ^{73} $ —rather than solely data scientists. For example, the decisions on the cut-points, which necessarily impact fairness measures such as false positive rates and positive predictive values, appear to have been made by the researchers and based on arbitrary fractions or multiples of the recidivism rates. $ ^{74} $ Yet, where those decisions affect moral and political outcomes with real-world consequences to individuals, they should instead be made by a multidisciplinary team that has the authority and direct interest in such consequences. Risk tool developers have a natural incentive to focus on overall accuracy. However, accuracy may need to yield to other important goals, such as differential validity, group fairness, and individual rights. Selecting the right tradeoff between these sometimes competing goals are more rightly within the power of policymakers and stakeholders. Here, it appears the cut-points were established somewhat arbitrarily without regard to such consequences as the false discovery rate and false omission rate (the reciprocals of positive predictive value and negative predictive value) and equal calibration, among other validity and fairness measures discussed above. Because PATTERN was developed to meet the obligations of the FSA, a preferable method for setting cut-points would be attuned to the goal of maximizing incentives for participation in rehabilitative programs and courses. Increasing the cut-point between low and medium would be more suitable to achieve this goal. Relatedly, information is needed regarding the process, and who was involved, in setting the rules governing the combined (final) RLC. The current rule dictates that the highest risk category from the general and violent scales will be used to set the final RLC. Different choices could have been made that would be more suitable to achieve the FSA’s goal of incentivizing and rewarding more individuals to complete programs and courses. For example, a person who scores low or minimum on one scale and medium on the other should have a final RLC of low. And a person who scores high risk on one scale, yet medium risk on another should be classified for purposes of the final RLC as medium. Additional information is also needed regarding the process for deciding on the definition of "recidivism." This is a policy decision that requires identifying the scope of conduct that should be included, consistent with the purpose of the FSA to successfully reintegrate individuals in the community. For example, what was the process for deciding to include all revocations, including 73 See Partnership on AI, Report on Algorithmic Risk Assessment Tools in the U.S. Criminal Justice System 31 (2019), https://www.partnershiponai.org/wp-content/uploads/2019/04/Report-on-Algorithmic-Risk-Assessment-Tools.pdf (suggesting an oversight body including “legal, technical, and statistical experts, current and formerly incarcerated individuals, public defenders, public prosecutors, judges, and civil rights organizations”); Danielle Kehl et al., Algorithms in the Criminal Justice System 34 (2017), https://dash.harvard.edu/bitstream/handle/1/33746041/2017-07_responsivecommunities_2.pdf?sequence=1&isAllowed=y. 74 See DOJ Report at 50. 75 See DOJ Report at 50-51. 16 EFTA00100209 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 technical violations, and for looking to arrests, despite the literature showing the serious racially disparate impact of looking to arrests, rather than convictions? $ ^{76} $ In light of the FSA's purpose, a more limited definition of recidivism focused on serious offending would be more appropriate than the broad definition used to develop PATTERN. ## B. Transparency & Accountability: Implementation Transparency and accountability are both mandated and essential in the implementation of PATTERN. While much remains unknown in this area, we already have several questions which warrant the attention of a multidisciplinary team as PATTERN is implemented. ## 1. Privacy/Confidentiality It appears that several of the factors in PATTERN, and the yet-to-come needs assessment, may require interviews and be based at least partially on self-reporting. This raises several questions and concerns. Additional information is needed on what protections will be in place to honor an individual’s right to be free of self-incrimination. More information is needed on what protections will be in place to prohibit the use of any interview admissions against an individual, either in a new prosecution or prison disciplinary proceeding. Information is also needed regarding how scores and information obtained in the scoring process will be maintained and confidentiality protected. And information is needed on the data retention policies for risk scores, needs assessments, and information obtained to complete the tools. ## 2. Challenges As discussed above, PATTERN scores and accompanying risk categories will directly affect how much time many individuals spend in prison. Information is needed on the procedures for contesting individual scores and category assignments. Risk assessment is unique enough that treating a challenge like any other grievance is not a sufficient process. Potential concerns include discovering factual errors, contesting judgment calls, challenging an override decision, and correcting a scoring miscalculation. To equip individuals to assess and challenge their PATTERN scores we expect individuals will be provided not only with their final PATTERN score and related risk category, but also scores on each of the individual factors, and information on the limitations of the scores, including the warnings set forth below. And individuals challenging their PATTERN score and category will need more. Indeed, much of the information individuals will need to challenge their scores tracks the information requested above regarding the development, validation and bias testing of PATTERN. In addition, among other information, individuals will need codebooks and scoring sheets, training materials, and inter-rater reliability scores for those scoring the tool. Additional information is needed regarding the plans to ensure adequate information and processes are provided to individuals challenging their PATTERN scores. 76 See Jennifer Eaglin, Constructing Recidivism Risk, 67 EMORY L.J. 59, 94 (2017). 17 EFTA00100210 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 ## 3. Risk Communication Information is needed on the manner in which risk scores and categories will be reported both within and outside the BOP. Studies show that risk communication format matters in how decision-makers understand the results and can be manipulated. $ ^{77} $ We are concerned that the scores and categories will not be communicated with sufficient context to make the scoring and results translatable to those who were not deeply involved in the development of the tool. To that end, we recommend reporting risk results as the ordinal bins plus that bin’s relevant observed (a) recidivism rate and (b) success rate (1-recidivism rate). The communication should also include the definition of recidivism to contextualize the meaning of the rates. In addition, we recommend including a set of warnings to ensure users of the scores and categories understand the tool's limits. $ ^{78} $ The following list includes ideas on the warnings we believe appropriate in light of our current understanding of PATTERN: - PATTERN is based on group statistics and cannot assess an individual's probability of reoffending; - (as relevant) PATTERN disproportionately judges minorities at higher risk than whites; - PATTERN relies on arrest data, which may merely replicate biases in policing practices; - PATTERN does not include all protective or promotive factors that may reduce the individual's risk prediction; - PATTERN does not predict the aspects of risk regarding imminence, frequency, severity, or duration; - PATTERN's rankings of risk (minimum, low, medium, high) are merely relative to the population studied; - PATTERN's score includes criminal history measures that did not require conviction and thereby may overestimate risk because of faulty data; - PATTERN's score may be higher based on evidence of juvenile offending; - PATTERN may increase risk when the individual does not engage in various types of programming; however, such programs may not have been made available to this individual for reasons not within the individual's control; - (as relevant) PATTERN factors can count the same events twice or multiple times; 77 See Ashley B. Batastini et al., Does the Format of the Message Affect What Is Heard? A Two-Part Study on the Communication of Violence Risk Assessment Data, 19 J. FORENSIC PSYCHOL. RES. & PRAC. 44, 46 (2019); Daniel A. Krauss et al., Risk Assessment Communication Difficulties: An Empirical Examination of the Effects of Categorical Versus Probabilistic Risk Communication in Sexually Violent Predator Decisions, 36 BEHAV. SCI. & L. 532, 534 (2018); Nicholas Scurich, The Case Against Categorical Risk Estimates, 36 BEHAV. SCI. & L. 554, 558 (2018). 78 See Wisconsin v. Loomis, 881 N.W.2d 749, 765 (Wis. 2016) (identifying necessary cautions, that may evolve, before considering risk assessment at sentencing). 18 EFTA00100211 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 - (as relevant) this PATTERN score represents an override of the algorithm and the reason for the override. ## 4. User Buy-In Research studies and anecdotal evidence indicate that users (e.g., those scoring the tool and relevant decision-makers who receive scores) tend to distrust, and find ways to deviate from, algorithmic risk results if they are not included enough in the process and program. $ ^{79} $ Information is needed on the methods planned to achieve sufficient user buy-in to improve compliance and consistency in order to achieve the FSA's goals in this endeavor. ## II. NEEDS ASSESSMENT A core purpose of Title I of the FSA is to help prisoners succeed in their communities upon release and thereby reduce recidivism. The Act contemplates accomplishing this by providing all individuals in prison evidence-based programming that is designed to help them succeed upon release and that has been shown by empirical evidence to reduce recidivism. $ ^{80} $ We are deeply concerned that the DOJ has not yet released the needs assessment required by the FSA. We understand from DOJ’s Report that the needs assessment is in the works, and there will be an opportunity to comment on that aspect of the DOJ's FSA obligations at a later time. In light of that, we raise only a few critical issues here. ## 1. Programs Evidence-based programming is the bedrock of the FSA. Other aspects of the risk and needs assessment system only make sense if there is programming. Assessing (and reassessing) needs and assigning (and reassigning) individuals to programming based on those needs require that appropriate and available programming exist. $ ^{81} $ In addition, the incentives and rewards identified in the law are contingent on participation in appropriate and available programming. $ ^{82} $ DOJ’s Report, however, suggests there are few programs or courses available, as indicated by the relatively few individuals who were scored on them in the developmental sample. $ ^{83} $ This is consistent with other information that waitlists to participate in BOP programs are long: 25,000 inmates are currently 79 See Jean-Pierre Guay & Geneviève Parent, Broken Legs, Clinical Overrides, and Recidivism Risk: An Analysis of Decisions to Adjust Risk Levels with the LS/CMI, 45 CRIM. JUST. & BEHAV. 82, 83-84 (2018). 80 See FSA at Title I, § 101(a) (codified at 18 U.S.C. §§ 3632, 3635(3)) and § 102(a) (codified at 18 U.S.C. § 3621(h)). 81 See FSA at Title I, § 101(a) (codified at 18 U.S.C. § 3632(a)(3)-(4)). $^{82}$ See FSA at Title I, $\§ 101$(a) (codified at 18 U.S.C. $\§ 3632$(a)(6), (a)(7), (d)). 83 See DOJ Report at 47, tbl.1 (showing almost half (49%) of the developmental sample had completed no programs, a vast majority had no technical/vocational courses (82%) or federal industry employment (92%) and well over half (57%) had not had drug treatment while incarcerated despite indication of need). 19 EFTA00100212 Federal Public & Community Defenders Legislative Committee 52 Duane Street, 10th Floor New York, NY 1007 Tel: (212) 417-8738 waiting to be placed in prison work programs, at least 15,000 are waiting for education and vocational training, and at least 5,000 are awaiting drug abuse treatment. More information is needed on how programming will be expanded to ensure the goals of the FSA are met. ## 2. BOP Current Needs Assessment The DOJ Report indicates the BOP is using its current needs assessment until one is developed pursuant to the FSA. More information is needed on BOP's current needs assessment and processes. ## 3. Responsivity Information is needed about how responsivity will be considered in connecting needs to programs. Relatedly, additional information is needed on the availability of culturally-sensitive programming (e.g., programs in Spanish for those with weak English skills and modification of 10 Step-like programs for non-Christians). ## III. CONCLUSION PATTERN is a high-stakes tool that directly affects how much time many people will spend in prison. High levels of transparency, accountability and auditability are both required and critical. We appreciate the opportunity to share our questions and concerns and hope there will be additional opportunities for feedback and dialogue after we have received the information identified above. Very truly yours, /s David Patton Executive Director, Federal Defenders of New York Co-Chair, Federal Defender Legislative Committee 84 See BOP: UNICOR, Federal Bureau of Prisons, https://www.bop.gov/inmates/custody_and_care/unicor_about.jsp (estimating the participation rate at 8%). 85 See Oversight of the Federal Bureau of Prisons Before the H. Subcomm. on Crime, Terrorism, Homeland Security and Investigations of the H. Comm. on the Judiciary, 115th Cong. 20 (2018) (BOP Director Inch). 86 See Dep’t of Justice, Bureau of Prisons, Drug Abuse Treatment Program, 81 Fed. Reg. 24484, 24488 (Apr. 26, 2016) (“over 5,000 inmates waiting to enter treatment”); Colson Task Force, at 36 (“at the end of FY 2014, more than 12,300 people systemwide were awaiting drug abuse treatment”). 20 EFTA00100213 # EXHIBIT B