The Wider Index

How to read the ratings

Each source carries three 1–5 ratings from our census of 2026-08-28. They measure the source as a text provider, not the underlying documents.

Fidelity —

how trustworthy the OCR text is: engine quality, redaction handling, provenance. A source that deliberately rewrites text (for example, PII-redacted exports) scores below a verbatim engine.

Coverage —

how much of the known corpus the source covers. The denominator is the DOJ release plus the House estate pages.

Usability —

whether you can actually get and use the text: license, bulk access, structure, and whether rows map back to EFTA numbers or filenames.

Scale numbers are the maintainer's own claim unless we marked them verified against the files. Where the two censuses disagreed, we kept the figure we could check. "Undisclosed" engine means the maintainer never named it — not that there is no engine.

Cross-validation

Ratings describe sources; cross-validation tests them. We sample a few hundred pages that several sources have all OCR'd, align each pair's texts, and measure a normalized character error rate between them. We add two symbolic checks that need no judgment: whether digit runs (dates, amounts, EFTA stamps) survive, and whether the expected stamp regexes still match. Pairwise error is symmetric and reference-free: a high rate means the two engines disagree, and the heatmap shows who disagrees with whom.

We only believe a claimed problem when we can verify it independently — by checking a third source's take on the same page or by reading the page image itself. Findings that survive that check appear here; the rest stay out.

Current battery: 450 sampled pages in 4 strata, 24 pair-stratum rows scored, seed 42. Alignment on raw image basenames: House NOV/OCT filenames are shared exactly by marker md stems and greg (22,903 nov / 8,718 oct rows); teyler keys are 'IMAGES-<dir>-<STEM>.txt' (covers NOV 100%); jmail567 doc_id equals the OCT image stem (3,680 rows, all OCT); kabass HouseOversightEstate rows carry renumbered DOJ-OGR keys with no filename mapping, so they join via the HOUSE_OVERSIGHT_###### stamp found inside their own OCR text (6,794 unique stems; doc-level rows were cut to the '--- page N ---' segment containing the stem). DOJ joins on EFTA numbers: marker frontmatter eftaNumber vs ishumilin document_id vs kabass DataSet1 file_key vs jmail-doj doc_id prefix. Four batteries: nov (160 pages in 4 sources + 40 in 3), oct (80 in 4 + 20 in 3), doj (100 pages from the 3,145-doc 4-source intersection, photo-skewed) and doj-text (all 50 text-rich docs, marker ocrChars>=1000, in the same 4-source intersection). Normalization: lowercase, collapse whitespace, strip soft hyphens (U+00AD/U+2010); CER = rapidfuzz Levenshtein / max(len); digit Jaccard over multisets of digit runs >=3; stamps matched case-insensitively (EFTA\d{8} | HOUSE_OVERSIGHT_\d{6}). RNG seed 42; sampling code in /tmp/crossval.

What we hold

Mirrored copies behind this site: the full Jmail open-data parquet set, the ishumilin and kabasshouse Hugging Face corpora, the small specialist sets (svetfm's Textract FBI files, the teyler 20k lineage, LovenSar's GLM output, the 567-labs annotated family), rhowardstone's structured-data mirror, and the only surviving copy of the ep-nov-12.greg.technology OCR database. We also hold both House Oversight official containers — the NOV container includes its TEXT and NATIVES; the OCT container does not. Ask before redistributing any of these: several carry no license or a non-commercial one, and this page records that rather than deciding it.

Method and provenance

The census ran on 2026-08-28 against live sites, APIs (Hugging Face, CourtListener, DocumentCloud, Wayback), and the maintainer's own documentation. Every claim in the underlying research carries its source URL, and screenshots show each site as we captured it. The catalog regenerates fromdata/catalogs/ecosystem/ via scripts/build_ecosystem_catalog.py; the ratings methodology follows the fleet's controlled-prose rules for evidence grading. The engine provenance of our own served text is documented indocs/ocr-engine-provenance.md. Corrections and new sources are welcome — the EFTA number or filename is the universal key, so adding a source is usually just metadata.