We reduced the false-positive rate of an automated source-fidelity checker for news production by identifying and correcting the two causes of its underperformance, and we confirm the improvement with a controlled before-and-after evaluation. The Verification Desk checks whether machine-drafted news articles faithfully represent their sources before publication. Over a defined period it held a large number of correctly-sourced articles as defective. We trace this to two independent causes and address both. First, model selection: on an identical, fixed body of evidence, replacing an open-source model with Anthropic's Opus 4.8 reduced the blocking-objection rate from 83% to 37% (measured over 204 and 107 articles). Second, evidence assembly: we identify and correct five mechanisms by which the system supplied incomplete evidence to the model without recording it had done so, with the result that 13 working links to established news and government sites, previously classified as fabricated, drop to 0, and attachment text in 4% of source records (360 of 8,975), previously unread, is now read. Thirty-seven of thirty-nine identified changes were implemented and verified. The corrections additionally yield a stateable guarantee for a fabrication-prone class of claim: within its scope, a fabricated or altered direct quotation cannot be recorded as verified, because confirmation is decided by a deterministic presence check that overrides the model. A verification control—an article containing a genuine discrepancy—is correctly still held, distinguishing a real correction from an indiscriminate loosening of the checker. We conclude that the elevated false-positive rate arose from incomplete evidence rather than from excessive strictness, and that correcting the evidence pipeline while preserving the checker's decision criteria both reduces incorrect objections and retains genuine detection.
Bushwick Daily is an independent local newsroom whose publishing pipeline drafts article text with large language models, working from source material the newsroom has gathered: press releases and newsletters received by email, frequently with PDF or Word attachments, and articles retrieved from other outlets. Because a language model can produce fluent text that is not fully supported by its sources, every draft passes through an automated review stage, the Verification Desk, before it may be published.
The Verification Desk checks a single, specific property called source fidelity: whether each factual claim in a draft—every figure, date, name, quotation, and attribution—faithfully represents the sources the newsroom holds or cites. This is deliberately distinct from truth. If a source makes a false assertion and the article reports that assertion accurately and attributes it correctly, the Desk passes it; evaluating the underlying assertion is a matter for editorial reporting, not automated verification. Restricting the system's question to faithfulness-to-source is what makes it automatable: fidelity to a held document is mechanically checkable, whereas truth about the world generally is not.
This report documents a reliability problem in that system—a period of elevated false-positive rate, in which correctly-sourced articles were held as defective—its analysis, the corrective changes made, and a before-and-after evaluation of their effect. It is written to be read without prior familiarity with the system.
For each article the Desk assembles the relevant evidence and evaluates the article against it. The evidence has two forms. The corpus is the source material the newsroom holds for the story—the text of the relevant emails, their attachments, and any retrieved articles—assembled into a single reviewable body of text. The cited pages are the external web pages the article links to; the Desk retrieves each so that a claim attributed to a page can be checked against that page's actual content.
A language model reads the draft alongside this evidence and produces a claim ledger: one entry per checkable claim, each carrying a verdict (for example confirmed, unsupported, or inaccurate) and a severity. A second, deterministic layer applies fixed rules over the model's output. A publish gate then reads the ledger and permits publication only if no unresolved claim of blocking severity remains. The design is intentionally conservative: under uncertainty the Desk holds an article for human review rather than releasing it.
One distinction organizes the remainder of this report. The model supplies judgment; the surrounding system supplies the model's evidence. A verdict can be no better than the evidence the system placed before the model. The problem analyzed here is a problem of evidence, not of judgment.
In mid-July 2026 a large batch of finished, publish-ready articles was held by the Desk, each marked as having a sourcing problem. Reviewing a sample of these holds against the underlying sources established that a substantial majority were false positives: the sourcing was faithful and the article correct, and the Desk's objection was mistaken.
A false positive in a publish gate has a specific cost profile. It does not expose readers to an error—the conservative design prevents that. Instead it consumes editorial time and withholds correct work from publication, and, at sufficient frequency, erodes operator trust in the checker. For a system whose value rests entirely on its objections being credible, an elevated false-positive rate is the failure mode most worth correcting. The objective of this work was to reduce that rate without reducing the Desk's ability to detect genuine problems.
Analysis identified two independent contributors, and separating them was the first result. The first was model selection. The held batch had been checked by an open-source model adopted temporarily under a cost-reduction experiment. The second was evidence handling: in a meaningful fraction of cases the system assembled incomplete evidence, and the model evaluated claims against it. The second contributor is model-independent, because it acts upstream of the model's judgment.
To separate the two, the held articles were re-checked with a stronger model (Opus 4.8) using the same evidence. Corpus assembly does not depend on the model, and the source material was fixed at the time of both runs, so both models read identical evidence. Table 1 isolates the model's contribution.
| Configuration | Articles | Passed | Held |
|---|---|---|---|
| Open-source model, original evidence pipeline | 204 | 34 (17%) | 170 (83%) |
| Opus 4.8, same evidence pipeline | 107 | 67 (63%) | 40 (37%) |
This addresses an apparent paradox: if the evidence was sometimes incomplete, how could the stronger model perform so much better on it? It did not have better evidence; it read the same evidence and judged it more accurately. In every examined case where the open-source model reported a claim as unsupported, the supporting text was in fact present in the corpus. The difference was calibration on shared evidence. Accordingly, the open-source routing experiment was retired and production checking now runs on Opus 4.8. That change alone does not resolve the second contributor, because a stronger model cannot compensate for evidence it is never shown; the evidence-handling corrections in Section 5 were made independently.
Five distinct mechanisms were traced to specific behavior in the code and confirmed against live data. In each, the system evaluated a claim against evidence that was incomplete and did not record the incompleteness. Table 2 summarizes them.
| Mechanism | Behavior | Effect on a claim |
|---|---|---|
| Cited-page truncation | Each retrieved page was truncated to its first 12,000 characters under a shared budget consumed in citation order; pages cited later could receive a fragment or nothing, with no logic preserving the passage a claim relied on. | A quotation located beyond the cutoff was invisible to the model, which reported the article's accurate quotation as a misquotation. |
| Fetch-failure classification | When a cited page could not be retrieved—commonly because the site blocks automated requests, or the request timed out—it was classified as fabricated rather than as unreachable. | Working links to established outlets were reported as invented citations. |
| Unread attachments | The corpus builder read the body of each source email but not the text already extracted from its PDF or Word attachments. | A press release placing its figures in an attached fact sheet had those figures reported as unsupported. |
| Alt-text loss | The HTML-to-text routine discarded image alt-text. One transit agency's newsletter encodes subway-line letters as small logo images; discarding the alt-text turned “the G and B trains” into “the and trains.” | Source text was corrupted before an article was drafted from it. |
| Partial quotation matching | The deterministic check confirming a supplied quotation against a page compared only the first 80 characters. | A quotation with a correct opening but an altered ending could pass. |
The most instructive change concerns validation of cited links, because the original design conflated two different questions. The first is whether a link is real—a language model can invent a plausible URL, and citing a nonexistent source is a genuine fidelity problem. The second is whether the page's content supports the claim attributed to it. The original implementation answered both by fetching the page and treated any fetch failure as evidence that the link was fabricated. A fetch can, however, fail for reasons unrelated to whether the link is real: the site may block automated requests, require a subscription, or respond slowly.
The redesign separates the two questions. Link validity is now determined by a Domain Name System (DNS) lookup: a link is treated as fabricated only when its domain does not resolve, confirmed against a second resolver. This does not retrieve the page, so a site's request-blocking cannot affect it, and it is the only condition that blocks an article on link grounds. Content support is still assessed by retrieving the page, which remains the majority of the Desk's link-related work; but when retrieval fails, the claim is marked unverifiable at minor, non-blocking severity, because the fabrication question has already been answered independently. The change reduced complexity: it removed any need for the fetch to succeed in order to validate a link. After it, every domain that had previously produced a false fabricated-link objection is correctly non-blocking when its content cannot be retrieved, while a genuinely non-resolving domain is still identified.
The mechanisms in Section 4.2 share a root property: the system based verdicts on partial evidence without recording that the evidence was partial. The corrective principle adopted is that the Desk may bound how much evidence it places in a single model request, but must never evaluate a claim as if the evidence were complete when it was not; access to the full source material must be preserved, and only the presentation to the model is bounded.
Three mechanisms enforce this. Cited pages are retrieved in full and then reduced by selecting the passages surrounding each claim, rather than by cutting to a fixed length, so a claim's supporting text is not lost due to its page's position in the citation list. The complete, uncut source text is retained for each check and used by the deterministic rules, so a determination such as “this quotation does not appear in the source” is made against the entire source even when the model was shown an excerpt. And any claim whose supporting evidence could not be fully examined is explicitly marked as incomplete and routed for further handling rather than receiving a final verdict. Incompleteness is thereby represented as an explicit state rather than an unrecorded condition.
The mechanisms above yield a specific, stateable property for one important and fabrication-prone class of claim: direct quotations. For a quotation attributed to a source the newsroom holds or successfully retrieves, the system does not rely on the language model's judgment to confirm it. After the model assigns a verdict, a deterministic check re-tests every quotation the model marked as confirmed against the complete, retained source text. Confirmation is upheld only if the quoted text is present in that source; where it is not, the deterministic check overrides the model and records the claim as a misquotation at blocking severity, irrespective of the model's assertion.
Within its scope—a quotation grounded in a source that is available to the system—a fabricated or altered quotation cannot be recorded as verified. The model's confirmation is necessary but not sufficient; deterministic presence of the quoted text in the source is required. The check is not a heuristic or a model judgment but a fixed rule executed after the model and empowered to override it.
Presence is evaluated after normalization of whitespace, quotation-mark style, and Unicode form, so that formatting differences do not cause false rejections; for long passages it tolerates internal ellipsis while still requiring each segment of the quotation to be present, so that a legitimate excerpt passes but a fabricated continuation does not. We verified the property directly: a supplied quotation consisting of a genuine opening followed by a fabricated remainder is rejected by the check, whereas the prior implementation—which compared only the opening characters of the quotation—accepted it.
Scope and non-claims. This is a narrow and absolute guarantee, not a broad and probabilistic one, and its limits are as important as its content. It applies to verbatim quoted material for which the cited source is available. It does not extend to paraphrase, numeric inference, or characterization, which are not verbatim strings and are assessed by the model together with other checks; and it does not apply when the cited source cannot be retrieved, in which case the claim is recorded as unverifiable rather than confirmed. The system therefore does not claim to detect every possible hallucination. What it establishes is stronger for being bounded: for the class of claim it covers, a confirmed verdict cannot be produced by a fabrication, because confirmation is decided by deterministic presence in the source rather than by the model's assertion.
Thirty-nine issues were identified and confirmed against the live code and database; thirty-seven were implemented, each as an isolated change with an automated test, integrated in a defined order with re-testing at each step. Two were deliberately not shipped and are noted as such: one did not reproduce in the current code, and one prompt-level protection had its principal component deferred. Most changes are to deterministic components—parsers, classifiers, passage selection, hashing—and were verified by automated tests and by replay against real records. The composed system was then evaluated end-to-end.
| Measurement | Before | After |
|---|---|---|
| Blocking-objection rate on identical evidence (open-source model → Opus 4.8) | 83% | 37% |
| Established-outlet links classified as fabricated (live sample) | 13 | 0 |
| Source records with attachment text read into the corpus (of 360 carrying attachments) | 0 | all |
| Held articles re-verified as correctly sourced under the corrected system | 0 of 180 | sweep in progress |
| Verification control (article with a genuine discrepancy) | held | still held |
The 180 articles that had been held and then released under manual attestation during remediation are being re-checked by the corrected Desk on Opus 4.8, replacing the manual attestations with fresh, system-generated verdicts. The corrected system returns a clean pass on articles whose holds had been caused purely by the corrected evidence artifacts, and retains holds where a genuine content discrepancy remains. Articles that remain held do so for genuine content discrepancies identifiable on inspection, not for the evidence-handling artifacts corrected here. (Re-verification sweep in progress; final figures will be recorded here on completion.)
A single result illustrates the intended outcome most directly. One held article, previously blocked by a false fabricated-link objection, produced no such objection under the corrected system—and, on the same run, independently surfaced a genuine content discrepancy (a date attribution) and was correctly still held. The false objection cleared while the real one did not. An evaluation in which every case passes would indicate the checker had been weakened; a retained, correct block is what distinguishes a correction from an over-correction.
Several limitations bound these results. The reported false-positive rates are measurements over defined article batches rather than an exhaustive census; a single global rate is not claimed. The controlled model comparison (Section 4.1) is drawn from runs of 204 and 107 articles that overlap but are not identical in membership, so it isolates the model's effect on shared evidence rather than a matched-pair rate. The changes described address the evidence supplied to the model; the residual rate at which the model itself misjudges a claim given complete evidence is a separate, model-level property, unchanged by this work and reported independently in the project's recall and calibration records. Finally, one prompt-level protection against adversarial content embedded in retrieved pages was only partially deployed; its remaining component is a known, scheduled item.
The elevated false-positive rate arose from two separable causes: a weaker model, since replaced, and an evidence pipeline that supplied incomplete evidence to the model without recording the incompleteness. The corrective work preserved the Desk's decision criteria unchanged and instead ensured that the evidence reaching the model is complete, or, where it cannot be, that its incompleteness is explicit rather than silent. The measured effect is a substantial reduction in incorrect objections with no loss of the Desk's ability to detect genuine problems, as confirmed by a retained control. For an automated source-fidelity checker, the general lesson is that a verdict rendered against silently-partial evidence is the primary avoidable error, and that making incompleteness a first-class, visible state is what prevents it. The same corrections yield a bounded but absolute guarantee for direct quotations, a class in which fabrication is both common and consequential: a confirmed verdict on a quotation cannot be produced by a fabrication.
This report draws on the project's internal change plan, two root-cause analyses, an independent design review, and the per-run verification records, with all figures carried across unchanged. Every mechanism described was confirmed against the live code and database, July 2026.