NYC News Digest

New York publishes in more languages than any city in the country. Its community and ethnic press, the three hundred outlets of the City's community media directory, reports every day in Spanish, Bengali, Urdu, Russian, Nepali, Malayalam, and a dozen more. Almost none of that reporting is read outside its own audience, and none of it could be seen in one place.

So questions that should be simple were unanswerable. Which story did the whole city converge on today? Which community newsroom broke something original that the dailies missed? Which outlets produce original journalism week over week, and which republish wire copy under a local name? I wanted to read the city whole. As far as I know, nothing that does this existed, so I built it.

The digest runs daily for the Bushwick Daily newsroom, where it opens the morning meeting: the day's edition is reviewed and its most consequential stories are argued over before assignments go out.

Each edition collects everything the city's press published in the past day, typically 1,200 or more articles from over ninety outlets, and sets it on one page that loads instantly and reads like a paper. Every publication in the batch is present. The list re-sorts four ways: by outlet, by story, chronologically, and by language, and the whole edition is searchable, full text, from one box.

The by-story view is the aggregate doing its work: coverage of the same event is grouped across outlets and across languages, so an English daily, a Spanish paper, and an Urdu weekly covering the same rent decision appear as one story with three tellings.

Language is detected per article, not per outlet, so a new publication in a new language needs no configuration. Headlines from the non-English press are machine-translated into English within minutes of ingestion, which is what lets stories cluster across languages and lets an editor scan the whole city at once. The original is never more than a keystroke away: translated articles carry an English/Original toggle, right-to-left scripts render right-to-left, and every non-English article links to a full translation of its source page.

The translation layer is engine-agnostic and records model provenance on every row. It currently runs M2M-100, an open multilingual model, entirely on newsroom hardware: no article text leaves the building.

Every edition opens with the numbers an editor actually wants: articles, outlets, words, total reading time, the non-English share and its translation coverage. Beneath them, the desk notes: the longest reads, outlets newly publishing or newly silent, the language mix, publish-to-scrape freshness, syndication, and byline concentration.

Read daily, these are a briefing. Read across weeks, they are a study of the city's press itself. The syndication watch separates original reporting from wire copy outlet by outlet, so it becomes visible which community publications produce original local journalism and which do not, a question the directory listing alone cannot answer. The most-covered clusters, tracked over time, map which conversations move through the whole city and which stay inside one language.

• CorpusA Postgres corpus of 1.8 million scraped articles feeds the digest; each edition reads the trailing day's batch. The scrape layer was left untouched: the digest and its translation store are purely additive.
• Language detectionStatistical detection per article, reinforced with Unicode script analysis for languages the detector lacks and a Devanagari disambiguation pass separating Nepali from Hindi. Verified against live output from each language community.
• Machine translationM2M-100 1.2B, an open-licensed multilingual model, quantized to int8 and served through CTranslate2 on CPU. Headline translation for a full edition completes in about a minute. Every translation row records its engine and model version, so the engine can be swapped without losing provenance.
• Story clusteringTF-IDF over translated headlines and ledes with union-find agglomeration, computed fresh at generation: 1,200 articles cluster in 0.6 seconds with no model dependency. Identical-headline detection separately flags syndicated copy.
• The artifactEach edition renders to a single self-contained HTML file: typefaces embedded, no external requests, working offline. All four views, full-text search, keyboard navigation, and per-reader progress run client-side. First paint is immediate; the document holds the full text of every article in the batch.
• DistributionEditions publish to a credentialed, unlisted, no-index path with atomic replacement, so the day's link never serves a torn file. A stable address always holds the latest edition.

The digest is an internal editorial tool. It republishes nothing; it reads the press in aggregate so a newsroom can exercise judgment about it. Access is credentialed and limited to the newsroom, and the editions shown here are illustrations from a live morning.