The Problem
News articles are long, written for general audiences, and hard to evaluate at scale without human raters. The goal was a system that could fetch live articles on any topic, rank them by relevance, summarize the top ones with Claude, score output quality automatically, and if quality fell short, refine the query and try again without any human intervention.
Graph Architecture
START → scrape → relevance → summarize → evaluate → router
|
quality low? → refine → scrape (up to 3x)
quality OK? → END
01scrapeFetches articles from NewsAPI and scrapes full text from each URL via BeautifulSoup. Passes raw article text into the graph state.
02relevanceClaude scores each article against the search topic. Top 3 by relevance score are kept for summarisation. Low-signal articles are dropped before any expensive summarize call.
03summarizeClaude Haiku summarises each top article in 2–4 sentences. Prompt-engineered for factual fidelity with no padding.
04evaluateTwo LLM-as-Judge chains score faithfulness and conciseness independently. ROUGE-L measures content recall against the source. All three scores stored per article in graph state.
05router (conditional edge)If average quality meets the threshold, exit. If below threshold and iterations are under 3, route to refine. Max iterations guard prevents infinite loops.
06refineClaude rewrites the search query to be more specific before the next scrape iteration. The graph adapts the query rather than retrying blindly.
Key Findings
LLM-as-Judge and ROUGE-L are complementary, not redundant
ROUGE-L measures surface-level content recall and penalises paraphrasing even when meaning is preserved. LLM-as-Judge catches faithfulness violations and padding that ROUGE scores as fine. Running both catches failure modes that neither finds alone.
Relevance filtering is the highest-leverage step
Ranking articles before passing them to Claude meant the summariser was always working on the right material. Removing this filter degraded faithfulness scores significantly. The model was faithful to the wrong content.
Query refinement outperformed prompt tuning
When summaries scored low, rewriting the NewsAPI query produced a bigger quality jump than adjusting the summarisation prompt. Bad input was the bottleneck. The model was doing its job; the search was pulling weak articles.
What I'd improve
Swap ROUGE-L for BERTScore for semantic similarity beyond surface overlap. Add parallel summarisation across articles to cut latency. Extend the eval rubric to cover source attribution. Add streaming output in the Streamlit UI.
