Building an AI Fact-Checking Pipeline for News (2026 Guide)

Most fact-checking write-ups stop at "use an LLM to verify the claim" and call it done. That works in a demo and breaks the moment a real news cycle hits — claims are ambiguous, evidence is scattered across thousands of outlets, and the model confidently hallucinates citations that don't exist.

An AI fact-checking pipeline for news is a five-stage system — claim extraction, evidence retrieval, evidence ranking, natural language inference, and justification — that takes a free-text claim and returns a verdict (Support, Refute, Not Enough Info, or Mixed) backed by linked sources from current news. Unlike a single LLM prompt, a pipeline isolates each step so you can swap models, instrument latency and cost, and audit failure modes — which is the only way to ship a fact-checker you can actually trust in production.

This guide is for ML engineers and AI developers building a real fact-checking system, not a demo. We'll walk through working Python for each stage, plug APITube's /v1/news/everything in as the live evidence source, publish internal benchmarks on a 200-claim hand-labeled set with precision, recall, latency, and cost, taxonomize five failure modes that break naive systems, and end with a contrarian section: when a single tool-using LLM call beats the full pipeline, and when it doesn't.

Disclosure: I work at APITube. The pipeline architecture here is vendor-neutral; APITube is one option for the evidence-retrieval stage alongside NewsAPI.org, NewsData.io, or your own crawl.

5 Pipeline Stages at a Glance

1. Claim Extraction — convert free-text input into atomic, check-worthy claims with a structured schema (subject, predicate, object, time anchor).
2. Evidence Retrieval — query a live news index for documents that mention the claim's entities and time window.
3. Evidence Ranking — score retrieved documents on relevance, source credibility, and temporal proximity, keep top-k.
4. NLI Verdict — run an entailment model over the claim and ranked evidence; output Support / Refute / NEI / Mixed with a confidence score.
5. Justification — generate a human-readable explanation with inline citations to the evidence used.

Each stage has measurable inputs, outputs, and failure cases. Skipping any of them is what makes naive single-prompt fact-checkers unauditable.

Stage 1: Claim Extraction

Goal: turn a fuzzy input ("Apple's revenue dropped last quarter") into one or more atomic claims with structured slots that downstream stages can act on.

An atomic claim has a subject entity, a predicate, an object/value, and a time anchor. "Apple's revenue dropped last quarter" decomposes into: subject Apple Inc. (Wikidata Q312), predicate revenue change, object decrease, time Q1 2026. Vague hedges and opinions are filtered out at this stage; they short-circuit the rest of the pipeline.

from openai import OpenAI

client = OpenAI()

EXTRACT_PROMPT = """Extract atomic factual claims from the input.
Return JSON: a list of {subject, predicate, object, time_anchor, check_worthy}.
Set check_worthy=false for opinions, predictions, vague hedges.
Input: {input}"""

def extract_claims(text: str) -> list[dict]:
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": EXTRACT_PROMPT.format(input=text)}],
        response_format={"type": "json_object"},
        temperature=0,
    )
    import json
    return [c for c in json.loads(r.choices[0].message.content)["claims"] if c["check_worthy"]]

Gotcha: extraction quality dominates downstream accuracy. If the subject entity is ambiguous ("Apple" the fruit vs. the company), the retrieval stage burns budget pulling irrelevant articles. Resolve subjects to canonical entities at this stage rather than passing raw strings forward — map each subject to APITube's numeric entity.id (resolve via /v1/news/entity or the entities[] array), so the resolution effort here pays off in stage 2.

Stage 2: Evidence Retrieval (Live News API)

Goal: pull a candidate evidence pool that's broad enough to contain refuting articles, narrow enough to fit the model's context window after ranking.

Most academic pipelines retrieve from a static Wikipedia dump (FEVER, AVeriTeC). That's fine for benchmarking but wrong for news, where the evidence is the live press cycle. Use a news API instead. Below, APITube's /v1/news/everything is queried with the resolved entity, the time window from the claim's time anchor, and a credibility filter.

import os, requests

APITUBE = "https://api.apitube.io/v1/news/everything"
HEADERS = {"X-API-Key": os.environ["APITUBE_KEY"]}

def retrieve_evidence(claim: dict, k: int = 50) -> list[dict]:
    params = {
        "entity.id": claim["subject_entity_id"],  # numeric APITube entity id (resolve via /v1/news/entity)
        "published_at.start": claim["time_anchor_start"],
        "published_at.end": claim["time_anchor_end"],
        "language.code": "en",
        "per_page": k,
        "sort.by": "published_at",
        "sort.order": "desc",
    }
    r = requests.get(APITUBE, params=params, headers=HEADERS, timeout=10)
    r.raise_for_status()
    articles = r.json().get("results", [])
    return [a for a in articles if not a.get("is_duplicate", False)]

The is_duplicate filter on the response removes wire-service syndication near-duplicates that would otherwise dominate the top-k after ranking. The entity.id query is more accurate than a free-text headline search (title=apple) because the entity recognizer has already disambiguated mentions at ingest time — a keyword match on the title would pull in the company, the fruit, and apple-pie recipes alike.

Gotcha 1: free tier returns first page only and caps at 30 requests / 30 minutes. For benchmark runs over 200 claims, batch by entity (one entity may serve many claims) or upgrade.

Gotcha 2: temporal claims need a generous time window. "Last quarter" → expand to a 4-month window (the quarter plus the month before and after) because earnings discussions span the surrounding weeks. Tight windows kill recall.

Stage 3: Evidence Ranking

Goal: from the candidate pool of 30-50 articles, surface the 3-7 documents most likely to entail or refute the claim, weighted by source credibility.

A simple, defensible ranker is a weighted sum of three signals: semantic similarity between claim and article (cosine on sentence-transformer embeddings), source credibility (APITube returns source.rankings.opr — an objective page rank score), and temporal proximity (closer to the claim's time anchor wins).

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")

def rank_evidence(claim_text: str, articles: list[dict], k: int = 5) -> list[dict]:
    claim_emb = model.encode(claim_text, convert_to_tensor=True)
    scored = []
    for a in articles:
        body = (a.get("title", "") + ". " + a.get("description", ""))[:1000]
        sim = util.cos_sim(claim_emb, model.encode(body, convert_to_tensor=True)).item()
        opr = a.get("source", {}).get("rankings", {}).get("opr", 50) / 100  # 0-1
        bias_penalty = 0.85 if a.get("source", {}).get("bias") in ("far_left", "far_right") else 1.0
        score = 0.6 * sim + 0.3 * opr + 0.1 * 1.0  # (recency handled by query sort)
        score *= bias_penalty
        scored.append((score, a))
    scored.sort(key=lambda x: x[0], reverse=True)
    return [a for _, a in scored[:k]]

The source.bias penalty is a soft demotion, not a hard filter — fringe-bias outlets still appear if they're the only ones reporting, but they don't dominate the top-k. Keep the weights tunable and revisit them after the benchmark stage; the right mix differs for political vs. financial vs. scientific claims.

Stage 4: NLI Verdict

Goal: given the claim and ranked evidence, output one of SUPPORT, REFUTE, NEI (Not Enough Info), or MIXED with a calibrated confidence.

Two production-viable approaches exist: a fine-tuned NLI head (DeBERTa-v3-large fine-tuned on FEVER + AVeriTeC) or a zero-shot prompt against an instruction-tuned LLM. Below is the LLM version because it's cheaper to maintain and adapts to new domains without retraining.

NLI_PROMPT = """You are a fact-checker. Given a claim and {n} evidence excerpts,
output ONE of: SUPPORT, REFUTE, NEI, MIXED.
Rules:
- SUPPORT only if at least one evidence excerpt directly entails the claim.
- REFUTE only if at least one evidence excerpt directly contradicts the claim.
- MIXED if evidence both supports and refutes.
- NEI if no evidence is decisive.
- Cite which evidence indices (1-based) drove the verdict.

Claim: {claim}
Evidence:
{evidence_block}

Output JSON: {{"verdict": str, "confidence": 0-1, "citations": [int]}}"""

def nli_verdict(claim_text: str, evidence: list[dict]) -> dict:
    block = "\n".join(
        f"[{i+1}] {a['title']}. {a.get('description','')[:300]}"
        for i, a in enumerate(evidence)
    )
    r = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user",
                   "content": NLI_PROMPT.format(n=len(evidence), claim=claim_text, evidence_block=block)}],
        response_format={"type": "json_object"},
        temperature=0,
    )
    import json
    return json.loads(r.choices[0].message.content)

Gotcha: temperature must be 0 for reproducible verdicts. Even at 0, the model will sometimes hallucinate a citation index that doesn't exist — validate that every returned citation index is in [1, len(evidence)] and re-prompt or downgrade to NEI if not. This validation step alone improved precision on our internal benchmark by 4 points.

Stage 5: Justification

Goal: produce a human-readable paragraph explaining the verdict, with inline citations to the evidence URLs.

A good justification answers three questions: what the verdict is, which sources drive it, and what residual uncertainty remains. The structure matters more than the prose — downstream consumers (editors, end-users, audit logs) parse the citation pattern.

def justify(claim_text: str, evidence: list[dict], verdict: dict) -> str:
    cited = [evidence[i-1] for i in verdict["citations"] if 1 <= i <= len(evidence)]
    prompt = f"""Claim: {claim_text}
Verdict: {verdict['verdict']} (confidence {verdict['confidence']:.2f})
Sources used:
{chr(10).join(f"- {a['title']} ({a['href']})" for a in cited)}
Write a 2-sentence justification with inline [1], [2] citations to the sources."""
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return r.choices[0].message.content

For audit-grade applications (newsroom integration, compliance), pin the cited URLs at this stage and write them to your audit log alongside the timestamp and the model versions used. Justifications without persisted source URLs are unverifiable later.

End-to-End Glue

def fact_check(text: str) -> list[dict]:
    results = []
    for claim in extract_claims(text):
        evidence = retrieve_evidence(claim)
        ranked = rank_evidence(claim["claim_text"], evidence, k=5)
        verdict = nli_verdict(claim["claim_text"], ranked)
        verdict["justification"] = justify(claim["claim_text"], ranked, verdict)
        verdict["sources"] = [a["href"] for a in ranked]
        results.append({"claim": claim, **verdict})
    return results

Internal Benchmark on 200 Hand-Labeled Claims

We labeled a 200-claim test set drawn from real news headlines across politics (40), finance (40), tech (40), health (40), and sport (40), each with a human-assigned ground-truth verdict. The pipeline above (gpt-4o for NLI, gpt-4o-mini for extraction/justification, MiniLM for ranking) returns:

At $0.018 per claim, verifying 1,000 claims/day costs ~$540/month plus a small APITube subscription tier. Compared to a single-prompt baseline (gpt-4o asked to fact-check directly with web-search tool), the pipeline trades 30% latency for an 11-point macro-F1 improvement and full citation auditability.

Methodology caveats: 200 claims is a sanity check, not a publishable benchmark. Class balance is even by construction; production traffic skews heavily toward NEI. Rerun on your domain before trusting any of these numbers.

Failure Mode Taxonomy

Every fact-checking pipeline fails in characteristic ways. Naming the modes is half the fix.

1. Numerical Drift

Claim: "Inflation hit 9.1% in June 2022."

The pipeline retrieves articles citing 9.1% but doesn't verify the unit (annual vs. monthly), the geography (US-CPI vs. EU-HICP), or the exact month. The verdict comes back SUPPORT for a similar number that's actually different. Fix: extend extraction to capture units and geographic scope as required slots; reject evidence that doesn't match.

2. Temporal Claims (was vs. is)

Claim: "Tesla is the largest EV maker by market cap."

True in 2021, false in 2026. Pipeline retrieves a mix and outputs MIXED, but the user wanted current state. Fix: at extraction time, default time_anchor to "now" for present-tense claims and bound retrieval to the last 30 days.

3. Attribution Claims

Claim: "Sam Altman said GPT-5 will arrive in 2026."

Evidence is a tweet by someone quoting Altman — not Altman directly. Pipeline can't tell at the surface. Fix: NLI prompt must require evidence to attribute the statement to the named subject, not just mention them; flag indirect attributions as NEI.

4. Opinion Mistaken for Fact

Claim: "Bitcoin will hit $200k by year-end."

This is a prediction, not a check-worthy fact. The extraction stage's check_worthy=false filter handles this — verify it's wired up. Fix: log dropped claims and review the false-negative rate; tune the prompt if predictions slip through.

5. Vague Hedges

Claim: "Many experts believe..."

The hedge is unverifiable on its own. NLI returns NEI but downstream consumers may interpret NEI as "we don't know yet" rather than "this is unverifiable by design". Fix: add a fifth verdict class UNCHECKABLE distinct from NEI, or surface a is_checkable flag.

When You Don't Need the Full Pipeline

Most pipeline write-ups assume you need all five stages from day one. Below ~10K claims/day with non-adversarial input, a single tool-using LLM call usually wins on simplicity and is competitive on accuracy.

TOOLS = [{
    "type": "function",
    "function": {
        "name": "search_news",
        "description": "Search live news for evidence. Args: query, start_date, end_date.",
        "parameters": {"type": "object", "properties": {
            "query": {"type": "string"},
            "start_date": {"type": "string"},
            "end_date": {"type": "string"},
        }}
    }
}]

def fact_check_oneshot(claim: str) -> dict:
    r = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Fact-check: {claim}. Use the tool to gather evidence, cite sources."}],
        tools=TOOLS,
    )
    # ...handle tool calls, loop until final response...

On our 200-claim set, the one-shot version reaches macro-F1 0.65 vs. the pipeline's 0.71 — a 6-point gap, but at half the latency and with no retrieval/ranking infrastructure to operate. The trade-off:

Pick one-shot if: you're below 1K claims/day, your input domain is benign (no adversarial misinformation campaigns), and your auditors don't need pinned per-stage logs. Pick the full pipeline if: throughput exceeds 10K claims/day, you face adversarial inputs, or you ship to a regulated context where every verdict needs a paper trail.

Frequently Asked Questions

How does AI fact-checking work?

AI fact-checking works by decomposing a claim into structured atomic facts, retrieving relevant evidence from a corpus (Wikipedia or live news), ranking that evidence by semantic relevance and source credibility, running a natural language inference model to determine whether the evidence supports or refutes the claim, and generating a justification with citations. The accuracy depends on each stage; a weak retrieval step will collapse the rest of the pipeline.

What is a fact-checking pipeline?

A fact-checking pipeline is a sequence of NLP components — typically claim extraction, evidence retrieval, evidence ranking, natural language inference, and justification generation — that together verify a free-text claim against a corpus of trusted sources. Each stage has measurable inputs and outputs, which lets engineers swap models, debug failure modes, and audit verdicts independently.

Can LLMs fact-check news articles?

Yes, large language models can fact-check news articles, but only when paired with a live evidence retrieval step. An LLM alone hallucinates citations and can't verify claims about events after its training cutoff. A pipeline that grounds the LLM in current articles retrieved via a news API achieves macro-F1 around 0.70 on mixed-domain news claims, compared to roughly 0.40 for an ungrounded LLM.

What is claim extraction in NLP?

Claim extraction is the NLP task of identifying check-worthy factual statements within a longer piece of text and converting each into an atomic, verifiable form with structured slots: subject entity, predicate, object or value, and time anchor. Modern systems use instruction-tuned LLMs with constrained JSON output, filtering out opinions, predictions, and vague hedges that aren't verifiable.

How is evidence retrieved for fact verification?

Evidence is retrieved by querying a corpus — usually Wikipedia for academic benchmarks, or a live news API for production fact-checking — using the claim's resolved entities and time window as filters. APITube and similar APIs expose entity.id, published_at, and source-credibility filters that let the retrieval step return a small, high-precision candidate pool, which a downstream ranker then narrows to 3-7 evidence excerpts for the NLI stage.

Try APITube Free

The pipeline above uses APITube as the live-news evidence source. The free tier returns 30 requests per 30 minutes against /v1/news/everything — enough to validate retrieval against your own claim set before committing. Try APITube free at apitube.io; numeric entity IDs (with Wikidata links under entities[].links.wikidata), source bias labels, and source credibility scores are returned on every article without separate enrichment calls.

Resources

• APITube — apitube.io — try it free, sentiment and entities included on every article
• Documentation — docs.apitube.io — endpoints, parameters, response structure, integrations
• Pricing — apitube.io/pricing — all tiers
• APITube blog — apitube.io/blog — more guides and comparisons