Entity Extraction from News Articles with spaCy

Pulling people, organizations, and places out of news text is a 5-minute spaCy demo and a 5-week production project. The demo is "load en_core_web_sm, call nlp(text), print doc.ents." The production project is everything after: which model to ship, how to disambiguate "Apple," when to retrain on your domain, and whether to write the pipeline at all when an API already returns entities per article.

Entity extraction from news articles with spaCy identifies spans of text that refer to real-world entities (people, organizations, locations, dates, monetary values) and labels them with a category, using spaCy's transition-based neural NER pipeline trained on OntoNotes 5. News-domain entity extraction has to handle dense ORG mentions, ambiguous proper nouns ("Washington," "Apple," "Jordan"), and a steady stream of new entities the pre-trained model has never seen — which means model selection, evaluation, and entity linking matter more than the extraction call itself.

This guide is for data scientists who already know what NER is and want the news-domain answers the generic tutorials skip.

What you'll build

A production-grade entity extraction news articles spaCy pipeline with six deliverables:

1. Working extraction code on a real news article
2. A model-variant decision (en_core_web_sm vs _lg vs _trf) backed by a benchmark
3. Per-entity-type evaluation (precision/recall for PERSON, ORG, GPE)
4. Wikidata-based entity linking that resolves "Apple" to the right QID
5. A minimal custom-NER training run with spacy train
6. A build-vs-buy decision matrix with numeric thresholds

Disclosure

We're APITube. Our /v1/news/everything endpoint already ships entities with Wikidata links per article. We still wrote this guide because rolling your own spaCy pipeline is the right call sometimes — you need custom entity types, you have a strict latency budget, or you can't ship article text to a third party. We'll tell you exactly when. The patterns below work standalone or alongside any news source.

Prerequisites

• Python 3.10+
• spacy>=3.7, pandas, requests
• A small set of news articles to test on (use APITube's free tier — 30 requests per 30 minutes — or paste from a URL)
• ~5 GB free disk if you plan to download en_core_web_trf (Transformer model)
• Familiarity with token-level NLP and basic precision/recall

Install the three default English models so we can compare them later:

pip install spacy pandas requests
python -m spacy download en_core_web_sm
python -m spacy download en_core_web_lg
python -m spacy download en_core_web_trf

Step 1 — Extract entities from a news article

Start with a small function that takes article text and returns entities grouped by type. Use a real news headline + lede so the example reflects the domain.

import spacy
from collections import defaultdict

nlp = spacy.load("en_core_web_sm")

ARTICLE = """
The U.S. Department of Justice on Wednesday announced charges against
two former Tesla engineers accused of leaking battery designs to BYD.
Attorney General Merrick Garland said the indictment, returned in the
Northern District of California, marks the largest trade-secrets case
involving Apple's electric-vehicle competitors this year.
"""

def extract_entities(text):
    doc = nlp(text)
    grouped = defaultdict(list)
    for ent in doc.ents:
        grouped[ent.label_].append(ent.text)
    return dict(grouped)

print(extract_entities(ARTICLE))
# {'ORG': ['The U.S. Department of Justice', 'Tesla', 'BYD', 'Apple'],
#  'DATE': ['Wednesday', 'this year'],
#  'PERSON': ['Merrick Garland'],
#  'GPE': ['the Northern District of California']}

Two things to notice. First, "the Northern District of California" gets tagged GPE — borderline correct, depends on your downstream task. Second, "Apple" is tagged ORG without disambiguation; you don't know if it means Apple Inc. or the apple producer mentioned in a fruit-export article from yesterday. Both points are why the next four steps exist.

What is named entity recognition in spaCy?

Named entity recognition (NER) in spaCy is a token-classification task that labels contiguous spans of text with predefined categories such as PERSON, ORG, GPE, DATE, and MONEY. The default English pipelines use a transition-based neural model trained on OntoNotes 5; you call nlp(text) and read doc.ents to get spans, character offsets, and labels back as Span objects.

Step 2 — Choose a model: en_core_web_sm vs _lg vs _trf

Most tutorials pick en_core_web_sm and never mention there are two better options. The cost is real — _trf is ~50× slower and ~50× larger — but for news, where headlines are short and stakes per article are high, it's frequently worth it.

Numbers are indicative — verify on your data before committing. The right answer is rarely "always _trf": for a streaming feed of 1,000 articles per minute, _sm on CPU beats a queue backup on _trf. For a quarterly report extracting 500 named individuals, _trf saves a human review pass.

What is the difference between en_core_web_sm, _lg, and _trf?

The three default English spaCy pipelines differ in architecture and resource cost: en_core_web_sm is a small CNN tagger with no pre-trained word vectors (~50 MB, ~5 ms/doc, F1 ~0.84); en_core_web_lg is the same architecture plus 685k GloVe vectors (~750 MB, ~15 ms/doc, F1 ~0.86); en_core_web_trf swaps the architecture for a RoBERTa-base transformer (~2 GB, ~200 ms/doc on CPU, F1 ~0.91). The transformer lifts F1 by roughly 5 points over _sm but costs about 50× the latency and 30× the memory.

Step 3 — Evaluate accuracy on your own news data

Don't trust generic benchmarks. Evaluate on your data because news varies wildly — a markets feed has tickers and CFOs, a sports feed has players and clubs, a politics feed has officials and bills. The right model is the one with the highest F1 on the entity types you actually care about.

Hand-label 100 sentences from your source. Use the spaCy Prodigy-style annotation if you can; if not, JSONL with [(start, end, label), ...] works. Then:

from spacy.scorer import Scorer
from spacy.training import Example

def evaluate(nlp, gold_data):
    examples = [Example.from_dict(nlp.make_doc(t), {"entities": ents}) for t, ents in gold_data]
    return Scorer().score(examples)["ents_per_type"]

Indicative results on a 100-sentence political-news sample (yours will differ):

ORG is consistently the weakest type across all three models — that's where retraining (Step 5) earns its cost.

How accurate is spaCy NER on news text?

On standard news corpora such as CoNLL-2003 and OntoNotes, default spaCy English pipelines achieve approximately 0.84 F1 (en_core_web_sm), 0.86 F1 (_lg), and 0.91 F1 (_trf). On real news, accuracy degrades 3–8 points depending on domain shift; ORG is typically the worst-performing type, missing niche companies, financial tickers, and political bodies that didn't appear in the training set.

Step 4 — Disambiguate entities with Wikidata

Extraction labels "Apple" as ORG but doesn't tell you which Apple. For news, that's a real problem — your dataset will mix Apple Inc. (Q312), the fruit (Q89), and Apple Corps (Q13219). Entity linking resolves the surface form to a stable Wikidata QID.

The simplest approach: for each ORG/PERSON span, query Wikidata Search and pick the highest-ranked candidate that has a P31 (instance of) compatible with your label.

import requests

WIKIDATA_API = "https://www.wikidata.org/w/api.php"

def link_entity(name, expected_type):
    params = {
        "action": "wbsearchentities",
        "search": name,
        "language": "en",
        "format": "json",
        "limit": 5,
    }
    candidates = requests.get(WIKIDATA_API, params=params).json().get("search", [])
    for c in candidates:
        if expected_type.lower() in (c.get("description") or "").lower():
            return {"qid": c["id"], "label": c["label"], "description": c["description"]}
    return None

print(link_entity("Apple", "company"))
# {'qid': 'Q312', 'label': 'Apple Inc.', 'description': 'American multinational technology company...'}

For production, replace this with spaCy's built-in EntityLinker and a knowledge-base graph (KnowledgeBase class) — better recall, no rate limits, but a heavier setup. Also cache aggressively: Wikidata API allows generous use but the network round-trip dominates. APITube returns Wikidata links directly in results[].entities[].links, so if you ingest from there you skip this step entirely.

Step 5 — Train a custom NER on news

Default models miss what your domain considers important — drug names in pharma news, military equipment in conflict reporting, financial tickers in markets coverage. Add a custom label or improve an existing one with spacy train.

Minimal config.cfg (generate the full version with python -m spacy init config):

[paths]
train = "./train.spacy"
dev = "./dev.spacy"

[nlp]
lang = "en"
pipeline = ["tok2vec", "ner"]

[components.ner]
factory = "ner"

[training]
max_epochs = 30
patience = 5
eval_frequency = 200

Convert your hand-labeled data into spaCy's binary format:

from spacy.tokens import DocBin
import spacy

nlp = spacy.blank("en")
db = DocBin()

TRAIN = [
    ("Pfizer reported quarterly revenue of $13.2 billion.",
     {"entities": [(0, 6, "ORG"), (38, 51, "MONEY")]}),
    # ... 200+ more examples for a meaningful run
]

for text, ann in TRAIN:
    doc = nlp.make_doc(text)
    ents = [doc.char_span(s, e, label=l) for s, e, l in ann["entities"]]
    doc.ents = [e for e in ents if e]
    db.add(doc)
db.to_disk("./train.spacy")

Then train and evaluate:

python -m spacy train config.cfg --output ./out --paths.train ./train.spacy --paths.dev ./dev.spacy

On a 200-sentence pharma news set, expect to lift ORG F1 from ~0.74 to ~0.85 after 20 epochs — but the gain is entirely on the labels you trained, and you risk regressing PERSON or GPE if your data is one-sided. Always keep a held-out evaluation set, and prefer fine-tuning the existing model (base_model = "en_core_web_lg") over starting from blank unless you have thousands of examples.

Should you train a custom spaCy NER model for news?

Train a custom spaCy NER model for news only when default models fail your specific entity types and you have at least 200 hand-labeled sentences for training plus 50 for held-out evaluation; below that threshold the model overfits and underperforms en_core_web_lg or _trf out of the box. For most general-news pipelines, model selection (Step 2) and entity linking (Step 4) deliver more gain than custom training.

Step 6 — End-to-end pipeline: load → extract → link → store

Tie it together. Pull articles from APITube, run them through your chosen spaCy model, link to Wikidata, deduplicate by QID, and persist:

import requests, spacy, sqlite3
from collections import Counter

nlp = spacy.load("en_core_web_lg")
db = sqlite3.connect("entities.db")
db.execute("CREATE TABLE IF NOT EXISTS mentions (qid TEXT, name TEXT, type TEXT, article_id TEXT, count INT)")

def fetch_articles(query, api_key, limit=20):
    res = requests.get(
        "https://api.apitube.io/v1/news/everything",
        params={"title": query, "language.code": "en", "per_page": limit},
        headers={"X-API-Key": api_key},
    )
    res.raise_for_status()
    return res.json()["results"]

def process(article):
    doc = nlp(article["title"] + ". " + (article.get("description") or ""))
    counts = Counter((ent.text, ent.label_) for ent in doc.ents
                      if ent.label_ in {"PERSON", "ORG", "GPE"})
    for (name, label), count in counts.items():
        linked = link_entity(name, label.lower())
        qid = linked["qid"] if linked else None
        db.execute(
            "INSERT INTO mentions VALUES (?, ?, ?, ?, ?)",
            (qid, name, label, article["id"], count),
        )
    db.commit()

for a in fetch_articles("election", api_key="..."):
    process(a)

At this scale (20–100 articles), this runs in seconds on _lg. At 10,000 articles/day, batch with nlp.pipe(texts, batch_size=64), cache the Wikidata calls, and consider switching to APITube's prebuilt entities field — which brings us to the build-vs-buy question.

How do you extract entities from a news article in Python?

To extract entities from a news article in Python, install spaCy and an English model with pip install spacy && python -m spacy download en_core_web_lg, load the pipeline with nlp = spacy.load("en_core_web_lg"), run it on the article text, and iterate doc.ents for spans with .text (surface form), .label_ (entity type), and .start_char / .end_char (offsets). For news, prefer _lg or _trf over _sm because ORG accuracy is materially higher.

Build vs buy: decision matrix

Honest take from the API vendor: don't build the pipeline yourself when a news API ships entities for free with each article. Build it when one of these four axes pushes you off the buy column:

At fewer than ~5,000 articles per day with no custom-type need and Wikidata linking required, the news-API path is faster to ship and cheaper to run. Beyond ~50,000 articles per day with custom types, the spaCy pipeline pays for itself within a quarter. The grey zone in between is where a hybrid wins — use the API's entities as a seed, run spaCy only on articles that fail a confidence threshold or contain custom labels.

Try it free

Run this guide's code against APITube's free tier — 30 requests per 30 minutes is enough to extract entities from a couple hundred recent articles and decide whether you need to build your own pipeline. Every result already includes entities with Wikidata links, so you can compare the API's output against your spaCy run side by side. Try APITube free at apitube.io.

Ressources

• APITube — apitube.io — essayez-le gratuitement, sentiment et entités inclus sur chaque article
• Documentation — docs.apitube.io — points de terminaison, paramètres, structure de réponse, intégrations
• Pricing — apitube.io/pricing — all tiers
• Blog APITube — apitube.io/blog — plus de guides et de comparaisons

Related guides:

• News Sentiment Analysis in Python: VADER, FinBERT, TextBlob