Implementing News Vector Search with Pinecone

Last updated: 2026-05-03.

A user types "EU climate deal" into your news search box. Your Postgres LIKE '%EU climate deal%' query returns nothing. The headlines that actually matter say "Brussels methane talks collapse" and "European Commission delays carbon framework" — different words, same story. That gap is the entire reason semantic search exists, and it's why news products end up reaching for vector databases.

This guide is for ML engineers shipping a real semantic-search-over-news system. We'll build the pipeline end-to-end against Pinecone Serverless using a live news API, settle the metadata schema that actually matters for news, address the freshness story top-3 vendor docs ignore, and run the cost math at 100K, 1M, and 10M articles so you can take it to a budget call.

Disclosure: I work on APITube. The pipeline below uses APITube as the data source, but the architecture is portable to any news API that exposes article body, source authority, and timestamps.

Key takeaways

• News vector search is a six-step pipeline (pull → chunk → embed → metadata → upsert → query+rerank), not a pinecone.upsert one-liner — and the metadata layer is where news-specific value lives.
• Vectors-are-forever doesn't fit news. Default to a 30-day hot window with time-decay scoring at query time and a TTL deletion policy for everything older.
• At 1M news articles in index, expect roughly $20-40/month in storage on Pinecone Serverless plus $50-200/month in reads at moderate query volume, using 2026 Standard pricing.

What News Vector Search Actually Is

News vector search is a retrieval system that converts each article into a high-dimensional embedding, stores those embeddings in a vector database alongside metadata, and answers user queries by finding semantically similar articles instead of literal keyword matches. Operationally, it's three layers: an embedding model (OpenAI, Cohere, Voyage), a vector database (Pinecone in our case), and a metadata schema rich enough to filter by source authority, recency, language, and topic without round-tripping through your application database.

The reason news pushes back on the standard vector-search tutorial is that articles aren't static documents. They're time-stamped, language-stamped, source-ranked, and they go stale. A movie-review tutorial gets to ignore all of that; a production news pipeline cannot.

The 6-Step Pipeline

This is the order of operations. Each step gets its own H2 below; here's the map.

1. Pull articles from the news API with pagination
2. Chunk article bodies to fit the embedding model's context window
3. Generate embeddings in batches with cost-aware batch sizing
4. Design the metadata schema — what to store alongside each vector
5. Upsert into Pinecone with the right batch size and namespace strategy
6. Query with metadata filters and a time-decay rerank

If you skip step 4 and ship a {"text": "..."} metadata, you'll come back to this article in two weeks. Don't.

Step 1: Pull Articles from APITube

The data layer. You need full article bodies (not just titles), reliable pagination, and the metadata fields that will power your filters. APITube's /v1/news/everything returns all of that on every record.

import os, time, requests
from typing import Iterator

API_BASE = "https://api.apitube.io/v1/news/everything"
HEADERS = {"X-API-Key": os.environ["APITUBE_KEY"]}

def fetch_articles(query: str, lang: str = "en", per_page: int = 100) -> Iterator[dict]:
    params = {"title": query, "language.code": lang, "per_page": per_page}
    url = API_BASE
    while True:
        for attempt in range(3):
            r = requests.get(url, params=params if url == API_BASE else None,
                             headers=HEADERS, timeout=15)
            if r.status_code == 200:
                break
            if r.status_code == 429:
                time.sleep(2 ** attempt * 5)
                continue
            r.raise_for_status()
        data = r.json()
        for article in data.get("results", []):
            yield article
        if not data.get("has_next_pages"):
            break
        url = data["next_page"]
        params = None

For a daily ingest, run this against your topic queries and pipe the output straight into the chunk → embed → upsert chain. For a one-time backfill, persist articles to disk first — embedding cost dominates and you don't want to repeat it on a pipeline retry.

Step 2: Chunk Article Bodies

OpenAI's text-embedding-3-large takes up to 8,191 tokens per input. Most news articles fit, but long-form investigations and wire-reports-with-context regularly exceed it. More importantly, embedding a 6,000-token article into a single vector blurs the meaning — a query about the third paragraph competes against the embedding of all twelve.

The fix is overlapping chunks of ~500 tokens with ~50-token overlap. Each chunk becomes its own vector, with a chunk_id and parent article_id in metadata so you can collapse on the way out.

import tiktoken

ENC = tiktoken.encoding_for_model("text-embedding-3-large")

def chunk_text(text: str, chunk_tokens: int = 500, overlap: int = 50) -> list[str]:
    tokens = ENC.encode(text)
    if len(tokens) <= chunk_tokens:
        return [text]
    chunks = []
    step = chunk_tokens - overlap
    for start in range(0, len(tokens), step):
        chunk = tokens[start:start + chunk_tokens]
        chunks.append(ENC.decode(chunk))
        if start + chunk_tokens >= len(tokens):
            break
    return chunks

500 tokens is a reasonable default for news. Smaller chunks lose narrative context; larger chunks bury the signal you're trying to retrieve.

Step 3: Generate Embeddings

Batch size matters here for throughput and cost. OpenAI accepts up to 2,048 inputs per call but 100-500 is the practical sweet spot — large enough to amortise the round-trip, small enough to keep retries cheap when one input has a problem.

from openai import OpenAI

client = OpenAI()
EMBED_MODEL = "text-embedding-3-large"
EMBED_DIM = 3072

def embed_batch(texts: list[str]) -> list[list[float]]:
    resp = client.embeddings.create(model=EMBED_MODEL, input=texts)
    return [d.embedding for d in resp.data]

At $0.13 per 1M tokens for text-embedding-3-large in 2026, embedding 1M articles averaging 800 tokens of body text costs roughly $104 one-time. We'll factor this into the cost table later.

Step 4: Designing the Metadata Schema

This is the section that separates a news pipeline from a hello-world. Pinecone metadata is what powers your query filters without round-tripping to Postgres — and the right schema means you can answer "high-quality English-language coverage of methane policy from the last 7 days" with a single query.

Pinecone's metadata limits cap individual values at ~40KB and total metadata at the same per-record. A long entities list will eat that budget — keep it to the top 5-10 entities by frequency, not every name spaCy finds.

Step 5: Upsert to Pinecone

Pinecone Serverless takes up to 1,000 vectors per upsert call, but the network round-trip dominates beyond ~100, so 100-200 per batch is the sweet spot. Use namespaces to partition by language or by index version (when you re-embed with a new model, you'll thank yourself).

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key=os.environ["PINECONE_KEY"])
INDEX_NAME = "news-vector-search"

if INDEX_NAME not in pc.list_indexes().names():
    pc.create_index(
        name=INDEX_NAME,
        dimension=EMBED_DIM,
        metric="cosine",
        spec=ServerlessSpec(cloud="aws", region="us-east-1"),
    )

index = pc.Index(INDEX_NAME)

def upsert_articles(articles: list[dict], namespace: str = "en"):
    vectors = []
    for article in articles:
        chunks = chunk_text(article.get("body") or article.get("title", ""))
        embeddings = embed_batch(chunks)
        for i, (chunk, emb) in enumerate(zip(chunks, embeddings)):
            entities = [e["name"] for e in article.get("entities", [])[:8]]
            vectors.append({
                "id": f"{article['id']}_chunk_{i}",
                "values": emb,
                "metadata": {
                    "article_id": article["id"],
                    "chunk_id": i,
                    "source_domain": article["source"]["domain"],
                    "source_opr": (article["source"].get("rankings") or {}).get("opr", 0),
                    "published_at": int(time.mktime(time.strptime(
                        article["published_at"], "%Y-%m-%dT%H:%M:%SZ"))),
                    "language": article.get("language", {}).get("code", "en"),
                    "story_id": (article.get("story") or {}).get("id", ""),
                    "entities": entities,
                    "title": article["title"][:200],
                },
            })
    for batch_start in range(0, len(vectors), 100):
        batch = vectors[batch_start:batch_start + 100]
        index.upsert(vectors=batch, namespace=namespace)

The title field in metadata is a small luxury — it lets you show search results without a separate fetch from your application database. Trim to 200 chars to stay well within the 40KB metadata cap.

Step 6: Query with Metadata Filters and Time-Decay Rerank

Cosine similarity finds semantically close articles. For news, that's necessary but not sufficient — a perfect semantic match from three years ago is rarely what the user wants. The fix is a time-decay rerank applied after Pinecone returns the top-K candidates.

import math

def search_news(query: str, days_back: int = 30, top_k: int = 50,
                final_k: int = 10) -> list[dict]:
    query_emb = embed_batch([query])[0]
    cutoff = int(time.time()) - days_back * 86400
    res = index.query(
        vector=query_emb,
        top_k=top_k,
        namespace="en",
        filter={
            "published_at": {"$gte": cutoff},
            "source_opr": {"$gte": 50},
            "language": {"$eq": "en"},
        },
        include_metadata=True,
    )
    now = int(time.time())
    HALF_LIFE_DAYS = 7
    decay_const = math.log(2) / (HALF_LIFE_DAYS * 86400)
    seen_articles = {}
    for match in res["matches"]:
        meta = match["metadata"]
        age_seconds = now - meta["published_at"]
        decay = math.exp(-decay_const * age_seconds)
        score = match["score"] * decay
        article_id = meta["article_id"]
        if article_id not in seen_articles or seen_articles[article_id]["score"] < score:
            seen_articles[article_id] = {"score": score, "metadata": meta}
    ranked = sorted(seen_articles.values(), key=lambda x: -x["score"])
    return ranked[:final_k]

The half-life of 7 days means an article 7 days old gets half the cosine score; 14 days old gets a quarter. Tune the half-life to your product — a research crawler wants 90 days, a breaking-news app wants 1 day. Also note the dedup-by-article_id step: chunks of the same article competing for slots is the most common avoidable failure in news vector retrieval.

Freshness, TTL, and Re-Embedding

News vectors have a useful life. Three policies that nobody else discusses but everyone needs:

Hot window: 30 days. Most news search queries care about the last month. Beyond that, both query volume and embedding relevance drop. Set this as your default published_at filter floor; require an explicit override to search older.

Time-decay scoring at query time. The half-life code above. Cosine similarity alone treats 2026 and 2023 articles identically; with decay, recency wins ties. This is a query-time concern, not a storage concern — old vectors stay in the index until you actively delete them.

TTL deletion policy. A nightly job that deletes vectors with published_at < now - 90 days. Pinecone Serverless storage costs $0.33/GB/mo, and a 3072-dim float vector with metadata runs ~13KB. At 10M articles, that's 130GB = $43/mo just for storage you may not be querying. The deletion job is one filter query and a bulk delete; run it nightly, not weekly.

When to re-embed. Only when you change embedding models. Switching from text-embedding-3-large to a future model means new dimensions, new vector space — they don't cohabitate. Pattern: create a second index in parallel, dual-write for a week, cut over, delete the old. Don't try to "upgrade in place."

Cost at News-Scale: 100K, 1M, 10M Articles (2026)

The math nobody runs. Assumptions: 3,072-dim vectors (text-embedding-3-large), ~3 chunks per article on average, ~13KB per vector including metadata, query rate of 100k/day with metadata filters, all using 2026 Pinecone Serverless Standard pricing ($0.33/GB/mo storage, $4.50/1M write units, $18/1M read units).

A query with metadata filtering counts as 5-10 read units, not 1, per Pinecone's cost docs. The reads number above assumes 6 RU/query; tighter filters get cheaper, looser get more expensive. Embedding cost is one-time at ~$0.13/1M tokens (text-embedding-3-large) — the 1M-article column adds roughly $104 of one-time embedding cost the month you backfill.

Two failure modes inflate this bill: leaving the half-life loose so users page deep into stale results (more reads per query), and skipping the TTL deletion job so storage compounds. Both are policy fixes, not infra fixes.

Embedding Model Decision Matrix for News

Three serious contenders in 2026. The right pick is not "best MTEB score" — it's the model whose strengths match your news use-case.

Recommendation by use-case: English-only daily news app → OpenAI text-embedding-3-large with truncation to 1,536 dimensions to halve storage. Multilingual archive across European or Asian languages → Cohere embed-v4.0; the multilingual gap on every other model widens past 5 supported languages. High-volume English crawler with millions of articles ingested daily → Voyage-3, where the per-token cost differential compounds enough to matter.

Switching cost is real (full re-embed of the index), so make this decision once with intent.

Pinecone vs pgvector vs Qdrant for News

Pinecone is the right answer when you want managed serverless economics, sub-100ms p95 query latency at any scale without ops, and metadata filters as a first-class feature. pgvector is the right answer if your news app already runs on Postgres, your scale is under 1M vectors, and avoiding a second data store matters more than the 5-10× latency gap at scale. Qdrant is the right answer if you want self-hosted control with serverless-like ergonomics, especially for a research crawler where cost-per-vector at 50M+ scale beats Pinecone meaningfully. For a production product with team-led news ingest at 1M-10M articles, Pinecone Serverless is the pragmatic default.

Unlike a general-purpose document store, a vector index is single-purpose infrastructure that buys you semantic recall and pays you back in user satisfaction, which means the lock-in cost of moving between vector DBs is real but bounded — you re-embed once and re-upsert. Choose deliberately, but don't over-think it.

FAQ

What is vector search?

Vector search is a retrieval method that represents documents and queries as high-dimensional numerical vectors (embeddings) and finds matches by computing distance — usually cosine similarity — between the query vector and stored document vectors. For news, this finds semantically related articles even when the wording differs ("EU climate deal" matches "Brussels methane talks").

How do I embed news articles?

Embedding news articles works by sending each article's body — or chunked sub-sequences for bodies over 8,000 tokens — to an embedding model API like OpenAI text-embedding-3-large or Cohere embed-v4. Each article becomes a vector of 1,024-3,072 floats. Batch 100-500 articles per API call for throughput, then store the result in a vector database with metadata for filtering.

Is Pinecone better than Weaviate for news?

Pinecone wins on managed serverless economics and zero-ops latency at scale; Weaviate wins on hybrid search out of the box and self-hosted flexibility. For most production news teams running 1M-10M articles with daily ingest and metadata filters, Pinecone Serverless is the lower-friction choice. For self-hosted requirements or built-in keyword + vector hybrid, Weaviate is competitive.

How much does Pinecone cost for 1 million articles?

For 1 million news articles with 3 chunks each (3M vectors at 3,072 dimensions ≈ 12GB) on Pinecone Serverless 2026 pricing, expect roughly $12/month in storage, ~$0.50/month in writes for daily ingest of 1k new articles, and $50-100/month in reads at 100k queries/day with metadata filters — total around $65-120/month. One-time embedding cost adds ~$104 if using OpenAI text-embedding-3-large.

Should I use OpenAI or Cohere embeddings for news?

For English-only news, OpenAI text-embedding-3-large is the safe default — strong retrieval scores, Matryoshka dimension truncation for cost control. For multilingual news (5+ languages), Cohere embed-v4.0 outperforms on cross-lingual retrieval and costs ~25% less per million tokens. Both are production-grade; the deciding factor is your language footprint, not raw English benchmark scores.

Resources

• APITube — apitube.io — try it free, full article bodies, source rank, story clustering and entities included on every record
• Documentation — docs.apitube.io — endpoints, parameters, response structure, integrations
• Pricing — apitube.io/pricing — all tiers
• APITube blog — apitube.io/blog — more guides and comparisons