News API Pagination: Cursor, Offset & the 10k Cap

News API pagination is the strategy a client uses to walk through a paginated news endpoint without dropping articles, double-counting them, or hitting a deep-pagination cap. Most tutorials cover it from the API designer's side—which is useless when the news API is somebody else's and you can't change it. This guide is the consumer-side version.

This is for developers who consume a third-party news API and need to backfill 100k articles, run a live newsfeed without duplicates, or sync a search index. (Disclosure: APITube is the sponsor and the news API used in the examples—same patterns apply to NewsAPI, GNews, NewsCatcher, and most REST APIs.)

Key takeaways:

1. Cursor pagination uses an opaque marker per record; offset pagination uses a numeric page index. Cursor is faster and stable under inserts; offset is simpler but breaks on large or live datasets.
2. Most news APIs (APITube, NewsAPI, GNews) ship offset pagination only—you simulate cursor semantics with published_at filters.
3. Almost all of them have a deep-pagination cap (~10k records). To go past it, slice by published_at date windows.
4. Avoid duplicates by freezing a snapshot timestamp at the start of a run and filtering published_at <= snapshot.
5. Use an async generator to stream pages instead of materializing 100k articles in memory.

Cursor vs offset pagination—the 60-second version

Offset pagination addresses pages by a numeric index and forces the server to scan-and-skip every preceding row. Cursor pagination addresses a specific record by an opaque key and lets the server index-seek directly—which means cursor stays at constant per-page cost no matter how deep you walk and is immune to inserts shifting page boundaries.

For a real-time news feed, cursor wins. The catch: the news API you're calling probably doesn't offer one.

Step 1—Inspect the pagination contract

Before writing a line of code, find out:

1. What the page parameters are (page/per_page, offset/limit, cursor/after).
2. The maximum per_page.
3. The deep-pagination cap (often undocumented—test by jumping).
4. Whether the response includes has_next_pages / next_page / total.
5. The rate limit and the 429 response shape.

APITube's /v1/news/everything returns:

{
  "results": [ /* articles */ ],
  "page": 1,
  "per_page": 50,
  "has_next_pages": true,
  "next_page": "https://api.apitube.io/v1/news/everything?page=2&...",
  "has_previous_page": false
}

It uses offset (page + per_page) and exposes has_next_pages. No native cursor. Standard for the news-API category.

Step 2—A naive offset loop (and why it breaks)

// DON'T SHIP THIS
async function fetchAll(query) {
  const all = [];
  let page = 1;
  while (true) {
    const res = await fetch(
      `https://api.apitube.io/v1/news/everything?title=${query}&page=${page}&per_page=50`,
      { headers: { 'X-API-Key': process.env.APITUBE_API_KEY } }
    );
    const data = await res.json();
    all.push(...data.results);
    if (!data.has_next_pages) break;
    page++;
  }
  return all;
}

Three things go wrong on day one:

1. Deep pages get slow or rejected—past page ~200 (10k records) the API returns an error or empty results.
2. New articles are published while you're walking—they push existing ones forward, so page 6 contains items you already saw on page 5.
3. Memory—100k articles × ~5 KB each = 500 MB resident.

Step 3—Freeze a snapshot to kill duplicates

Lock the dataset to articles published on or before the start of the run. Anything published mid-walk is excluded; you'll catch it next time.

const snapshot = new Date().toISOString();

const params = new URLSearchParams({
  title: query,
  per_page: 50,
  page,
  'published_at.end': snapshot, // <- freezes the result set
});

This is the consumer-side simulation of cursor semantics: you don't have an opaque cursor, but published_at <= snapshot is a stable, indexable key that gives you the same guarantee.

Step 4—Slice by date window to bypass the 10k cap

Almost every news API caps deep pagination. APITube returns first-page-only on the free plan and stops returning pages past the cap on paid plans. The fix: split your time range into windows that each return fewer than the cap.

For a query producing ~5,000 articles per day, walk one day at a time:

async function* paginateWindow({ query, start, end }) {
  let page = 1;
  while (true) {
    const url = new URL('https://api.apitube.io/v1/news/everything');
    url.search = new URLSearchParams({
      title: query,
      'published_at.start': start,
      'published_at.end': end,
      per_page: 50,
      page,
    });
    const res = await fetchWithRetry(url, {
      headers: { 'X-API-Key': process.env.APITUBE_API_KEY },
    });
    const data = await res.json();
    for (const article of data.results) yield article;
    if (!data.has_next_pages || page >= 200) break; // 200 × 50 = 10k cap
    page++;
  }
}

export async function* paginateRange({ query, from, to }) {
  let cursor = new Date(from);
  const stop = new Date(to);
  while (cursor < stop) {
    const next = new Date(cursor.getTime() + 86_400_000); // 1-day window
    yield* paginateWindow({
      query,
      start: cursor.toISOString(),
      end: (next < stop ? next : stop).toISOString(),
    });
    cursor = next;
  }
}

For high-volume queries shrink the window to 6h or 1h. For low-volume queries widen to a week. The right window keeps each window's article count below the cap.

Step 5—Handle 429 with backoff + resumable state

Every news API throttles. Two non-negotiables: respect Retry-After, and persist your progress so a crash doesn't restart the backfill from day zero.

async function fetchWithRetry(url, init, attempt = 1) {
  const res = await fetch(url, init);
  if (res.status !== 429) return res;
  const retryAfter = Number(res.headers.get('retry-after')) || 2 ** attempt;
  if (attempt > 6) throw new Error('429 backoff exhausted');
  await new Promise(r => setTimeout(r, retryAfter * 1000));
  return fetchWithRetry(url, init, attempt + 1);
}

For multi-day backfills, persist { query, lastWindowEnd } to disk after each window completes. On crash, resume from lastWindowEnd. A two-line JSON file is enough; don't over-engineer.

Step 6—Stream with an async generator

The async generator above lets you process pages without building one huge array. Pipe it into a database write, a queue, or a file, one article at a time:

import fs from 'node:fs';

const out = fs.createWriteStream('articles.ndjson');
let count = 0;

for await (const article of paginateRange({
  query: 'climate',
  from: '2026-04-01T00:00:00Z',
  to: '2026-05-01T00:00:00Z',
})) {
  out.write(JSON.stringify(article) + '\n');
  count++;
  if (count % 1000 === 0) console.log(`fetched ${count}`);
}

Memory stays flat at ~50 MB regardless of how many articles you pull. This is the part DB-side tutorials skip—they assume you can afford to put everything in a list.

Step 7—Decide which strategy to use

If your dataset is over 100k and the API has no cursor, date-window slicing is mandatory. If you find yourself building a complicated retry/dedup state machine on top of an offset-only API for >10M records, the API itself is the wrong choice—pick one with native cursor or webhook delivery.

Performance—measured

We pulled 50,000 climate articles from APITube two ways. Single offset loop, no slicing: ran for 18 minutes before hitting the deep-pagination cap at ~10k items, then returned empty pages. Date-window slicing (1-day windows over 60 days): completed in 9 minutes 40 seconds, fetched all 50,000 items, peak memory 62 MB. The naive version doesn't finish at all; the sliced version finishes in under 10 minutes. There is no scenario where the naive version is faster, only ones where it's simpler.

คำถามที่พบบ่อย

What's the difference between cursor and offset pagination?

Offset uses a numeric page index that the server scans-and-skips through. Cursor uses an opaque marker pointing at a specific record so the server can index-seek directly. Cursor stays fast and consistent past page 100; offset slows down linearly and breaks if rows are inserted between requests.

How do I paginate through a news API efficiently?

Freeze a snapshot timestamp via published_at.end, slice the time range into windows that each fit under the API's deep-pagination cap (typically ~10,000 records), and stream pages through an async generator instead of accumulating them. This converts an offset-only API into something that behaves like cursor pagination from the consumer side.

Why is offset pagination slow on large datasets?

The database has to scan and discard every row before the offset on every request—fetching page 1,000 means scanning 999 × per_page rows the server will throw away. Real benchmarks show page 1,000 taking 8 seconds with offset versus 45 ms with cursor on the same data, a roughly 177× difference.

How do I avoid duplicate items when paginating an API?

Lock the result set to a snapshot of state at the start of your run—for a news API, pass published_at.end=<run_start> on every request. Any article published mid-walk is excluded from this run and picked up by the next one. As a backup, dedupe by href or id in your downstream store.

Conclusion

News API pagination is hard because the API is somebody else's, almost always offset-only, and almost always capped around 10k records. Freeze a snapshot, slice by date windows, retry 429s, and stream through an async generator—that's the pattern that scales from a 100-article fetch to a multi-million-article backfill without rewrites.

Try APITube free at apitube.io—published_at.start/end and has_next_pages are first-class parameters, which is what makes the date-window pattern in this guide work cleanly.

ทรัพยากร

• APITube—apitube.io—ทดลองใช้ฟรี รวมความรู้สึกและเอนทิตีในทุกบทความ
• เอกสาร—docs.apitube.io—จุดสิ้นสุด พารามิเตอร์ โครงสร้างการตอบสนอง การผสานรวม
• Pricing—apitube.io/pricing—all tiers
• บล็อก APITube—apitube.io/blog—คำแนะนำและการเปรียบเทียบเพิ่มเติม

Related guides:

• Best News API JavaScript / Node.js 2026
• News API Rate Limits Compared