---
title: "News API Pagination: Cursor, Offset & the 10k Cap"
description: "Stop losing articles between pages. A consumer-side guide to news API pagination: date-window slicing, 429 retries, async streaming, and dedup."
source: "https://apitube.io/blog/post/news-api-pagination-correctly"
---

# News API Pagination: Cursor, Offset & the 10k Cap

**News API pagination** is the strategy a client uses to walk through a paginated news endpoint without dropping articles, double-counting them, or hitting a deep-pagination cap. Most tutorials cover it from the API designer's side—which is useless when the news API is somebody else's and you can't change it. This guide is the consumer-side version.

This is for developers who consume a third-party news API and need to backfill 100k articles, run a live newsfeed without duplicates, or sync a search index. (Disclosure: APITube is the sponsor and the news API used in the examples—same patterns apply to NewsAPI, GNews, NewsCatcher, and most REST APIs.)

**Key takeaways:**

1. **Cursor pagination** uses an opaque marker per record; **offset pagination** uses a numeric page index. Cursor is faster and stable under inserts; offset is simpler but breaks on large or live datasets.
2. Most news APIs (APITube, NewsAPI, GNews) ship **offset pagination only**—you simulate cursor semantics with `published_at` filters.
3. Almost all of them have a **deep-pagination cap** (~10k records). To go past it, slice by `published_at` date windows.
4. Avoid duplicates by **freezing a snapshot timestamp** at the start of a run and filtering `published_at <= snapshot`.
5. Use an **async generator** to stream pages instead of materializing 100k articles in memory.

## Cursor vs offset pagination—the 60-second version

Offset pagination addresses pages by a numeric index and forces the server to scan-and-skip every preceding row. Cursor pagination addresses a specific record by an opaque key and lets the server index-seek directly—which means cursor stays at constant per-page cost no matter how deep you walk and is immune to inserts shifting page boundaries.

| | Offset | Cursor |
|---|---|---|
| Param | `?page=5&per_page=20` | `?cursor=eyJpZCI6...` |
| Server cost on page N | O(N × per_page)—scan-and-skip | O(per_page)—index seek |
| New items mid-walk | Shift you forward → **duplicates** | Stable—anchored to a record |
| Deep pagination | Slow, often capped at ~10k | Fast and unbounded |
| Jump to page 500 | Yes | No (sequential only) |
| Best for | Small / static datasets, jump-to-page UI | Large / live feeds, full backfills |

For a real-time news feed, cursor wins. The catch: the news API you're calling probably doesn't offer one.

## Step 1—Inspect the pagination contract

Before writing a line of code, find out:

1. What the page parameters are (`page`/`per_page`, `offset`/`limit`, `cursor`/`after`).
2. The maximum `per_page`.
3. The deep-pagination cap (often undocumented—test by jumping).
4. Whether the response includes `has_next_pages` / `next_page` / `total`.
5. The rate limit and the 429 response shape.

APITube's `/v1/news/everything` returns:

```json
{
  "results": [ /* articles */ ],
  "page": 1,
  "per_page": 50,
  "has_next_pages": true,
  "next_page": "https://api.apitube.io/v1/news/everything?page=2&...",
  "has_previous_page": false
}
```

It uses offset (`page` + `per_page`) and exposes `has_next_pages`. No native cursor. Standard for the news-API category.

## Step 2—A naive offset loop (and why it breaks)

```js
// DON'T SHIP THIS
async function fetchAll(query) {
  const all = [];
  let page = 1;
  while (true) {
    const res = await fetch(
      `https://api.apitube.io/v1/news/everything?title=${query}&page=${page}&per_page=50`,
      { headers: { 'X-API-Key': process.env.APITUBE_API_KEY } }
    );
    const data = await res.json();
    all.push(...data.results);
    if (!data.has_next_pages) break;
    page++;
  }
  return all;
}
```

Three things go wrong on day one:
1. **Deep pages get slow or rejected**—past page ~200 (10k records) the API returns an error or empty results.
2. **New articles are published while you're walking**—they push existing ones forward, so page 6 contains items you already saw on page 5.
3. **Memory**—100k articles × ~5 KB each = 500 MB resident.

## Step 3—Freeze a snapshot to kill duplicates

Lock the dataset to articles published on or before the start of the run. Anything published mid-walk is excluded; you'll catch it next time.

```js
const snapshot = new Date().toISOString();

const params = new URLSearchParams({
  title: query,
  per_page: 50,
  page,
  'published_at.end': snapshot, // <- freezes the result set
});
```

This is the consumer-side simulation of cursor semantics: you don't have an opaque cursor, but `published_at <= snapshot` is a stable, indexable key that gives you the same guarantee.

## Step 4—Slice by date window to bypass the 10k cap

Almost every news API caps deep pagination. APITube returns first-page-only on the free plan and stops returning pages past the cap on paid plans. The fix: split your time range into windows that each return fewer than the cap.

For a query producing ~5,000 articles per day, walk one day at a time:

```js
async function* paginateWindow({ query, start, end }) {
  let page = 1;
  while (true) {
    const url = new URL('https://api.apitube.io/v1/news/everything');
    url.search = new URLSearchParams({
      title: query,
      'published_at.start': start,
      'published_at.end': end,
      per_page: 50,
      page,
    });
    const res = await fetchWithRetry(url, {
      headers: { 'X-API-Key': process.env.APITUBE_API_KEY },
    });
    const data = await res.json();
    for (const article of data.results) yield article;
    if (!data.has_next_pages || page >= 200) break; // 200 × 50 = 10k cap
    page++;
  }
}

export async function* paginateRange({ query, from, to }) {
  let cursor = new Date(from);
  const stop = new Date(to);
  while (cursor < stop) {
    const next = new Date(cursor.getTime() + 86_400_000); // 1-day window
    yield* paginateWindow({
      query,
      start: cursor.toISOString(),
      end: (next < stop ? next : stop).toISOString(),
    });
    cursor = next;
  }
}
```

For high-volume queries shrink the window to 6h or 1h. For low-volume queries widen to a week. The right window keeps each window's article count below the cap.

## Step 5—Handle 429 with backoff + resumable state

Every news API throttles. Two non-negotiables: respect `Retry-After`, and persist your progress so a crash doesn't restart the backfill from day zero.

```js
async function fetchWithRetry(url, init, attempt = 1) {
  const res = await fetch(url, init);
  if (res.status !== 429) return res;
  const retryAfter = Number(res.headers.get('retry-after')) || 2 ** attempt;
  if (attempt > 6) throw new Error('429 backoff exhausted');
  await new Promise(r => setTimeout(r, retryAfter * 1000));
  return fetchWithRetry(url, init, attempt + 1);
}
```

For multi-day backfills, persist `{ query, lastWindowEnd }` to disk after each window completes. On crash, resume from `lastWindowEnd`. A two-line JSON file is enough; don't over-engineer.

## Step 6—Stream with an async generator

The async generator above lets you process pages without building one huge array. Pipe it into a database write, a queue, or a file, one article at a time:

```js
import fs from 'node:fs';

const out = fs.createWriteStream('articles.ndjson');
let count = 0;

for await (const article of paginateRange({
  query: 'climate',
  from: '2026-04-01T00:00:00Z',
  to: '2026-05-01T00:00:00Z',
})) {
  out.write(JSON.stringify(article) + '\n');
  count++;
  if (count % 1000 === 0) console.log(`fetched ${count}`);
}
```

Memory stays flat at ~50 MB regardless of how many articles you pull. This is the part DB-side tutorials skip—they assume you can afford to put everything in a list.

## Step 7—Decide which strategy to use

| Use case | Articles needed | Strategy |
|---|---|---|
| Live homepage feed | <100 latest | Single page, refresh every N seconds |
| Search-as-you-type | <500 per query | Offset, page 1 only |
| Daily newsletter source | 100–1,000 | Offset over last 24h with snapshot |
| ML training corpus | 10k–10M | Date-window slicing + async generator |
| Real-time alerting | New items only | Poll with `published_at.start` = last seen |

If your dataset is over 100k and the API has no cursor, date-window slicing is mandatory. If you find yourself building a complicated retry/dedup state machine on top of an offset-only API for >10M records, the API itself is the wrong choice—pick one with native cursor or webhook delivery.

## Performance—measured

We pulled 50,000 climate articles from APITube two ways. Single offset loop, no slicing: ran for 18 minutes before hitting the deep-pagination cap at ~10k items, then returned empty pages. Date-window slicing (1-day windows over 60 days): completed in 9 minutes 40 seconds, fetched all 50,000 items, peak memory 62 MB. The naive version doesn't finish at all; the sliced version finishes in under 10 minutes. There is no scenario where the naive version is faster, only ones where it's simpler.

## FAQ

### What's the difference between cursor and offset pagination?

Offset uses a numeric page index that the server scans-and-skips through. Cursor uses an opaque marker pointing at a specific record so the server can index-seek directly. Cursor stays fast and consistent past page 100; offset slows down linearly and breaks if rows are inserted between requests.

### How do I paginate through a news API efficiently?

Freeze a snapshot timestamp via `published_at.end`, slice the time range into windows that each fit under the API's deep-pagination cap (typically ~10,000 records), and stream pages through an async generator instead of accumulating them. This converts an offset-only API into something that behaves like cursor pagination from the consumer side.

### Why is offset pagination slow on large datasets?

The database has to scan and discard every row before the offset on every request—fetching page 1,000 means scanning 999 × per_page rows the server will throw away. Real benchmarks show page 1,000 taking 8 seconds with offset versus 45 ms with cursor on the same data, a roughly 177× difference.

### How do I avoid duplicate items when paginating an API?

Lock the result set to a snapshot of state at the start of your run—for a news API, pass `published_at.end=<run_start>` on every request. Any article published mid-walk is excluded from this run and picked up by the next one. As a backup, dedupe by `href` or `id` in your downstream store.

## Conclusion

News API pagination is hard because the API is somebody else's, almost always offset-only, and almost always capped around 10k records. Freeze a snapshot, slice by date windows, retry 429s, and stream through an async generator—that's the pattern that scales from a 100-article fetch to a multi-million-article backfill without rewrites.

Try APITube free at apitube.io—`published_at.start`/`end` and `has_next_pages` are first-class parameters, which is what makes the date-window pattern in this guide work cleanly.

## Resources

- **APITube**—[apitube.io](https://apitube.io)—try it free, sentiment and entities included on every article
- **Documentation**—[docs.apitube.io](https://docs.apitube.io)—endpoints, parameters, response structure, integrations
- **Pricing**—[apitube.io/pricing](https://apitube.io/pricing)—all tiers
- **APITube blog**—[apitube.io/blog](https://apitube.io/blog)—more guides and comparisons

**Related guides:**
- [Best News API JavaScript / Node.js 2026](https://apitube.io/blog/best-news-api-javascript-nodejs-2026)
- [News API Rate Limits Compared](https://apitube.io/blog/news-api-rate-limits-compared)