Batch Capturing at Scale: Concurrency Patterns Beyond Rate Limits

Respecting rate limits and backing off on 429 responses — covered in Rate Limits, Retries, and Batching — keeps a batch job from failing outright. It does not, by itself, make a batch of 10,000 URLs finish in a reasonable amount of time. That is a separate problem: how many requests should actually be in flight at once, and how should a client be structured to keep that number steady without babysitting it.

This is about the client-side architecture of a large batch job, not the server-side limits it has to respect.

Why "Just Loop Over the List" Doesn't Scale

The naive version — a for loop that awaits each request before starting the next — is correct but slow. Every request pays full round-trip latency serially, and a 10,000-URL job that takes 2 seconds per capture would take over five and a half hours end to end, even though the API could easily be handling dozens of these in parallel.

The other naive version — firing all 10,000 requests at once with Promise.all — is fast until it isn't: it can trip rate limits immediately, exhaust local sockets or memory, and turn one slow or failed request into noise that's hard to isolate from the other 9,999.

The Fix: A Bounded Concurrency Pool

The practical middle ground is a worker pool: a fixed number of requests in flight at any time (say, 10-20), with a new one starting as soon as a slot frees up. In Node.js, this looks like:

async function runWithConcurrency(items, limit, worker) {
  const results = [];
  let index = 0;

  async function next() {
    while (index < items.length) {
      const current = index++;
      results[current] = await worker(items[current]);
    }
  }

  await Promise.all(Array.from({ length: limit }, next));
  return results;
}

const urls = [/* ... thousands of URLs ... */];

const results = await runWithConcurrency(urls, 15, async (url) => {
  const response = await fetch("https://api.prod.ss.snapshot-site.com/api/v1/screenshot", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-snapshotsiteapi-key": "YOUR_API_KEY",
    },
    body: JSON.stringify({ url, format: "png", width: 1440, fullSize: true, hideCookie: true }),
  });
  return response.json();
});

This keeps exactly limit requests in flight at all times, without needing a queue library or a job scheduler.

Picking the Concurrency Number

There is no universally correct number — it depends on what your account's rate limit actually allows and what "the rest of your infrastructure" can absorb (open sockets, memory for buffered responses, downstream storage writes). A reasonable starting point is 10-20 concurrent requests, watched for 429 responses; if those start appearing, the concurrency limit is set too high relative to the account's rate limit, and the retry/backoff logic from the rate-limits post should still be layered on top as a safety net, not a replacement for tuning this number down.

Isolating Failures in a Large Batch

A worker pool has a natural advantage over Promise.all: because each item resolves independently, one failed or slow capture does not block the rest, and a per-item try/catch inside the worker function can log exactly which URL failed without losing the whole batch's progress. For genuinely large jobs, write completed results incrementally (append to a file or a database row per item) rather than holding everything in memory until the end — that way a crash partway through doesn't lose completed work, and the job can resume from where it left off.

When to Reach for the CLI Instead

If the batch job is closer to a one-off operational task than a piece of application code, the Snapshot Site CLI is often a faster path — pairing it with a shell loop and a modest xargs -P concurrency flag gets most of this same behavior without writing a worker pool at all.

Getting the concurrency shape right once means every future batch job — a migration, an audit, a scheduled resync — runs at a predictable speed instead of however fast a naive loop happens to go, and Snapshot Site handles the same request the same way whether it's the 1st or the 10,000th in a batch.