Screenshot Every Page of a Website from Its sitemap.xml

A client hands you a site and asks for a redesign. Or a migration went live at 2 a.m. and someone needs to confirm that all 400 URLs still render. Or the quarterly report needs a visual inventory of what the site currently looks like, page by page, before anyone touches it. In all three cases the job is the same: capture every page the site says it has, once, reproducibly, into a folder you can hand to a designer or diff against next month.

The site already tells you what it has. That's what sitemap.xml is for. What follows is a working Node script that reads it — including the nested-index case that quietly defeats most tutorials — filters it, captures it with bounded concurrency, and survives dying halfway through.

Why the naive sitemap parser captures nothing

Here is the thing people get wrong, and it fails silently rather than loudly.

A small site's sitemap.xml is a <urlset> containing <url><loc> entries. That's the shape every tutorial handles. But past a few thousand URLs — and on essentially every WordPress, Shopify, or enterprise CMS install regardless of size — sitemap.xml is instead a sitemap index: a <sitemapindex> containing <sitemap><loc> entries that point at further sitemaps. /post-sitemap.xml, /page-sitemap.xml, /product-sitemap-1.xml, and so on. Frequently those children are served gzipped, as .xml.gz.

A parser that only looks for <url> elements finds zero of them in a sitemap index. It doesn't error. It reports "0 URLs found", you assume the client's sitemap is broken, and you go and write a crawler instead. The fix is about fifteen lines: detect which document type you got, and recurse.

Gzip has its own trap. fetch transparently decompresses a response carrying Content-Encoding: gzip, but a .xml.gz file is usually served as gzip content with Content-Type: application/gzip — the bytes arrive still compressed and you have to unpack them yourself. The reliable test is the gzip magic number at the start of the buffer, which works whichever way the server chose to do it.

The script

Node 20 or newer, no dependencies. The blocks below are one file, sitemap-capture.mjs, in order.

// sitemap-capture.mjs
import { appendFile, mkdir, readFile, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";
import { gunzipSync } from "node:zlib";
import path from "node:path";

const API_KEY = process.env.SNAPSHOT_SITE_API_KEY;
const ENDPOINT = "https://api.prod.ss.snapshot-site.com/api/v1/screenshot";

const OUT_DIR = "captures";
const MANIFEST_FILE = "captures/manifest.ndjson";
const CONCURRENCY = 8;
const MAX_SITEMAP_DEPTH = 2;

const CAPTURE_OPTIONS = {
  format: "png",
  width: 1440,
  fullSize: true,
  hideCookie: true,
  delay: 2,
};

delay is in seconds, with an accepted range of 0 to 10 — a much narrower window than the millisecond field most browser-automation APIs expose, which is why habit tends to win here and produce a validation error on the very first URL. Two is a reasonable settle time for a site you don't control.

Fetching and parsing, including the index case

async function fetchSitemap(url) {
  const response = await fetch(url, {
    headers: { "user-agent": "sitemap-capture/1.0" },
  });
  if (!response.ok) throw new Error(`${url} -> HTTP ${response.status}`);

  const buffer = Buffer.from(await response.arrayBuffer());
  // A .xml.gz file arrives as gzip *content*, which fetch does not unwrap.
  // Test the magic number rather than trusting the content type.
  const isGzip = buffer[0] === 0x1f && buffer[1] === 0x8b;
  return (isGzip ? gunzipSync(buffer) : buffer).toString("utf8");
}

function decodeXml(value) {
  return value
    .replace(/<!\[CDATA\[([\s\S]*?)\]\]>/g, "$1")
    .replace(/&lt;/g, "<")
    .replace(/&gt;/g, ">")
    .replace(/&quot;/g, '"')
    .replace(/&apos;/g, "'")
    .replace(/&amp;/g, "&");
}

function parseEntries(xml, tag) {
  const blocks = new RegExp(
    `<(?:\\w+:)?${tag}\\b[^>]*>([\\s\\S]*?)</(?:\\w+:)?${tag}>`,
    "gi",
  );
  const locPattern = /<(?:\w+:)?loc\b[^>]*>([\s\S]*?)<\/(?:\w+:)?loc>/i;
  const lastmodPattern =
    /<(?:\w+:)?lastmod\b[^>]*>([\s\S]*?)<\/(?:\w+:)?lastmod>/i;

  const entries = [];
  for (const block of xml.matchAll(blocks)) {
    const body = block[1];
    const loc = body.match(locPattern);
    if (!loc) continue;
    const lastmod = body.match(lastmodPattern);
    entries.push({
      loc: decodeXml(loc[1].trim()),
      lastmod: lastmod ? lastmod[1].trim() : null,
    });
  }
  return entries;
}

async function collectUrls(sitemapUrl, depth = 0, seen = new Set()) {
  if (depth > MAX_SITEMAP_DEPTH || seen.has(sitemapUrl)) return [];
  seen.add(sitemapUrl);

  const xml = await fetchSitemap(sitemapUrl);

  // A <sitemapindex> yields <sitemap> children; a <urlset> yields <url>.
  const children = parseEntries(xml, "sitemap");
  if (children.length > 0) {
    const collected = [];
    for (const child of children) {
      try {
        collected.push(...(await collectUrls(child.loc, depth + 1, seen)));
      } catch (error) {
        console.warn(`skipping ${child.loc}: ${error.message}`);
      }
    }
    return collected;
  }

  return parseEntries(xml, "url");
}

Two deliberate choices. The regex allows an optional namespace prefix, because plenty of real sitemaps emit <sm:url> or similar, and \b stops <urlset> from being mistaken for a <url> block. And a failing child sitemap warns and continues rather than aborting the run — one dead /product-sitemap-7.xml should not cost you the other six.

Regex parsing of XML is a compromise, and worth naming as one. Sitemaps are machine-generated, extremely regular, and constrained by a published schema, so it holds up in practice. If you're parsing something that isn't a sitemap, use a real parser.

Filtering the list

The sitemap is the site's claim about itself, not your capture list. Archives, tag pages and feeds inflate the count without adding anything a designer or a stakeholder will look at.

const EXCLUDE = [
  /\/page\/\d+/i,
  /\/tag\//i,
  /\/author\//i,
  /\/feed\/?$/i,
  /[?&]replytocom=/i,
  /\.(?:xml|json|pdf|jpe?g|png|webp|gif|svg)$/i,
];

function keepUrl(entry, since) {
  if (EXCLUDE.some((pattern) => pattern.test(entry.loc))) return false;
  if (since && entry.lastmod && new Date(entry.lastmod) < since) return false;
  return true;
}

The --since filter uses <lastmod> to skip pages that claim not to have changed since your last pass, which on a large site is the difference between 400 captures and forty. Treat it as an optimisation rather than as truth: lastmod is self-reported, plenty of static-site generators stamp every URL with the build timestamp, and some CMSes never update it at all. Note that entries with no lastmod at all are kept, which is the safe default.

Be clear about what this filter is for: it's a duplicate-cost guard, not change detection. Its sibling on the client side is the caching pattern for avoiding duplicate capture costs, which hashes the capture parameters to avoid paying twice for a request you already made — and which is explicitly the wrong tool for asking whether a page changed, since that question needs a fresh capture by definition. If "what changed since last quarter?" is the actual deliverable, that's the compare endpoint, and it gets its own section at the end of this article.

Deterministic output filenames

This is the small detail that decides whether two runs are comparable. Derive the filename from the URL path, not from a counter or a timestamp, and the same page lands at the same path every single time — which means a run in August and a run in November can be diffed directory against directory.

function outputPath(pageUrl) {
  const { hostname, pathname, search } = new URL(pageUrl);
  const slug =
    `${pathname}${search}`
      .replace(/\/index\.html?$/i, "/")
      .replace(/[^a-zA-Z0-9]+/g, "-")
      .replace(/^-+|-+$/g, "")
      .toLowerCase() || "home";

  // The slug is lossy — it lowercases, and collapses every run of
  // non-alphanumerics to one dash — so always suffix a hash of the full URL.
  // Deterministic, and no two URLs can land on the same file.
  const digest = createHash("sha1").update(pageUrl).digest("hex").slice(0, 8);
  const stem = slug.slice(0, 110).replace(/-+$/, ""); // truncation can land on a dash
  const safe = `${stem}-${digest}`;

  return path.join(OUT_DIR, hostname, `${safe}.${CAPTURE_OPTIONS.format}`);
}

https://example.com/blog/how-we-migrated/ becomes captures/example.com/blog-how-we-migrated-<hash>.png, and the homepage becomes captures/example.com/home-<hash>.png, where <hash> is the first eight hex characters of SHA-1 over the full URL. Both filenames are identical next month, which is the whole point — the slug keeps the name readable, the suffix keeps it unique.

The hash suffix is not decoration. Slugifying is lossy in three ways that bite on real sites: .toLowerCase() merges /About and /about, collapsing non-alphanumeric runs merges /services/seo with /services-seo, and the query string merges /search?q=shoes with /search-q-shoes. Each collision means the second capture silently overwrites the first, two manifest lines point at one file, and your directory-against-directory diff quietly compares the wrong pages. Eight hex characters of SHA-1 over the full URL costs nothing and removes the entire class of problem.

The manifest, written as you go

A 400-page audit that dies at page 300 because a laptop slept, a token expired, or the office wifi dropped should cost you one page of work, not three hundred. Write each result the moment it lands.

let writeChain = Promise.resolve();

function record(entry) {
  // Serialise manifest writes: eight workers appending concurrently to one
  // file is a race, and a chained promise is cheaper than a lock.
  // The .catch() keeps one failed append from poisoning the chain for
  // every later write — losing one manifest line beats losing the run.
  writeChain = writeChain
    .then(() => appendFile(MANIFEST_FILE, `${JSON.stringify(entry)}\n`))
    .catch((error) => console.warn(`manifest write failed: ${error.message}`));
  return writeChain;
}

async function loadManifest(file) {
  try {
    const text = await readFile(file, "utf8");
    const done = new Map();
    for (const line of text.split("\n")) {
      if (!line.trim()) continue;
      const entry = JSON.parse(line);
      done.set(entry.url, entry); // last write wins
    }
    return done;
  } catch (error) {
    if (error.code === "ENOENT") return new Map();
    throw error;
  }
}

Newline-delimited JSON rather than one big JSON object, because an append-only log has no read-modify-write step to lose. Last-write-wins on load means a URL that failed on Monday and succeeded on Tuesday reads as succeeded, so re-running the script is also how you retry failures.

Capturing, with retries

async function capture(pageUrl) {
  const response = await fetch(ENDPOINT, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Accept: "application/json",
      "x-snapshotsiteapi-key": API_KEY,
    },
    body: JSON.stringify({ url: pageUrl, ...CAPTURE_OPTIONS }),
  });

  if (!response.ok) {
    const error = new Error(`capture HTTP ${response.status}`);
    error.status = response.status; // so withRetry can tell 429 from 401
    throw error;
  }

  const payload = await response.json();
  if (!payload.link) throw new Error(payload.message || "no link in response");
  return payload.link;
}

async function download(link, destination) {
  const response = await fetch(link);
  if (!response.ok) throw new Error(`download HTTP ${response.status}`);
  await mkdir(path.dirname(destination), { recursive: true });
  await writeFile(destination, Buffer.from(await response.arrayBuffer()));
}

async function withRetry(task, attempts = 3) {
  let lastError;
  for (let attempt = 1; attempt <= attempts; attempt += 1) {
    try {
      return await task();
    } catch (error) {
      lastError = error;
      // Don't burn attempts on errors a retry can't fix: a bad key,
      // a rejected parameter, a 404. Retry 429, 408, and 5xx only.
      const status = error.status;
      if (status && status < 500 && status !== 429 && status !== 408) throw error;
      if (attempt < attempts) {
        await new Promise((resolve) => setTimeout(resolve, 2 ** attempt * 1000));
      }
    }
  }
  throw lastError;
}

The v1 response carries a hosted link to the rendered image; the download step pulls it into your folder so the audit is a set of local files rather than a set of URLs.

The status check in withRetry matters more than the backoff does. Without it, a mistyped API key means 400 URLs each retried three times with six seconds of waiting in between, and a run that takes half an hour to tell you something it knew on the first request. A 401, a 403, a 404 or a rejected parameter will return exactly the same answer on attempt three as on attempt one, so those fail immediately; only 429, 408 and 5xx are worth a second look. Beyond that, the backoff here is deliberately minimal — enough to ride out a transient blip, not a full-strength policy. If your runs are large enough to see sustained 429 responses, the proper treatment of backoff, jitter and which errors are worth retrying at all is in rate limits and retries for high-volume usage. Do bear in mind that every attempt is a billed request, not every success.

Bounded concurrency, and the run itself

async function runPool(items, limit, worker) {
  let index = 0;
  async function next() {
    while (index < items.length) {
      const current = index;
      index += 1;
      await worker(items[current]);
    }
  }
  await Promise.all(
    Array.from({ length: Math.min(limit, items.length) }, next),
  );
}

async function main() {
  const sitemapUrl = process.argv[2];
  if (!sitemapUrl || !API_KEY) {
    console.error(
      "usage: SNAPSHOT_SITE_API_KEY=... node sitemap-capture.mjs https://example.com/sitemap.xml [--since=2026-07-01]",
    );
    process.exit(1);
  }

  const sinceArg = process.argv.find((arg) => arg.startsWith("--since="));
  const since = sinceArg ? new Date(sinceArg.slice("--since=".length)) : null;
  if (since && Number.isNaN(since.getTime())) {
    console.error(`--since is not a valid date: ${sinceArg}`);
    process.exit(1);
  }

  const entries = await collectUrls(sitemapUrl);

  // Child sitemaps overlap more often than you'd think — the same URL listed
  // in both /page-sitemap.xml and /product-sitemap.xml would otherwise be
  // captured twice, billed twice, and raced by two workers onto one file.
  const targets = [
    ...new Map(
      entries.filter((entry) => keepUrl(entry, since)).map((e) => [e.loc, e]),
    ).values(),
  ];
  console.log(`${entries.length} URLs found, ${targets.length} after filtering`);

  const done = await loadManifest(MANIFEST_FILE);
  const pending = targets.filter(
    (entry) => done.get(entry.loc)?.status !== "ok",
  );
  console.log(
    `${targets.length - pending.length} already captured, ${pending.length} to go`,
  );

  await mkdir(OUT_DIR, { recursive: true });

  let completed = 0;
  await runPool(pending, CONCURRENCY, async (entry) => {
    const at = new Date().toISOString();
    let destination = null;
    try {
      // Inside the try: outputPath() calls new URL(), which throws on the
      // relative and unescaped <loc> values that broken sitemaps emit.
      destination = outputPath(entry.loc);
      const link = await withRetry(() => capture(entry.loc));
      await download(link, destination);
      await record({ url: entry.loc, file: destination, status: "ok", link, at });
    } catch (error) {
      await record({
        url: entry.loc,
        file: destination,
        status: "failed",
        error: error.message,
        at,
      });
    }
    completed += 1;
    if (completed % 25 === 0) console.log(`${completed}/${pending.length}`);
  });

  console.log("done — check manifest.ndjson; re-run to retry failures");
}

await main();

Eight workers, each pulling the next URL as soon as it finishes one. Not Promise.all over 400 URLs, and the reason is worth being blunt about: unbounded fan-out opens 400 simultaneous connections, trips rate limits within the first second, and hands you back a pile of 429s where you wanted screenshots. It's also slower in wall-clock terms once you count the retries. Eight to fifteen in flight is the range that behaves; the tuning logic, and the difference between "respecting the limit" and "finishing quickly", is worked through in batch capturing at scale.

Note where the try starts. Everything that can throw for a single URL — including outputPath, which calls new URL() and will throw on a relative /about or an unescaped space — has to sit inside it. Misconfigured sitemap plugins emit exactly those values, and post-migration sites are where you meet them. If one escapes the worker it rejects the pool, rejects main(), and takes the process down with the other seven workers' captures already billed and never written to the manifest: one bad <loc> costing you the run is the precise failure this script exists to avoid.

One payload for the whole site

Because every page goes through the same CAPTURE_OPTIONS object, per-site quirks get fixed once. hideCookie: true deals with the consent banner that would otherwise sit across the top of all 400 images. fullSize: true gives you the whole page rather than the fold, which is what a redesign inventory needs. width: 1440 is a sensible desktop audit width and accepts anything from 100 to 8000.

If the site has a sticky chat bubble, a promo bar or a floating cookie widget that survives hideCookie, switch ENDPOINT to /api/v2/screenshot and add hide — a comma-separated list of CSS selectors removed before rendering, available on every plan. Replace the two declarations in the config block above with these:

const ENDPOINT = "https://api.prod.ss.snapshot-site.com/api/v2/screenshot";

const CAPTURE_OPTIONS = {
  format: "png",
  width: 1440,
  fullSize: true,
  hideCookie: true,
  delay: 2,
  hide: "#intercom-container, .promo-bar, .newsletter-modal",
};

The v2 endpoint also accepts javascriptCode, a script run before the capture, for sites that need a scroll nudge or a state set first — that one is gated to Ultra and above. Either way, you are not running or maintaining a browser to do any of this, which is the part that gets expensive; the arithmetic behind that claim is in what self-hosting Puppeteer on Lambda actually costs.

Quota arithmetic

One capture is one request, and the script makes one capture per page per pass. So a 400-page site is 400 requests every time you run it in full.

Against the published plans, that works out as follows. Ultra is €20 for 15,000 requests a month, which is roughly 37 full passes over a 400-page site. Mega AI is €39 for 30,000, or about 75 passes. Those are the two tiers this kind of work lands on; the lower ones and the per-1,000 rates are in the cost breakdown linked below.

Two adjustments to make to that figure. Retries are billed too, so leave some headroom above your URL count. And --since cuts the recurring number hard: an incremental weekly pass over the pages that actually changed is usually a few dozen requests, not four hundred. For the retry margin, the per-1,000 rates and where the plan boundaries land, see what a screenshot API costs per 1,000 captures.

The n8n variant

If you'd rather not run a script, the same job fits in a workflow. Install n8n-nodes-snapshot-site — it's a verified community node, so it installs directly on n8n Cloud as well as self-hosted — and add a Snapshot Site API credential.

The shape is five nodes. A Schedule or Manual Trigger, then an HTTP Request node fetching the sitemap URL with the response format set to text, then a Code node that runs the same parseEntries logic over items[0].json.data and returns one item per URL, then a Loop Over Items node with a batch size of one, and inside the loop the Snapshot Site node on its Screenshot operation. Send the result wherever the client expects it — Google Drive, S3, a Sheets row per URL.

Three honest caveats. The loop gives you serial execution rather than a tuned worker pool, so a 400-page site takes a while; that's usually fine for a scheduled overnight job and painful for anything interactive. There's no manifest, so a failed execution restarts from the top unless you add a "captured" flag to a sheet and filter on it. And gzipped child sitemaps are awkward to unpack in an n8n Code node, so if the client's sitemap.xml is an index pointing at .xml.gz files, run the script instead. Also worth knowing: the node's delay field is labelled "Delay (ms)" in the UI even though the API reads it as seconds, so keep the value between 0 and 10 there too.

What bulk website screenshots are actually for

Four jobs, in rough order of how often they come up.

Pre-redesign inventory. Before anyone opens Figma, you need to know what exists. A folder of full-page captures organised by URL path is a more honest brief than a page list in a spreadsheet, because it shows the six one-off landing pages from 2023 that nobody remembered and the template that's used exactly twice.

Post-migration verification. Run the script against the old sitemap before cutover and the new one after. The manifest is the audit trail: every URL, its status, and where its image landed. A failed line with an HTTP 404 is a redirect somebody didn't write. This is the case where deterministic filenames earn their keep, because the two directories line up.

Client reporting. A quarterly deck showing the current state of the site, generated rather than assembled by hand.

A baseline for visual comparison. Once you have a stable, reproducible set, the next pass isn't just another set of images — it's a diff. That's a different tool: the compare endpoint takes a live URL against a stored image and returns a mismatch percentage plus a diff image, which is what turns "here are 400 screenshots" into "here are the eleven pages that changed". The visual diff API covers how that works, and comparing staging against production before a release is the same mechanism pointed at a deploy rather than at a calendar.

Capture sites you're authorised to capture — your own, or a client's with sign-off in writing, which on an agency engagement you'll want anyway. Check the target's robots.txt and terms before pointing a script at every URL it publishes, and keep the concurrency modest: eight in flight is courteous to a client's shared hosting, forty is a load test nobody asked for. Past that, an audit is one script run away, and the free tier is 50 captures a month — enough to prove the whole pipeline against a slice of a client's sitemap before you size a plan, so grab a key from Snapshot Site and point it at your own site first.