Architecture of a visual regression testing SaaS, how I built VizProof

Building a visual regression testing SaaS means capturing, comparing and storing thousands of screenshots deterministically and at scale. Here is the full architecture of VizProof: frozen capture under Playwright, a perceptual diff engine (pixelmatch, SSIM), baseline management, BullMQ worker queues, immutable S3 storage, and a GitHub Action that blocks the PR, with the code behind the key decisions.

What visual regression testing solves

Unit tests validate logic, but they can't see that a button turned grey, a margin collapsed, an icon vanished, or a component overflows on mobile. Visual regression testing captures a reference image (the baseline), compares it on every build, and flags the pixels that changed.

The challenge isn't comparing two images: any pixel diff does that. The real challenge is doing it reliably, fast and at scale, without drowning the team in false positives. A tool that cries wolf on every build gets disabled within two weeks. Everything in the architecture below exists to keep one promise: a flagged diff means a real visual change worth a human glance.

Architecture overview

Layer	Role	Stack
API	Run intake, auth, quotas	Node.js + Fastify
Orchestration	Job queues, retries, concurrency	BullMQ (Redis)
Workers	Browser render + capture	Playwright
Diff engine	Perceptual comparison	pixelmatch + SSIM
Storage	Screenshots & baselines	S3 (immutable objects)
Front	Diff review, approval	Angular (signals + OnPush)
CI	Trigger & status	GitHub Action / webhook

The guiding principle: every screenshot is an immutable object, identified by a content hash. We never mutate an image, we create a new one. That simplifies caching, error recovery, deduplication and auditing. We can always answer "what did this screen look like at commit abc123?".

import { createHash } from 'node:crypto';

// A snapshot's identity = metadata + a hash of the PNG content.
// Two identical captures point to the same S3 object (free dedup).
function snapshotKey(meta: SnapshotMeta, png: Buffer): string {
  const contentHash = createHash('sha256').update(png).digest('hex');
  const { projectId, story, viewport, commit } = meta;
  return `snapshots/${projectId}/${story}/${viewport}/${commit}/${contentHash}.png`;
}

Takeaway: separate the logical identity (project/story/viewport/commit) from the content hash. The first is for retrieval and comparison; the second deduplicates storage and guarantees immutability.

1. Deterministic capture

The first source of false positives is non-determinism: animations, unloaded fonts, blinking carets, dynamic dates, image lazy-loading, scrollbars. Capture has to freeze all of it before the shot.

async function captureDeterministic(page: Page, opts: CaptureOpts): Promise<Buffer> {
  // 1. Freeze animations, transitions and caret via an injected stylesheet
  await page.addStyleTag({
    content: `*, *::before, *::after {
      animation-duration: 0s !important;
      animation-delay: 0s !important;
      transition-duration: 0s !important;
      caret-color: transparent !important;
    }`,
  });

  // 2. Freeze the clock: Date.now() and dynamic dates become stable
  await page.clock.setFixedTime(new Date('2025-01-01T00:00:00Z'));

  // 3. Wait for the network to settle and fonts to be ready
  await page.waitForLoadState('networkidle');
  await page.evaluate(() => document.fonts.ready);

  // 4. Playwright's animations: 'disabled' flag also freezes in-flight animations
  return page.screenshot({ fullPage: true, animations: 'disabled' });
}

A few concrete pitfalls I hit on VizProof:

• Web fonts: without document.fonts.ready, the screenshot fires with the fallback font, then the real font swaps in on the next render: a guaranteed diff on every run. Bundling fonts locally instead of via a CDN also removes variable latency.
• Dates and clock: page.clock.setFixedTime() (Playwright ≥ 1.45) freezes Date.now(), setTimeout and requestAnimationFrame in the page. Essential for any component that shows "3 minutes ago" or a real-time chart.
• Lazy-loading: a fullPage shot of a long page won't necessarily trigger loading of below-the-fold images. Force a programmatic top-to-bottom scroll before the shot.

Takeaway: 80% of false positives vanish by freezing animations, pinning the clock and awaiting document.fonts.ready. Stabilize capture before optimizing the diff. A perceptual diff engine will never rescue an unstable capture.

2. The diff engine: pixel, perceptual and structural

Strict pixel-by-pixel comparison is too naive: a sub-pixel rendering difference between two machines (GPU, antialiasing) triggers an alert on visually identical content. The right approach combines a per-pixel perceptual tolerance with a global threshold on how much of the surface actually changed.

Method	Detects	False positives	CPU cost
Strict pixel diff (equality)	Any difference, to the bit	Very high	Low
pixelmatch (threshold + anti-AA)	Differences visible to the eye	Low	Low
SSIM (structure, luminance, contrast)	Structural changes	Very low	Medium

pixelmatch is the base building block. Two parameters matter and are often confused:

• threshold (0 to 1): the per-pixel color distance above which two pixels are deemed different. Lower means more sensitive. Default: 0.1.
• includeAA: at false (default), pixelmatch detects and ignores anti-aliased pixels, exactly what we want to absorb text-rendering differences between machines.

import pixelmatch from 'pixelmatch';

// diff.data receives the difference image (changed pixels highlighted)
const changedPixels = pixelmatch(
  baseline.data, current.data, diff.data,
  width, height,
  { threshold: 0.1, includeAA: false }, // per-pixel sensitivity + ignore AA
);

// GLOBAL threshold, distinct from the per-pixel threshold: how much moved?
const changedRatio = changedPixels / (width * height);
const status = changedRatio > project.tolerance ? 'changed' : 'passed';

Keeping the two thresholds distinct is essential: threshold decides whether a pixel changed, changedRatio decides whether the screenshot changed. The global tolerance (0.2% by default) is configurable per project: a marketing landing page tolerates less drift than a dense data dashboard where a few pixels move constantly.

Beyond the pixel: SSIM and regions

Pixelmatch tells you how many pixels changed, not where or whether it matters. Two refinements change the game:

• SSIM (Structural Similarity Index) compares luminance, contrast and structure over sliding windows rather than pixel by pixel. It tells a genuine structural change (a block that shifts) apart from uniform rendering noise. More expensive, so we reserve it for screens with high false-positive rates.
• Region detection: instead of a global ratio, group the changed pixels into connected bounding boxes. A diff concentrated in a single region is more meaningful than a percentage, and lets us point the human reviewer straight at the affected area.

Masking dynamic regions

Some areas are irreducibly non-deterministic: an ad carousel, a live counter, a user avatar, an interactive map. We mask them before the diff. Playwright can paint native masks at capture time, neutralizing them at the source:

await page.screenshot({
  fullPage: true,
  animations: 'disabled',
  mask: [page.locator('[data-vrt-ignore]'), page.locator('.live-ticker')],
  maskColor: '#FF00FF', // constant mask color → never a diff on these areas
});

The data-vrt-ignore convention in the markup lets developers declare the areas to ignore themselves, without touching the SaaS configuration.

3. Baseline management and approval workflow

A detected diff isn't necessarily a bug: it might be an intended change. The heart of the product is the human approval loop that promotes a new render into a baseline. We model it as an explicit state machine, with allowed transitions and nothing else.

type SnapshotStatus =
  | 'pending'    // captured, not yet compared
  | 'passed'     // identical to baseline (within tolerance)
  | 'changed'    // diff detected, awaiting human review
  | 'approved'   // accepted → becomes the new baseline
  | 'rejected';  // confirmed regression → red build

// Allowed transitions: anything else is rejected by the service.
const TRANSITIONS: Record<SnapshotStatus, SnapshotStatus[]> = {
  pending:  ['passed', 'changed'],
  passed:   [],
  changed:  ['approved', 'rejected'],
  approved: [],
  rejected: ['changed'], // re-opened after a new push
};

function transition(snap: Snapshot, to: SnapshotStatus, actor: UserId): Snapshot {
  if (!TRANSITIONS[snap.status].includes(to)) {
    throw new Error(`Illegal transition: ${snap.status} → ${to}`);
  }
  return { ...snap, status: to, reviewedBy: actor, reviewedAt: new Date() };
}

Promoting a baseline never overwrites the old image: we write a new immutable object and move a simple baseline → snapshotId pointer. History stays intact, and a rollback is just a pointer change.

On the Angular side, diff review uses signals and OnPush to stay smooth even with hundreds of screenshots:

@Component({ changeDetection: ChangeDetectionStrategy.OnPush })
export class ReviewBoardComponent {
  private snapshots = signal<Snapshot[]>([]);
  filter = signal<SnapshotStatus>('changed');

  // Memoized recompute: only re-renders snapshots for the current filter
  visible = computed(() =>
    this.snapshots().filter((s) => s.status === this.filter()),
  );

  pendingReview = computed(() =>
    this.snapshots().filter((s) => s.status === 'changed').length,
  );
}

4. Scaling with a worker queue

A project can generate 500 screenshots per build (stories × viewports × browsers). Running them serially would take several minutes. The fix: a BullMQ job queue with parallel, concurrency-limited workers.

import { Queue, Worker } from 'bullmq';

const connection = { host: 'redis', port: 6379 };
const captureQueue = new Queue('capture', { connection });

new Worker(
  'capture',
  async (job) => {
    const { url, viewport, runId } = job.data;
    const png = await captureDeterministic(/* ... */);
    const key = await uploadToS3(png, runId);
    await enqueueDiff(runId, key); // chains into the diff queue
  },
  {
    connection,
    concurrency: 4, // 4 parallel browser contexts per worker
  },
);

An important subtlety: in BullMQ, concurrency is set per worker instance, and the docs recommend values of 100 to 300 for purely I/O-bound jobs. Here we're the opposite: each job spins up a real browser, heavy on CPU and RAM (200-400 MB per Chromium context). Too much concurrency thrashes the machine and slows everything down. The right strategy:

• keep concurrency low per worker (4-6 depending on available vCPUs);
• scale horizontally by adding worker instances (containers), not by inflating concurrency;
• reuse a single browser instance and create only a fresh browserContext per job (isolation without the cost of a cold start).

5. One-step CI integration

Adoption depends on friction. The goal: a single action in the pipeline that blocks the PR on an unapproved regression and posts a direct link to the review.

- name: Visual regression
  uses: vizproof/action@v1
  with:
    token: ${{ secrets.VIZPROOF_TOKEN }}
    build: ${{ github.sha }}
    base: ${{ github.event.pull_request.base.sha }}
    fail-on: changed   # blocks the PR if a diff isn't approved

The Action calls the API, waits for the run result, then exits with a non-zero code if any snapshots are changed. On GitHub, a status check "VizProof: visual regression" becomes blocking via branch protection rules. The PR comment links to the review board: one click to approve or reject, and the check goes green.

# Exit code driven by the run status → blocks or allows the merge
exit_code=$(curl -s -H "Authorization: Bearer $VIZPROOF_TOKEN" \
  "https://api.vizproof.io/runs/$RUN_ID/exit-code")
exit "$exit_code"

Mastering false positives

This is the metric that makes or breaks the product. A checklist of levers, from highest payoff to finest tuning:

• Freeze animations, transitions and caret (injected stylesheet).
• Wait for fonts (document.fonts.ready) and self-host them.
• Pin the clock (page.clock.setFixedTime) to neutralize dates and timers.
• Ignore anti-aliasing (includeAA: false) and tune threshold per project.
• Mask dynamic regions (data-vrt-ignore, native Playwright mask).
• Lock the render environment: same browser version, same viewport, same deviceScaleFactor, identical container across baseline and build.
• Per-project global tolerance rather than one threshold for the whole SaaS.

Takeaway: a false positive costs more than a false negative. Every ignored alert erodes the team's trust in the tool. Better to miss a diff occasionally than to ship a tool people end up disabling.

Cost and scalability

The dominant cost driver of a visual regression SaaS isn't the diff (a few ms of CPU): it's browser rendering and storage.

Item	Why it costs	Lever
Browser rendering	Chromium = CPU + RAM per capture	Bounded concurrency, horizontal scaling, autoscaling on queue depth
S3 storage	Thousands of immutable PNGs piling up	Content-hash dedup, lifecycle policy, Infrequent Access class
Bandwidth	Downloading images to diff	Diff on a worker close to the bucket, no client round-trip
Redis (BullMQ)	Queue state	Short jobs, automatic cleanup (removeOnComplete)

Two decisions radically change the bill:

• Content-hash deduplication: if a screen doesn't move between two commits, the identical PNG is stored only once. On a stable project, the dedup rate often exceeds 90%.
• Queue-depth-driven autoscaling: size the number of workers on the BullMQ backlog, not on a fixed count. Outside build hours, scale to zero workers. You only pay for compute during runs.

Results

• Capture time: 500 screenshots in ~90s (vs ~6 min serially).
• False positives: cut 12x after freezing animations, pinning the clock and applying a perceptual threshold.
• Storage deduplication rate: > 90% on a mature project, with no purge.
• Visual regressions caught before prod: the product's real value. A visual bug costs far more in production than a red check in CI.

Learn more

I covered the product context, the trade-offs and the business metrics in the full case study: VizProof, Visual Regression Testing SaaS.

Building a SaaS or an internal tool and want to challenge its architecture (capture, job queues, storage, CI integration)? Let's talk.