Safari vs Firefox Regression Matrix on macOS

Rent one M-series Mac from Figment (entry price $95/month, exact chip and RAM confirmed in the quote — see Figment Mac hosting), run the WebKit and Firefox Playwright projects against the same commit on that host, and ship a single divergence-matrix.html artefact that highlights where the two engines disagree. The config, the reporter and the CI workflow are below. When you are ready to provision, go to /mac-hosting/start.

Running Playwright on only one engine hides WebKit-specific regressions until a user files them. A same-host matrix removes that blind spot without introducing hardware drift between the two runs.

Why Same-Host Parity Testing Catches Real Bugs

When WebKit and Firefox run on the same macOS machine at the same commit, any pass/fail difference is a genuine engine divergence — not a flaky network, a timezone drift, or a mismatched system font. That eliminates the most common source of false positives in cross-browser CI.

Playwright's own documentation states that its WebKit build is derived from recent WebKit sources, often ahead of what ships in Apple Safari, and that for the closest-to-Safari experience — in particular for video playback — you should run WebKit on a Mac (playwright.dev/docs/browsers). Apple's release history currently lists Safari 27.2 (released 26 October 2026) as the most recent stable and the Safari 27.3 beta from 3 November 2026 (developer.apple.com/documentation/safari-release-notes). Playwright's WebKit is not Safari, but running it on macOS keeps media codec availability and font rendering much closer to what real Safari users see than running WebKit on Linux CI.

Playwright Project Configuration: WebKit and Firefox Side by Side

Both projects share the same testDir, baseURL, timeout and retry policy, and diverge only on the use block. Pin a Playwright version in package.json and record npx playwright --version so the shipped WebKit build is reproducible between developer and CI runs.

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  timeout: 30_000,
  retries: 1,
  reporter: [
    ['html', { open: 'never', outputFolder: 'playwright-report' }],
    ['./reporters/divergence-reporter.ts'],
  ],
  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:3000',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
  projects: [
    {
      name: 'webkit',
      use: {
        ...devices['Desktop Safari'],
        locale: 'en-US',
        timezoneId: 'America/New_York',
        viewport: { width: 1280, height: 800 },
      },
    },
    {
      name: 'firefox',
      use: {
        ...devices['Desktop Firefox'],
        locale: 'en-US',
        timezoneId: 'America/New_York',
        viewport: { width: 1280, height: 800 },
      },
    },
  ],
});

With identical locale, timezoneId and viewport, any date-formatting or layout difference between the two projects is an engine signal, not an environment mismatch.

Aligning Viewport, Timezone and Locale

Divergence reports become noise when the two browsers start from different baselines. Lock these four settings:

Add a second project for colorScheme: 'dark' only when you actually need to test theme divergence — duplicating the project list roughly doubles the total runtime.

The Divergence Reporter: Flagging Safari-Only and Firefox-Only Failures

The built-in HTML reporter shows pass/fail per test but does not cross-reference results between projects. The reporter below does that in a post-processing step and writes a separate divergence-matrix.html next to the standard report. Test titles are HTML-escaped before interpolation so a title containing <, > or & cannot break the rendered table or inject markup from the test source into the artefact.

// reporters/divergence-reporter.ts
import type {
  Reporter,
  TestCase,
  TestResult,
} from '@playwright/test/reporter';
import * as fs from 'fs';
import * as path from 'path';

type BrowserStatus = 'pass' | 'fail' | 'skip';

interface Row {
  title: string;
  webkit: BrowserStatus;
  firefox: BrowserStatus;
}

const ESC: Record<string, string> = {
  '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#39;',
};
const escapeHtml = (s: string): string => s.replace(/[&<>"']/g, c => ESC[c]);

export default class DivergenceReporter implements Reporter {
  private results = new Map<string, Partial<Record<string, BrowserStatus>>>();

  onTestEnd(test: TestCase, result: TestResult): void {
    const project = test.parent.project()?.name ?? 'unknown';
    // titlePath() = [root, project, file, ...describes, test].
    // Drop root + project so the same test has the same key across projects.
    const title = test.titlePath().slice(2).join(' \u203a ');
    const existing = this.results.get(title) ?? {};
    existing[project] =
      result.status === 'passed' ? 'pass'
      : result.status === 'skipped' ? 'skip'
      : 'fail';
    this.results.set(title, existing);
  }

  async onEnd(): Promise<void> {
    const rows: Row[] = [];
    for (const [title, browsers] of this.results) {
      rows.push({
        title,
        webkit: (browsers['webkit'] ?? 'skip') as BrowserStatus,
        firefox: (browsers['firefox'] ?? 'skip') as BrowserStatus,
      });
    }

    const classify = (r: Row): string => {
      if (r.webkit === 'fail' && r.firefox !== 'fail') return 'safari-only';
      if (r.firefox === 'fail' && r.webkit !== 'fail') return 'firefox-only';
      if (r.webkit === 'fail' && r.firefox === 'fail') return 'both-fail';
      return 'pass';
    };

    const badge = (s: BrowserStatus): string =>
      `<span class="badge ${s}">${s.toUpperCase()}</span>`;

    const body = rows.map(r => {
      const cls = classify(r);
      return `<tr class="${cls}">
    <td>${escapeHtml(r.title)}</td>
    <td>${badge(r.webkit)}</td>
    <td>${badge(r.firefox)}</td>
    <td>${cls}</td>
  </tr>`;
    }).join('\n');

    const html = `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Browser Divergence Matrix</title>
<style>
  body { font-family: system-ui; padding: 1.5rem; }
  table { border-collapse: collapse; width: 100%; }
  th, td { border: 1px solid #ccc; padding: .4rem .7rem; text-align: left; }
  tr.safari-only  td:first-child { background: #fff3cd; }
  tr.firefox-only td:first-child { background: #cfe2ff; }
  tr.both-fail    td:first-child { background: #f8d7da; }
  .badge { font-size: .75rem; font-weight: bold; padding: .15rem .4rem;
           border-radius: 3px; }
  .pass { background:#d1fae5; color:#065f46; }
  .fail { background:#fee2e2; color:#991b1b; }
  .skip { background:#f3f4f6; color:#6b7280; }
</style>
</head>
<body>
<h1>Browser Divergence Matrix</h1>
<p>Generated ${new Date().toISOString()}</p>
<table>
<thead><tr><th>Test</th><th>WebKit (Safari engine)</th><th>Firefox</th><th>Classification</th></tr></thead>
<tbody>
${body}
</tbody>
</table>
</body>
</html>`;

    const outDir = 'playwright-report';
    fs.mkdirSync(outDir, { recursive: true });
    fs.writeFileSync(path.join(outDir, 'divergence-matrix.html'), html, 'utf8');
    console.log(`Divergence matrix written to ${outDir}/divergence-matrix.html`);
  }
}

Classifications used in the output:

This reporter has not been run end-to-end as part of this article; wire it against a small real test file first and confirm that the totals in divergence-matrix.html match the counts in the standard playwright-report/index.html before trusting the classifications in CI.

Generating the Matrix Artefact in CI on Every Push

A GitHub Actions workflow that uploads both reports as a named artefact against a self-hosted Mac runner:

# .github/workflows/browser-matrix.yml
name: Browser regression matrix
on: [push, pull_request]

jobs:
  matrix:
    runs-on: [self-hosted, macOS]   # match the labels on your Figment runner
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install webkit firefox

      - name: Run regression matrix
        run: npx playwright test
        env:
          BASE_URL: ${{ secrets.STAGING_URL }}

      - name: Upload matrix artefact
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: browser-matrix-${{ github.sha }}
          path: playwright-report/
          retention-days: 30

The if: always() clause uploads the artefact even when tests fail — which is exactly when the report is most useful. Point this workflow only at trusted branches; GitHub's current guidance warns that self-hosted runners should almost never be used with pull requests from public forks and recommends ephemeral or just-in-time runners instead (docs.github.com/en/actions/reference/security/secure-use#hardening-for-self-hosted-runners).

Storing Baselines for Nightly Diff Comparison

For the nightly run against main, write the matrix JSON alongside the HTML so the next run can diff which classifications changed. Append this inside onEnd() in divergence-reporter.ts, after the HTML writeFileSync:

const jsonOut = rows.map(r => ({
  title: r.title,
  webkit: r.webkit,
  firefox: r.firefox,
  classification: classify(r),
}));
fs.writeFileSync(
  path.join(outDir, 'divergence-matrix.json'),
  JSON.stringify(jsonOut, null, 2),
  'utf8'
);

In the nightly workflow, download the previous browser-matrix-* artefact, run a sorted diff on the two JSON files, and fail the job when any test moves from pass to safari-only or firefox-only. That surfaces new engine regressions without needing a human to scan the HTML.

Manual Acceptance Worksheet Before Trusting the Report

Before relying on the matrix in a required CI check, walk through this once on the Mac host and tick each item. This is the only way to be sure the reporter counts match reality — the code above is a worked example, not a benchmarked pipeline.

1. Install Node matching your CI image. Run npx playwright install webkit firefox on the Mac. Record the Playwright version from npx playwright --version and pin the exact @playwright/test version in package.json.
2. Create three deliberately divergent test files: one that passes on both engines, one that fails on WebKit only (for example, a CSS or Date-parsing quirk that behaves differently), and one that fails on both.
3. Run npx playwright test locally. Open playwright-report/index.html and note the pass/fail counts per project.
4. Open playwright-report/divergence-matrix.html. Confirm the number of rows equals the number of distinct tests and that the counts of safari-only, firefox-only, both-fail and pass reconcile with step 3.
5. Add a test with the title title with <script>alert(1)</script> and rerun. The HTML artefact must render the literal text in the first column, not execute anything. If a script runs, the escaping step was skipped.
6. Push the branch, confirm the self-hosted Mac runner picks up the job, and download the artefact from the Actions run summary. Compare its contents against your local artefact.

Only after those six steps pass should you wire the nightly diff into a blocking required check.

Picking the Right Figment Mac for This Matrix

The $29 tier is a virtual Intel macOS environment on Linux/KVM with no GPU acceleration and no audio. Playwright's WebKit and Firefox install and run there, which is enough to shake the config down, but codec-sensitive specs (video playback, WebRTC, hardware-accelerated canvas) will not match what Safari users on real Apple hardware see. Use it for reporter development, not for the production matrix. Access to the VM is through the browser console; outbound network from the VM is available so the GitHub Actions runner agent can register against your repository.

An M-series Mac from $95/month is the right target for the recurring matrix. The exact chip, RAM and disk are confirmed when Figment quotes available capacity; physical M-series provisioning is currently request-based, so send the configuration you need and Figment will quote delivery. If you want the CI pipeline, backups and monitoring handled for you, Managed Build Mac at $249/month includes one M-series Mac, CI setup, backups, monitoring, and 30 minutes of monthly support. Managed Mac Team at $999/month adds a second M-series Mac, a second runner, team setup, and one hour of monthly support. For fleets larger than that, request an enterprise quote — fleet size and delivery are quoted per request, not pre-committed.

Start a quote at Figment Mac hosting or provision directly at /mac-hosting/start.

Related guides

• Safari content blocker rule testing on macOS
• iOS Simulator Face ID biometric test
• GitHub Actions self-hosted runner on a Mac