Running Safari Tests Across Multiple macOS Versions

Recommendation: rent two Macs — one per macOS version you need to cover — and run the same Playwright WebKit suite against each. Two Figment M-series Macs from $95/month each ($190/month total) is the direct-cost path; the Managed Mac Team at $999/month (two M-series Macs, two CI runners, team setup, backups, monitoring and one hour of monthly support) is the right tier if you want the CI plumbing handled for you. Figment M-series plans are request-based: the exact chip, RAM, disk, macOS version per machine and SSH access are confirmed in the configuration quote at /mac-hosting/start. Verify current Safari/macOS pairings in Apple's Safari release notes before you commit.

You cannot downgrade macOS on a running machine to get an older Safari, and Apple does not ship older Safari installers for a newer OS. The only reliable way to hold two specific macOS/Safari combinations at once is to hold two separate machines.

Why Two Machines, Not One

Each macOS major version is paired with a Safari major version that updates along a single track. From Apple's Safari release notes as of 2026-11-10:

Confirm the exact Safari minor your rented Mac reports with safaridriver --version and sw_vers; do not assume the table above matches a specific image until the machine is provisioned. If you upgrade one Mac to the newest track, you lose the older environment — hence the two-machine approach.

Picking the Two Versions to Cover

Pull Safari OS-version share from your analytics for the last 90 days and pick the two non-current versions with the largest share. A compact one-liner against a CSV export:

# analytics-export.csv columns: date, path, ua_family, os_version
awk -F',' '$3 == "Safari" { print $4 }' analytics-export.csv \
  | sort | uniq -c | sort -rn | head -10

Retire a dedicated test Mac for a version once it drops below roughly 2% of your Safari sessions. The marginal bugs it catches rarely justify the monthly cost at that point.

Provisioning Two Figment M-series Macs

Two Figment M-series Macs from $95/month each totals from $190/month. $95 is the M-series entry price; the specific chip generation, RAM and disk come back on the quote and are not implied by the headline price. When you file the request at /mac-hosting/start, include:

• The macOS version you want installed on each machine (for example, Sequoia 15 on Mac 1 and Sonoma 14 on Mac 2). Figment's $29 virtual Intel macOS VM is being installed with Sequoia and is not a target here — it is Intel on Linux/KVM without inbound SSH, so use physical M-series for the per-OS pair.
• Inbound SSH on each physical Mac, with your CI public key authorised, confirmed in the quote before you commit.
• Any inbound firewall exceptions for your artifact store or internal test URL.

Worked cost over a year, two self-managed M-series Macs: 2 × $95 × 12 = $2,280/year from Figment's entry price. The Managed Mac Team at $999/month totals $11,988/year, a $9,708 premium that buys two configured CI runners, team setup, backups, monitoring and one hour of monthly support. Only pick Managed Team if that premium is cheaper than what your team would otherwise spend building and maintaining the same pipeline.

If you only need a single Mac and plan to buy a full year upfront, a competing provider lists a lower annual: CloudMac publishes $99.99/month and $999/year for a dedicated M4/16 GB/256 GB on its pricing page and home page. That $999/year is $141 less than 12 × $95 for a specific M4/16 GB config, so if an M4/16 GB spec is exactly what you need and you are happy with one machine, price-check before you decide. Figment's $95 entry does not guarantee M4/16 GB — the quote names the exact hardware — and the Managed Team work is a different scope, not a hardware match.

SSH, Sync and Playwright on Both Macs

Add your CI public key to each Figment Mac at provisioning (or append it to ~/.ssh/authorized_keys on the machine), then verify connectivity before wiring up the Makefile:

ssh -o BatchMode=yes -o ConnectTimeout=10 user@mac-sequoia.example.com \
  'sw_vers -productVersion && /usr/bin/safaridriver --version'
ssh -o BatchMode=yes -o ConnectTimeout=10 user@mac-sonoma.example.com \
  'sw_vers -productVersion && /usr/bin/safaridriver --version'

You should see the macOS version strings (15.x, 14.x) and the matching Safari build. If either times out, confirm Remote Login is on under System Settings → General → Sharing on that Mac, and that your network path reaches it.

The Makefile below uses rsync to push your repository to both Macs, ssh to run npx playwright test --project webkit on each, and rsync to pull reports back. Playwright's generic CI setup, including the --reporter flag and HTML report output path, is documented at playwright.dev/docs/ci.

The Makefile: Parallel Cross-OS Safari Runner

# cross-os-safari.mk — run the same Playwright WebKit suite on two Macs
# Usage:
#   MAC_V1=user@host1 MAC_V2=user@host2 TEST_URL=https://staging.example.com \
#     make -f cross-os-safari.mk test-both

MAC_V1     ?= user@mac-sequoia.example.com
MAC_V2     ?= user@mac-sonoma.example.com
LABEL_V1   ?= macOS Sequoia 15 - Safari 18
LABEL_V2   ?= macOS Sonoma 14 - Safari 17
REMOTE_DIR := safari-cross-os
REPORT_DIR := reports
REPORT_V1  := $(REPORT_DIR)/v1
REPORT_V2  := $(REPORT_DIR)/v2

.PHONY: dirs sync-v1 sync-v2 test-v1 test-v2 test-both merge clean

dirs:
	mkdir -p $(REPORT_V1) $(REPORT_V2)

sync-v1:
	rsync -az --delete --exclude node_modules --exclude .git \
	  ./ $(MAC_V1):$(REMOTE_DIR)/
	ssh $(MAC_V1) "cd $(REMOTE_DIR) && npm ci --quiet && npx playwright install webkit"

sync-v2:
	rsync -az --delete --exclude node_modules --exclude .git \
	  ./ $(MAC_V2):$(REMOTE_DIR)/
	ssh $(MAC_V2) "cd $(REMOTE_DIR) && npm ci --quiet && npx playwright install webkit"

test-v1: dirs sync-v1
	ssh $(MAC_V1) "cd $(REMOTE_DIR) && TEST_URL='$(TEST_URL)' \
	  npx playwright test --project=webkit --reporter=html,json" \
	  || { echo 'Mac V1 run failed'; exit 1; }
	rsync -az $(MAC_V1):$(REMOTE_DIR)/playwright-report/ $(REPORT_V1)/
	rsync -az $(MAC_V1):$(REMOTE_DIR)/test-results.json  $(REPORT_V1)/results.json

test-v2: dirs sync-v2
	ssh $(MAC_V2) "cd $(REMOTE_DIR) && TEST_URL='$(TEST_URL)' \
	  npx playwright test --project=webkit --reporter=html,json" \
	  || { echo 'Mac V2 run failed'; exit 1; }
	rsync -az $(MAC_V2):$(REMOTE_DIR)/playwright-report/ $(REPORT_V2)/
	rsync -az $(MAC_V2):$(REMOTE_DIR)/test-results.json  $(REPORT_V2)/results.json

# -j2 runs test-v1 and test-v2 in parallel
test-both:
	$(MAKE) -f cross-os-safari.mk -j2 test-v1 test-v2
	$(MAKE) -f cross-os-safari.mk merge

merge: $(REPORT_V1)/results.json $(REPORT_V2)/results.json
	node scripts/merge-reports.js \
	  --v1 $(REPORT_V1) --label-v1 "$(LABEL_V1)" \
	  --v2 $(REPORT_V2) --label-v2 "$(LABEL_V2)" \
	  --out $(REPORT_DIR)/comparison.html
	@echo "Open $(REPORT_DIR)/comparison.html"

clean:
	rm -rf $(REPORT_DIR)

The --reporter=html,json output path and the test-results.json filename follow Playwright's documented defaults; configure reporter in playwright.config.ts if you have customised them. The example is illustrative — run it against your own project and read the exit codes, do not treat a green make as a benchmark.

Merging the Two HTML Reports

scripts/merge-reports.js reads both JSON result files, pairs specs by title, and writes one HTML table linking to each full report:

#!/usr/bin/env node
'use strict';
const { readFileSync, writeFileSync } = require('fs');
const { join } = require('path');

const args = process.argv.slice(2);
const get = (f) => { const i = args.indexOf(f); return i !== -1 ? args[i + 1] : null; };

const v1dir = get('--v1');
const v2dir = get('--v2');
const labelV1 = get('--label-v1') || 'V1';
const labelV2 = get('--label-v2') || 'V2';
const outFile = get('--out') || 'comparison.html';

const v1 = JSON.parse(readFileSync(join(v1dir, 'results.json'), 'utf8'));
const v2 = JSON.parse(readFileSync(join(v2dir, 'results.json'), 'utf8'));

function flatten(suites, path = []) {
  const out = [];
  for (const s of suites || []) {
    const next = [...path, s.title].filter(Boolean);
    for (const sp of s.specs || []) {
      out.push({ key: [...next, sp.title].join(' > '), ok: sp.ok });
    }
    out.push(...flatten(s.suites, next));
  }
  return out;
}

const map1 = new Map(flatten(v1.suites).map((r) => [r.key, r.ok]));
const map2 = new Map(flatten(v2.suites).map((r) => [r.key, r.ok]));
const keys = [...new Set([...map1.keys(), ...map2.keys()])].sort();

const cell = (v) => v === undefined ? '-' : v ? 'PASS' : 'FAIL';
const rows = keys
  .map((k) => `<tr><td>${k}</td><td>${cell(map1.get(k))}</td><td>${cell(map2.get(k))}</td></tr>`)
  .join('\n');

writeFileSync(outFile, `<!doctype html><html lang="en"><head><meta charset="utf-8">
<title>Safari Cross-OS Comparison</title>
<style>body{font-family:system-ui,sans-serif;margin:2rem}
table{border-collapse:collapse;width:100%}th,td{border:1px solid #ccc;padding:.4rem .75rem}
th{background:#f4f4f4}td:nth-child(2),td:nth-child(3){text-align:center;font-weight:600}</style>
</head><body>
<h2>Safari Cross-OS Comparison</h2>
<p>Generated ${new Date().toISOString()}</p>
<table><thead><tr><th>Test</th><th>${labelV1}</th><th>${labelV2}</th></tr></thead>
<tbody>${rows}</tbody></table>
<p>Full reports: <a href="v1/index.html">${labelV1}</a> &middot;
<a href="v2/index.html">${labelV2}</a></p>
</body></html>`);
console.log('Written:', outFile);

Any test that is PASS in one column and FAIL in the other is an OS-level Safari regression worth filing immediately — those are the bugs you cannot reproduce without a dedicated older-OS Mac.

Running It, and Retiring a Rental

Archive reports/comparison.html as a dated CI artifact per build. When a user reports a scroll or layout bug in an older Safari, you can grep historical artifacts for the first build where the row flipped from PASS to FAIL, which is far faster than reproducing against a one-off rental.

Re-run the analytics share check every quarter. When a covered macOS version drops below 2% of your Safari sessions, cancel that Mac and redirect the ~$95/month to coverage on whatever Safari track has climbed in its place.

Start a configuration request at /mac-hosting/start or review the plans at /mac-hosting.

Related reading

• Simctl screenshot regression archive
• Screen recording in XCUITest
• Self-hosted GitHub Actions runner on macOS