xcrun simctl Screenshot Pipeline for Regression

A screenshot regression pipeline catches pixel-level drift before it lands in main. With xcrun simctl, ImageMagick's compare, and a small Makefile you can wire one into a pull-request check in an afternoon. This page delivers that Makefile, a matching GitHub Actions job, and a direct Figment plan recommendation for the runner.

What xcrun simctl Gives You

xcrun simctl is the command-line interface to the iOS Simulator bundled with Xcode. Apple documents the Simulator and its control commands in Running your app in Simulator or on a device and the archived Interacting with the iOS Simulator guide, which shows the xcrun simctl io booted screenshot form used below.

The sub-commands this pipeline uses:

The exact flags accepted by io screenshot depend on your installed Xcode. Run xcrun simctl help io on your own machine to confirm the flags available there before scripting them into CI; keep PNG as the default, since it's lossless and the only sensible input for a numeric diff.

The Simulator requires macOS with Xcode installed. Figment's $29 virtual Intel macOS plan runs a compatible Intel Xcode 26 or earlier toolchain (Apple's porting guidance says to use Xcode 26 or earlier on Intel hardware), with no GPU or audio. For snappy simulator boots and parallel devices, M-series is the better fit — see the recommendation at the end.

Enumerate Target UDIDs

Before the Makefile runs, pick a stable UDID per device-OS pair. Run this once:

xcrun simctl list devices --json \
  | python3 -c "
import json, sys
for runtime, devices in json.load(sys.stdin)['devices'].items():
    for d in devices:
        if d.get('isAvailable'):
            print(d['udid'], d['name'], runtime)
" | grep -E 'iPhone 15|iPad Pro'

Hardcode the UDIDs you want into a SIMULATORS variable. Querying dynamically at build time can silently pick a different runtime after an Xcode upgrade.

The Makefile

Save this as Makefile at your repo root. It assumes your app is already installed and in the right state on each simulator (via your existing xcodebuild test or xcrun simctl launch step); it only owns boot, screenshot, diff, archive, promote and clean.

# ── Configuration ────────────────────────────────────────────────────────────
THRESHOLD    := 0.02                 # max ImageMagick NRMSE distortion
BASELINE_DIR := regression/baselines
ARCHIVE_ROOT := regression/runs
RUN_DIR      := $(ARCHIVE_ROOT)/$(shell date +%Y%m%d_%H%M%S)

# Space-separated list: UDID|DeviceName
SIMULATORS := \
  AAAAAAAA-0000-0000-0000-000000000001|iPhone15Pro \
  BBBBBBBB-0000-0000-0000-000000000002|iPadPro13

UDIDS := $(foreach p,$(SIMULATORS),$(firstword $(subst |, ,$(p))))

.PHONY: all boot screenshots diff archive promote clean
.SHELLFLAGS := -eu -o pipefail -c
SHELL       := /bin/bash

all: boot screenshots diff archive
	@echo "Regression run complete: $(RUN_DIR)"

$(RUN_DIR):
	@mkdir -p $@ $(BASELINE_DIR)

boot:
	@for u in $(UDIDS); do \
	  xcrun simctl boot $$u 2>/dev/null || true; \
	  until xcrun simctl list devices | grep $$u | grep -q Booted; do sleep 1; done; \
	done

screenshots: | $(RUN_DIR)
	@for p in $(SIMULATORS); do \
	  u=$${p%|*}; n=$${p#*|}; \
	  xcrun simctl io $$u screenshot --type=png $(RUN_DIR)/$$n.png; \
	done

diff: screenshots
	@fail=0; \
	for p in $(SIMULATORS); do \
	  n=$${p#*|}; \
	  baseline=$(BASELINE_DIR)/$$n.png; current=$(RUN_DIR)/$$n.png; \
	  if [ ! -f "$$baseline" ]; then \
	    echo "No baseline for $$n – seeding from this run."; \
	    cp "$$current" "$$baseline"; continue; \
	  fi; \
	  err=$$(compare -metric NRMSE "$$baseline" "$$current" \
	           $(RUN_DIR)/$$n.diff.png 2>&1 >/dev/null || true); \
	  echo "$$n NRMSE=$$err"; \
	  over=$$(awk -v e="$$err" -v t="$(THRESHOLD)" 'BEGIN{print (e+0>t+0)?1:0}'); \
	  if [ "$$over" = "1" ]; then \
	    echo "FAIL: $$n ($$err > $(THRESHOLD))"; fail=1; \
	  else \
	    rm -f $(RUN_DIR)/$$n.diff.png; \
	  fi; \
	done; \
	exit $$fail

archive:
	@tar -czf $(RUN_DIR).tar.gz -C $(ARCHIVE_ROOT) $(notdir $(RUN_DIR))
	@echo "Archived: $(RUN_DIR).tar.gz"

promote:
	@latest=$$(ls -1dt $(ARCHIVE_ROOT)/*/ | head -1); \
	cp $$latest/*.png $(BASELINE_DIR)/
	@echo "Promoted $$latest to baselines."

clean:
	@for u in $(UDIDS); do xcrun simctl shutdown $$u 2>/dev/null || true; done
	@rm -rf $(ARCHIVE_ROOT)

Notes on the choices above:

• compare -metric NRMSE baseline current diff.png is the documented ImageMagick form; it writes err to stderr and the diff image to the named output. convert with a -compare flag is not the right tool for a scored diff.
• .SHELLFLAGS := -eu -o pipefail -c means recipe lines stop on the first failure, so a failed screenshot aborts the run instead of being logged as a fast success.
• The pipeline uses real for loops in one shell, not $(foreach …) expanded into stacked ; lines, so variables like fail and u behave the way you'd expect.
• promote is manual. The first run seeds any missing baselines; after an intentional visual change, run make promote and commit the updated PNGs.

Integrating as a CI Pre-Merge Step

A GitHub Actions job running on a self-hosted Mac runner:

# .github/workflows/screenshot-regression.yml
name: Screenshot Regression
on: [pull_request]

jobs:
  screenshot:
    runs-on: [self-hosted, macOS]
    steps:
      - uses: actions/checkout@v4
      - name: Restore baselines cache
        uses: actions/cache@v4
        with:
          path: regression/baselines
          key: baselines-${{ github.base_ref }}
      - name: Ensure ImageMagick
        run: command -v compare || brew install imagemagick
      - name: Run screenshot regression
        run: make
      - name: Upload diff archive on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: screenshot-diffs
          path: regression/runs/*.tar.gz

If you register the runner on a hosted Mac, follow GitHub's current hardening guide for self-hosted runners and keep the runner restricted to trusted repositories.

Archive Folder Layout

Each run lands in regression/runs/YYYYMMDD_HHMMSS/:

regression/
  baselines/
    iPhone15Pro.png
    iPadPro13.png
  runs/
    20261108_143022/
      iPhone15Pro.png
      iPadPro13.png
      iPhone15Pro.diff.png   # only on threshold breach
    20261108_143022.tar.gz

Keeping baselines in git and run archives as CI artifacts lets you diff baselines themselves across commits — useful when a designer intentionally ships a visual change.

Sizing and the Right Figment Plan

A worked example for storage, not a benchmark. Assume a 350 KB PNG per device, 10 devices per run, 500 runs:

• Raw PNGs: 350 KB × 10 × 500 = 1,750,000 KB ≈ 1.75 GB
• Gzipped tarballs (rule-of-thumb ~0.3× for screenshots): ≈ 525 MB
• Baselines on disk: 10 × 350 KB ≈ 3.5 MB

Even an entry virtual plan has room for active runs if you prune or offload archives.

Direct recommendation for this job:

• Default pick: a Figment M-series Mac from $95/month for the runner. Exact chip, RAM and disk are confirmed on the quote, so request the spec you need and ask about availability on the start page.
• If you want the pipeline configured and watched: Managed Build Mac $249/month includes one M-series Mac, CI setup, backups, monitoring, and 30 minutes of monthly support. Team-scale with two Macs and two runners is Managed Mac Team $999/month.
• $29 virtual Intel macOS (2 vCPU, 8 GiB RAM, 80 GiB disk, browser console access) runs Intel Xcode 26 or earlier and can drive iOS Simulator for smoke-level screenshot checks. It has no GPU or audio and is not an Apple silicon machine, so parallel simulator fan-out and arm64 execution should go to M-series.
• Physical Intel ($49 i3) and physical M-series inventory is request-based; the start page records interest until supply confirms. Sequoia is being provisioned on the VM; Tahoe is not promised. There is no public trial, SLA, or discount beyond what is written into the quote.

Pricing details live on the mac hosting page; taxes and any setup fees are stated on the actual offer at checkout. Full plan with CI setup and monitoring: start a request at /mac-hosting/start.

Related reading

• xctest-failure-screenshot-ci — capture Xcode failure screenshots inside XCTest
• safari-webauthn-passkey-test — test passkey flows in a real Safari on macOS
• github-actions-self-hosted-runner — register a hosted Mac as a GitHub Actions runner