iOS Simulator ATS Exception Verification Test

Start here. App Transport Security (ATS) decisions are read from the running app's Info.plist, not toggled at runtime, so the only honest way to prove which exception keys permit which loads is to build one scheme per configuration, drive a URLSession request from an XCTest, and record whether the response was an HTTP status, an ATS block (NSURLErrorDomain code -1022), or a different network error. This article gives you a compact XCTest, a fill-in worksheet you can paste into a review packet, and a plan pick for the Figment Mac that fits the job.

A Figment virtual Intel macOS instance at $29/month (2 vCPU, 8 GiB RAM, 80 GiB disk; browser-console access, no inbound SSH) is a reasonable host when your project builds with Xcode 26 or earlier and uses a Simulator runtime that still supports Intel. If you need Xcode 27, visionOS, ARM-only runtimes, or several Simulator devices in parallel, pick the Figment M-series Mac from $95/month. Exact chip, RAM, disk and macOS version are set in the quote; physical plans collect requests until supply is confirmed.

The keys the test has to cover

The authoritative list of keys lives on Apple's NSAppTransportSecurity property-list reference. Your truth table should have one row per key you actually set, plus a control row with no exceptions at all.

The two keys that cost the most review time are NSAllowsArbitraryLoadsInWebContent (teams expect it to cover URLSession; it does not) and per-domain entries that omit NSIncludesSubdomains and therefore silently miss API subdomains.

Endpoints you can assert against honestly

The test host must reach two endpoints whose reachability is controlled by you, otherwise an ATS block becomes indistinguishable from DNS failure, a captive portal, or a certificate error. Two options work well on an iOS Simulator running on a Mac:

1. Local servers on the host, reached through the Simulator's loopback. Add NSAllowsLocalNetworking (or a per-domain exception for localhost) when the scheme you are proving needs the local row to succeed.
2. A domain you own with a plain-HTTP vhost and a TLS vhost on the same name, published through DNS. This lets you prove the base-domain and subdomain rows for NSIncludesSubdomains.

If you use a local HTTPS listener with a self-signed certificate, install the root into the booted Simulator with xcrun simctl keychain <device> add-root-cert <cert.pem> so a URLSession failure signals ATS rather than certificate-trust rejection. Record the Simulator runtime version you used; ATS behaviour for local networking has changed across major OS versions and your evidence should pin the row to a runtime.

A compact XCTest that records the actual outcome

The example below is written for Xcode 16.4's XCTest and is not an executed benchmark. Replace the placeholders with endpoints your app actually hits, make sure the containing scheme's test-host Info.plist has the ATS keys you want to prove, and run the scheme locally before you rely on the asserted outcomes. The async pattern follows Apple's XCTestExpectation reference.

// ATSTruthTableTests.swift
import XCTest

final class ATSTruthTableTests: XCTestCase {

    struct ATSRow: Encodable {
        let scheme: String
        let label: String
        let url: String
        let expected: String   // "allow" or "block"
        let observed: String   // "allow", "ats-block", or "other-error"
        let httpStatus: Int?
        let errorDomain: String?
        let errorCode: Int?
    }

    // Supply these from the scheme's environment variables.
    // Each scheme supplies its own matrix matching its Info.plist keys.
    private func matrix() -> [(label: String, url: String, expectAllow: Bool)] {
        let env = ProcessInfo.processInfo.environment
        let httpHost  = env["ATS_HTTP_HOST"]  ?? "localhost:8080"
        let httpsHost = env["ATS_HTTPS_HOST"] ?? "localhost:8443"
        return [
            ("control-https", "https://\(httpsHost)/ping", true),
            ("control-http",  "http://\(httpHost)/ping",   false),
        ]
    }

    func testATSTruthTable() throws {
        let schemeName = ProcessInfo.processInfo.environment["ATS_SCHEME"] ?? "ATSDefault"
        let session = URLSession(configuration: .ephemeral)
        var rows: [ATSRow] = []

        for tc in matrix() {
            guard let url = URL(string: tc.url) else {
                XCTFail("Invalid URL: \(tc.url)"); continue
            }
            let done = XCTestExpectation(description: "load \(tc.label)")
            var observed = "other-error"
            var status: Int? = nil
            var errDomain: String? = nil
            var errCode: Int? = nil

            let task = session.dataTask(with: url) { _, response, error in
                defer { done.fulfill() }
                if let http = response as? HTTPURLResponse {
                    observed = "allow"
                    status = http.statusCode
                } else if let e = error as NSError? {
                    errDomain = e.domain
                    errCode = e.code
                    if e.domain == NSURLErrorDomain && e.code == -1022 {
                        observed = "ats-block"
                    }
                }
            }
            task.resume()
            wait(for: [done], timeout: 15)

            rows.append(ATSRow(
                scheme: schemeName, label: tc.label, url: tc.url,
                expected: tc.expectAllow ? "allow" : "block",
                observed: observed, httpStatus: status,
                errorDomain: errDomain, errorCode: errCode
            ))

            // Only assert ATS behaviour. An "other-error" row is logged but
            // neither passes nor fails: fix the endpoint and re-run.
            if observed != "other-error" {
                XCTAssertEqual(
                    observed == "allow", tc.expectAllow,
                    "ATS mismatch [\(tc.label)]: expected \(tc.expectAllow ? "allow" : "block"), observed \(observed)"
                )
            }
        }

        let encoder = JSONEncoder()
        encoder.outputFormatting = [.prettyPrinted, .sortedKeys]
        let data = try encoder.encode(rows)
        let attachment = XCTAttachment(
            uniformTypeIdentifier: "public.json",
            name: "ats-truth-table-\(schemeName).json",
            payload: data
        )
        attachment.lifetime = .keepAlways
        add(attachment)
    }
}

Points worth reading twice:

• The other-error branch means the row is not usable evidence. A DNS failure, a TLS trust failure, or a timeout all land there. Fix the endpoint before you claim ATS proved anything.
• Only -1022 in NSURLErrorDomain counts as an ATS block. Other codes (for example -1200 secure-connection-failed) can be a trust-store or TLS-negotiation problem rather than an ATS decision.
• A per-scheme ATS_SCHEME variable keeps the attachment filenames distinct so xcrun xcresulttool merge output is easy to diff.

Scheme matrix and manual acceptance worksheet

ATS keys live in the app's Info.plist, so cover the key matrix by building one scheme per configuration and reusing the same test class. For each scheme, run the test once on your target Simulator runtime and paste the filled worksheet into the review packet.

Worksheet rows (fill per run, attach beside the xcresult):

• Xcode version and build (xcodebuild -version): _______
• Simulator runtime (device + iOS version): _______
• Scheme name: _______
• ATS_HTTP_HOST / ATS_HTTPS_HOST: _______
• For each matrix row — label, observed, httpStatus, errorDomain, errorCode: _______
• Reviewer sign-off and date: _______

Run each scheme in CI, merge the attached JSON with xcrun xcresulttool merge, and commit the merged truth table alongside your review notes. The combination of the merged JSON and a signed worksheet is the artifact reviewers can actually diff between releases.

Running it on a Figment Mac

Per Apple's porting-to-Apple-silicon guide, running Xcode on an Intel processor requires Xcode 26 or earlier; Xcode 27 itself is Apple silicon only. Not every current Simulator runtime still supports Intel — check the Xcode release notes for the exact pair before you commit to the VM.

The Figment virtual Intel macOS instance at $29/month is a real macOS environment reached through a browser console. It is enough for a single Simulator and the matrix above if your toolchain fits Xcode 26 or earlier. There is no inbound SSH to the VM; your tests either run in the desktop session or your CI runner connects outbound to the service you already use.

For Xcode 27, visionOS, parallel Simulator devices, or ARM-only dependencies, the Figment M-series Mac starts at $95/month and the exact configuration (chip, RAM, disk, macOS version) is confirmed in the quote. Physical plans collect requests until supply is confirmed, so state your Xcode version, required Simulator runtimes, and parallelism up front.

Run ATS verification on a Figment Mac or request a configuration quote and describe the test matrix you plan to run so the plan matches the job.

Related testing guides:

• Safari and Firefox Regression Matrix
• Safari Crash Log Evidence Collection
• GitHub Actions Self-Hosted Runner on Mac