Resume Hosted Mac Jobs After a Network Failure

A hosted Mac automation job that exits silently during a network blip leaves you with two problems: get the job running again, and figure out whether the previous run committed side effects you now need to reconcile. This guide solves both with launchd supervision, an idempotent wrapper, and a decision tree for reconciliation.

Buying recommendation. For a script-only agent (Python, Node.js, curl, shell) that calls external APIs, start on the Figment virtual Intel macOS environment at $29/month. If the job must execute ARM-native binaries or Apple-silicon-only apps, request an M-series Mac starting at $95/month; the exact chip, RAM and disk are set in the quote. Physical i3 and M-series plans are provisioned on request, so confirm availability before you cut over. Start a configuration.

Choose a Mac plan and define the job

Pick the plan that matches what your job actually executes. Current Figment monthly rates:

If your job is a standard script — polling an API, transforming JSON, pushing artifacts — the $29 virtual Intel macOS runs it without issue. It is a full Intel macOS environment, not an SSH hop, and it can host supported Intel toolchains (Xcode 26 or earlier). ARM execution, GPU acceleration, audio and Xcode 27 require Apple silicon, so route those jobs to an M-series. Physical plans collect requests until supply is confirmed; ask on the configuration form.

Access to the virtual Mac is through the browser console. Outbound connections from the VM — CI agents, API calls, webhooks — work normally. Inbound public SSH to the VM is not currently implemented, so run commands in the desktop terminal or through an outbound CI agent that reaches back to your orchestrator.

Inputs and required permissions

Before writing a single recovery handler, define four things:

• Trigger — a cron schedule via launchd, an inbound webhook delivered through an outbound agent, a file watch, or a shortcuts run call from a remote orchestrator.
• State boundary — which side effects are write-once (an API debit, a sent message) versus safely retried on duplicate submission with the same idempotency key.
• Credentials — stored in your CI secret manager and injected as environment variables at run time. Do not export the whole environment to disk; it can contain unrelated secrets.
• Idempotency key — a unique token (run ID + UTC timestamp) written at job start and queryable from the downstream API at resume.

The macOS Shortcuts CLI lets any remote orchestrator fire a named shortcut. Per Apple's Run shortcuts from the command line, shortcuts run "ProcessBatch" --input-path /data/batch.json returns 0 on success and 1 on failure, which a wrapper script can capture before writing a state flag.

Implement the concrete workflow

The pattern: a shell wrapper tracks job state in a file; launchd supervises the wrapper and restarts it when the network returns.

#!/usr/bin/env bash
# /usr/local/bin/run-agent-job.sh
# Example — adapt the shortcut name, API URL and paths to your project.
set -euo pipefail

STATE_DIR="$HOME/agent"
STATE_FILE="$STATE_DIR/job.state"
LOCK_DIR="$STATE_DIR/job.lock.d"
mkdir -p "$STATE_DIR"

# Prevent concurrent reruns (mkdir is atomic)
if ! mkdir "$LOCK_DIR" 2>/dev/null; then
  echo "Job already running" >&2; exit 1
fi
trap 'rmdir "$LOCK_DIR" 2>/dev/null || true' EXIT

# Reconcile any prior partial run
if [[ -f "$STATE_FILE" ]]; then
  prior=$(cat "$STATE_FILE")
  if [[ "$prior" != "COMPLETE" ]]; then
    echo "Prior run ended at: $prior — running reconciliation" >&2
    /usr/local/bin/reconcile-job.sh "$prior" || exit 1
  fi
fi

echo "STARTED_$(date -u +%Y%m%dT%H%M%SZ)" > "$STATE_FILE"

# Confirm the target API is reachable before proceeding
curl -sf --max-time 10 https://api.example.com/health \
  || { echo "API not reachable yet" >&2; exit 1; }

# Main job
shortcuts run "ProcessBatch" --input-path /data/batch.json
echo "COMPLETE" > "$STATE_FILE"

Save this plist as ~/Library/LaunchAgents/com.yourteam.agentjob.plist and load it with launchctl load ~/Library/LaunchAgents/com.yourteam.agentjob.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.yourteam.agentjob</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/bin/run-agent-job.sh</string>
  </array>
  <key>KeepAlive</key>
  <dict>
    <key>NetworkState</key><true/>
    <key>SuccessfulExit</key><false/>
  </dict>
  <key>ThrottleInterval</key><integer>60</integer>
  <key>StandardOutPath</key><string>/tmp/agentjob.log</string>
  <key>StandardErrorPath</key><string>/tmp/agentjob.err</string>
</dict>
</plist>

Per the launchd.plist reference on launchd.info, KeepAlive → NetworkState: true starts the job while any network interface is available, and SuccessfulExit: false keeps relaunching until the wrapper exits 0. ThrottleInterval: 60 enforces a 60-second pause between invocations, preventing a tight crash loop. For rate-limited downstream APIs, raise it to 300 seconds.

launchd considers the network available as soon as any interface holds an IP address. That does not guarantee internet reachability or that your specific API is healthy — the explicit curl probe inside the wrapper closes that gap.

Network recovery decision tree

After an unexpected job exit, walk this tree to pick the right recovery action:

Job exited with non-zero status
│
├── STATE_FILE = STARTED_<timestamp>   ← partial run
│   ├── Are all side effects idempotent for this run ID?
│   │   ├── YES → let launchd restart; wrapper retries safely
│   │   └── NO  → run reconcile-job.sh <timestamp>
│   │             query the API for records already committed
│   │             de-duplicate, then allow restart
│   └── Lock dir stale (process gone, lock dir present)?
│       └── rmdir "$LOCK_DIR"; run reconciliation; restart
│
├── STATE_FILE = COMPLETE              ← clean exit
│   └── Check upstream trigger; no reconciliation needed
│
└── STATE_FILE missing                 ← job never started
    └── Restart is clean; no reconciliation needed

Side-effect reconciliation checklist:

• Query the downstream API for records tagged with the current idempotency key.
• Delete or skip duplicates before resubmitting.
• Rotate or void any one-time tokens consumed during the failed run.
• Append a log entry: timestamp, prior state, reconciliation action, outcome.

Failure and recovery checks

After the network returns, confirm the job recovered:

1. tail -50 /tmp/agentjob.log — look for a gap of at least ThrottleInterval seconds between the failed entry and the restart. A shorter gap means a non-network bug.
2. launchctl list | grep agentjob — a numeric PID column means the job is active; the exit-status column holds the last exit code.
3. cat ~/agent/job.state — COMPLETE means the last full run succeeded. A STARTED_ timestamp that persists across two restarts points to a deterministic failure, not a transient drop.
4. Direct probe: curl -sf https://api.example.com/health && echo ok — distinguishes a machine-level network issue from an upstream API outage.

Cost and operating limits

$29 is a monthly flat rate on the virtual Intel macOS. A two-week run still costs the full month; there is no published daily or pro-rated term. If your automation runs continuously and you want someone watching it, the $249 Managed Build Mac is one M-series Mac with CI setup, backups, monitoring, and 30 minutes of monthly support included — useful when you want stalled jobs surfaced before you discover them hours later. For a two-runner team setup with a one-hour monthly support allowance, the Managed Mac Team plan is $999.

Factor downstream rate limits into ThrottleInterval. A quota of 60 requests per minute should not be retried every 10 seconds during recovery — 60 to 120 seconds keeps you within the window while still restoring quickly.

Start or migrate the workload

Request a Figment Mac on the configuration form, copy the plist and wrapper into place, inject credentials from your CI secret manager, and load the agent with launchctl load. Confirm hardware availability in the quote before cutting over.

Get started with a hosted Mac →

Related guides:

• Mac agent log redaction
• Mac agent run audit
• Mac GitHub Actions self-hosted runner