Send Useful Failure Alerts from Hosted Mac Jobs

When an automation job fails silently on a hosted Mac the problem compounds: the queue backs up, an upstream service stalls, and you hear about it hours later from a downstream complaint. The fix is not another dashboard. It is a small alert helper every job can call in one line, with a deduplication window so a stuck failure does not flood the channel.

This guide gives you a working helper script, a payload structure that fits how Slack incoming webhooks actually behave, and a short acceptance worksheet so you can verify the helper on your own hosted Mac before you depend on it.

Choose a Figment plan for this job

Alert delivery — a JSON POST via curl to a Slack incoming webhook — is negligible work. Any Figment plan handles it, so the real decision is about the companion workload the alerter sits next to.

• $29/month virtual Intel macOS (2 vCPU, 8 GiB RAM, 80 GiB disk) is enough if alerting lives beside a light cron/launchd pipeline that is already happy on Intel. It is a virtual Intel macOS environment on Linux/KVM — no GPU acceleration, no audio, no ARM execution. VM capacity is being verified, so confirm availability when you order.
• M-series from $95/month fits if the alerter rides along with device builds, Xcode simulators that need Apple silicon, or arm64 execution. The $95 entry is an unspecified M-series configuration; the exact chip, RAM, and disk are confirmed in your quote, and physical M-series supply is request-based.
• Managed Build Mac $249/month adds CI setup, backups, monitoring, and 30 minutes of monthly support on one M-series Mac — useful when you want someone else to wire the launchd plist and the alerting side together.

Outbound HTTPS to hooks.slack.com (or your own incident endpoint) is the only network requirement. The $29 VM is reached through its browser console; there is no public inbound SSH, so if a receiving system restricts source IPs you update its allowlist from the egress IP you observe on the host. Slack does not publish a per-workspace webhook source range, and Apple does not publish one for its developer services either.

Pick a plan and start →

What incoming webhooks actually accept

Two details trip people up, and both are documented on Slack's incoming webhooks page:

• The destination channel is bound to the webhook URL when the app is installed. A channel field in the JSON body is ignored. If you want #ops-alerts and #ops-critical to receive different alerts, install the app twice (once per channel) and store two webhook URLs.
• The minimum required field is text. Rich layout is done with blocks; the older attachments array still works but the docs now call it legacy. The script below sends a short top-level text for notification previews plus a blocks array for readable formatting.

Those two facts drive the payload design that follows.

Inputs and required permissions

The script needs very little:

• SLACK_WEBHOOK_URL (normal severity) and optionally SLACK_WEBHOOK_URL_CRITICAL (installed on a separate channel), both stored in your CI service's secret manager — GitHub Actions secret, Vault, 1Password CLI — and injected only into the job that posts the alert. Do not export the entire environment to a plaintext file; it can leak unrelated keys. Never commit the URLs.
• curl and jq on the Mac. curl ships with macOS; install jq with brew install jq so JSON escaping is safe for arbitrary error text.
• Write access to a local dedup state directory under $HOME.
• No TCC prompts — the script only hits the network and reads/writes files in $HOME.

To create each webhook, follow Slack's incoming webhooks guide and paste each URL into your secret manager. Load it into an interactive shell with read -s SLACK_WEBHOOK_URL; export SLACK_WEBHOOK_URL so the literal never lands in shell history.

The alert payload structure

A useful alert answers four questions immediately: what broke, where, when, and what to look at first.

Implement the alert helper

Save alert.sh to ~/scripts/alert.sh. This is an example to review and execute against your own webhook in a test channel; treat the acceptance worksheet below as the gate before depending on it.

#!/usr/bin/env bash
# alert.sh — structured Slack alert with dedup window and escalation count.
# Usage: alert.sh <job> <severity> <step> <message> [extra-context]
# Requires: SLACK_WEBHOOK_URL (and optionally SLACK_WEBHOOK_URL_CRITICAL).
set -euo pipefail

JOB="$1"
SEVERITY="$2"
STEP="$3"
MESSAGE="$4"
CONTEXT="${5:-}"

: "${SLACK_WEBHOOK_URL:?Set SLACK_WEBHOOK_URL in environment}"
command -v jq   >/dev/null || { echo "jq not installed (brew install jq)"   >&2; exit 2; }
command -v curl >/dev/null || { echo "curl not installed"                   >&2; exit 2; }

# Sanitize identifiers that become filenames.
safe() { printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_'; }
JOB_S=$(safe "$JOB")
STEP_S=$(safe "$STEP")

HOST=$(hostname -s)
RUN_ID=$(date +%Y%m%d-%H%M%S)
SEVERITY_UP=$(printf '%s' "$SEVERITY" | tr '[:lower:]' '[:upper:]')

umask 077
DEDUP_DIR="${HOME}/.alert-dedup"
mkdir -p "$DEDUP_DIR"

# --- Deduplication: suppress if same job+step fired within the last 5 minutes.
DEDUP_LOCK="${DEDUP_DIR}/${JOB_S}__${STEP_S}.lock"
if [[ -f "$DEDUP_LOCK" ]]; then
    LAST_FIRED=$(stat -f %m "$DEDUP_LOCK")
    NOW=$(date +%s)
    if (( NOW - LAST_FIRED < 300 )); then
        echo "SUPPRESSED (dedup ${JOB_S}/${STEP_S}, $(( 300 - (NOW - LAST_FIRED) ))s left)" >&2
        exit 0
    fi
fi
touch "$DEDUP_LOCK"

# --- Escalation count: consecutive failures for this job+step.
ESCALATION_FILE="${DEDUP_DIR}/${JOB_S}__${STEP_S}.count"
COUNT=1
if [[ -f "$ESCALATION_FILE" ]]; then
    COUNT=$(( $(cat "$ESCALATION_FILE") + 1 ))
fi
echo "$COUNT" > "$ESCALATION_FILE"

# --- Route by severity.
# Slack binds the channel to the webhook URL at install time, so a "channel"
# field in the payload is ignored. Install the app twice if you want two
# channels and set SLACK_WEBHOOK_URL_CRITICAL to the second URL.
WEBHOOK="$SLACK_WEBHOOK_URL"
if [[ "$SEVERITY" == "critical" ]] || (( COUNT >= 3 )); then
    WEBHOOK="${SLACK_WEBHOOK_URL_CRITICAL:-$SLACK_WEBHOOK_URL}"
fi

# --- Build the payload with jq so arbitrary error text is escaped correctly.
PAYLOAD=$(jq -n \
  --arg sev   "$SEVERITY_UP" \
  --arg job   "$JOB" \
  --arg step  "$STEP" \
  --arg msg   "$MESSAGE" \
  --arg ctx   "$CONTEXT" \
  --arg host  "$HOST" \
  --arg run   "$RUN_ID" \
  --arg count "$COUNT" \
  '{
    text: ("[" + $sev + "] " + $job + " > " + $step + ": " + $msg),
    blocks: [
      { type: "header",
        text: { type: "plain_text",
                text: ("[" + $sev + "] " + $job + " > " + $step) } },
      { type: "section",
        fields: [
          { type: "mrkdwn", text: ("*Message*\n" + $msg) },
          { type: "mrkdwn", text: ("*Context*\n" + (if $ctx == "" then "-" else $ctx end)) },
          { type: "mrkdwn", text: ("*Host*\n" + $host) },
          { type: "mrkdwn", text: ("*Run ID*\n" + $run) },
          { type: "mrkdwn", text: ("*Escalation*\n#" + $count) }
        ] }
    ]
  }')

RESPONSE_FILE=$(mktemp -t alert-last.XXXXXX)
HTTP_STATUS=$(curl -sS -o "$RESPONSE_FILE" -w "%{http_code}" \
  -X POST -H 'Content-type: application/json' \
  --data "$PAYLOAD" "$WEBHOOK")

if [[ "$HTTP_STATUS" != "200" ]]; then
    echo "ALERT_SEND_FAILED: HTTP ${HTTP_STATUS} body=$(cat "$RESPONSE_FILE")" >&2
    rm -f "$RESPONSE_FILE"
    exit 1
fi
rm -f "$RESPONSE_FILE"
echo "Alert sent: ${JOB_S}/${STEP_S} escalation=#${COUNT}"

Make it executable with chmod +x ~/scripts/alert.sh. Clear the counter and lock on a clean run:

alert_reset() {
    local job step job_s step_s
    job="$1"; step="$2"
    job_s=$(printf '%s' "$job"  | tr -c 'A-Za-z0-9._-' '_')
    step_s=$(printf '%s' "$step" | tr -c 'A-Za-z0-9._-' '_')
    rm -f "${HOME}/.alert-dedup/${job_s}__${step_s}.count" \
          "${HOME}/.alert-dedup/${job_s}__${step_s}.lock"
}

Deduplication and escalation rules

The 5-minute dedup window (300 seconds) prevents alert storms when a flaky upstream causes rapid repeated failures. Match it to your batch cadence — a 10-minute OCR job should use 600.

Apply these rules in your Slack workflow or PagerDuty integration. Because channels are per-webhook, "post to a different channel" means "send to a different webhook URL":

Auto-disable at 10 protects you from an unbounded series of alerts when a job cannot recover without a human:

if (( COUNT >= 10 )); then
    launchctl unload "${HOME}/Library/LaunchAgents/com.yourdomain.failed-job.plist" || true
    SLACK_WEBHOOK_URL="${SLACK_WEBHOOK_URL_CRITICAL:-$SLACK_WEBHOOK_URL}" \
      ~/scripts/alert.sh "$JOB" "critical" "auto-disabled" \
        "Disabled after 10 consecutive failures. Re-enable manually."
fi

Add alerting to any job in one line

Use trap ERR so unexpected non-zero exits fire an alert automatically. set -E makes the trap inherit into functions and subshells:

#!/usr/bin/env bash
set -eEuo pipefail
trap 'rc=$?; ~/scripts/alert.sh "pdf-check" "error" "unexpected-exit" \
      "Script failed at line ${LINENO} (rc=${rc})" \
      "$(tail -n 20 /var/log/pdf_check/err.log 2>/dev/null || echo no-log)"' ERR

# ... job body ...

# On a clean success path, clear the counter and lock.
source ~/scripts/alert_reset.sh
alert_reset "pdf-check" "unexpected-exit"

For expected failure points where you want a specific message:

~/scripts/pdf_check.sh ~/pdf-spool ~/pdf-results || \
    ~/scripts/alert.sh "pdf-check" "error" "batch-run" \
      "PDF check batch exited non-zero" \
      "$(tail -n 20 /var/log/pdf_check/err.log 2>/dev/null || echo no-log)"

Watch the alerter itself

The alert helper can fail: network down, webhook revoked, SLACK_WEBHOOK_URL unset, jq missing. Protect against a silent alerter:

• Redirect stderr to a dedicated log: 2>>"${HOME}/Library/Logs/alert_errors.log".
• Add a separate periodic launchd job using StartInterval 3600 that checks whether alert_errors.log grew in the last hour and falls back to mail or an osascript -e 'display notification "..."' call.
• Do not let alert.sh retry itself on failure; pick a quiet secondary path instead.

Acceptance worksheet

Run these checks on the hosted Mac you intend to use. Record the outcome; do not mark the helper production-ready until each row passes.

1. brew install jq succeeds; jq --version prints a version.
2. SLACK_WEBHOOK_URL=... ~/scripts/alert.sh smoke warning startup "routing ok" returns Alert sent: ... and the message appears in the installed channel.
3. Run the same command twice within 5 minutes; the second call prints SUPPRESSED to stderr and sends no second message.
4. Wait 5 minutes, re-run; a second message appears and the escalation counter increments (you will see #2 in the payload).
5. Pass a message argument containing double quotes, a newline ($'\n'), and a backslash. Confirm Slack renders the message without breaking the JSON.
6. Temporarily rename the webhook URL to force a 404. The script exits non-zero and writes ALERT_SEND_FAILED to stderr. The dedup lock is still created (expected — this also rate-limits a failing sender); delete it manually if you want an immediate retry.
7. Simulate the critical path: set SLACK_WEBHOOK_URL_CRITICAL to a second webhook installed in a different channel and call alert.sh job critical step msg. Confirm the second channel receives it.
8. Call alert_reset for the job+step and confirm the count file and lock file are gone from ~/.alert-dedup/.

Cost and operating limits

Alert delivery adds no meaningful load to any Figment plan. The $29 virtual Intel macOS handles it comfortably next to a launchd cron workload. Slack's incoming webhooks page does not publish a specific per-webhook rate limit, so if a failure loop pushes more than a few alerts per second, insert sleep 1 or push through a small queue. At that point the real fix is the upstream job, not the alerter.

Start or migrate the workload

1. Pick a plan at /mac-hosting. The $29 virtual Intel macOS is enough for alert delivery; request the M-series from $95 only if the companion workload needs ARM execution or more RAM. Ask for a configuration quote if you need specific chip, RAM, or disk.
2. Create a Slack app with an incoming webhook for #ops-alerts and, if you want a separate critical channel, install a second webhook for #ops-critical.
3. Store SLACK_WEBHOOK_URL and SLACK_WEBHOOK_URL_CRITICAL in your CI secret manager and inject them into the launchd or CI job that sends alerts.
4. Copy alert.sh to ~/scripts/ and chmod +x it.
5. Walk through the acceptance worksheet above.
6. Add trap ERR or explicit alert.sh calls to each automation script; call alert_reset on successful completion paths.

Configure your plan and start →

Related guides: Reboot recovery for Mac agents · Task queue for Mac agents · Self-hosted GitHub Actions runner on Mac