Gadget - Delays in background action processing – Incident details

Delays in background action processing

ResolvedDegraded performance9 hours 31 minutes

Affected

API writes

Degraded performance from 8:00 AM to 5:31 PM

  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    p95 start lag has recovered to normal values for all queues. We'll continue to monitor to ensure all is well. Please let us know in Discord if you are still seeing issues.

  • UpdateUTC

    Background action start latencies continue to improve, with the high, default, and low priority background action queues having fully recovered. Shopify webhook processing is still delayed for some applications.

  • IdentifiedUTC

    We are seeing improvements on queue latency for background jobs and are continuing to mitigate the issue.

  • InvestigatingUTC

    We are currently investigating issues with elevated background action queue processing times.