Gadget - Delays in background action processing – Incident details

All systems operational

Delays in background action processing

Resolved
Degraded performance
Started 7 days agoLasted about 10 hours

Affected

API writes

Degraded performance from 8:00 AM to 5:31 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    p95 start lag has recovered to normal values for all queues. We'll continue to monitor to ensure all is well. Please let us know in Discord if you are still seeing issues.

  • Update
    Update

    Background action start latencies continue to improve, with the high, default, and low priority background action queues having fully recovered. Shopify webhook processing is still delayed for some applications.

  • Identified
    Identified

    We are seeing improvements on queue latency for background jobs and are continuing to mitigate the issue.

  • Investigating
    Investigating

    We are currently investigating issues with elevated background action queue processing times.