Gadget - Delayed background actions – Incident details

All systems operational

Delayed background actions

Resolved
Partial outage 30 %
Started 5 months agoLasted about 8 hours

Affected

API writes

Degraded performance from 10:00 AM to 10:42 AM, Partial outage from 10:42 AM to 2:13 PM, Operational from 2:13 PM to 5:56 PM

Updates
  • Resolved
    UTC
    Resolved

    Some apps queues for first-party background actions with default priority continued to be delayed due to an overloaded internal component. This issue has been corrected and all workers are working hard to catch up.

  • Update
    UTC
    Update

    The background action start latency has recovered. We are continuing to monitor but the impact looks to be mitigated.

  • Update
    UTC
    Update

    We are still seeing elevated latencies for back ground action starts and are applying more mitigations.

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    UTC
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.