Gadget - Delayed background actions – Incident details

Delayed background actions

ResolvedPartial outage 30%7 hours 56 minutes

Affected

API writes

Degraded performance from 10:00 AM to 10:42 AM, Partial outage from 10:42 AM to 2:13 PM, Operational from 2:13 PM to 5:56 PM

  • ResolvedUTC

    Some apps queues for first-party background actions with default priority continued to be delayed due to an overloaded internal component. This issue has been corrected and all workers are working hard to catch up.

  • UpdateUTC

    The background action start latency has recovered. We are continuing to monitor but the impact looks to be mitigated.

  • UpdateUTC

    We are still seeing elevated latencies for back ground action starts and are applying more mitigations.

  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • IdentifiedUTC
    We are continuing to work on a fix for this incident.
  • InvestigatingUTC
    We are currently investigating this incident.