Gadget - Major outage – Incident details

All systems operational

Show current status

Major outage

ResolvedMajor outage3 hours 3 minutes

Affected

API writes

Degraded performance from 6:12 PM to 6:23 PM, Partial outage from 6:23 PM to 6:52 PM, Major outage from 6:52 PM to 7:25 PM, Operational from 7:25 PM to 9:15 PM

API reads

Degraded performance from 6:12 PM to 6:23 PM, Partial outage from 6:23 PM to 6:52 PM, Major outage from 6:52 PM to 7:25 PM, Operational from 7:25 PM to 7:47 PM, Degraded performance from 7:47 PM to 8:32 PM, Operational from 8:32 PM to 9:15 PM

AI Assistant

Degraded performance from 6:12 PM to 6:23 PM, Partial outage from 6:23 PM to 6:52 PM, Major outage from 6:52 PM to 7:25 PM, Operational from 7:25 PM to 9:15 PM

Shopify Plugin

Degraded performance from 6:12 PM to 6:23 PM, Partial outage from 6:23 PM to 6:52 PM, Major outage from 6:52 PM to 7:25 PM, Operational from 7:25 PM to 8:04 PM, Degraded performance from 8:04 PM to 8:32 PM, Operational from 8:32 PM to 9:15 PM

Code editing

Degraded performance from 6:12 PM to 6:23 PM, Partial outage from 6:23 PM to 7:25 PM, Operational from 7:25 PM to 9:15 PM

  • ResolvedUTC

    This incident has been resolved -- all platform components are nominal.

  • MonitoringUTC

    All internal services with the exception of the Log Viewer have recovered and are operational. We're monitoring closely and engineers are working on restoring the logging system now.

  • IdentifiedUTC

    Further service disruptions are occurring in GCP us-central1 -- we're working on mitigations and will continue to post updates.

  • UpdateUTC

    Some auxiliary components in GCP us-central1 are still not operational, which is making Gadget's live queries and search indexing fall behind. Webhooks are processing at full volume. We're continuing to monitor and will update as things change.

  • MonitoringUTC

    GCP and Shopify have largely recovered. Gadget app error rates are down and have recovered as well. We've deployed some fixes to ensure Gadget apps catch up on the backlog of Shopify webhooks and syncs, and we'll continue to monitor.

  • UpdateUTC

    We're seeing some recovery from GCP and Cloudflare but not a full one yet -- we're continuing to monitor closely and prepared to recover ASAP.

  • UpdateUTC

    There are reports of GCP and Cloudflare having an incident at the moment. We are being affected by this.

    We are continuing to work on a fix for this incident.

  • UpdateUTC
    There are reports of GCP having an incident at the moment. We are being affected by this.
  • IdentifiedUTC
    We are currently investigating this incident.