Gadget - Major outage – Incident details

Code editing experiencing major outage

Major outage

Resolved
Major outage
Started 11 days agoLasted about 3 hours

Affected

API writes

Degraded performance from 6:12 PM to 6:23 PM, Partial outage from 6:23 PM to 6:52 PM, Major outage from 6:52 PM to 7:25 PM, Operational from 7:25 PM to 9:15 PM

API reads

Degraded performance from 6:12 PM to 6:23 PM, Partial outage from 6:23 PM to 6:52 PM, Major outage from 6:52 PM to 7:25 PM, Operational from 7:25 PM to 7:47 PM, Degraded performance from 7:47 PM to 8:32 PM, Operational from 8:32 PM to 9:15 PM

AI Assistant

Degraded performance from 6:12 PM to 6:23 PM, Partial outage from 6:23 PM to 6:52 PM, Major outage from 6:52 PM to 7:25 PM, Operational from 7:25 PM to 9:15 PM

Shopify Plugin

Degraded performance from 6:12 PM to 6:23 PM, Partial outage from 6:23 PM to 6:52 PM, Major outage from 6:52 PM to 7:25 PM, Operational from 7:25 PM to 8:04 PM, Degraded performance from 8:04 PM to 8:32 PM, Operational from 8:32 PM to 9:15 PM

Code editing

Degraded performance from 6:12 PM to 6:23 PM, Partial outage from 6:23 PM to 7:25 PM, Operational from 7:25 PM to 9:15 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved -- all platform components are nominal.

  • Monitoring
    Monitoring

    All internal services with the exception of the Log Viewer have recovered and are operational. We're monitoring closely and engineers are working on restoring the logging system now.

  • Identified
    Identified

    Further service disruptions are occurring in GCP us-central1 -- we're working on mitigations and will continue to post updates.

  • Update
    Update

    Some auxiliary components in GCP us-central1 are still not operational, which is making Gadget's live queries and search indexing fall behind. Webhooks are processing at full volume. We're continuing to monitor and will update as things change.

  • Monitoring
    Monitoring

    GCP and Shopify have largely recovered. Gadget app error rates are down and have recovered as well. We've deployed some fixes to ensure Gadget apps catch up on the backlog of Shopify webhooks and syncs, and we'll continue to monitor.

  • Update
    Update

    We're seeing some recovery from GCP and Cloudflare but not a full one yet -- we're continuing to monitor closely and prepared to recover ASAP.

  • Update
    Update

    There are reports of GCP and Cloudflare having an incident at the moment. We are being affected by this.

    We are continuing to work on a fix for this incident.

  • Update
    Update
    There are reports of GCP having an incident at the moment. We are being affected by this.
  • Identified
    Identified
    We are currently investigating this incident.