Gadget - Notice history

We implemented a fix and are seeing apps return to normal.

We have identified the issue and are deploying mitigation strategies.

We have identified the issue and are deploying a fix.

This incident has been resolved. This incident was caused by test improvements to Gadget's load balancing strategy made to a test environment impacting the production environment unexpectedly. We've reverted the offending change, and will prioritize work to fully isolate this test environment where it cannot impact production at all. We apologize for any inconvenience this downtime has caused!

This incident has been resolved -- all platform components are nominal.

All internal services with the exception of the Log Viewer have recovered and are operational. We're monitoring closely and engineers are working on restoring the logging system now.

Further service disruptions are occurring in GCP us-central1 -- we're working on mitigations and will continue to post updates.

Some auxiliary components in GCP us-central1 are still not operational, which is making Gadget's live queries and search indexing fall behind. Webhooks are processing at full volume. We're continuing to monitor and will update as things change.

GCP and Shopify have largely recovered. Gadget app error rates are down and have recovered as well. We've deployed some fixes to ensure Gadget apps catch up on the backlog of Shopify webhooks and syncs, and we'll continue to monitor.

We're seeing some recovery from GCP and Cloudflare but not a full one yet -- we're continuing to monitor closely and prepared to recover ASAP.

There are reports of GCP and Cloudflare having an incident at the moment. We are being affected by this.

We are continuing to work on a fix for this incident.