Gadget - Notice history

This incident has been resolved -- all platform components are nominal.

All internal services with the exception of the Log Viewer have recovered and are operational. We're monitoring closely and engineers are working on restoring the logging system now.

Further service disruptions are occurring in GCP us-central1 -- we're working on mitigations and will continue to post updates.

Some auxiliary components in GCP us-central1 are still not operational, which is making Gadget's live queries and search indexing fall behind. Webhooks are processing at full volume. We're continuing to monitor and will update as things change.

GCP and Shopify have largely recovered. Gadget app error rates are down and have recovered as well. We've deployed some fixes to ensure Gadget apps catch up on the backlog of Shopify webhooks and syncs, and we'll continue to monitor.

We're seeing some recovery from GCP and Cloudflare but not a full one yet -- we're continuing to monitor closely and prepared to recover ASAP.

There are reports of GCP and Cloudflare having an incident at the moment. We are being affected by this.

We are continuing to work on a fix for this incident.

The fix has been completed and performance has recovered. We're monitoring to ensure everything remains stable.

We’ve resolved an issue that impacted the availability of the Gadget Web Editor.

Earlier today, we deployed a maintenance job intended to apply to a subset of development environments. Unfortunately, due to a bug, this job triggered a rebuild of all development environments, which overwhelmed our servers and caused them to spike our CPU usage.

As a result, the servers responsible for hosting the Gadget Web Editor were unable to start new workloads. Importantly, the core infrastructure that handles application reads and writes was unaffected, as it runs on a separate set of servers.

To mitigate the issue, we immediately canceled the maintenance job and moved the Web Editor workloads to a dedicated set of servers. This allowed the Gadget Web Editor to recover and will help prevent similar incidents going forward.

As part of our follow up action items we will also investigating additional safeguards to catch issues like this earlier and ensure greater isolation between background jobs and customer-facing services.

Thanks for your patience, and we’re sorry for the disruption.

Our servers that handle editor requests has been rebooted and is running again.

We've identified the issue and latencies have recovered to normal. We'll continue to monitor.

We have identified the issue and are working to re-establish performance

This incident is now resolved. Reads, writes, and background actions are all processing with nominal latencies.

System performance is now nominal. We're now working through all queued background actions to execute them as fast as possible without degrading performance.

We are seeing instances of degraded performance as the system recovers

We've resolved the issue and will are continuing to monitor. We'll follow up with more details.

We've identified the issue and are working to remediate as soon as possible.

the fix has been rolled out

we have identified the issue and a fix is underway

We've fixed the issue and are monitoring going forward.