Gadget - Notice history

The fix has been completed and performance has recovered. We're monitoring to ensure everything remains stable.

We’ve resolved an issue that impacted the availability of the Gadget Web Editor.

Earlier today, we deployed a maintenance job intended to apply to a subset of development environments. Unfortunately, due to a bug, this job triggered a rebuild of all development environments, which overwhelmed our servers and caused them to spike our CPU usage.

As a result, the servers responsible for hosting the Gadget Web Editor were unable to start new workloads. Importantly, the core infrastructure that handles application reads and writes was unaffected, as it runs on a separate set of servers.

To mitigate the issue, we immediately canceled the maintenance job and moved the Web Editor workloads to a dedicated set of servers. This allowed the Gadget Web Editor to recover and will help prevent similar incidents going forward.

As part of our follow up action items we will also investigating additional safeguards to catch issues like this earlier and ensure greater isolation between background jobs and customer-facing services.

Thanks for your patience, and we’re sorry for the disruption.

Our servers that handle editor requests has been rebooted and is running again.

We've identified the issue and latencies have recovered to normal. We'll continue to monitor.

We have identified the issue and are working to re-establish performance

This incident is now resolved. Reads, writes, and background actions are all processing with nominal latencies.

System performance is now nominal. We're now working through all queued background actions to execute them as fast as possible without degrading performance.

We are seeing instances of degraded performance as the system recovers

We've resolved the issue and will are continuing to monitor. We'll follow up with more details.

We've identified the issue and are working to remediate as soon as possible.

the fix has been rolled out

we have identified the issue and a fix is underway

We've fixed the issue and are monitoring going forward.