Increased re-buffering and error rates on live streams
Incident Report for Mux
Resolved
We have moved impacted live stream traffic to an alternate region where live streams are stable at this time.

We continue to analyse the impact of the Google Cloud Storage outage on live streams that were active at the time of the issue.

Please contact our support team if you continue to see issues.
Posted Oct 24, 2020 - 18:49 UTC
Update
We have completed our failover to an alternate region, and are seeing healthy metrics from the new region.

If you continue to see issues on your live stream, please disconnect your encoder, call the /complete API (https://docs.mux.com/reference#signal-live-stream-complete), and reconnect your encoder.

We continue to monitor the underlying Google Cloud Issue: https://status.cloud.google.com/incident/storage/20005
Posted Oct 24, 2020 - 17:30 UTC
Monitoring
We have completed our failover to an alternate region, and are seeing healthy metrics from the new region.

If you continue to see issues on your live stream, please disconnect your encoder, call the /complete API (https://docs.mux.com/reference#signal-live-stream-complete), and reconnect your encoder.
Posted Oct 24, 2020 - 17:19 UTC
Update
We have made a change on our side to attempt to route around the poorly performing GCS region (at this point we believe it is one region, though in transparency we are not certain that this isn’t global for GCS) for live.mux.com and global-live.mux.com. This has a few impacts after the rollout:

1) New live streams that were not active this morning should work without issue
2) Live streams that are currently streaming will continue to work, but with the degraded performance seen before this change
3) Any live stream that tries to reconnect and cannot will need the /complete endpoint called, but after that should be able to connect and live stream, without a change in the encoder configuration
Posted Oct 24, 2020 - 16:51 UTC
Identified
We are experiencing increased re-buffering and error rates on live streams caused by performance issues with Google Cloud Storage.
Posted Oct 24, 2020 - 15:54 UTC
Investigating
We are currently investigating an issue causing increased re-buffering and error rates on live streams.
Posted Oct 24, 2020 - 15:14 UTC
This incident affected: Mux Video - Live.