Increased re-buffering and error rates on live streams

Incident Report for mux

Resolved

We have moved impacted live stream traffic to an alternate region where live streams are stable at this time.

We continue to analyse the impact of the Google Cloud Storage outage on live streams that were active at the time of the issue.

Please contact our support team if you continue to see issues.

Posted Oct 24, 2020 - 18:49 UTC

Update

We have completed our failover to an alternate region, and are seeing healthy metrics from the new region.

If you continue to see issues on your live stream, please disconnect your encoder, call the /complete API (https://docs.mux.com/reference#signal-live-stream-complete), and reconnect your encoder.

We continue to monitor the underlying Google Cloud Issue: https://status.cloud.google.com/incident/storage/20005

Posted Oct 24, 2020 - 17:30 UTC

Monitoring

Posted Oct 24, 2020 - 17:19 UTC

Update

We have made a change on our side to attempt to route around the poorly performing GCS region (at this point we believe it is one region, though in transparency we are not certain that this isn’t global for GCS) for live.mux.com and global-live.mux.com. This has a few impacts after the rollout:

1) New live streams that were not active this morning should work without issue
2) Live streams that are currently streaming will continue to work, but with the degraded performance seen before this change
3) Any live stream that tries to reconnect and cannot will need the /complete endpoint called, but after that should be able to connect and live stream, without a change in the encoder configuration

Posted Oct 24, 2020 - 16:51 UTC

Identified

We are experiencing increased re-buffering and error rates on live streams caused by performance issues with Google Cloud Storage.

Posted Oct 24, 2020 - 15:54 UTC

Investigating

We are currently investigating an issue causing increased re-buffering and error rates on live streams.

Posted Oct 24, 2020 - 15:14 UTC

This incident affected: Mux Video - Live.