This incident has been resolved. Any live streams started after 2023-04-23 at ~03:00:00 UTC (approximately three and a half days ago) are working correctly and have been since that time.
Posted Apr 26, 2023 - 22:16 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 25, 2023 - 22:45 UTC
Investigating
At approximately 18:00 UTC on April 18, 2023, a regression was introduced into Mux Video Live Streaming which had to do with how slate insertion was handled. This regression was patched at approximately 03:00 UTC on April 23, 2023. During this time, some live streams failed that had slate insertion configured _and_ experienced live ingest patterns that required slate insertion (i.e. due to encoder disconnects or dropped/delayed frames from the source encoder).
The majority of affected live streams were disconnected, but the encoder was able to reconnect almost immediately. Due to the nature of the disconnect, however, any configured reconnect_window did not apply at this time, resulting in a new asset upon reconnect rather than a continuation of the original asset.
A number of live streams that were affected resulted in assets that were not cleanly terminated. This resulted in a few issues. First off, the encoder would have been unable to reconnect cleanly. Additionally, the resulting assets were placed in a non-playable state, either having been truncated or completely unplayable with 4xx or 5xx errors being returned. Our team is actively investigating options to recover the content of these affected assets.
During this time, live streams that were not configured for slate insertion, live streams that did not trigger slate insertion, and all on-demand ingest and playback were unaffected.