We encountered an issue starting 2021-06-25 18:40 UTC where Mux Video events were potentially delayed heavily, repeated, or never sent at all. We implemented a fix at 2021-06-25 20:36 UTC, at which point new events started being delivered as expected. During the time from 18:40 to 20:36 UTC, any events that happened may not show up on assets or live streams in the dashboard, and may not have been delivered to any configured webhooks.
We are not able to resend events that were not sent in this timeframe, due to issues that may occur on configured webhook endpoints which would receive events that are out of order. However, reach out to support@mux.com and reference this incident, and we can provide a report on any objects that changed during this time, allowing reconciliation with your database.
Posted Jun 25, 2021 - 22:33 UTC
Identified
We have deployed a mitigation restoring event and webhook delivery from 2021-06-25 20:36 onward.
For events that happened from 2021-06-25 18:40 UTC until 2021-06-25 20:36 UTC, those will be missing from the dashboard and will not have been sent to configured webhooks, and our team is working on addressing those missing events.
Posted Jun 25, 2021 - 20:43 UTC
Update
The delay to events and webhook delivery of the events remains, and may be significant at times. Our team is actively working on resolving this, and we'll post updates as we have them.
Posted Jun 25, 2021 - 19:48 UTC
Investigating
We are currently investigating an issue where some webhook events are delayed. You may see recent events not show up on assets or live streams in the dashboard, and events may be delayed in being delivered to configured webhooks.
We will provide updates as we have them.
Posted Jun 25, 2021 - 19:10 UTC
This incident affected: Mux Dashboard, Mux Video - VOD, and Mux Video - Live.