Delayed Webhooks and Realtime Dashboard Outage

Incident Report for mux

Resolved

This incident has been resolved.
Posted Aug 17, 2020 - 07:13 UTC

Monitoring

We experienced a DOS attack on our services that affected webhook delivery and realtime dashboard availability. We have mitigated the issue, and all services are functioning as expected. We will continue to monitor the services.

During this time, some webhooks were delayed in being delivered, and at this time, any backlog should have been processed
Posted Aug 17, 2020 - 06:30 UTC

Update

We have implemented a fix for newly created Webhooks and Realtime Dashboard. However, delivery of webhooks created during the outage continue to be delayed. We are continuing to work on a fix for the delayed webhooks.
Posted Aug 17, 2020 - 02:50 UTC

Update

This issue is confirmed to be impacting all webhook delivery across Video and Data products. We are continuing to work to mitigate the issue.
Posted Aug 17, 2020 - 02:35 UTC

Identified

We have identified the issue and are working to mitigate.
Posted Aug 17, 2020 - 02:35 UTC

Investigating

We are currently investigating this issue.
Posted Aug 17, 2020 - 02:26 UTC
This incident affected: Mux Dashboard, Mux Video - VOD, and Mux Video - Live.