What Went Wrong

Around 0:20 UTC on April 20th, our data collection agents began rejecting beacons due to their inability to reach our servers responsible for validation and authorization. Service was restored around 2:20 UTC.

Impact

• The majority of beacons were dropped between 0:20-2:20 UTC.
• Any views during this time were either not reported at all or only have partial data.
• The Mux Dashboard will display a gap in data for the 1:00 UTC hour. The 0:00 and 2:00 UTC hours will display a data point from the partial data that was received during those hours.

Technical Details

The failure occurred as a result of a specific subdomain used for internal traffic returning 503s to the data collection agents. While reconfiguring the validation and authorization system to improve monitoring, the configuration for the subdomain used for internal traffic was accidentally neglected and the load balancers set up on that subdomain were no longer able to communicate with the appropriate servers. Caching mechanisms in the collection agents meant that failures slowly increased between 0:20-1:20 UTC. Complete failure occurred at 1:20 UTC. The service was restored by reconnecting the load balancers nearly an hour later.