We are marking this incident as resolved as we have not seen any issues since midday. Follow up from the vendor with details about the MPLS outage will be added to the post mortem follow up of this incident.
We apologize for the inconvenience and strive to do better. Internally we are re-evaluating some of the routing options available so that failover to alternative routes is faster and customer impact during an outage is reduced.
If you need to reach us regarding this incident, please reach out to our support team at customerservice@soundofdata.nl or via +3110 4291411
Posted Feb 29, 2024 - 13:59 CET
Monitoring
We have received unofficial confirmation that the issue has been found but no official RFO has been shared. Root cause appears to be a faulty software upgrade on the carrier's MPLS network switches which has been reverted.
We will start testing to see if the issue is indeed resolved and will update around 1 pm with our results.
Posted Feb 29, 2024 - 11:50 CET
Update
Vendor has identified a problem on their MPLS network which is causing problems for various services. Engineers are working to mitigate and resolve.
Our engineers have rerouted all outbound traffic to other carrier routes and outbound traffic is no longer affected. Inbound traffic is also working but we are still working with our team to verify all national networks (KPN/Odido/Vodafone/etc) with all number ranges (31800 / 3190x / geographical numbers).
Next update around 1 pm or sooner if new information becomes available.
If you need to reach us regarding this incident, please reach out to our support team at customerservice@soundofdata.nl or via +3110 4291411
Posted Feb 29, 2024 - 11:06 CET
Identified
We've been seeing issues with one of our partners and have begun remediation steps. As of now, we have been informed that they are experiencing a mayor issue on their EU network, impact traffic for BE, NL & DE customer connections, MPLS, IP sec VPN's and voice traffic. Engineers are working and incident has been raised to p1 incident.
On our side we have begun with rerouting all traffic to different partners for our outbound calls. These steps are nearly complete and are expected to be completed in the next 15 minutes.
For inbound we are currently assessing the impacted source networks, numbers and customers. We are marking inbound currently as a partial outage.
Next update at 11 am, or sooner if new information becomes available.
If you need to reach us regarding this incident, please reach out to our support team at customerservice@soundofdata.nl or via +3110 4291411
Posted Feb 29, 2024 - 10:10 CET
This incident affected: ITSP | Services (Voice Services).