Recently we experienced an issue with one way audio through one of our vendors. This vendor is used to host inbound calls and also for some of the outbound traffic.
During the outage we experienced one way audio on inbound and outbound calls.
After working with the vendor and the networking team we can now share the root cause of the incident to be a faulty router used in the MPLS network.
A Cisco router in the MPLS core, to which the IPsec servers for WVoIP are also connected, was causing problems that corrupted the VLAN routing. By resetting previous routing tables, Operations was able to fix the problems. A program is also underway to replace the Cisco's with new equipment.
As the issue was found to be outside of our span of control, our own efforts have been focused on rerouting as fast as possible.
Currently the re-routing of traffic is still a manual intervention by the networking team when an outage like this happens. We are working on creating a process in which we can more easily remove a vendor route globally.
We will work with the vendor regarding future maintenance so that we can be more pro-active about maintenance in these potenial high impact changes.