Hi all. Sorry for the disruption today, we had 2 major server outages. Voice calls were unaffected for customers using Conduit3 PABX systems as they failed over to our Auckland server.

Web services failed over to a hot standby, albeit without dynamic content.

While the outages took out our main data server and our main ISDN trunk servers, the distributed nature of VentureVoIP's systems meant that voice calls were unaffected for people who are using ConduIT3 PABX systems.

All systems have now been restored and are functioning normally.

The cause of the outages was a distributed denial of service attack, which would have normally taken out any VoIP provider, but because of the nature of the VentureVoIP network, we were able to shut down the affected servers while relying on back up servers.

Thanks for your patience.

A short explanation of how we managed 0 minutes of voice downtime

Basically all of our customers who use ConduIT3 PABX systems are registered to multiple servers at the same time. On each connection the IAX qualify parameter is used so that calls only go through servers which have low latency.

When the DDoS started, ping times to the affected servers increased which was sufficient to route voice calls via other servers.

Once we realized that we were under attack we pulled the voice servers down completely so that all qualifies to those servers would fail.

In extensions.conf on the customers machines they sequentially try each of our voice servers before failing over to PSTN based calling.

Once we pulled down the voice servers which were affected by the attack, all calls were guaranteed to travel via different paths, thereby maintaining voice quality.

All billing information is stored on all servers. This meant that once we had the systems back up and going we just resynced the data to the servers which had been shut down and brought them up again.

We are in the process of setting up a hot failover system for dynamic web content and this should be completed within the next couple of weeks.

While we had set up systems so that we would not be taken out in this exact situation, it had never been tested in a real world attack, so even though it was a hard day, we got to the end of it with some measure of relief.

Comments (Click to post)

Name:
Subject:
Website:
Message: