High-Availability Cloud Based Voice of IP Voice Communications

Written by Ahsan Baig, Mike Carvalho, Jim Willows, and Philip Bockrath

In this article we describe our experience from the implementation of a high-availability multi Cloud-based VoIP system to AC Transit, the third-largest public bus system in California. The AC Transit service area includes portions of Alameda and Contra Costa Counties, serving 13 cities and unincorporated areas from San Pablo down to South Fremont. AC Transit also provides commuter service across the Bay to San Francisco, San Mateo and Santa Clara counties. The Service consists of more than 151 bus routes throughout a 364-square mile service area serving 1.5 million people.


Real-time voice communications between Operations Control Center (OCC), Bus Operators, and Field Supervisors are critical components of maintaining service reliability and practicing safety. Historically, the voice communications at AC Transit were achieved by leveraging the traditional Land Mobile Radio (LMR) system. At the critical decision making time, the options were to invest millions of dollars into the replacement and upgrade of the LMR system, including the long-haul communication hardware infrastructure and end-user radio equipment, or deploy the newly developed Voice Over IP (VoIP) software based technology using the commercial cellular 5G network. As the computer industry moves towards Internet Protocol (IP) based voice connectivity, the IP protocols provide device interoperability, system reliability, spectrum efficiency, wide coverage areas at a cost-effective price. Based on extensive market research and assessment of technology maturity, AC Transit concluded that the VoIP communications option was more stable, reliable, and inexpensive than traditional analog radio communication solutions.

 

Network Architecture

AC Transit designed its cloud infrastructure utilizing industry best practices. The two privately-owned data center sites are set up to host the VoIP technology infrastructure for high availability and resiliency purposes, with full redundancy in connectivity and failover capability. This technology stack is the foundation of our private VoIP Cloud infrastructure. Additionally, the corporate office hosts the Radio over IP (RoIP) equipment and provides the core switch for Local Area Networks (LAN) devices. One of the Bus Division hosts the UHF Land Mobile Radio equipment, which is set up as a secondary backup. The Microsoft AZURE Cloud environment is also utilized to host the third-party Apps for remote connectivity and emergency communications.

Scalable, secure, and resilient network infrastructure is key to the successful operation of this VoIP system. Each layer of our network technology has a precise role; system reliability, adequate bandwidth, Quality of Service, and same-day hardware and software service contracts allow for continuous, reliable operation. Each Local Area Network (LAN) provides connectivity for our servers, dispatch consoles, mobile devices, and user workstations. Multiple LANs are configured within the VoIP system, and each LAN is comprised of core switches, distribution switches, access network switches, wiring plants, and fiber optic backbones. These LANs are located at our data centers, corporate office, bus divisions, and even on our buses. Each network device must be able to provide Quality of Service (QoS) technology.  QoS configurations provide traffic prioritization and resource reservation controls by elevating VoIP network traffic above all other IP based network traffic. Speed and reliability are essential to providing reliable VoIP communications. The Wide Area Network (WAN) provides connectivity between our data centers, corporate offices, and bus divisions. This high-speed backbone provides continuous, reliable communications for all our VoIP technology. This technology includes point-to-point circuits, leased ethernet circuits, point-to-point fiber optic connections, and virtual private networks (VPN) over the Internet.

 

Figure 1 - AC Transit Wide Area VoIP Communications
Figure 1 - AC Transit Wide Area VoIP Communications Solution

 

Hybrid Cloud Implementation

Hybrid Cloud setup, using Public and Private Clouds, enables resiliency and a cost-effective way of managing resources, as these are the most critical parts of our VoIP implementation. Various backup technologies and systems are in place to provide reliable operations in the event of hardware failure, network outages, or software issues. A dual data center configuration is used to provide maximum system uptime. Both data centers have all the server and network infrastructure to independently operate the VoIP system. The data centers utilize an N+2 redundancy scheme for power conditioning, power generating, UPS battery backup, and equipment cooling. The data centers are geographically located with different power grids and experience different natural disasters.

 

Routing Topology

AC Transit operates a highly resilient routed network consisting of an EIGRP core with OSPF branches for vendor-neutral compatibility and several static and BGP-based networks redistributed into the core at strategically chosen points. This fully autonomous system self-corrects any single point of failure for our most critical traffic. Automatic mechanisms, such as IP Service Level Agreement (SLA) with object tracking and customized administrative distances for static routes, are employed to preconfigure the routing system for any condition and avoid the need for human interaction to correct for a link or device failure. This extends to the VoIP remote data centers, which span two geographically distinct locations and utilize dual BGP circuits that do not directly integrate into the organization's core EIGRP and OSPF protocols. Despite those technical complications, the automatic routing mechanisms in place perfectly facilitate failover of these networks from AC Transit's HQ just as easily and smoothly as the failover mechanisms built directly into EIGRP.

 

End to End Quality of Service (QoS)

To ensure reliable voice service even when the network is heavily saturated, the organization utilizes a single comprehensive QoS marking, queueing, and shaping policy similar to our routing policy. The tool we use to accomplish this is Differentiated Service Code Point (DSCP) tags. Initially set up and installed as part of the organization's original VoIP phone implementation more than ten years ago, the QoS marking and queueing scheme has been updated over the years as capacity improved and required throughput increased. AC Transit proudly achieves latency values of about 50ms or less for most internal voice calls and jitter less than 20ms, which is extremely adequate for high-quality voice and video calls.

We can achieve excellent voice quality in terms of latency and jitter by employing a similar policy for Quality of Service. Despite the difficulties of integrating remote data centers that utilize different routing protocols and different link technologies, AC Transit operates a single autonomous system that is highly available for all voice communications, thanks to our strict adherence to these important policies.

 

System Performance - Where the Rubber Meets the Road

With the design of any safety-critical communication solution, many factors come into play including security, reliability, redundancy, flexibility, scalability, and authentication. Like many other Public Transit agencies, AC Transit deals with around-the-clock incidents from mundane to life-threatening daily. Therefore, when critical events transpire, it is necessary to ensure that reliable voice communication is in place for situational awareness that can meet demand even when failures occur.

 

Figure 2 - AC Transit Service Area
Figure 2 - AC Transit Service Area

 

AC Transit's VoIP communication system has been in production use for almost two years. Thorough network design, implementation, and validation testing are critical in ensuring end-users experience high-reliability communications. Throughout the design, evaluation, and deployment segments of the process, all links in the network were evaluated for latency, jitter, and packet loss throughout the process. This included fixed-end networks, broadband links, and on-vehicle solutions.

The AC Transit service area was evaluated using ITU G.107 standards-based testing using 0.25 mi x 0.25 mi grids. Data collected was evaluated using both spatial and temporal tools to ensure that QoS DSCP fields were adhered to for expedited forwarding of critical packet streams.

 

Conclusion

A multi Cloud-based VoIP implementation, based on geographically diverse data centers, provides redundancy and high availability for the network. The Quality of Service design elements and engineering the tight time-delay metrics for voice quality are obtained through detailed network traffic engineering. A unique perspective in this implementation is the complete paradigm shift for a Public Transit agency, whose focus is in providing safe and reliable mobility services, instead of spending sacrce technical resources in developing and maintining the private LMR systems.

 

 

 

This article was edited by Shafi Khadem

For a downloadable copy of the August 2021 eNewsletter which includes this article, please visit the IEEE Smart Cities Resource Center.

Ahsan Baig
Ahsan Baig provides strategic direction, policy guidance, and responsible for overseeing and coordinating all IT investments, including innovation, digital transformation roadmap, infrastructure modernization, development of zero-emission bus energy management software platform, and implementation of an enterprise cybersecurity framework. Ahsan has over 27 years of private and public sector Information Technology experience, leading and managing high-performing teams in critical, high-visibility projects. Ahsan has extensive experience in designing, deploying and maintaining several intelligent transportation and smart transit related projects, both nationally and internationally.  Ahsan earned a bachelor’s degree from Pakistan; and a Master of Science in Electrical Engineering from the City College, City University of New York (CUNY).  Mr. Baig is a Chairperson of the California Transit Association IT Committee. He is an active member of IEEE, APTA, and CTA. He is a public speaker and has published research and technical papers in various journals and magazines.
Mike Carvalho
Mike Carvalho is the Head of IT Infrastructure and has been at AC Transit for the last 12 years. A USAF veteran and a graduate of Southern Illinois University at Carbondale he has been designing and implementing IT technology and infrastructure for 35 years. While at AC Transit, Mike has redesigned and upgraded all the datacenter, network, wireless, server, and storage systems using reliable next generation technology hardware and software solutions.
Many of these new solutions have been implemented using cloud technologies. His unique “always live” disaster recovery and business continuity solutions have maintained reliable and redundant backup IT systems for many years. AC Transit was one the very first companies to roll out Office 365 email, implement Pure Technology SSD based storage solutions, and rollout out Clever Devices CAD/AVL and VOIP solutions on our entire bus fleet. Mike was key to finding and introducing to AC Transit these revolutionary new technologies. 
Jim Willows
Jim Willows is a lead engineer on the AC Transit network infrastructure team, where he is primarily responsible for managing routing and switching infrastructure equipment. His duties including analyzing, designing, and implementing all new network transmission solutions. He has worked on a variety of large projects, including most recently the Tempo Bus Rapid Transit (BRT) project in East Oakland, where he was a primary design architect for routed systems. Jim has a bachelor’s degree in Network and Communications Management, and has held a variety of certifications including Cisco, CompTIA, and The Linux Foundation. Jim lives in Oakland, CA with his partner and their dog.
Phillip Bockrath
Philip Bockrath leads a group of innovative embedded systems, applications and communications engineers to create exciting next generation wireless technologies. He has spent 27 years working at Motorola, Harris and various government contractors. Philip has an extensive background in the design, development and deployment of cross platform communications using POC, MCPTT, LTE, LMR, TETRA, Cellular, DMR, point to point and point to multipoint wireless communications. Current design and project implementations allow for seamless switching of disparate voice and data communications allowing for graceful degradation in the event of catastrophic failures or circumstances.

Past Issues

To view archived articles, and issues, which deliver rich insight into the forces shaping the future of the smart cities, please visit the IEEE Smart Cities Resource Center.