Large-Scale Transit Authority Tracks Train Fare Processing Issue With NETSCOUT

Customer Profile

This North American transit authority is responsible for transporting more than 175 million passengers by rail and 120 million passengers by bus every year. This authority relies on the performance of the rail and bus infrastructure and their committed personnel to transport residents and travelers to their offices, schools, appointments, and other local destinations. The authority’s trains travel across more than 100 miles of track, and they manage a fleet of 1,000+ buses. 

The Challenge

The transit authority is not merely a “nice-to-have” travel option for local ridership. Geography, population density, financial, and regional roadway congestion all combine to position trains and buses as a singular travel option for many riders on a day-in, day-out basis. As with so many other transit services, passengers swipe re-usable, pre-paid smart cards to pay for each ride. When the monetary balance of the card runs out, the passengers can add to the account with cash, credit cards, or debit cards at train stations, bus stops, or online. This is one of several critical application services that the transit authority depends on every day. 

A fare processing issue impacting the ability for riders to automatically “re-load” their fare payment cards at station kiosks surfaced across a two-month period, with this problem periodically impacting: 

• Timely travel for those customers now requiring exact change for manual fares 
• Toll collecting efficiency for the transit authority, with increased personnel now involved in manual fare processing and managing some instances of fare dodging 
• Public perception of the transit authority 
• The IT team’s internal reputation across the organization 

The IT team dedicated significant organizational resources to monitoring and troubleshooting this issue, involving their own Application, Networking, Network Operations Center teams, as well as cycles spent communicating with several of their technology vendors. 

While there were differing opinions over root cause across the organization, there was one point on which all IT teams agreed – they needed better visibility into the fare system’s financial application and underlying technology supporting that app’s operation across the transit authority’s railway and bus station network. Absent that visibility, finger-pointing between IT teams and technology vendors would continue without a satisfactory identification of root cause. 

In this same timeframe, IT leadership engaged in a strategic project that involved extending visibility farther into the authority’s data center for further service assurance coverage. With cost containment guidelines in place, leadership was looking for innovative solutions that would improve network visibility and real-time application monitoring, while extending the value of their legacy IT toolsets. 

Solution in Action

Sometimes, a months-long financial processing performance issue inconveniencing riders and giving transit authority business the proverbial “black eye” can come down to a relatively simple IT issue – in this case, a few dropped network packets related to a single server blade on a network router. That’s what NETSCOUT’s smart visibility uncovered for the transit authority’s IT team and their responsible technology vendor. 

After other solutions had already tried and failed to shed light on root cause, the IT team used a NETSCOUT InfiniStreamNG (ISNG) appliance typically instrumented elsewhere in its environment to provide packet captures used by the nGeniusONE performance analytics platform to troubleshoot the issue. The IT team was able to deploy ISNG real-time, packet-based monitoring for analysis by nGeniusONE. In this case, with the ability to decipher the packet-level details, their nGeniusONE solution identified the specific blade in an 18-blade router chassis that was dropping a few network packets per every million packets processed, which was causing the problem with the fare payment service. 

With the NETSCOUT solution’s success in to addressing the high-profile fare payment issue, the transit authority was able to secure appropriated capital expenses to move forward with their strategic visibility project, which included deployment of software-based ISNG data sources and packet brokers. NETSCOUT’s software-based approach met the agency’s cost containment guidelines, enabling IT to deploy ISNG Smart Visibility appliances and nGenius Packet Flow Operating System (PFOS) Software at primary and secondary data center locations, improving performance reliability and redundancy of the transit authority’s real-time monitoring solution. 

The Results

Without NETSCOUT smart visibility and nGeniusONE’s performance analytics and on-board packet analysis, the IT team may still have been looking for the cause of its fare processing issues. 

With this issue resolved, the IT team was able to: 

• Reconcile the fare processing issue inconveniencing their customers and regional visitors 
• Restore the agency’s reputation with local riders 
• Satisfactorily address this performance challenge with an important technology partner 

The transit authority is positioned to improve service performance and reliability to local ridership, with NETSCOUT’s end-to-end visibility providing their IT teams with the solution required for real-time transit application performance, including dedicated service dashboard and service monitor views.