- SCADA system brownouts caused blind spots across energy transport network
- Lost network transactions impacted SCADA quality of service
- nGeniusONE® Service Assurance platform
- InfiniStreamNG® (ISNG) smart visibility appliances
- nGenius®PULSE with nPoint sensors
- NETSCOUT® Premium Support Services
- Successful SCADA troubleshooting of energy transport network issue without downtime
- Avoided outages and associated revenue impacts
This company is a leading transporter of natural gas and oil. Recent innovations have enabled the company to enjoy an expanded presence in the renewable energy generation business segment.
For years, the company’s Network Operations (NetOps) team had looked to NETSCOUT as a trusted business partner that delivered service assurance and visibility solutions across the business, with past projects assisting efforts to migrate applications to Software-as-a-Service (SaaS) platforms.
The company had enhanced operations efficiencies through numerous ongoing digital technology initiatives. In one such project, the company’s operations team improved the performance of their controller environment by using the Supervisory Control and Data Acquisition (SCADA) system to identify operational changes (e.g., pressure drops) to alert whether there was a possible transport network leak. The company also used the SCADA system to monitor sump levels, vapor concentrations, pump-seal failures, and equipment vibration leaks. In the context of the company’s IT Operations, SCADA was monitored along the network peripheral, with Programmable Logic Controllers (PLCs) remotely installed across the delivery network communicating operational status to the SCADA database.
Over time, NetOps monitoring of company’s SCADA environment showed they were experiencing electrical brownouts (e.g., power blips) on their wide area network (WAN) satellite links. As these brownouts occurred, the PLCs would initiate health check requests every 10 seconds. While the satellite links did not go down during this time, the SCADA system stopped running on the affected WAN links, which placed controller environment operations at risk. For every five-minute interval, an estimated 600 network transport transactions would be impacted when SCADA was not operating.
Since IT Operations wanted to avoid costly, disruptive transport network shutdowns to troubleshoot this SCADA brownout issue, the company prioritized solutions that would enable on-the-fly root cause analysis and resolution without interrupting the natural gas and oil network used to deliver always-needed energy resources to millions of customers.
Solution in Action
In successfully troubleshooting and resolving this issue, NetOps collaborated with their long-contracted NETSCOUT Premium Support Services (PSS) Engineer to take advantage of the end-to-end visualization across their SCADA environment provided by their already-in-production:
- nGeniusONE Service Assurance analytics
- InfiniStreamNG (ISNG) smart visibility appliances
- nGeniusPULSE with nPoint sensors
The PSS Engineer and NetOps team collaborated to devise a troubleshooting workflow using the following NETSCOUT service elements:
- nGeniusPULSE with nPoints were used to generate synthetic testing to assess the performance of the Modbus data communications protocol used by the PLCs remotely installed across the delivery network. Since the PLCs were initiating health check requests every 10 seconds, the PSS Engineer concurred that could be an indication of issues occurring on the line.
- Through nGeniusPULSE synthetic testing, it was determined that monitoring transactions and TCP Retransmissions provided the most reliable indicator of issues.
- An nGeniusONE alert profile was configured to send email updates on the potential indicators of issues occurring between the PLCs and SCADA servers.
- With those alerts integrated into a customized nGeniusONE Service Dashboard view, NetOps was provided with a real-time snapshot into the health of the Modbus data communications protocol traffic.
The NetOps team then leveraged nGeniusONE Service Dashboard’s ability to visualize realtime performance of the SCADA environment and corresponding analysis into transactions, response time, TCP retransmissions, and failures to determine root cause related to oversaturated network links. In addition to the graphical representation on the Service Dashboard, an email alert was configured to be sent to the NOC. Since NetOps had only a very short timeframe to determine whether to shut down the pipeline to address this issue, this was a critically important NETSCOUT feature used to assure uninterrupted transport network service delivery.
As a result, NetOps made a configuration change to switch traffic to a secondary link, which returned the PLCs and SCADA environment to high-quality performance.
Given the expanse of the company’s energy business, any extended transport network issue could result in environmental concerns involving potential leaks or pipeline shutdowns, both of which could lead to lost production, industry-related fines, adverse publicity, and unhappy customers. In this case, the NetOps team helped the company avoid these damaging scenarios by customizing their NETSCOUT smart visibility into their remote PLC operations to determine that the mean-time-to-repair this issue involved a straightforward network configuration change.