Reducing Manufacturing Plant Loss With Service Edge Visibility

Customer Profile

This global manufacturer navigated pandemic-related business challenges by expanding their e-commerce channels to supplement traditional distribution to brick-and-mortar retailers. The company’s earlier investments in factory robotics helped sustain manufacturing operations during the company’s own remote workforce transition.

During that time, the company also successfully completed a strategic information technology (IT) operations initiative, where services formerly managed by a business processing provider (BPP) were transitioned to an insourced model that improved data center operations. Their newly deployed NETSCOUT solution was a key factor in assuring this transition success, providing service edge visibility and performance analytics necessary to assure data center and business services were not impacted by this move.

The Challenge

The company operates hundreds of manufacturing plants, with this footprint enhancing global production capacity, regional delivery, and architectural redundancy. Wanting to assure operations consistency, many of the plants were configured with small data center environments that relied on Microsoft® solutions (including SQL Server and Active Directory), VMware™ Software Defined Data Center (SDDC) network elements, as well as Programmable Logic Controllers (PLCs) for manufacturing automation.

The company also made the strategic decision to deploy an Internet of Things (IoT) visibility application in their manufacturing plant environments. This manufacturing operations application was particularly critical to on-the-floor plant workers, as the business relied on the app’s reliable performance for:

• Packaging products
• Readying finished goods for distribution
• Resolving product deficiencies through lot number tracking
• Assuring compliance with federal standards governing manufacturing operations

The manufacturing application was deployed alongside an associated SQL server that used Active Directory (AD) to authenticate thousands of plant floor operators logging into the app as part of their daily workflows.

Over time, the Network Operations team (NetOps) was made aware of acute performance latency involving this manufacturing application at one U.S. plant, with the underlying IT assumption that those problems related to the network. In an important distinction for NetOps, the team knew this application communicated with many other manufacturing apps and business services across the network in order to function as designed. While root cause analysis had eluded IT Operations during their earlier outsourced IT contract, as well as the monitoring tools being used by that BPP, NetOps believed the NETSCOUT nGeniusONE analytics and smart visibility enhancements already realized across their service edges could offer the same successes in troubleshooting and resolving this application performance issue.

Solution in Action

Taking advantage of the service edge visibility delivered by vSTREAM virtual agents deployed at several plant locations, as well as InfiniStreamNG (ISNG) appliances installed at Equinix Co-lo’s in U.S. and Asia, NetOps begin anew their troubleshooting. Using real-time nGeniusONE Service Dashboards that offered drill-downs to contextual Service Alarm and Monitor views to determine root cause analysis, NetOps worked with NETSCOUT to determine the following contributors to poor manufacturing application performance at the U.S. plant:

• Latency in connecting to, and responses from AD servers.
• An asymmetric routing issue that involved an incorrect and under-reported IP address change made in the context of the BPP contract.
• Resultant, oversubscribed AD server calls from this one plant causing authentication failures in 60% of requests originating from the plant floor.

The nGeniusONE DNS Service Monitor provided additional performance metrics for Equinix Co-lo’s, as well as the U.S. plant, which showed authentication traffic was getting routed to an Active Directory instance in Asia, rather than a U.S. Co-lo that was the logical destination for these AD authentication transactions.

Unbeknownst to the collective NetOps, IT Operations, Active Directory, Problem Management, and Incident Response teams, these latency and authentication issues had been occurring for some time. The issue was particularly problematic in the Active Directory environment, as locational awareness functionality used to identify the location of other AD resources was impacted.

If it were not for NETSCOUT, these teams would likely have been unable resolve this issue by making a standard IP configuration change, because related error messages were not being generated in a manner contributing to root cause analysis.

The Results

In environments as vast as this company’s global footprint, manufacturing plant performance is business-critical.

With NETSCOUT sharing additional findings about the larger impact of applications latency and authentication issues, the IT Leadership team was pleased to realize there were correlations between improved performance and reduced plant loss.

Without improved service edge visibility and nGeniusONE analytics, these problems may have persisted and caused end-user experience issues, productivity frustrations, and additional plant loss.