Decryption Is Key for Enhanced Security and Monitoring

But how does TLS encryption and decryption work, and why is it secure?

Words in light blue with dark background
Erik Hjelmstad

In Part 1 of my series on Transport Layer Security (TLS) decryption, I went over a few basics of encryption, discussed TLS 1.2, and concluded by outlining the improvements TLS 1.3 provided. In this second installment, I dive into TLS decryption in versions TLS 1.2 and 1.3.

TLS Decryption

To decrypt TLS sessions, there are a few requirements. One of the options is to be either on the client or on the server. The client and server must be able to decrypt the session at some point to use the information. For some scenarios, this may be all that is needed, but this, unfortunately will not scale well.

Another option is to have a copy of the server’s private key and use it to listen to the traffic as it traverses the network. This works in some scenarios and will allow sending all the decrypted traffic to a security or monitoring tool for further analysis. There are two problems with this solution. The first is that it will work only if you control the server. It will never work to decrypt internal client traffic out to the internet. The second is that it will not work for TLS 1.3, even if you have the server’s secret key. TLS 1.3 uses ephemeral keys, so you can’t passively listen to the traffic and then decrypt it.

To decrypt sessions from multiple clients to multiple servers, it is necessary to be a person-in-the-middle so that traffic can be modified between the client and the server. In this situation, the initial state for a TLS 1.2 session is a little different. NETSCOUT nGenius Encryption Appliance (nDA) needs to be able to see all the traffic between the client and the server. The client needs a certificate authority (CA) certificate and public key, but now this certificate needs to be generated by nDA. The server initial state remains the same. nDA has its own self-created CA certificate and public and private keys, the CA’s certificate and public key, and its own server public and private keys.

TLS 1.2 Decryption

So, what is the process for decrypting TLS 1.2 sessions? The start of this TLS session is the same: The client sends a Hello message that includes a list of the cipher suites it supports. The server responds a with a Hello message, picks one of the cipher suites, and includes its server certificate and its public key. This message is intercepted by nDA. nDA replaces the server public keys with its own and re-signs, using its CA private key. This is a key the client trusts, so it goes through the same steps as before: The client authenticates this server certificate by decrypting it with the CA public key. It then generates a premaster secret, encrypts it using the server public key, and sends it to the server. This message is intercepted by nDA as well. It uses its own private key to decrypt this message, recovers the premaster secret, and is then able to generate the session key from the premaster secret.

TLS 1.2 Decryption Diagram

nDA now does two more things. First, it processes the server Hello message, authenticates the server, and encrypts the premaster secret, just like the client would, and sends this to the server. It also responds to the client with an encrypted handshake message, which establishes TLS session 1. The server decrypts the message it receives, generates the session key from the premaster secret, and sends its handshake message. nDA receives this message and knows TLS session 2 is also active. The client, nDA, and server all now have the same session key, and nDA will be able to decrypt all messages between the client and server.

The overall process for decrypting TLS 1.3 is similar, but there are fewer overall steps, because a few of the above steps are combined. nDA acts as both the client and the server, modifying the keys as appropriate so that nDA winds up with two TLS session keys—one with each side.

To Be Concluded

In my third and final installment, I’ll demonstrate a few ways the client can determine if this is happening on its connections and show why TLS is still secure, even when nDA is used to decrypt transactions.

Read Part 1 of my TLS decryption series —and stay tuned for Part 3.