by Thomas Pore, director of security products at LiveAction
It’s no secret that malicious actors are hiding malware in encrypted traffic and that this trend has surged over the last 18 months. In fact, according to a recent report, 91.5% of malware arrived over encrypted traffic in Q2’21. While advanced encryption protocols (such as AES, TLS, QUIC, RDP, Secure Shell and more) offer the guarantee of data confidentiality and integrity, which provides privacy and security for network traffic, it also comes at a huge cost. It prevents SOC teams from easily gaining visibility into network traffic and properly securing the organization. In this article, I’d like to explore how organizations can identify malware in encrypted traffic and why it’s important.
Traditional security tools attempt to stop malware by using deep packet inspection or rules-based monitoring on unencrypted traffic. Encrypted traffic creates security blind spots and renders most security devices useless – including those that use deep packet inspection techniques to examine the packet’s payload to detect threats. This approach is no longer sufficient and has given rise to the field of Encrypted Traffic Analysis (ETA), which helps preserves data privacy and offers insights into encrypted traffic by observing a variety of data features through passive monitoring – all without the need for decryption proxies. Advanced ETA methods use Deep Packet Dynamics (DPD) to enable behavioral anomaly and machine learning techniques for detecting complex attacks. It helps organizations ensure compliance and protect against threats such as malware, MITM attacks, command and control activity, and data exfiltration.
Complicating matters, many applications and services rely on Cloud and web-based platforms that use encryption, such as Secure Sockets Layer/Transport Layer Security (SSL/TLS), to ensure user privacy and data confidentiality (not to mention additional regulatory standards like PCI DSS and GDPR). While decryption proxies and Man-in-the-Middle boxes can be used to monitor encrypted traffic, they apply bulk decryption, analysis, and re-encryption that can negatively impact performance and latency. Hence the need for more advanced ETA methods.
What does ETA allow you to guard against? Here are four reasons you need to see into encrypted traffic, and how ETA can help:
- The goal of encryption is to provide confidentiality between a connection. Decrypting secure transactions challenges the basic intent of protecting messages and ensuring privacy. Breaking encryption removes the privacy of end-users and violates organizational compliance obligations. Not to mention, the management of deployments can be incredibly complex with intermediary certificates, rolling out agents, and is only operationally effective on a subset of corporate assets. ETA solves this problem by finding a balance between network visibility and security while maintaining the confidentially of end users. This is done using Deep Packet Dynamics (DPD).DPD passively monitors network traffic collecting simple, enhanced, and advanced behavioral characteristics about each network connection, without having to decrypt a packet. It restores lost visibility through insights provided from machine learning and fingerprinting by combining traditional flow tuple information (such as IP Address, ports, protocols, enhanced metadata, and much more). This form of network visibility reinforces privacy by eliminating the complexities of decrypting and inspecting traffic.
- Threat actor activity continues to increase, and as encrypted traffic increases, so do the attacks hidden within it. There is a consistent need by threat actors to gain remote access to your organization and they are evading detection by hiding in plain sight, among the encrypted traffic. The detection of activity in network traffic associated with the attacker’s existing malware is paramount to organizational security. ETA solves this problem by maintaining an inventory of encrypted traffic seen within a network. Tracking the use of formal encryption services (TLS, SSH, etc.) including handshakes and connection metadata over time provides a baseline inventory of what assets are using encryption, what types, resources used, and who they’re talking to. ETA leverages machine learning to alert analysts on priority risk events where encryption anomalies exist, delivering an improved security posture by increasing visibility.
- Improper use of encryption can create a false sense of security, leading to increased risk and exposure for organizations. Historically there has been an inherent trust with supply chain vendors – assuming they’re experts at securing their products. However, those products could contain weaknesses unknown to the customer, such as inadequate encryption strength. Simply enabling encryption is not enough today. Enabling an encryption scheme that is theoretically sound, but not strong enough to provide adequate protection, may allow an attacker to intercept and steal sensitive information. ETA solves this problem by providing network-wide visibility into encryption use for regular and proactive security protocol assessments.
- Encryption assurance and the ability to identify unauthorized changes (often driven by Shadow IT) are critical to the security posture of an organization. Many enterprises use large-scale VPN circuits to connect users with data, and most are using encrypted VPN tunnels for remote access, without any assurance that the data is encrypted. ETA provides assurance that traffic intended to be encrypted is encrypted. Additionally, ETA allows for the detection of new remote access encryption communication, which is crucial to identify where IT professionals have potentially bypassed formal authentication channels enabling remote uses by enabling RDP or other unapproved remote access applications.
When looking to implement an ETA solution, there are some key areas to consider. First, most attacks are still occurring against on-premises systems. Despite organizations moving production workloads to the cloud, attackers are leveraging phishing and stolen credentials for remote access into organizations. Future proofing your encrypted traffic strategy should include ETA visibility from the core, to the edge, and in multi-cloud. Next, organizations need to look beyond JA3(S). Often considered the first approach to encrypted traffic analysis – and not be excluded from your security posture – it does share the same limitations as signature-based solutions, which are reliant on pre-identified threats or blacklists.
Furthermore, encryption certificate analysis is useful and can be helpful in a forensic response. However, advanced adversaries will go out of their way to ensure that their hacking infrastructure is using valid certificates, whether that means setting up a reputable infrastructure on their own, obtaining certificates for domains they do not own, or hacking servers with legitimate certificates. And finally, Consider an ETA strategy where machine learning is leveraged to help improve the detection, response time, and prioritization of threats and risks, that would otherwise go undetected.
Unfortunately, network visibility is eroding as encrypted protocols increase. But the industry is working to address these challenges with advanced ETA that provides alternative techniques for security professionals to gain visibility into this traffic. Combining DPD with machine learning is revitalizing classic approaches to cryptanalysis by applying powerful algorithms to identify patterns in network data. To learn more, check out these resources.
About the Author
Thomas Pore is the director of security products at LiveAction. He was previously vice president of technical services and the director of IT and services at Plixer. Thomas can be reached online through LinkedIn and through the LiveAction website.
