by Dr Leila Powell, Lead Security Data Scientist, Panaseer
An oft-quoted business maxim is “if you can’t measure it, you can’t manage it”. Despite its straightforward logic, this truism has its detractors. These people have clearly never met a compliance auditor or the board-level sponsor of an enterprise cybersecurity strategy. If they had, the importance of metrics to the management of cyber risk would be abundantly clear, as would the criticality of being able to prove your position beyond doubt.
That’s because at the heart of any effective security programme are metrics: the objective measurements that answer key questions about how well the organisation is managing controls coverage and security risks.
When done right, metrics help enterprises create a stronger security posture by ensuring a control failure does not turn into a security incident.
The value of metrics
Cyber metrics facilitate decision making, support governance oversight and accountability, and improve performance. How else can you decide your priorities or assist internal and external compliance? How else can you determine how well you’re doing, and how much better you need to be?
Metrics feed a growing appetite for reports,dashboards and big data insights from a wide range of stakeholders; from risk management and compliance departments to executive leadership, shareholders, external auditors and regulators.
hey can start a cycle. Metrics beget metrics. Their delivery inevitably spurs demand for yet more metrics going into finer and more granular detail. For example, knowing how many logins there were today is the predecessor to knowing how many failed.
As this cycle continues, producing some metrics can be tougher than it looks.Sooner or later, the tools you use aren’t up to the job. How about knowing the number of failed logins from machines that weren’t patched, a week last Tuesday?
Factors impacting CISOs’ ability to deliver metrics at scale
We all know about the growing number of unfilled cybersecurity roles, which makes it doubly frustrating to see what finite resources within security teams are being asked to do with their time. We recently commissioned a study that found one-third of an enterprise security team’s time is spent on manual reporting in response to measurement requirements. A third!?
What makes matters worse is that much of that reporting could be questionable, with a strong majority having concerns about the lack of visibility and insights the data produced actually provides. The cause of these concerns is rooted in reliance on manual analysis, which can be subjective or qualitative, incomplete, or, as is often the case, prone to human error. And yet another kick in the teeth for CISOs is that the challenge is set to deepen on two fronts: the rising intensity, complexity and frequency of cyber threats, and more searching reporting requirements from regulators.
For example, as of 6 August 2020, the Singapore financial regulator MAS Cyber Hygiene Notice requires that organisations under its jurisdiction must ensure that security controls are in place on ‘every system’, thus showing a need for a continuous 360° view of every asset across multiple controls. As cyber threats persistnd regulators increase their expectations for reporting coverage, we can all expect more of the same.
Top 4 challenges when producing security metrics
The first challenge when producing security metrics is accuracy, the fundamental cornerstone for any measurement or communication of data. These metrics must be trusted beyond question, and yet there are signs they may not be. We asked more than 400 CISOs about their challenges in producing metrics and 37% of them cited “trust in the data” as their number one. This eats away at the very foundations of risk management; in effect making it ‘risky’.
Second is the time and resources required. All metrics have to be delivered by a certain deadline but take a certain amount of time to produce. Timeliness is a real issue when these two parameters cannot easily be reconciled. Do the metrics end up showing the current position, or the position from several days, or even months, ago? This feeds back to the principal question of trust. The threat landscape is constantly evolving, and it is very hard to keep pace with a metrics programme that is trusted and manually driven.
The third challenge is the sheer number of requests. As detailed above, these come both from internal sources who want reassurance that the security team is doing everything it can within an acceptable risk appetite, and external regulatory sources that need to validate compliance status to their standards. There is no sign that the volume of metrics requests is likely to decrease and relieve the burden it places on security teams.
The fourth challenge is not knowing what to measure. It’s hard enough for many teams to know exactly what to measure right now, let alone what metrics they’ll be asked for in the future. It’s an understandable cause of consternation. This also reflects the tendency of security measurement requests to be made reactively rather than teams being able to collect, prioritise and manage measurement needs upfront.
Why security metrics programmes lack maturity
Failure to overcome the challenges outlined above demonstrates a lack of a mature security metrics programme. Our research found around one-third of CISOs would score their own programme as “basic”, “elementary” or “intermediate”. Based on my conversations with many security, risk and compliance leaders, anything below “mature” would account for the majority of even the largest organisations.
The issue revolves around the reliance on painfully manual dataollection, collation and processing. This takes a long time. It can also be reliant upon subjective and ‘representative’ (i.e. incomplete) findings from – for example – qualitative questionnaires.
An overabundance of tools (our research found on average 57 separate security tools in use), creates difficulty producing cross-cutting, contextual insight. That extends to a seemingly rare ability to produce metrics very quickly, with complete confidence in their integrity
ar better would be a continuous and automatic metrics programme with little to no manual intervention. One that allows CISOs to get onto the front foot and literally ‘see what they’re missing’, enabling cyber risks to be identified by surfacing the context that emerges when they are able to combine multiple metrics, and prioritising these according to business impact.
Getting on the road to metric maturity
Each enterprise will be somewhere on its journey to maturing a metrics programme. The starting point has got to be identifying which metrics to measure, and the starting point for that is typically a recognised framework such as NIST CSF or CIS CSC. While frameworks provide broad coverage, good structure, and a common language, they aren’t designed to solve every challenge around security measurement. Reliance on them sometimes causes organisations to try to measure too much at once or lose focus on their priorities. Plus, they don’t take into account the significant challenges around data collection.
Inevitably, real-world metrics programmes step outside of these frameworks. Without a better underlying solution in place, the problem becomes cyclical: the list of metrics increases as demands evolve, with the design and implementation of each new metric stretching the capabilities of internal processes, tooling and sheer manual effort.
CISOs seem aware of these issues and determined to put them right. Sixty-five percent of those surveyed have targets in place to achieve the kind of metrics maturity described above by early-mid 2021.
Getting there is a straight choice between internal development or investing in platforms such as Continuous Controls Monitoring (CCM).
The latest Gartner Hype Cycle for Risk Management (July 2020) is its first to identify CCM as an emerging technology that addresses the security metrics issue and related challenges. In fact, the technology itself has been under development for 6 years and in live deployments since early 2017.
Gartner defines CCM as: “a set of technologies that automates the assessment of operational controls’ effectiveness and the identification of exceptions.” Also, that it is “runtime and transaction-level monitoring and is most useful for operational controls.”
Another way of understanding CCM is as a single source of truth organisations can use to address all security metrics requirements.It helps to restore stakeholder trust in the measures, whether that is trust in the validity of the metrics themselves, or in the security or compliance posture of the organization in general. With CCM, data is cleaned, normalized, aggregated, de-duplicated and correlated as part of the entity resolution process. By unifying disparate data, it can identify previously unknown or unmanaged assets and control coverage gaps in near real-time.
Crucially, the platform provides self-service access to current and historical data so that time-bound regulatory requests can be accurately and efficiently fulfilled – a blessed relief for CISOs and their security teams who would otherwise look to tomorrow’s security metrics workload with significantly more trepidation.
About the Author
Dr Leila Powell started out as an astrophysicist, using supercomputers to study the evolution of galaxies. Now she tackles more down-to-earth challenges! As the Lead Data Scientist at Panaseer, she helps information security functions in global organisations understand and reduce their cybersecurity risk exposure. She’s an advocate for diversity and inclusion in tech and co-created the WEDS (We Empower Diverse Startups) Network with other women in cyber tech startups to champion inclusive practices beyond her own team.
