The CISO’s Guide to Metrics that Matter
By Colin O’Connor, Chief Operating Officer, ReliaQuest
One of the most challenging tasks for today’s CISO is quantifying the success and the value of security investments. In the era of vendor sprawl, in which the majority of CISOs have purchased more than five technologies in the past year alone, effective reporting may seem arduous and daunting. But while security leaders tend to have a good sense of their department’s performance, many executives and board members complain that those metrics fail to provide adequate insight. Without the right security metrics, organizations could fail to communicate their security objectives and expectations, lack the ability to earn adequate budgets, and provide false confidence in security.
Security metrics must demonstrate business value and communicate the successes and needs of the entire program holistically. Traditional metrics fall short because they’re not actionable, they’re centered on tools, and don’t address people, processes and technology. For a CISO to communicate effectively with their peers and board, they must address the performance and output of their security model. Relevant security metrics help close the communications gap between boards and security teams so both can speak the same meaningful language. With that in mind, here are three metrics that should matter most to CISOs in 2020:
Visibility
Turnkey metrics reported over time — like number of events or MTTR — indicate progress through a siloed lens. These types of metrics tell an incomplete story. If a CISO can only see 50% of their environment, how is it possible to accurately represent the organization’s risk profile? Instead of building funnel metrics, CISOs must communicate how much of the environment is visible, and whether the proper controls are in place to efficiently defend and respond against threats. Visibility is difficult to capture without an accurate Configuration Management Database, but as a starting point, gaps in visibility across environment types may be compared. For example, in the case of multiple cloud environments, do teams have the same level of visibility as they do with on-premise data centers?
Measuring visibility across spectrums by environment, tech controls, and attack surfaces provides a greater understanding of risk, and better captures improvements from onboarding new data sources and analytics in a scoring that the board can understand.
Tool Efficacy
CISOs invest heavily into a critical tool set – such as SIEM, EDR, and UEBA. As such, it’s relevant to the bottom line to ensure ROI is maximized on these technologies. CISOs should examine efficacy through the lens of tool health and tool maturity. Number of outages, instances of usage problems, vendor tickets or drops in functionality can provide a quantitative reflection of effectiveness. Many security tools come with a variety of features and functionality, maturity should measure how much of what a CISO bought are they using? CISOs should dig into performance around feature integrations, projections for environment growth and architecture recommendations to understand if tools are being optimized and being used effectively.
Visibility metrics expose gaps in the security program, while tool efficacy metrics how well you are using your investment and help to determine whether these gaps can be filled by an existing tools capability, or if a new investment is needed. For example, gaps in SaaS activity often lead to investment in cloud access security brokers (CASBs) if these gaps can’t be filled by optimizing existing technology while many companies look to invest in a UEBA without realizing a traditional SIEM has the UEBA capability they need, it’s just not being optimized.
Team Performance
For many CISOs, team performance is the most challenging metric to quantify. However, finding the right metrics will unearth gaps in resource allocation or processes that could improve overall performance and organizational security. CISOs must examine where teams are spending time (look to false positive rates); how well the team understands its environment (examine true positive rates and anomalous safe rates); and how fast it’s resolving issues (such as MTTR). Discrepancies or anomalous findings will allow for immediate changes that will improve how teams engage with tools and protect the environment.
By reviewing trends for combined visibility, tool efficacy, and team performance scores, CISOs can better understand their level of protection, with the ability to drill down into specific coverage areas or threat types to explain if protection meets risk tolerance levels.
CISOs must surface metrics that convey meaning to executives and board members in a mutually-shared language. Using metrics that matter will allow CISOs to effectively highlight opportunities to improve security, connect security investments to business outcomes, and demonstrate and mitigate relevant risks to the enterprise. The right kinds of metrics allow CISOs to have a direct impact on the bottom line and improve both team and investment performance.
About the Author
Colin O’Connor oversees all technical operations as chief operating officer at ReliaQuest, and previously served as vice president of security operations leading detection and response, security engineering, threat management, and business analytics.
