ISM3 Brings Greater Standardization to Security Measurement Across Enterprise IT

Updated: February 06, 2010

The security metrics panel and sponsored podcast discussion are coming to you from The Open Group's Enterprise Architecture Practitioners Conference in Seattle on Feb. 2, 2010. The goal is to determine the strategic imperatives for security metrics, and to discuss how to use them to change the outcomes in terms of IT's value to the business.

Our panel consists of a security executive from The Open Group, as well as two experts on security who are presenting at the consortium's Security Practitioners Conference: Jim Hietala, Vice President for Security at The Open Group; Adam Shostack, co-author of The New School of Information Security, and Vicente Aceituno, director of the ISM3 Consortium. The discussion is moderated by Dana Gardner, principal analyst at Interarbor Solutions.

Here are some excerpts:

Hietala: We think there's a contribution to make from The Open Group, in terms of developing the ISM3 standard and getting it out there more widely. [Being a data-driven security organization means] using information to make decisions, as opposed to what vendors are pitching at you, or your gut reaction. It's getting a little more scientific about gathering data on the kinds of attacks you're seeing and the kinds of threats that you face, and using that data to inform the decisions around the right set of controls to put in place to effectively secure the organization.

A presentation we had today from an analyst firm talked about people being all over the map [on security practices]. I wouldn't say there's a lot of rigor and standardization around the kinds of data that's being collected to inform decisions, but there is some of that work going on in very large organizations. There, you typically see a little more mature metrics program. In smaller organizations, not so much. It's a little all over the map.

... The important outputs of a good metrics program can be that it gives you a different way to talk to your senior management about the progress that you're making against the business objectives and security objectives.

That's been an area of enormous disconnect. Security professionals have tended to talk about viruses, worms, relatively technical things, but haven't been able to show a trend to senior management that justifies the kind of spending they have been doing and the kind of spending they need to do in the future. Business language around some of that is needed in this area.

Shostack: We have an opportunity to be a heck of a lot more effective than we have been. We can say, "This control that we all thought was a really good idea -- well, everyone is doing it, and it's not having the impact that we would like." So, we can reassess how we're getting real, where we're putting our dollars.

The big change we've seen is that people have started to talk about the problems that they are having, as a result of laws passed in California and elsewhere that require them to say, "We made a mistake with data that we hold about you," and to tell their customers.

We've seen that a lot of the things we feared would happen haven't come to pass. We used to say that your company would go out of business and your customers would all flee. It's not happening that way. So, we're getting an opportunity today to share data in a way that's never been possible before.

Aceituno: The top priority should be to make sure that the things you measure are things that are contributing positivity to the value that you're bringing to business as a information security management (ISM) practitioner. That's the focus. Are you measuring things that are actually bringing value or are you measuring things that are fancy or look good?

Because metrics are all about controlling what you do and being able to manage the outputs that you produce and that contribute value to the business ... you can use metrics to manage internal factors.

I don't think it brings a bigger return on investment (ROI) to collect metrics on external things that you can't control. It's like hearing the news. What can you do about it? You're not the government or you're not directly involved. It's only the internal metrics that really make sense.

Basically, we link business goals, business objectives, and security objectives in a way that's never been done before, because we are painfully detailed when we express the outcomes that you are supposed to get from your ISM system. That will make it far easier for practitioners to actually measure the things that matter.

Business value approach


Shostack: Vicente's point about measuring the things you can control is critical. Oftentimes in security, we don't like to admit that we've made mistakes and we conceal some of the issues that are happening. A metrics initiative gives you the opportunity to get out there and talk about what's going on, not in a finger pointing way, which has happened so often in the past, but in an objective and numerically centered way. That gives us opportunity to improve.

Hietala: There's some taxonomy work to be done. One of the real issues in security is that when I say "threat," do other people have the same understanding? Risk management is rife with different terms that mean different things to different people. So getting a common taxonomy is something that makes sense.

The kinds of metrics we're collecting can be all over the map, but generally they're the things that would guide the right kind of decision making within an IT security organization around the question, "Are we doing the right things?"

Today, Vicente used an example of looking at vulnerabilities that are found in web applications. A critical metric was how long those vulnerabilities are out there before they get fixed by different lines of business, by different parts of the business, looking at how the organization is responding to that. We're trying to drive that metric toward the vulnerabilities being open for less time and getting fixed quicker.

Shostack: We've seen over the last few years that those security programs that succeed are the ones that talk to the business needs and talk to the executive suite in language that the executives understand.

We've seen over the last few years that those security programs that succeed are the ones that talk to the business needs and talk to the executive suite in language that the executives understand.



The real success here and the real step with ISM3 is that it gives people a prescriptive way to get started on building those metrics.

You can pick it up and look at it and say, "Okay, I'm going to measure these things. I'm going to trend on them." And, I'm going to report on them."

As we get toward a place, where more people are talking about those things, we'll start to see an expectation that security is a little bit different. There is a risk environment that's very outside of people's control, but this gives people a way to get a handle on it.


Featured Research