Metrics focused on driving the right behavior

5.4Metrics focused on driving the right behavior

With a foundation of robust knowledge management in place, together with intelligent automation using industry tools and internally developed custom tools and scripts, we need to make sure we are driving the right behavior with the engineers and enabling them to succeed. We do this by defining carefully selected metrics which are not only actionable but also drive positive reinforcement.

It’s easy to come up with metrics such as number of bugs or number of fixes to measure application quality (security or otherwise). However, that can be a rather antiquated model that worked better with traditional engineering practices. In the modern engineering model, where the pace of innovation and continuous assurance are the core drivers, we focused on mean-time based metrics instead such as mean-time-to-triage (MTTT) and mean-time-to-fix (MTTF).

These metrics are driven by mean-time and measure how fast the engineers respond to security issues by triaging them first and then by fixing them in their release cycles. Enabling our engineers with automation is critical, but we also need to measure how fast they are leveraging the automation to drive continuous assurance.

Automation metrics is one dimension we considered while developing an all up Application Security (AppSec) Health Index for an application or for a portfolio of applications managed by a team. We’ll discuss the details of our application security health index or AppSec Health Index later in the paper, but first we want to discuss the concept of the “four stages of competence.” This forms the backdrop for why we developed the AppSec Health Index in the form we did.

The four stages of competence model (developed by psychologist Noel Birch in the 1970s) is a helpful illustration of the stages that an individual goes through to learn a new skill. We’ve applied it to the process that an engineer progresses through as they learn how to achieve security hygiene with the AppSec Health Index.

When learning a new skill, individuals or teams progress through each phase from “unconscious incompetence,” (an unskilled state) to “unconscious competence” (a state of expertise).

We think this illustration of the progression of skills is a great concept and is a basis for what we wanted to achieve when creating a health index for our AppSec program.

The AppSec Health Index measures the impact of our SDL in maintaining a secure posture of an application portfolio. It is a qualitative metric that is used to indicate the application security health of a team or organization. The index is derived from four underlying quantitative metrics: 

Table AppSec Health Index qualitative metrics

It’s important to note that the targets for each metric should be based on an organization’s maturity and needs. For example, in some organizations, having SDL compliance for as little as 50% of the portfolio may be a reasonable target. In other organization, aiming for 100% compliance may seem realistic. Determining the right target is not trivial and can often take many iterations to converge. However, it’s important to stress that the above numbers are examples only and should not be treated as targets for any company that wants to achieve the most ideal application security posture.

If we apply the four stages of competence to our AppSec program, and factor in the AppSec Health index metrics, the progression of skill might look like the following:


Figure stages progression

Finally, to calculate the single qualitative AppSec Health Index, we analyze the four quantitative metrics to determine the conditions that set its value. One such example is to define the overall AppSec Health Index into three tiers of Green (on target), Yellow (close to target) and Red (off target) and set the value based on this:

Figure AppSec Health Index

5.4.1.1Driving competition

The concept of using metrics to drive competition or gamification is not new in security. Microsoft engineering teams love to compete to be at the top and to be the best among their peers. We approached this by publishing the AppSec Health Index quarterly on a scorecard for all engineering teams in Microsoft IT. This drove a sense of competition and encouraged teams to mimic the best practices of the teams who are on the top of the scorecard.

5.4.1.2Positive reinforcement

Security teams traditionally point out negative metrics and gaps in the scorecard for the engineering teams. Although these are required at times to drive action by the teams and to reduce risk, consistently calling out only negative metrics does not promote the positive behavior we want to ultimately instill. We used the natural alternative approach where the metric also reflected positive reinforcement for the teams who were doing better. We weren’t surprised to learn that showcasing positive behavior goes even further if highlighted in the right way. Our AppSec Health Index helps us surface and drive positive reinforcement. For example, we started publishing the AppSec Health Index by using a quarterly risk scorecard across all our business groups. This initiative not only drove a sense of competition and positive behavior, but it also increased health indexes across those groups.

5.4.1.3Auxiliary metrics

Finally, we also developed an auxiliary metric specifically around trending information for “Fortify Top 10.” As more and more applications began using automated static security scanning, we wanted to identify what the top trending Fortify issues were across our engineering community and within specific engineering teams. This metric helps uncover potential pain points such as a high number of unfixed issues in a certain category or in a certain pocket of the business which may require special attention. We have driven multiple targeted campaigns and programs to deal with the pain points this metric revealed.

Figure Sample - Fortify Top 10 Vulnerabilities

We also found “Bug density data” valuable to determine an engineering team’s security maturity and to enable the conversations with the teams to figure out corrective actions.

Figure Sample - Bug Density (per 10k LOC)

5.5User experience

So far we have discussed having the right knowledge management, deploying in-place intelligent automation, and finally capturing it all with effective metrics to drive the right behavior within our engineering community. All of this could be in vain if we don’t take a very explicit look at overall user experience. Users in this context are our software engineers who interact with and use this solution.

Oftentimes, security teams undermine the importance of end-to-end user experience while creating and deploying security solutions. Although as an industry, we’ve made significant progress, we think the security industry as a whole has been behind on this – just look at the plethora of security solutions in the industry that treat user experience as an afterthought! Security is critical, but it doesn’t mean we have an excuse to ignore user experience. We focused on simplicity and user experience when thinking about our end-to-end solution. Some of the concepts that helped orient our work on this included:

5.5.1.1If it compiles, it complies

How can we integrate security seamlessly with the engineering experience so it does not feel like a bolted-on solution but is instead part of the experience? For example, using security static analysis in build environments, we can define the rules that automate TCP verification as part of the compile and build processes. If all controls have been implemented accurately, the code just complies cleanly. It’s important to note that this is an aspiration state. We don’t believe we are yet at a point where all known security issues can be effectively detected during build in an automated capacity.

5.5.1.2The secure way should be an easy way

Always look for opportunities to define the secure way as an easy way. For example, using automation is the easy way. Running Fortify in build environments instead of employing a manual assessment saves weeks of time. And instead of hiring pen testers to test an app over several weeks, periodic WebInspect scans can save time and resources.

User experience is a journey and we are still working toward achieving the optimal end-to-end user experience for our process. If engineers are treated like customers and if they are engaged during tool evaluations, process redesign, and overall user experience discussions, we believe this can yield very powerful results.

5.5.2Taking security to engineers

5.5.2.1Fortify integration with IDE

One of the reasons engineers value Fortify is that it is in their Integrated Development Experience (IDE); namely, Visual Studio (VS). Visual Studio is where engineers spend most of their time, and having security plugged right into the environment is very convenient for them to use and interact with. They can view the results of the scan in their familiar interface and can triage and collaborate by responding to the issues within the same UI. The following is a screenshot taken from our training material which we use to articulate the value of Fortify to our engineering teams.



Figure Fortify Plugin for Visual Studio

5.5.2.2Fortify build-integration

By integrating Fortify in the build process, we took another step to take security to engineers directly. This integration promotes transparency and helps security become even more a part of the overall engineering experience.

5.5.2.3Reducing unnecessary security questionnaire

We looked closely at the SDL process and the underlying security attestation questionnaire that engineering teams complete for their application. We uncovered many duplicate and unnecessary questions and saw an opportunity to reduce noise by keeping only the questions we really need. For example, when we evaluated our initial risk assessment process, it had grown to an alarming 93 questions just to determine the initial risk score. Those 93 questions spanning two questionnaires were reduced to two questions in one questionnaire. The end result was an efficient and quicker security process which reduces friction even further.

We went a step further by also making the questionnaire intelligent. When the engineering team opts to run static and dynamic security tools, certain control questions (that validate the implementation of TCPs) are automatically hidden, which makes the questionnaire shorter and smarter, and therefore, enhances the user experience.

5.5.2.4Reducing unnecessary security and privacy requirements

Similar to reducing security friction from a process point of view, we continuously look for ways to ensure that our TCP set (security and privacy requirements) are refreshed and streamlined on an ongoing basis. During the last process enhancement we did in ISRM, we were able to reduce 106 varying technical requirements (TCPs) to 73 requirements.

5.5.2.5Reducing time to engage security

How easy is it to engage your security team? What is the average time it takes to engage the security team for the organization? These were the questions we asked when we looked at the very first step of our process: engagement. We realized that our initial intake process, from discovering a new application to giving them SDL guidance, could take several days. It was obvious that this had to drastically change. We discarded the interim processes and a tool set that weren’t needed or that didn’t add any value and reduced the overall time to initiate the SDL process by 83%. We enabled our engineers to self-provision the SDL process without waiting for the central team to provision one for them.
There is much more work we need to do as we integrate security more closely to engineering systems and processes. One of the big initiatives we are driving is evaluating how we can use Visual Studio Team Systems (VSTS) work items to drive security activities more effectively and efficiently.

6Future of application security

The scope of application security has to be redefined. Traditionally, application security was seen as the practice of protecting the application code. In the modern engineering era, this is an extremely limited view. The same set of engineers who write the code have to be concerned with operational practices, host configuration, and network in addition to the code. In light of this, we are repositioning ourselves from virtual silos of “application security,” “infrastructure security,” and “network security,” to a single unit focused on “Secure Engineering.” The details of the strategy behind this shift is outside the scope of this paper, but it’s important to emphasize that we believe modern engineering practices such as those described in this paper require a shift toward holistic thinking about security engineering.

7Lessons learned

We have learned so much from our engineering teams and from our partnerships – and we have a lot more yet to learn. The following are some of the more important lessons we’ve learned so far.

7.1Partner with engineers

It’s critical to partner with your engineers. They need to be stakeholders in the process and not just consumers of the process. When you evaluate your tools, don’t evaluate just within your security team but extend it to the engineers. Pay special attention to what is working for them and what isn’t. They can offer important feedback on not only your process and requirements but also can identify gaps to share with your tool vendor to make the tool work even better for your engineers. Engineers need to love the tools even more than security does.

7.2Focus on the willing

Any security process relies on its participants to be successful. Adopting new methods and processes is not an easy endeavor, and it’s important to focus on the willing. Listen to your engineers who are positive and willing to adopt robust strategies for your security program and focus on their needs. As more engineers engage with the process and become stakeholders, others will follow.

7.3Be thoughtful about selecting technology

Because automation is such a significant part of our strategy to secure modern engineering, we learned key lessons along the way about evaluating and licensing third-party technology that are worth sharing. First, technology is not a one-size-fits-all solution, nor is it a quick fix in terms of creating a security process. Second, be mindful about customizing any tool to work for your engineering experience and don’t merely throw it over the wall to your engineers. Involve them in your vetting of any new third-party tool and listen to what works for them and what doesn’t before you make any decisions about onboarding a tool.

7.4Build your process first, then focus on tools

It’s a common mistake to jump right into technology. It’s tempting to think that a tool can solve all your security problems. The reality is that you need to have your process built first and then evaluate tools to support that process with the right metrics. The main reason is that it’s rare that you’ll need just one piece of technology. If you are not conscientious about how you bring in technology to support your process, every new technology you introduce could end up corrupting your process and leave you with a Frankenstein-like end result.

Avoid an overly architected, rigid process. In fact, the nimbler the process the better. For example, have at least your knowledge base well defined so you know exactly what you are evaluating and how it will help you streamline your process in driving software security assurance.

7.5Integrate your tools into the engineers’ world

If it complies, it compiles. If you start with this as your North Star, you quickly discover that you must bring security right into the engineers’ world – not the security world! This not only helps to reduce friction for your engineers, it also enables them to move faster by integrating your tools into the world in which they live and operate. If they don’t need to exit their IDE to use the tool, they are more likely to use the tool and tool fatigue is less of a road block.

This is our aspiration because there are security issues, such as poor authorization controls or design issues, that tools have a very hard time detecting. Nevertheless, it’s an aspiration worth driving towards – automating the detection of all those security issues that can be automated.

7.6Build a relationship with your vendor

A tool that meets your needs today may not be able to keep pace with the needs of tomorrow. We didn’t want to just license a tool for our security needs, but instead wanted to develop a relationship our tool vendor. To this day, we maintain a very open dialogue with HPE Fortify not only about how we use their security products, but also about what we would like to see to meet the ever-changing needs of our engineers. These could include obvious needs such as supporting newer technologies or frameworks. Another advantage of this relationship has been the opportunity to test features or new technology that HPE Fortify has developed in answer to other customer’s requirements.

7.7Be mindful of business impact

Keeping business impact always in perspective may seem obvious, but it’s not always the easiest to practice. Security solutions exist to enable business to achieve its goals in a secure and trusted manner. If security does not affect the business in a positive meaningful way, then it’s going to be an irritating speed bump that runs the risk of becoming a secondary consideration.

The easiest approach we found was to invite your business stakeholders (the engineering teams) into the discussion. This discussion should include not just the processes and technology, but the development of metrics such as mean-time-to-triage and mean-time-to-fix to measure success. If the business can align to your metrics and agree that driving to defined targets will help create an impact, you’ve got a natural win-win situation.

7.8Keep up with changing technology

Technology is incredibly fickle. Not only is it important to maintain your vendor relationships to continue to drive your requirements, but it’s important to constantly look at new players coming into the field who may have more novel solutions. Additionally, once you’ve built your security processes, take care not to become complacent with respect to changing technology. As important as a robust security process is, it’s equally important to keep an eye on the horizon of emerging technology, and to be aware that your security model may need to continually adapt to new innovations in the industry.

8Conclusion

The modern engineering world presents ongoing challenges for us as we adapt to a DevOps model while also navigating the changing security landscape. Our approach has been to enable our engineers with security solutions that reduce friction as much as possible. We want to improve and enhance the engineering experience for our engineers with respect to software security assurance. We feel very comfortable with our iterative approve on our solution because this is very much the agile way – to jump in and iterate on the solution over time. We’ve done the heavy lifting already of creating the building blocks for our foundational components of knowledge management, automation, metrics, and user experience. Making these strategic shifts was the difficult part, and now we can build on that momentum.

In addition to the challenges we talked about, we also see big opportunities for advancing application security in this age of modern engineering. There has never been a better time to push security automation and develop integrated security services for engineering teams as they think about operating in a DevOps environment and push for CI/CD scenarios. Similar to how development, test, and operation roles have merged to shape today’s modern engineer, we feel that a software security assurance program can yield much better results than before if the processes are baked seamlessly into the engineering process. Security teams should leverage this momentum of automation to further expand the scope and coverage in their organization in an effective and efficient manner.

9Appendix A: Resources

9.1.1SDL

Security Development Lifecycle

https://www.microsoft.com/sdl

Application security in the changing risk landscape

https://f5.com/Portals/1/PDF/security/f5-ponemon-report-2016.pdf

The Security Development Lifecycle: SDL: A Process for Developing Demonstrably More Secure Software (Developer Best Practices)

https://www.amazon.com/Security-Development-Lifecycle-Developing-Demonstrably/dp/0735622140/ref=sr_1_1?ie=UTF8&qid=1473087879&sr=8-1&keywords=security+development+lifecycle

9.1.2Modern engineering and DevOps

Creating a culture of modern engineering within Microsoft IT

https://msdn.microsoft.com/en-us/library/mt709101.aspx

DevOps—Application Lifecycle Management – Microsoft

https://www.microsoft.com/en-us/cloud-platform/development-operations

The science of DevOps decoded

http://www.gartner.com/smarterwithgartner/the-science-of-devops-decoded/

Directory: download
download -> Performance Tuning Guidelines for Windows Server 2008 May 20, 2009 Abstract
download -> Northern England’s set-jetting locations
download -> Bbc archives: Beyond the programme download
download -> Occupational health and safety
download -> Dynatest 3031 lwd light Weight Deflectometer
download -> Forth coming competitive exams
download -> English Olympiad Tasks: 10 th form Аудіювання, 10 клас. Текст
download -> Brush High School Morning Announcements Friday, September 05, 2014 all school
download -> Year 9 The Arts — Music
download -> Years 7 and 8 standard elaborations — Australian Curriculum: Music draft