5.3Implementation
There is no such thing as perfect security, but the pursuit of thoroughness is important because one vulnerability is all it takes to compromise an application. In ISRM, we use HPE Fortify SCA to conduct automated static security analysis of our applications. After evaluating many static analysis tools, we chose Fortify SCA because of its thoroughness, its ability to facilitate collaboration, and most importantly, its coverage against our TCPs.
Fortify SCA is a static application security solution. It traverses your code, identifies potential security issues, and compiles a list for you to review. A human determines which issues are serious flaws that must be fixed and which issues are false positives or low priority and can be deprioritized or ignored.
Fortify offers two desktop IDEs to view and aggregate issues. Our engineers use the Visual Studio plugin to view Fortify SCA results inside their development environment, while most of our analysts use the Fortify proprietary IDE, Audit Workbench. Everyone accesses scan results from the same Fortify Software Security Center (SSC) server, which ensures that all comments and reviews are preserved from one scan to the next. Build engineers will configure regular scans which will automatically upload results to the Fortify SSC server, ensuring everyone has recent scans to work from.
We started the journey with Fortify in 2014, and we use the following three capabilities with Fortify for our needs in ISRM:
This is our most preferred method. We’ve integrated Fortify SCA static analysis with the application build processes. Based on the schedule and release cadence, static analysis runs as well. All the results from the scan are then automatically uploaded to the Software Security Center (SSC) server.
Self-hosted scans are one-time scans. This provides a scanning method for those applications that either do not have a build environment for integration or that are one-time applications. Examples are marketing campaign applications that focus on a specific event or applications with a shelf life of less than six months, for example. These applications leverage self-hosted scanning to perform Fortify SCA static security scanning. Results are then uploaded to the SSC server. Engineers are encouraged to utilize this feature so they can see scan results within their IDE before their code is uploaded to build servers.
-
Static security analysis on-demand
On-demand static security analysis provides engineering teams with white glove treatment. A central security team scans the code on behalf of the engineering teams and helps the team triage the issues. This is similar to HPE’s Fortify on Demand service, but it is run within the organization by our central security team. We developed this capability primarily to handle venture integration scenarios. For example, this capability comes in handy when your company inherits LOB applications as a result of an acquisition and the engineering teams managing those applications have little or no exposure to static analysis. You will need to hand hold them to get them started and slowly move to self-hosted scans and finally to build integration.
5.3.2Fortify SCA and intelligent automation
Vetting Fortify SCA and enabling our development teams to use it is our best example of intelligent automation. Introducing automation into a development environment reduces the burden on engineers of using security tools and processes. Throwing a tool over the wall and expecting that a team will not only use the tool, but benefit from the data, may not set a development team up for success. Time is a precious commodity to engineers and onboarding a new tool can be far outside their scope. In this sense, a security team needs to become a “team enabler,” and must work with development teams to find the best methods to fit the tool into their workflow. It must become a part of their environment and must work within their established processes. To that end, and focusing on team enablement, we partnered with our engineering teams to help integrate the tool into their build processes.
5.3.3Fortify SCA implementation process
We enabled our engineering teams to succeed with Fortify SCA during implementation by designing two process tracks: one for the build teams and one for development teams. Error: Reference source not found illustrates the steps we’ve followed for each track, including the phases that are utilized for training.
5.3.3.1Build track
The build track is where the heavy lifting occurs. We work with the build owner to ensure that Fortify SCA is installed on their build environment and is configured and scheduled to run based on their release cadence. We then run the first scan and hand off to the development track.
5.3.3.2Development track
Once the build automation is complete, and we receive the first baseline results, we triage the results together with the engineering teams. We meet with all stakeholders and help them understand the review process so that they can begin to own the process in their environment.
5.3.4Fortify SCA deployment architecture
Our Fortify SCA deployment currently has nine Fortify SSC servers running with their own database on a shared database server. Each Fortify SSC server is reachable with a distinct URL and is independent of the others. One of the Fortify SSC servers is dedicated to our one-time scan activity to ensure that we have good separation before teams fully onboard the tool. We use views in each database to collect data for reporting, which is then migrated at regular intervals to the reporting warehouse. Based on our experience, we chose to use three separate User Acceptance Test (UAT) environments:
-
Environment we use for testing the current version of Fortify SSC.
-
Environment we use for all the patches Fortify provides us to test that our issues are fixed before they are rolled into a major release.
-
Environment we use for testing underlying operational changes or updates.
Figure UAT environments
5.3.5Shortfalls and opportunities
We faced several challenges when we began to deploy Fortify SCA in our large enterprise space. The following are some of the shortfalls, including ways that Fortify SCA is helping us address or will be addressing these shortfalls in the future.
Fortify SSC does not currently allow for load balancing and horizontal scaling. Because of the size of our organization, and because we were unable to balance the load of our Fortify SSC servers, we knew we would have to manually scale the solution. We currently manage nine Fortify SSCs in our environment for static analysis and two for dynamic analysis. Each group in Microsoft IT is so large (from an application portfolio perspective) that initially we chose to separate by group, which was an easy distinction to keep the load manageable. We further addressed load issues by identifying which group uses which Fortify SSC server in reports. Even through reorganizations and group name changes, we have been able to manage the load consistently by using this approach.
Achieving better horizontal scaling is still an opportunity. However, the performance of the Fortify SSC servers has improved greatly over the past several releases and made this scaling less necessary.
5.3.5.2Differential scanning
Currently Fortify SCA scans the entire code base with each scan to determine code changes and any security issues in the code. With Agile and DevOps picking up pace of development, security teams are looking for ways to deliver results in a shortened timeframe. The ability to perform differential scanning on only the code that has changed since the last scan is key to making scans run faster and more efficiently.
While using Fortify SCA as our analysis tool, we found an opportunity to openly communicate to the Fortify team about our needs as an enterprise organization along with our feedback. In doing so, we’ve found tremendous value in moving beyond a customer/vendor relationship. The partnership we’ve built with the Fortify team in our journey has yielded important advances in our ability to secure our code. We believe our relationship with the Fortify team, and continuously sharing information, has uncovered valuable new insights that not only helped us, but helped Fortify as well.
5.3.6VSTS integration
Much like the rest of the industry, Microsoft is moving many of our key resources into the cloud, including moving software engineering to Visual Studio Team Services (VSTS). This platform has become a focal point for our engineering teams. As a security organization, ISRM knew that it was necessary to find ways to integrate our programs and the scan results into this platform to help enable the engineering teams. With the support of the Microsoft Engineering group behind VSTS, we worked together with the Fortify team to design and build a VSTS extension that allows our engineering teams to remain in compliance with our internal security policies without slowing them down. This integration of Fortify within Microsoft Visual Studio Team Services highlights one of the most beneficial achievements to come out of our relationship with the Fortify team.
The goal with VSTS integration was not merely the basic use of Fortify in VSTS like you would use any source repository system. This was about building a solution that allows minimal input from the user and that would enable the teams to use Fortify with less friction in a deployment phase. We wanted to make the Fortify scan feel more like doing a hosted build in VSTS.
Another significant aspect is the ability to enable Fortify scanning through a virtual machine, which is preconfigured and ready for teams to use directly in Microsoft Azure. Teams can pick the Fortify build machine out of the Azure store and it will have Fortify ready to go. Scanning machines will spin up on demand instead of requiring dedicated machines that are only used periodically for scans. This on demand service improves the efficiency of both the engineering and security teams.
Figure Fortify SCA on VSTS
5.3.7Dynamic analysis
In ISRM we use HPE Fortify WebInspect to conduct dynamic web application security testing. It complements static code analysis performed through Fortify SCA, and both use a common Fortify SSC server for management and reporting.
Our approach for deploying dynamic security analysis was pragmatic. We rolled out WebInspect initially as an SDL requirement to help teams discover issues during application testing. We were very aware of the tool fatigue issue our engineers raised, so we focused on static security analysis first and then moved to dynamic analysis once we achieved the sufficient effective adoption of static analysis.
Figure WebInspect
In the first year, we encouraged all applications (with web interfaces) to run WebInspect by self-scanning and uploading the results for compliance. Multiple application teams used this model, and some lessons we learned included the following:
-
Low quality scans The quality of scans submitted to Fortify SSC were low. Either they were incomplete scans or misconfigured scans. We realized that engineers must be trained in the tool to run effective scans. For example, running a basic crawl without configuring macros can yield incomplete or duplicative scans of the same resource.
-
Resources for scans Applications teams complained that there was a lack of dedicated servers from which to conduct self-scans, particularly for larger applications and services.
These lessons paved the way to make changes in our dynamic security scanning strategy. We decided to onboard higher risk applications to a dynamic security scanning service. This service uses a security team to engage with the engineering team once and provide that team with automated, scheduled macro-based scans. Because the scans are led and configured by a security team, we are better assured of the quality of the scans and their results. We are also evaluating use of APIs to further streamline scans scheduling and kick off process.
5.3.8WebInspect deployment architecture
The diagram below describes the two main implementation methodologies for WebInspect at ISRM.
-
The right side of the diagram illustrates continuous dynamic scanning for high-risk applications. These scans are run as periodic scheduled scans on the WebInspect Enterprise sensors controlled from the WebInspect Enterprise server.
-
The left side illustrates dynamic scanning for all other applications. These scans are run by engineering teams on their own infrastructure and are uploaded to the WebInspect Enterprise server.
Figure WebInspect Deployment Architecture
5.3.9Runtime detection and protection
Currently there are two technology solution types to tackle application runtime security detection and protection challenges. These are Web Application Firewall (WAF) and Runtime Application Self-Protection (RASP). ISRM looked into both these technology solution types to address the runtime detection and prevention challenge. We believe RASP is the more promising of the two, although RASP technology is evolving and more work is needed before it becomes the mature solution we need.
5.3.9.1WAF
A WAF works by sitting in front of an organization’s web application stack. It analyzes all incoming traffic for attack patterns and blocks any suspected malicious requests from reaching the web application itself.
However, typical WAFs suffer from a number of limitations, including – but not limited to – the following:
-
Utilizes signature-based pattern matching and blacklist filtering, which can be bypassed as attacks become more sophisticated.
-
Lacks insight into application logic, data, and event flows because it does not have application context.
-
Has high false positive rate from list of detected malicious activity.
-
Has relatively limited vulnerability category coverage.
-
Requires extensive coverage and maintenance for firewall rules to stay in sync with application changes.
Even with these limitations, WAFs can provide a level of protection that justify their cost (including operational maintenance) and can be an effective control in many scenarios.
5.3.9.2RASP
RASP is a fairly new security technology. It is hooked into an application or application runtime environment and can be capable of controlling application execution to prevent real-time attacks as well as detection.
We have been evaluating a RASP technology from HPE Fortify called Application Defender to provide runtime detection and protection. Our evaluation for RASP has been focused around the following key features:
-
Enterprise management How does the technology scale for an enterprise deployment?
-
Performance and scalability How does the technology impact the application performance?
-
Agent installation and maintenance How easy is it to install and manage agent updates as new rules for attacks roll out? Does it require server restart?
-
Platform support Does the technology works seamlessly for on premise applications and cloud applications, particularly Platform as a Service (PaaS)?
-
Ruleset quality What is the current ruleset coverage when mapped to our TCPs, and what is the update frequency of the ruleset?
It’s clear that the deployment and evaluation of any tool that sits in a production environment needs to be carefully vetted from all aspects listed above. We have been working closely with the Fortify team to vet solution details and to work on bridging the gaps to make sure this technology is ready for our needs.
The following are two key areas we see that require enhancements:
Applications running in Infrastructure as a Service (IaaS) and PaaS (Web Roles) environments can be supported, but the deployment automation is yet to be determined.
App Defender also has challenges with uniquely identifying the Azure PaaS hosts which prohibits effective monitoring.
An update to the agent installed on the host application can require IIS and application server restart, depending on the application type. This can be a significant operational burden for engineering teams.
We plan to test the on-premise solution for App Defender. We are testing it against different scenarios and applications to identify what it would take to deploy this technology in the least disruptive way for the engineering teams. We intend to deploy an end-to-end scenario where App Defender is configured in “Detection Mode Only” at first, and then integrate the feed into a security information and event management (SIEM) technology such as HPE Security ArcSight. Our goal is to directly enable our engineering teams with the detection capabilities that can be incorporated into their telemetry systems to not only gauge the operational state of their application but also the security posture.
A solution like App Defender can provide very advanced application monitoring capability. It is a key component of enabling continuous assurance, and also helps share the responsibility of security with the engineering teams which enables our modern engineers to gain the appropriate security insight into their applications.
5.3.10 Automation factory
Having the ability to license third-party enterprise tools like Fortify SCA and WebInspect help security teams like ours focus on operational aspects of the service while ignoring the costs associated with engineering and support. But this isn’t to say that internally developed custom built tools or scripts aren’t useful. Much like the rest of the industry, Microsoft IT is moving its workloads and application portfolio to Azure. Azure has enabled DevOps to really flourish by effectively democratizing the infrastructure. Engineers can now control virtual networks and host configuration through code changes. This not only opens up tremendous opportunities for engineering teams but, if leveraged correctly, it enables security teams to drive more effective continuous assurance.
To address this shift, we’ve made it a priority to develop what we call the “Secure DevOps Kit for Azure” which contains a set of tools, extensions, code snippets, and other automations.
Most specifically, the Secure DevOps Kit for Azure contains the following components:
-
A package of scripts and programs that ensure more secure provisioning, configuration, and administration of an Azure subscription.
-
A set of extensions, plug-ins, templates, and PowerShell modules that empower a developer to create, check-in, and deploy code with greater security from the beginning.
-
A tool that can capture snapshots of “secure” state for a subscription, for the application resources, watch for drift, and that can enable operational security compliance.
-
A package that would grant the visibility to enterprise IT teams about the shape and form of all of the above along with extensions that can provide application layer events and alerts for individual service line teams.
Figure Secure DevOps Kit
In the spirit of agile development, this kit is a work in progress. We are hoping to refine many of the ideas as we iterate and enhance it over sprints with regular feedback from our engineering community.
Share with your friends: |