Tag: case study

In part one, we discussed how education content publisher Pearson applied ThreadFix to its AppSec request workflow needs. Here’s the rest of the story.

The ThreadFix Implementation

Senior software security engineer Matt Tesauro added ThreadFix to Pearson’s workflow in the following steps:

1. Using FPM, Tesauro created a Debian Linux package internally to deploy the ThreadFix Tomcat Package (ThreadFix also offers its own Debian package for this).
2. Tesauro hooked ThreadFix into each of Pearson’s toolsets using the tool scanner file formats ThreadFix understands.
3. Using the importers built into ThreadFix, Tesauro connected it to the Checkmarx, Veracode, BurpSuite, ZAP, WhiteHat, AppScan and Qualys testing tools Pearson uses.
4. To make that happen, Tesauro pointed ThreadFix to an API.
5. Tesauro then set up an API key.
6. The system was then set to fetch the test results.
7. If necessary, Tesauro wrote glue code to upload the results to ThreadFix.

Pearson used ThreadFix to normalize and de-dupe what had proceeded from manual assessments, dynamic scans on applications and static code analyses of source code, directing those outputs into ThreadFix. “So if a static tool and a dynamic tool both found the same cross-site scripting issue, we fed those into ThreadFix, which de-duped those and gave us just one issue to report so that we didn’t bury two teams with fixing the same single coding error,” says Tesauro.

“ThreadFix became the one source of knowledge for all of the output of our team, which for us is the security vulnerabilities we find in the applications that we either created, bought or had a third party create,” says Tesauro. That enabled Pearson’s AppSec team to feed tickets into JIRA of backlogs from development teams, which made it easier to produce metrics and do reporting. ThreadFix made it easier to feed data into governance, risk, and compliance tools such as RSA Archer. “That enables visibility into work in progress and workflow and into areas of the business that affect AppSec’s work,” Tesauro notes.

Additional Capabilities with ThreadFix

Tesauro eventually expanded ThreadFix to other departments beyond AppSec to benefit from its capabilities and the metrics it could produce regarding app progress.

Tesauro notes a point between the “Bag of Holding” and ThreadFix in the AppSec pipeline that required some work. He used a StackStorm orchestration and layered ChatOps on top to enable new developer capabilities at that point in the process.

“Now you can go into our private channel and use HipChat to ask the system, for example, to set up Checkmarx for a particular application in a particular business unit. You give it the URL pointer to that unit’s Git repository and off the system goes, setting up the application profile in the Bag of Holding and in ThreadFix and then talking to Checkmarx to set up weekly scans of the code that sits on the master branch. It set that all up in a minute,” says Tesauro.

HigherEd was the first additional group inside Pearson to benefit from ThreadFix metrics. One such metric was a 44 percent decrease in time-to-fix after adding a JIRA integration into the mix, according to Tesauro. “Once we started producing those numbers, we had a lot of other business units ask, ‘Hey, why does HigherEd get these great time-to-fix numbers? We want them too for our apps.’ It was a great way to get buy-in,” says Tesauro.

Further Results

AppSec saw a 5x increase in throughput for the request workflow and went from 44 application assessments in 2014 to fewer than 200 in 2015, Tesauro notes.

“In 2015, we lost two and a half people and still sped up the process,” he says.

Pearson is a publisher of education industry content to meet the needs of teachers and students from kindergarten/early learning through higher education and continuing education (for professionals). The company uses a mix of software, from legacy third-party software and “classic ASP apps that are on life support to auto-scaling systems on Amazon,” says Matt Tesauro, senior software security engineer at Pearson. “It’s a very diverse development environment.” Which meant it was no easy task keeping it secure.

Pearson Before DevOps

Prior to adopting DevOps, Pearson was bogged down by manual processes, especially in application security. “People sent requests buried in email or, if you were lucky, in a Google spreadsheet. There were very few clear lines and hand-offs between the different activities that were the responsibility of the AppSec group,” explains Tesauro. It was a very manual, custom approach in terms of request handling: someone would respond to a request with an activity, which would require a report that was then emailed.

At the time, Pearson had no single, well-established methodology or workflow for these activities. There was no particular look and feel or design for creating dynamic, manual or static assessments. There was no list of inputs into the process or outputs to expect at the end of the process. “At best, everything flowed in an ad hoc fashion,” says Tesauro.

Then, about two years ago, Tesauro and a senior-level peer at Pearson attempted to prioritize those apps the company most needed to assess for security and related issues. They had hoped and planned it would be about 10 apps, to keep the number and the workload manageable. For political reasons, they were able to settle on no fewer than 64 apps that needed appraisal. “We ended up being able to assess only 44 out of the 64 top apps. There are 2,000 and some-odd apps at Pearson, in rough numbers. Just the scale told us we were going to fail in the long run,” says Tesauro.

Ramping Up to Change

To resolve this app assessment challenge, Tesauro looked to institute DevOps and Build Pipeline approaches to lay the groundwork for eventually integrating security into the various development teams at Pearson. “I made equivalents of DevOps and Build Pipeline approaches for how we did work in the AppSec program at Pearson,” says Tesauro. These included standard methods of input and intake of service requests similar to how the business produces software development targets for a DevOps shop to hit. That’s when Pearson began to work on instituting and leveraging ThreadFix.

“ThreadFix was one of the first two parts of that pipeline,” he says. The other part of the AppSec pipeline solution was what Tesauro calls the “Bag of Holding,” a mechanism and repository for tracking all the team-to-team interactions and work assignments. That feeds into ThreadFix.

During that time, and for about the first half of 2015, Tesauro and a teammate automated a number of manual steps by creating code to perform the “mindless work” so developers could concentrate on their core business instead. “Now when the static code analysis tool Checkmarx finishes its analysis, it generates a report, washes the directory and pushes the report into ThreadFix,” says Tesauro. He also automated a number of other test applications and tools.

Stay tuned for part two of this case study, coming soon.