devopsy.kinsta.cloud

Author: David Geer

Trusting Continuous Delivery
DevOps, continuous delivery and the pipeline for faster software development are fundamentally different from traditional software development. It’s such a different approach and a different culture that it can take some time before Dev and Ops teams embrace and trust it—a necessary step for DevOps to make progress in any organization. By mixing containers and their value proposition into DevOps, this new software creation and deployment paradigm can readily foster that trust.

What Continuous Delivery Brings

DevOps and continuous delivery offer application development using containers, providing the required level of trust to get over that hump and move things forward. According to David N. Blank-Edelman, a DevOps veteran and currently technical evangelist at Apcera, the following container features lead to trust because they answer Dev and Ops concerns:
- Containers are resource-friendly replacements for virtual machines, built to encase the coder’s development environment
- Containers are the complete package format replacement, possessing everything you need to run an app
- Containers create an abstraction that enables development and its results to sit in a host in any of a number of infrastructures including on-premise and inside any of several public cloud providers
- Containers are the fastest means to launch short-lived, purpose-built testing sandboxes as part of a continuous integration process
- Containers are the end result of a build pipeline (artifact-to-workload)
- Containers form the boundary for loosely coupled components as part of a larger app or service. Such is the case, for example, when each part of a microservice-based app lives in its own container
According to Blank-Edelman, the aforementioned features imbue trust because they satisfy the following Dev and Ops queries:
- Is the container allocated to use only the resources it should?
- What resides in that container / workload?
- Where did that container come from? Is it OK from a security, coding and business requirements perspective?
- Where is the container permitted to live and run such as in what cloud, environment, country, and in proximity to other containers technically?
- Can the container communicate with only the things it should such as other containers and any hypervisor or other management resource? “If I move the container to another location, do the answers to these same questions still apply?” asks Blank-Edelman.
Though these questions express concerns in direct and simple language, experienced DevOps engineers who have felt the frustration of building development processes and attendant infrastructure understands these are difficult questions to satisfy. Without modern containerization efforts to sustain DevOps, development and production goals grow further out of reach.

The Importance of Policies

As containers undergird DevOps and continuous delivery, policies support containers. Developers input changes and innovations in code to the continuous delivery pipeline. Developers then respond to the needs of the growing code in a nurturing fashion, fixing errors and enabling new features. This pipeline works when automation, together with each progressive step in the development process, lays a foundation for the next step in a predictable and reliable manner.

“Here’s where it gets tricky: First, not only do you have to be able to answer the trust-related questions for each step, but it is entirely possible (and entirely likely in an enterprise environment) that different people/teams will share control of and input to each step on that path,” says Blank-Edelman. With so much collaborative effort in play juggling all these pieces, policies are necessary to configure these steps, address networking and other elements, and control the component parts of the process ahead of time and everywhere along the way.

Policies that make particularly good container watchdogs include those that set steps in the DevOps/continuous delivery pipeline apart, hemming them in on all sides. “Just like having your papers in order makes for an easier border crossing when you travel abroad, so, too, does having your policies reflect the trust required to move from one step to another,” says Blank-Edelman.
March 28, 2016
How Continuous Delivery Challenges Security

Security and continuous delivery each have implications in their effects on each other. While continuous delivery can demand new security measures, traditional security can slow continuous delivery or simply not address its security needs at all.

Micro Case Study Clarifies Security Conflict

Amrit Williams, now CTO of CloudPassage, shares an experience that exemplifies the challenges continuous delivery poses for security. Williams once worked with one of the largest national HMOs in the United States, which developed and supported a customer healthcare portal to enable customers to access health data and take advantage of other HMO services.

At the time of his service there, the company sought to enhance its portal’s performance to challenge other HMOs in the marketplace that were popular, in part, due to the strength of their portal.

“They wanted to add a bunch of new features and make continual improvements by way of moving from a development model of new releases every six-month period to new site releases and updates every couple of hours or about four times per day,” says Williams. But every time the Ops team would bring up new containers and compute resources to deploy new technologies and application code, success would count on repeated manual changes to firewall rules to appropriately pass data, enabling applications to talk to the portal and its technologies.

“They finally decided to allow the firewall rules to apply to the application at the server level rather than at the underlying network level so that they didn’t have to go changing those rules all the time,” says Williams.

In DevOps, Security Still in First Gear

“Traditionally in the enterprise, security has been about speed only when the company is potentially dealing with a compromise, a policy violation or some other threat to the business,” says Williams. The organization is not used to having to respond quickly with security simply to deliver code. Common security tools were not designed to shift and adapt to automated code development and deployment.

“For example, a lot of times the security teams will rely on endpoint security tools that typically require static IP addressing, which is antithetical to dynamic compute models,” explains Williams. These mammoth resource-hogging security applications ask too much and secure too little if they can even work effectively in DevOps shops at all. When you must stop to adjust these security tools, you may trigger change management processes that can slow the development process even further.

Why Not Fail Fast?

While DevOps makes it easier to isolate and fix coding errors quickly, this doesn’t appear to spill over to security. Fixing errors to make the code work doesn’t mean the code is more secure. These changes could even introduce new security flaws that were not there before.

Securing continuous delivery is about the security team surrounding dynamic processes with protections that keep up with the speed of the continuous delivery pipeline. This can be difficult when these processes are automated, bringing up resources on demand to create and deploy software.

“It is really the deployment and configuration of these environments that can lead to some of the biggest security concerns and not necessarily the code itself,” he says. “It is that aspect of DevOps that security is usually not very familiar with, which can really result in them being left out of the process because they try to apply a rule or a government policy that is again, antithetical to that process.”

Continuous Delivery Security Incidents Challenge Legacy Security

In this case example, Williams shares a particular type of threat that legacy security tools would not prevent in a continuous delivery environment.

“It was 2012, during Christmas and it happened to customers using a region of Amazon‘s public cloud services. Amazon rebooted a bunch of servers, and some of those servers were running local host-based firewall management tools that were reliant on connectivity back to the management consoles as well as a static IP address,” says Williams.

When those systems came back online, neither the agents on the hosts or the tools functioned properly. Amazon and its customers had to work to ensure the agent software on the hosts worked and communicated with the firewall management tools and that customer systems were secure. This was important because of the number of sales Amazon customers had to process securely at the time, which was just a few days before Christmas Eve.

Again, this is an example of the failure of legacy security tools inside an environment that was increasingly adding automation while approaching continuous delivery. The lesson here is that as DevOps and continuous delivery approaches are altogether unique from legacy development, so the security tools that protect these environments need to be new, unique and suited to continuous delivery by design, lest they fail to survive partial outages such the Amazon example.

March 24, 2016
Case Study: Sauce Labs Covers Animoto’s Testing Needs

Animoto’s web and mobile platforms enable businesses and consumers to drag and drop stock and user-defined visual and audio elements to create videos via a tiered monthly subscription service. Animoto updates service features with increasing frequency—some of them several times a day.

“We have a broad range of applications and features including an online video maker application, where we enable users to import videos and video elements from all the known, established sources such as Facebook and through Dropbox,” says Siddharth (“Sid”) Jain, VP of Engineering at Animoto. “They can then export videos out to a variety of venues such as YouTube.”

Animoto must iterate new innovations continuously into its application/platform for high-quality yet simple video creation to compete with similar solutions in the marketplace such as WeVideo and Stupeflix. This fast-paced development process is challenging, since Animoto must update separate versions of its software for the web, Android OS and Apple iOS (both iPhone and iPad). “The mobile apps are largely consumer-focused for very easy, on-the-go, frictionless video creation and sharing, while our web apps feature server audiences of professional photographers, educators and small businesses,” says Jain.

In addition to testing service capabilities of different versions of the video maker application, Animoto must test the on-boarding/registration flow and the free product experience, where users get their first taste of Animoto. “We do lots of interesting things such as testing support for multiple currencies around the globe and running continuous multiple AB tests to ensure that the free product experience is working,” Jain notes.

Notable Animoto testing includes user funnel testing in a simulated user flow environment starting at sign-up and continuing through video creation, feature exploration and the upgrade experience. “We test this several times a day on the different platforms,” Jain says, noting Animoto tests for the seamless success of import and export of videos and video elements to and from Facebook, Dropbox and YouTube. “The emphasis is on ensuring the quality and consistency of the user experience, i.e., making sure that nothing goes out that breaks revenue in terms of service.”

Porting to Sauce Labs Promotes Progress

Sauce Labs‘ cloud service helps Animoto scale up and out instantly to test Apple/iOS versions of its application so it can meet the high standards set by Apple and ensure fast inclusion of new app versions in the Apple App Store. This helps Animoto to compete by getting new innovations and features to market more quickly. “The biggest advantage to using Sauce Labs is that we don’t have to create or maintain a test lab internally,” says Jain.

This accelerated testing speed is especially important since the app must adapt to every new iOS version while maintaining backward compatibility. “It would be a very nightmarish thing to do this all manually,” he says.

In addition to faster testing, Sauce Labs is helping Animoto ensure that testing was delivering on its promises of software improvement through accurate feedback. “We used to be religious about doing extensive unit tests. Now we emphasize integration tests over unit tests because we got bit by unit tests that were all green even though the software functionality was broken,” says Jain. “Now we have a balance between unit and integration tests while frontloading the latter with tests of customer-facing functionality and collecting fine-grain feature use data.”

The comprehensive testing helps to ensure that users aren’t shying away from certain features and that features are not broken. Frontloading uncovers regressions very early so the Animoto developers dedicated to fixing them can catch them before they lead to breakage.

Sauce Labs helps Animoto shrink technical debt. “Debt rears its head in awkward ways. We do get surprised, but having good test coverage allows us to know where the app hot spots are and to have feedback to fix these and verify changes swiftly,” says Jain.

Results: More Stability, More Capacity

“We have been able to reduce testing time from three hours to under 30 minutes for some tests,” says Jain. “Thanks to Sauce Labs’ cloud, we added VM capacity, moving from 20 VMs to 45 VMs for the testing environment.”

March 10, 2016
Software Complexity Driving DevOps Automation

The drivers behind increasing software complexity also are the drivers behind expansive DevOps automation. As the saying goes, “Software is eating the world.” DevOps is gobbling up software, too, engulfing it in practices that are necessary to make it thrive under the pressure of the current environment.

That environment and those drivers are of keen interest to many. DevOps authority Andy Vitus, partner at Scale Venture Partners, has identified four drivers in particular that are creating the software complexity pushing DevOps automation.

Multiple Distributed Software Locations

According to Vitus, by moving software from one server to multiserver to multiple data center deployments, the industry has made software more elaborate. Applications now execute in a distributed manner and in parallel using multithreading inside multiple processor cores; among many servers running multicore processors; and in and between geographically dispersed data centers. This makes software more powerful and reliable, though it demands more DevOps automation to produce it.

Simply incorporating the ability for software to run in parallel using multithreading adds intricacies to the development process. To keep both the workload and the number of man hours needed during development manageable, automate as much of the additional coding duties and testing as possible.

“Running applications across servers and data centers requires that you ensure consistency in the data layer,” which will then need new databases to handle the load, says Vitus. Depending on the database, the developer must rewrite the software to be able to use it.

The entanglements don’t end there.

With mobile applications, mobile devices and back-end servers both store data. “You have to be a lot more careful about knowing what version of the data is the true one, unless you’ve got a synchronization layer built in,” says Vitus. Either way, programming annoyances can be introduced.

Getting Lost in Translation

Expanding development from a single programming language to multiple languages, even for the same project, has made software more convoluted. In legacy development shops, programmers typically used one language for everything regardless of whether it was the best language for their creations. These days teams are applying choice languages such as C++, Python and Go to the tasks to which they are best-suited. “This increases the complexity of security and patch management as you use six, seven or eight different languages all with their own vulnerabilities,” says Vitus.

While adopting several languages certainly adds to patch management and security headaches, understanding how each language can and must be compatible with the others for them to work together is an entirely new challenge. “The ways that they handle variables and time stamps changes over time, and you have to ensure that different services can still talk to each other,” says Vitus.

Stacks of Stacks

Moving from a simple three-tier stack to a distributed microservices architecture catches software development in another snag, Vitus says. The database layer has grown to include in-memory and other database types. As development breaks the application layer into microservices, each potentially with its own data store, the two layers combine to create this new stack.

Millions of Endpoints

Finally, moving from deploying servers to deploying servers, mobile and Internet of Things (IoT) takes software down new paths of mostly uncharted territory, says Vitus. “Today you deploy software changes to a back-end server that must talk to a mobile client using iOS or Android or to IoT devices such as dishwashers and light bulbs in smart homes,” he says. “You could be deploying code to millions of devices.” This certainly affects the construction of the code for use on so many different platforms.

Conclusion

These drivers help create a magnified need for DevOps automation. Considering the costs of hiring engineers and technicians to do the same development work manually, the world will be watching companies that are pushing for increased automation to displace those labor costs. “To the extent that you can offload some of that to regression tests, you end up saving, and the cost of building a new software product goes down,” says Vitus.

March 7, 2016
How DevOps Prevents DDoS Attacks

Distributed denial of service (DDoS) attacks are some of the most pervasive and difficult attacks to prevent. The attack uses many distributed endpoints and/or systems to flood a web domain, application or service with excessive service requests or application calls. There are limits to the amount of bandwidth or the number of service requests or application calls that the associated web-based resource can handle at any given time, and when attackers using these requests or calls overwhelm these Internet resources, service is denied to genuine users who need legitimate access.

“These attacks are designed to either melt down your network or squash your applications and servers. Industry experts identify them as L3/L4 attacks and L7 attacks accordingly,” says Stephen Gates, principal SE and senior technical expert at NSFOCUS IB, a global provider of network security and advanced analytics. L3/L4 and L7 refer to three of the seven layers in the Open Systems Interconnect (OSI) reference model: L3/L4 refers to the network and transport layers, respectively, while L7 refers to the application layer.

These DDoS attacks are a big concern because they prevent users from accessing services and websites to conduct their business, which impacts user experience, employee productivity and a company’s bottom line. “The largest L3/L4 DDoS attacks on record exceeded 400Gbps in size,” says Gates. “Organizations all over the world regularly feel the effects of L7 attacks,” amounting to a lot of service disruption for a lot of businesses.

Handling DDoS Attacks with Rugged DevOps

Although Rugged DevOps can’t defend against L3/L4 attacks, it could help mitigate L7 attacks, Gates notes. Since L7 attacks are so common, a Rugged DevOps approach that fixes many—if not most—of them is an important tool can be an enterprise’s arsenal against DDoS. For it to work, Rugged DevOps must address the several types of vulnerabilities that appear in development of applications that make successful L7 DDoS attacks possible.

Most of the vulnerabilities susceptible to L7 DDoS attacks unfortunately permit unwanted user behavior in or in conjunction with applications that are accessible via the Internet. “For example, if a client machine continues to do the same thing over and over again (repetitive HTTP GET), this would be indicative of an unusual client behavior,” explains Gates. An HTTP GET is a request to the HTTP (web) server for data. Applications must apply methods for disregarding the several forms of unwanted behaviors.

Attackers can also send DDoS attacks against the application layer by using malformed packets to leverage weaknesses in ill-coded software. Packets are a means for encapsulating data into packages or units for transfer across packet-switched networks. “For example, if an attacker launched the ApacheKiller script against a vulnerable version of Apache, the application would completely fail to operate,” illustrates Gates. Applications frequently display these and other vulnerabilities to specially crafted attack packets.

How Rugged DevOps Can Help

Developers could avoid many of these kinds of application coding errors and vulnerabilities early in development by applying Rugged DevOps, which can catch and fix security holes early in the software life cycle. “Then they will likely not propagate into later versions of the application or the plugins riding on top of these applications,” says Gates.

Running penetration tests on software early in the development process is one way to thwart holes that enable L7 DDoS attacks. When coding missteps are uncovered early in the process, developers can fix them early, before they become part of the software. Tools such as Gauntlt, Mittn and BDD Security are examples of software testing tools that developers can use inside the DevOps shop.

Failed tests require a response. One such response is to automatically fail to build the software when the software fails the test(s). If development can’t move forward without fixing the security holes, the security holes will be fixed.

There are well-worn methods that developers can rely on to fix the most common, most threatening application security vulnerabilities. Developers should not have to do a lot of digging to uncover these methods. Many organizations and resources such as the Open Web Application Security Project (OWASP) clearly set these approaches apart and label each of them on their own distinct Web pages. This SQL Injection Prevention Cheat Sheet is only one example.

DDoS and other forms of attacks are, unfortunately, a part of life. Being able to mitigate an attack before it can occur is a goal. Rugged DevOps is one way to help make that goal a reality.

March 1, 2016
How To Maintain Open Source Code Hygiene
To maintain good open source code hygiene in the development process, you must first establish what you mean by open source code hygiene, including good and bad hygiene. “By hygiene, we just mean is it appropriate? Is it clean for the intended use?” says Mike Pittenger, VP of Strategy at Black Duck Software.

Black Duck suggests three metrics it uses to determine whether it is appropriate to apply a given open source code project to a certain development endeavor:
- For software that will be distributed at large, avoid open source code that the development community has made available through a GPL license.
- For secure software, check your potential open source code for vulnerabilities in general and make sure to use the most secure version available.
- Monitor the code periodically to check for new vulnerabilities or other concerns.
As an example of the third point, Pittenger notes OpenSSL is no longer a viable option because of the vulnerabilities that have come to light in the form of Heartbleed.

Checking Source Code for Open Source Concerns

While the InfoSec Institute recommends at least 14 different open source vulnerability scanners for web applications alone, offerings such as Black Duck’s can serve as a shortcut to discovery of known bad open source components. “Instead of having to load Nessus, Metasploit or Nexpose with a Heartbleed rule and scanning every application, every server that they’ve got, they can just go into their Black Duck console, type in OpenSSL, and it would immediately show them every version they used in their software applications,” says Pittenger. Of course, you should always do your own due diligence to determine whether any scanning product is the best choice for your organization and aligns with your needs and requirements.

Due to the many open source code reuse scenarios active in enterprise DevOps shops, you will want to continually monitor code for newly registered vulnerabilities. Enterprises can view reusing existing assets as a built-in savings by not having to purchase or create new assets. In the case of reusing legacy source code, there is also a time savings, as developers want to shrink time to market for code-based projects.

“Let’s say you have a company that is pulling information from a database to display in pie charts. You have a rendering component that you will reuse. When you’re starting on a new product, do you build a new rendering library from scratch or do you use the existing one?” asks Pittenger.

Even if those components are open source, there can be great savings in reusing the library.

The fact that enterprises do this underlines the importance of continuing to check any code that leverages open source code, as this is how vulnerabilities can spread over time. It’s also important to check open source code, because it may be a form of low-hanging fruit for attackers. Open source code is fully viewable by attackers who build malware and launch attacks, whereas proprietary code is closed.

Since people constantly discover new vulnerabilities in open source code components that previously had tested free of such holes, it’s important to monitor those components for newly uncovered weaknesses, even as you reuse open software in new projects.

Containers are a great example. You may want to dismantle an image in varying degrees to build a new application. “If I have a base image and it’s built using a LAMP stack (i.e., a Linux OS, an Apache HTTP server, a MySQL RDBMS and PHP), I can strip down my base image, take away the existing app and take it all the way down to the Linux component, then build it up for the new application,” Pittenger said. “That is a typical, consistent reuse of open source components.”

During the process, recheck every component for vulnerabilities. Even if everything checks out doesn’t mean you have the best assurance of security for those components going forward. By monitoring those components, you can stay on top of updates to fix any newly discovered security holes.
February 24, 2016
DevOps Q&A: Datalink’s Dennis Vickers on MobileApps

Companies are custom-building mobile enterprise applications to enhance employee productivity on smartphones. devopsy.kinsta.cloud recently spoke with Dennis Vickers, director of Software Solutions at data center services and solutions provider Datalink about this trend and where DevOps fits in to the process.

devopsy.kinsta.cloud: Why does applying smartphones to business initiatives require so much additional custom-built software?

Vickers: Companies have developed and designed legacy software to group things that need to get done together into applications. With mobile application development the focus is more granular in terms of functionality. It’s not that the applications that we designed for the legacy environment don’t fit on the smaller screen, though they don’t. It’s really that the whole concept of having a group of functionalities in a single application doesn’t really work in a mobile environment. You need to rethink the way that not only your applications work but really the way that you do work. You need to think more in terms of granular work than accumulation of work.

In finance, for example, you might have an application that combines accounts receivable, accounts payable, even expense management. You would have an application that would have all of that functionality in it. For mobile apps, what would make sense is to break out the expense management part of that and make that a mobile app that allows users to be able to use their smartphones to actually capture expenses as they’re incurred, as opposed to having to save a receipt or remember what your expenses were and then go back to an application and fill that in later at the office.

devopsy.kinsta.cloud: How do you apply DevOps development approaches to provide usability on mobile devices for the most needed features and functions of in-house applications?

Vickers: Mobile applications tend to be single functions focused on getting something done. Users are much less tolerant of anomalies in the software, whether that anomaly is a bug, a poor user interface or anything that is undesirable about the application. They’re not nearly as willing to look for workarounds with a mobile app as they are with legacy applications.

You have to have a continuous delivery approach with mobile applications because the time frame between deliveries has to be accelerated. You have to automate that process. We’ve made good use of some of the automation tools, specifically Puppet, around doing continuous delivery. We use tools for source control and QA (quality assurance). Some type of automated delivery process is critical because the turn time for release has to be shortened. There’s just not a willingness for users to wait long periods between releases.

devopsy.kinsta.cloud: Can you discuss the custom-built software project where you developed a custom equipment maintenance app that feeds into the customer’s proprietary maintenance application?

Vickers: We developed that for an aerospace company that had a requirement to provide field service for one of the Department of Defense systems. They had to be able to maintain large pieces of equipment. We had to build an application that enabled them to be at the equipment and do the data collection that they needed at that time. It walked them through the data collection process right then.

Before that, they had a paper process where they would print out instructions, go out and write down the information that they needed to collect, and then go back and enter that into an application. This allowed them to be able to do that closer to the actual event, which cut transcription errors and data collection errors and made the whole process much more efficient. That was an extension of a legacy app that added quite a bit of value.

devopsy.kinsta.cloud: For that Department of Defense project, were there particular development tools or approaches in DevOps for the software?

Vickers: We used an agile methodology where we iteratively released the software and did test and review and then essentially layered the application. We provide a layering process as we build and add features to the application. We delivered to the customer through an automated process using Chef. We use Jenkins to do some release management, but we often revert back to Chef or Puppet.

What we haven’t done a lot of yet, which I think is important and I see growing in the mobile platform, is monitoring. I think that monitoring is an important part of DevOps. I haven’t seen that move into the mobile environment nearly as much as I think it needs to. I think it will happen in the short term. I think there’s some good monitoring tools out there that are coming online or that are maturing and are going to be adopted for mobile app dev.

devopsy.kinsta.cloud: There was another project where you custom-built a jobs app that enables union members to apply for posted jobs without going into the union hall.

Vickers: We developed a mobile app so union members can review jobs that are available and request the job through the app. They can receive job assignments and electronic tickets to go out and do that job. It saves traveling down to the union hall on a regular basis and makes the whole process much more efficient.

The only difference in that project is that we needed to integrate with the back end through an API. The tool we used to help us interface with that API and manage that API integration was Postman. We use it for testing, managing and documenting APIs we build or work with.

February 16, 2016
Case Study: Pearson Weaves ThreadFix into AppSec, Part 2
In part one, we discussed how education content publisher Pearson applied ThreadFix to its AppSec request workflow needs. Here’s the rest of the story.

The ThreadFix Implementation

Senior software security engineer Matt Tesauro added ThreadFix to Pearson’s workflow in the following steps:
1. Using FPM, Tesauro created a Debian Linux package internally to deploy the ThreadFix Tomcat Package (ThreadFix also offers its own Debian package for this).
2. Tesauro hooked ThreadFix into each of Pearson’s toolsets using the tool scanner file formats ThreadFix understands.
3. Using the importers built into ThreadFix, Tesauro connected it to the Checkmarx, Veracode, BurpSuite, ZAP, WhiteHat, AppScan and Qualys testing tools Pearson uses.
4. To make that happen, Tesauro pointed ThreadFix to an API.
5. Tesauro then set up an API key.
6. The system was then set to fetch the test results.
7. If necessary, Tesauro wrote glue code to upload the results to ThreadFix.
Pearson used ThreadFix to normalize and de-dupe what had proceeded from manual assessments, dynamic scans on applications and static code analyses of source code, directing those outputs into ThreadFix. “So if a static tool and a dynamic tool both found the same cross-site scripting issue, we fed those into ThreadFix, which de-duped those and gave us just one issue to report so that we didn’t bury two teams with fixing the same single coding error,” says Tesauro.

“ThreadFix became the one source of knowledge for all of the output of our team, which for us is the security vulnerabilities we find in the applications that we either created, bought or had a third party create,” says Tesauro. That enabled Pearson’s AppSec team to feed tickets into JIRA of backlogs from development teams, which made it easier to produce metrics and do reporting. ThreadFix made it easier to feed data into governance, risk, and compliance tools such as RSA Archer. “That enables visibility into work in progress and workflow and into areas of the business that affect AppSec’s work,” Tesauro notes.

Additional Capabilities with ThreadFix

Tesauro eventually expanded ThreadFix to other departments beyond AppSec to benefit from its capabilities and the metrics it could produce regarding app progress.

Tesauro notes a point between the “Bag of Holding” and ThreadFix in the AppSec pipeline that required some work. He used a StackStorm orchestration and layered ChatOps on top to enable new developer capabilities at that point in the process.

“Now you can go into our private channel and use HipChat to ask the system, for example, to set up Checkmarx for a particular application in a particular business unit. You give it the URL pointer to that unit’s Git repository and off the system goes, setting up the application profile in the Bag of Holding and in ThreadFix and then talking to Checkmarx to set up weekly scans of the code that sits on the master branch. It set that all up in a minute,” says Tesauro.

HigherEd was the first additional group inside Pearson to benefit from ThreadFix metrics. One such metric was a 44 percent decrease in time-to-fix after adding a JIRA integration into the mix, according to Tesauro. “Once we started producing those numbers, we had a lot of other business units ask, ‘Hey, why does HigherEd get these great time-to-fix numbers? We want them too for our apps.’ It was a great way to get buy-in,” says Tesauro.

Further Results

AppSec saw a 5x increase in throughput for the request workflow and went from 44 application assessments in 2014 to fewer than 200 in 2015, Tesauro notes.

“In 2015, we lost two and a half people and still sped up the process,” he says.
February 15, 2016
Case Study: Pearson Weaves ThreadFix into AppSec, Part 1

Pearson is a publisher of education industry content to meet the needs of teachers and students from kindergarten/early learning through higher education and continuing education (for professionals). The company uses a mix of software, from legacy third-party software and “classic ASP apps that are on life support to auto-scaling systems on Amazon,” says Matt Tesauro, senior software security engineer at Pearson. “It’s a very diverse development environment.” Which meant it was no easy task keeping it secure.

Pearson Before DevOps

Prior to adopting DevOps, Pearson was bogged down by manual processes, especially in application security. “People sent requests buried in email or, if you were lucky, in a Google spreadsheet. There were very few clear lines and hand-offs between the different activities that were the responsibility of the AppSec group,” explains Tesauro. It was a very manual, custom approach in terms of request handling: someone would respond to a request with an activity, which would require a report that was then emailed.

At the time, Pearson had no single, well-established methodology or workflow for these activities. There was no particular look and feel or design for creating dynamic, manual or static assessments. There was no list of inputs into the process or outputs to expect at the end of the process. “At best, everything flowed in an ad hoc fashion,” says Tesauro.

Then, about two years ago, Tesauro and a senior-level peer at Pearson attempted to prioritize those apps the company most needed to assess for security and related issues. They had hoped and planned it would be about 10 apps, to keep the number and the workload manageable. For political reasons, they were able to settle on no fewer than 64 apps that needed appraisal. “We ended up being able to assess only 44 out of the 64 top apps. There are 2,000 and some-odd apps at Pearson, in rough numbers. Just the scale told us we were going to fail in the long run,” says Tesauro.

Ramping Up to Change

To resolve this app assessment challenge, Tesauro looked to institute DevOps and Build Pipeline approaches to lay the groundwork for eventually integrating security into the various development teams at Pearson. “I made equivalents of DevOps and Build Pipeline approaches for how we did work in the AppSec program at Pearson,” says Tesauro. These included standard methods of input and intake of service requests similar to how the business produces software development targets for a DevOps shop to hit. That’s when Pearson began to work on instituting and leveraging ThreadFix.

“ThreadFix was one of the first two parts of that pipeline,” he says. The other part of the AppSec pipeline solution was what Tesauro calls the “Bag of Holding,” a mechanism and repository for tracking all the team-to-team interactions and work assignments. That feeds into ThreadFix.

During that time, and for about the first half of 2015, Tesauro and a teammate automated a number of manual steps by creating code to perform the “mindless work” so developers could concentrate on their core business instead. “Now when the static code analysis tool Checkmarx finishes its analysis, it generates a report, washes the directory and pushes the report into ThreadFix,” says Tesauro. He also automated a number of other test applications and tools.

Stay tuned for part two of this case study, coming soon.

February 12, 2016
QA in the DevOps Era

In software approaches that produce monolithic applications, it is not uncommon to find siloed teams, with development, testing and operations each an island unto themselves. The responsibilities of test and development are distinct from each other and are not meshed into a hybrid team as you might see in advanced DevOps shops.

“QA (quality assurance) always comes late in the game to verify or test the software developed by the development team against the predefined specifications know as software requirements,” says Venkata Vinay Pandella, team lead at Nisum Technologies Inc.

That’s the view inside and among these teams in which legacy development approaches still flourish. The rest of the enterprise sees these teams altogether differently from how they see themselves. “Even in a traditional software engineering organization, folks outside the engineering team generally consider Development, QA and Operations teams as a single group with a single responsibility to deliver a quality software product,” says Pandella.

Even when Dev and Ops do embrace each other, the rest of the enterprise often maintains the same POV it had when Dev, Ops and QA were three entities and not one. How do DevOps and QA see QA after the shift?

DevOps as a Foundation for the New QA

Enterprises, the business units, Dev and Ops all expect DevOps to propel software using continuous integration (CI), continuous delivery (CD) and ultimately continuous deployment techniques that are fueled by and gain traction using automation. The more processes that Dev and Ops teams are able to automate, the more that DevOps accelerates the process of software delivery and deployment to the delight of software consumers and the business.

DevOps enables teams to push software out weekly, daily and hourly through continuous software improvement. This negates the many obvious downsides of applying the traditional software development approach, which requires excessive planning, cross-team coordination, staging, testing and releases of monolithic blobs of software all at once. “Given that companies organically adopt DevOps together with agile software development practices, business managers are able to fulfill the objective to get high-quality software out to market faster,” says Pandella.

To meet the goals of DevOps, QA must feed into the automated development process and help teams to achieve increasing software quality and faster time to market.

QA: The Quality Icing on the DevOps Cake

According to Pandella, QA specialists and teams have a number of roles and responsibilities in DevOps:

• QA should prevent software defects at the earliest point during the software development process rather than find and report on software bugs after DevOps has produced the software. This will help to ensure that customers experience as few defects as possible.

• QA takes ownership of and responsibility for continuous software improvement and software quality tracking throughout the software development life cycle.

• Every QA team member is responsible for identifying problems not just in the product but also in the process, and recommending changes wherever they can to improve the continuous delivery of quality software.

• Tests must be in code, based on the designed release cadence to publish a new release every week, every day or every hour. There is no room for manual testing—QA must develop automation systems through code that can ensure quality standards are maintained.

• Automation rules: QA should automate anything that can be automated.

• QA testers are the quality advocates, influencing both developmental and operational processes. They don’t just find bugs; they look for any opportunity to improve the quality of software through the software development process.

• QA must move beyond functional testing, because even a small, nonfunctional defect introduced into a critical transaction process on an e-commerce website can bring a business to its knees. Many organizations have mature processes for automating functional testing; they are only beginning to apply these practices to other areas of testing such as security and stress testing. In particular, load testing and stress testing are critical disciplines for DevOps organizations that move at high velocity.

February 9, 2016