If companies are to reach their strategic goals—reducing time to market, boosting sales, improving product market fit and brand image, and cutting cybersecurity costs—then it’s time for a new outlook on software security.
Today’s business leaders must learn to see security for what it is: A differentiating factor. Companies with reputations for secure developmental processes and infrastructure will rise above those known for data breaches.
Security as a Business Strategy
The road to a more secure company—and realizing the perks associated with it—begins with lean security. This is an approach to information security similar to the Toyota principles of management and production that calls for environmentally aware engineering, simplified coding, automation of security checks and constant incorporation of feedback.
Although some business leaders believe organizational security is inherently expensive, lean security doesn’t mean massive costs. Much like DevOps, lean security champions process improvements and cultural changes above purchasing new tools.
Lean security does, however, require engineers to keep its principles in mind throughout the development process. In this way, security is much like sales: A sales representatives’ best strategy is to consider his goals from the beginning. Just as the salesperson isn’t trying to make any sale, the engineer isn’t trying to create just any product. For the salesperson, this means tweaking the sales process—perhaps by vetting leads before engaging them—to make profitable sales. For the software engineer, it means structuring the development process through lean security to create secure products.
Done right, lean security results in a self-defending, simple system created by people who are aware of a product’s real-world security risks and the company’s business goals. Lean security recognizes your company’s data is always at risk, no matter the size of the company or team, and it works to create value through systemic protection. Here’s how to start:
Improve Environmental Awareness
Lean security approaches risk management as a team effort. Think about a hospital. Everybody working there — from the cooks to the nurses to the executives — has a responsibility to take care of patients.
Software firms must function the same way: Everyone from engineers to salespeople to the CIO must keep data privacy, software security and business needs in mind throughout product development and a product’s life cycle.
Environmental awareness means addressing security with the same sensibilities that you would your own life. Just as you don’t hand your credit card to random people, neither should you hand the keys to a software engineer who doesn’t know the risks of insecure development.
More secure coding begins with a change of mindset. Once team members understand a product’s security risks, they start thinking about how to create a more secure environment. If a developer is creating a mobile app for a car, for instance, then they must recognize the app will be used in conjunction with a device that carries human life.
Simplify Engineering
Complexity is the enemy of security. Simplification, however, cannot just be limited to lines of code: It means organizing your teams—and thus, your entire system—in a way that improves time to market and tightens feedback loops.
DevOps is a great way that business leaders can improve security through sensible reorganization. While the integration of development and operations teams has been lauded as a security boosting measure by software leaders at companies including Dell, CA Technologies recently studied DevOps’ business benefits. It found companies that have implemented DevOps techniques are 2.5 times more likely to improve customer retention, twice as likely to grow their revenues and 3.4 times more likely to improve their market share.
In short, keep teams compact and agile and design with the end in mind. The more lines of code you add, the more complex teams you create and the more times you reinvent the wheel—and, ultimately, the less secure and less profitable your company’s products become.
Automate or Die
Automation is an essential component of lean security, and DevOps engineers are no strangers to automation. Threats are constantly increasing, and no engineer—or team of engineers—can discover and remedy them all manually.
Look to Netflix for a great example of automated security. Once upon a time, Netflix addressed malware alerts by manually creating a help desk ticket and assigning an engineer to investigate the problem. The time from alert generation to eventual resolution, according to Netflix, often spanned more than a week. As Netflix’s security challenges increased in both diversity and complexity, it found itself spending exponentially more time and money combating these threats.
Then, Robert Fry, Netflix’s senior information security architect, came up with a brilliant solution: He engineered FIDO, an orchestration layer that automatically evaluates, assesses and responds to security threats. FIDO detects threats, then scores them based on the attack’s intended target and other factors. FIDO then attempts to mitigate the event by closing a network port, ending a VPN session or disabling the account.
Tools such as Netflix’s FIDO automatically detect risks in seconds, while it could take a human months to stumble across them. Think of security automation like a home security system: ongoing, vigilant security without the need for continued manpower.
Measure Everything
Although its origins are disputed, many credit English mathematician Karl Pearson with quipping, “That which is measured improves. That which is measured and reported improves exponentially.”
Development teams should take Pearson’s wisdom to heart, and measurement must begin with identification of goals. Is your goal to reduce application downtime or decrease event response time? Then it’s essential to measure mean time to repair. Do you want to reduce code defect density? Begin by measuring the number of issues per thousand (or million) lines of code.
Once you’ve identified goals, use software to track progress over time. Etsy’s engineers built a tool they call StatsD. The software helps Etsy monitor everything from login failures to coffee availability. The data is then displayed in handy graphs to help the team make sense of the information.
Once you’ve taken baseline measurements, be sure to review your metrics on a regular basis. For instance, you might realize an application’s deployment times aren’t where you’d like them to be. Keep measuring results—and keep trying new things—to nudge those values into better territory. Just remember: You cannot improve that which you do not measure.
The Business of Lean Security
The value of a secure, simple system cannot be understated: It means getting products to market faster; it means better public perception and press; it means more satisfied customers who feel secure using your product; and it means fewer disruptions in engineers’ work schedules. At its core, lean security lowers a company’s overall exposure to risk and reduces its expenses.
Companies are already staking their futures—and consumers’ safety—on code that facilitates transactions, drives vehicles and manages power plants. The Internet of Things will soon run everything from toasters to jet engines, and security will grow even more important. Protect your company from within through lean security.
DevOps is all about increasing the feedback loop from idea to customer and back again. Tremendous gains have been achieved in automation and tools to drastically reduce the amount of manual labor involved in progressing down the release pipeline. But mission-critical applications in a highly competitive market cannot afford to go down. Complex applications, environments and infrastructure increase risk, which can cause failure.
The failure rate of production deployments is still too large. Studies show that the average cost of a production outage is roughly $5,000 per minute, while the average cost of a critical application failure per hour is $500,000 to $1 million.
Why so much failure? All too often we’re still seeing a lack of consistency across the release pipeline. Many times the provisioning and deployment processes are very different at the opposite ends of the pipeline. But by shifting left the operations concerns, you can achieve much more success in production.
Understanding How to Shift Left
The term “shift left” refers to a practice in software development in which teams focus on quality, work on problem prevention instead of detection, and begin testing earlier than ever before. The goal is to increase quality, shorten long test cycles and reduce the possibility of unpleasant surprises at the end of the development cycle—or, worse, in production.
Shifting left requires two key DevOps practices: continuous testing and continuous deployment. Continuous testing involves automating tests and running those tests as early and often as possible, along with service virtualization to mimic unavailable systems. Continuous deployment automates the provisioning and deployment of new builds, enabling continuous testing to happen quickly and efficiently.
The first and most obvious way to shift left operations is to work side by side with development in creating the deployment and testing processes. Failures observed in production often are not seen earlier in the life cycle. Many times these failures can be attributed directly to differences in deployment procedures. Development may create its own deployment procedures that are very different from those used by operations for production. Sometimes production procedures are much more manual and may even use different tooling. Operations and development need to take ownership in building standard deployment procedures. The deployment process is then practiced potentially hundreds of times in test environments before reaching production. You then have much more confidence that the production deployment will be successful.
Another way to reduce the failure rate is to make all environments in the pipeline look as much like production as possible. This is almost impossible using traditional provisioning processes. Production environments usually are much heavier and detailed than are needed by development. But by using cloud and pattern capabilities, you can get much closer.
The ability to use patterns to define consistent environments eliminates the failures that occur simply by configuration inconsistencies. This again requires operations and development to work together in creating the provisioning process so developers create their test environments the same way they are created in production. Using IBM UrbanCode as an example, this can be done by connecting UrbanCode Deploy to an automated self-service provisioning system such as OpenStack HEAT or IBM Cloud Orchestrator connected to OpenStack.
These standard patterns can be used to create environments for all stages of the pipeline. And by using cloud capabilities you can deprovision these environments as fast as you can create them. This also allows the enterprise to have confidence in their environment patterns. They are tested on a daily basis and, therefore, operations can feel confident that testing is done on production-like environments.
No one wants to experience production failures—and it is in everyone’s best interest to do all they can to prevent them. At the root of DevOps is the removal of those silos between development and operations. By applying these shift left operations principles you are well on your way to reducing your failure rate.
The ingredients to make shift left are real and available: automated testing and deployment tools, service virtualization technology, agile practices and cloud environments. The main obstacle that remains is the mindset of delivery organizations. Learn more from the newly updated Application Deployment and Release for Dummies ebook to find out more.
About the author / Dr. Darrell R. Schrag
Dr. Darrell R. Schrag (@devopsdrs) is an experienced software lifecycle consultant specializing in the IBM cloud solution, helping customers establish their cloud strategy across the enterprise. He’s focused on implementing continuous delivery of software applications through the delivery pipeline incorporating cloud and provisioning solutions. He’s experienced in helping clients transition to “born on the cloud” applications utilizing microservices to take advantage of a flexible and nimble cloud platform.
When it comes to deploying apps in the cloud, the conventional wisdom is that DevOps is not required if you’re a small or midsized company. The cloud is designed to enable virtually anyone to set up the infrastructure they need, quickly and easily, without a lot of experience in the intricacies of managing data center operations or hardware. But then, by the time the company realizes it needs DevOps, it’s almost too late—the cloud infrastructure has become complex, costs have exceeded expectations and precious time is being wasted.
To ensure you’re able to optimize cloud, or hybrid cloud, deployments—and avoid many of the problems that other companies have experienced—there are a number of best practices you must employ within your company.
Best Practice No. 1: Realize You Need DevOps from the Start
When you manage your own hardware, you know exactly what the deployment costs are and what resources are available, and have a clear understanding of how they are interrelated. The abstraction of the cloud makes the initial setup a snap. But typically there is no formal structure involved: Individual developers can initiate new instances whenever they need them, without considering the bigger picture of the resources already in use, how they’re used or what is being planned by the organization.
Spinning up unnecessary cloud resources not only costs a company more in the long run, it also leads to a cloud sprawl that requires time-consuming, manual processes to maintain. Think about the challenges your company would face if it had thousands of machines created by different processes but you didn’t know who created them, what they were being used for or even if they were still needed.
By employing DevOps from the start, you will be able to balance the needs of developers with the availability of resources in the cloud to ensure expenses do not run out of control and you understand exactly how the cloud infrastructure is being used now and into the future.
Best Practice No. 2: Have a Process and Purpose
As you create your cloud infrastructure, you need to have a clearly defined purpose, as well as a process in place for deployment and ongoing management. You will need to keep track of what resources are being used and their designated purposes. Some cloud vendors, such as AWS, offer tagging capabilities that help you keep track of what services each machine was created for and whether they are being used for production, backup, load balancing or testing.
Regardless of whether you use tagging capabilities or maintain a separate database of cloud resources, knowing what you have purchased and how it’s being used will provide needed visibility, ease of management and greater control of costs.
Best Practice No. 3: Leverage Automation
Automation is important—both in the cloud and on premises—to streamline routine practices, make sure they’re applied consistently and eliminate potential errors that are common with manual processes.
When deploying on multiple clouds or in a hybrid environment, it is important to understand that tags used for one cloud provider won’t work on another. The process is typically broken into three steps: inventory creation, system preparation and automatic deployment. Many cloud systems allow you to create a dynamic inventory directly from the servers, eliminating the need to create or maintain it manually. However, deployment over multiple clouds require an abstraction of this inventory; it is common to create a static inventory in a unified format from the various dynamic inventories.
Best Practice No. 4: Design the Network with Security in Mind
Poorly designed networks are almost impossible to secure. Problems often arise when companies allow their networks to grow organically rather than being planned out in advance. A number of companies have employed the best practice of segregating the services network and the management network. The services network includes all the components required to provide services to your customers and users, while the management network is comprised of what you need internally to manage the network. Separating these networks provides greater assurance that unauthorized users cannot gain access to your systems via the Internet.
Another part of security is key management. Depending on the cloud provider, keys can be widely available or have a root password that is open to the world. In fact, many hackers routinely scan Github for keys developers accidentally check in. With having to manage so many keys, you may struggle to ensure that they’re not exposed. DevOps plays an important role in keeping the keys locked away, then providing them to a select group of individuals who can use them only in production.
Best Practice No. 5: Be Strategic About Monitoring
When you’re deploying hardware on-premises, it’s a common practice to define in advance what to monitor. But monitoring in the cloud is quite different, with basic defaults that produce vast quantities of data—much of which may not be relevant to your organization.
Taking a cue from how you handle monitoring on-prem, you’ll want to design cloud monitoring to fit your company’s unique needs and filter out the information that isn’t pertinent to your network or operations.
Best Practices Go a Long Way
Keeping in mind these best practices and employing a DevOps model from the start will enable your company to get the most benefits from its cloud deployment regardless of its size, and avoid the costly and time-consuming challenges that arise without proper advanced planning and a strong focus on network design and processes.
About the Author/Roger Fulton
Roger Fulton is Iron.io’s director of Infrastructure and Operations. His background includes broad experience in software architecture and operational planning. He manages international teams to deliver highly available products for end users and highly performing in-house infrastructure. Prior to joining Iron.io, Roger served as head of application and infrastructure engineering at Kaybus. He has also held technical leadership roles at JRV Consulting Group, Cybrata, Invisible IT and Clarus Systems. Roger holds a Masters from the Queens University of Belfast, having graduated with highest honors.
“DevOps is a lifestyle, not a title.” This is a quote from Ron Melanson, of Amazon, and I hope he won’t mind me sharing it here. It struck a chord with me, and this blog explains why.
I have spoken about DevOps on a few occasions. Some audiences are not from a development background, so I explain the “why” of DevOps to them to help set context.
Chances are you are past the “why” stage and into the “how” stage. Even so, it is always useful to think about the “why” part, or even to revisit it—it is important to make sure you really are getting what you want from the DevOps journey.
In my DevOps presentations I vary between the transactional aspects of the “why” and the emotive aspects of the “why.” The transactional aspects are what the tooling can help with, and so that matters to a degree. However, since emotions influence us the most, especially from a lifestyle perspective, it is the emotional aspects that I want to focus on here.
The picture below captures, for me, the essence of why people care about DevOps: they want to be happy at work, and to sleep soundly at night. If you are not where you want to be with respect to those things, then you need to change the way you lead your life.
The DevOps lifestyle is a virtuous cycle
I highlight the “change the way you lead your life” aspect to illustrate contrasts among the differing approaches to DevOps that I have witnessed.
Now that DevOps has grabbed the popular imagination, it is quite common to see job advertisements for people to be part of a DevOps team.
One ex-colleague of mine, hired to lead a DevOps team, expressed some concern about whether that notion actually made sense. He saw his team off in a corner focused on tooling and processes, i.e. the transactional, while everyone else in the organization seemed to be doing what they always did.
When his team suggested a change to a process, or tried to explain how a tool could be adopted, they often were at a disadvantage. The people in development and operations, whose working practices would be affected, had not actually bought into the whole idea in practice, and/or simply did not have the bandwidth to think about it.
Consequently, with little or no time to pay attention to what the DevOps team were doing, there was no engagement from the people for whose supposed benefit the DevOps team was laboring. Thus, there was no driver for changes in behavior. They were all missing the emotive “why” part.
The challenge my colleague faced is common. Hiring someone to do DevOps does allow you to focus on getting people with the skills and experience for the transactional and technical aspects of DevOps. That is necessary, but it is far from sufficient.
If the benefits are to accrue to the people working in development and operations—and, indeed, the wider company and business they serve—then it is those people who have to own the idea, top to bottom.
Owning the idea means rethinking what you are doing, and why—in other words, changing your lifestyle. At least at work, anyway.
Production Docker, automated workflows, open source projects and microservices are just some of the ways that Booz Allen is helping the General Services Administration (GSA) modernize its IT operations. And frankly, after talking to Nirmal Mehta, the lead architect on its recent project, I became a little embarrassed for you DevOps-born startups.
In my circle of friends, the frustration and sarcasm used to describe government processes is a common pastime. And it’s totally fair. Even without being an insider, you can still easily approximate the magnitude of wasted money due to bad business processes and dated IT operations.
But things are turning around. Beyond the early negative press, the issues with healthcare.gov made very clear that the gap between large enterprise and government IT and the modern development shop is massive. And even more recently, the use of private sector hackers to unlock phones, and the continued efforts to recruit from and partner with Silicon Valley show that the government needs the best and the brightest technical help.
One government institution, the GSA, took the modernization challenge head-on. It selected Booz Allen Hamilton to lead the charge and build a new development platform—one that will ensure dev projects deliver results faster, at higher quality and are more systematic.
The GSA is an arm of the government responsible for managing the contracts and processes for vendors that wish to do work with the government. If you have sold to the government in any fashion, you are familiar with it. The administration oversees more than $1 trillion in transactions and more than 300,000 vendors and contractors. To manage the processes across a wide range of categories, it requires many web and mobile applications, with new ones coming on board every year.
Before working with Booz, GSA’s IT ops were about what you would expect—big, slow and risky. New projects took months to get started, with solutions that could take more than a year to complete, and they were ultimately subject to dreadfully low sub-30 percent success rates. (Watch this video and pay attention from the 8:30 point on).
The GSA came to a point where it fully recognized the hard cost of this way of operating, and the indirect cost of maintenance, tech support and unhappy customers on both the vendor and government-consumer sides. It knew something had to change.
Booz’s solution was not 100 percent unique, but its implementation was above and beyond. The firm decided to build a unified platform as a service (PaaS) that served as a portal for all new development projects. In some organizations, this is called “shared services.” But with Booz’s platform, it went beyond just application components.
The users of the solution would be development teams who were awarded projects. Once awarded, they would access the platform and create a service that consisted of applications and microservices. They would be given immediate access to a private repo via GitHub Enterprise, which would also include Docker Compose files.
Then, the teams were off and running. As soon as they made their first commit, the system—already wired up and driven by Jenkins—would pull, run tests and deploy containers to dedicated nodes in a continuous integration (CI) environment, run on FedRAMP-compliant AWS regions. Then the instances were automatically given IPs, where the development team could begin testing.
In addition to a prebaked development and deployment system, the platform provided the development team an entire DevOps stack, giving them “the entire application and lifecycle environment to build their application,” explains Mehta. That included everything from identity management and infrastructure-as-code to business intelligence (BI) and even release automation, so that after the applications were deployed, the teams had what they needed to run updates, measure and learn.
The idea is that a development team with this platform only needs to think about the logic of the application, not process. And now, at the organizational level, there is oversight, continuity and reliability—something which the GSA had never seen before. After the changes, instead of dealing with long contracts and RFP processes, the CTO could approve bids and get development up and running in days, and create solutions in months, not years.
Mehta’s team was responsible for architecting the system, putting it all together and vetting the “marketplace” of tools made available to developers. By doing so, people were able to make sure the proper security considerations were—and are—made, and that the tool configurations, Chef scripts and Docker images stay consistent.
The whole thing was built based on a set of principles. Cloud-first, secure foundation and, to my surprise, open standards and open source. The U.S. government, which fought cloud and open source for so long, is now one of its biggest champions. “I think 2015 was a huge tipping point in gov open source adoption. I think we have hit an inflection point for sure, and I’ve realized that DevOps really implies open source,” Mehta says.
If this is too hard to believe, just check out Booz Allen’s Jellyfish project (which has become part of the ManageIQ project), and this open source policy document in the GSA’s own public 18F repo, which encourages entities to use open source technologies it has developed.
This does not mean the GSA won’t spend money—that is not the driver of open source tools. The driver is cross-pollination, giving back and building better code through standards. Where warranted, such as with wanting full Jenkins support via CloudBees, the GSA will invest in technology. Jenkins is the heart of the system, and basically the home of all of the platform’s workflows and automation. Having the backing of CloudBees is important. Plus, CloudBees is a GSA vendor. It also has become one of Docker’s early customers. I can’t imagine the love it gets from those kids.
Often when I do these case studies, the solutions are too new to validate long-term success. But the platform has been up for six months now, with several development teams and beta applications deployed, which means the system works! The path to prod is still manual and belongs to a separate team, but there are goals to automate that as well.
A government agency is finally more focused on results than process. (If you have ever been through the RFP process, you know what I mean.) “The government is realizing that it is not sufficient to give the modern user, who expects technology that works [to stay with] legacy applications,” Mehta says. The platform also allows the GSA to have smaller contracts, and in theory, smaller and better development teams who don’t have to worry about development operations overhead.
When I asked Mehta if his project is the norm or just a fluke, he said “more and more companies are asking us about DevOps, not just cloud migration. We know that if they just do a cloud migration, they are going to create more problems faster.” He believes that even if DevOps is not explicitly stated, federal and local governments do care about modern IT ops—if not because they want to, but because they have to.
So, can you say you have a fully automated full-stack container-driven platform that spans multiple applications and multiple development teams? Chances are no, but you could, and you should. Mehta and team have proven that even the government can have a Ferrari of a development environment and stay on the cutting edge.
To learn more about Mehta’s project, watch his session at DockerCon2015 here. And check out this presentation from Navin Vembar.
How do you balance the need to “go fast” with the need to test everything and deliver high-quality software?
With applications the driving force in today’s economy, the quality and release cadence of your software are critical to your business and your bottom line. You want to get software updated in the hands of your end users as quickly as possible. However, even the fastest release cycle wouldn’t make a difference if your product ends up being buggy or malfunctioning, resulting in a bad user experience or service interruption.
I want to take a look at one of the most important (if not the most important) aspects of the software delivery process: testing! As software teams become more mature in their implementation of DevOps and continuous delivery to streamline their software delivery, testing takes a central role as a prerequisite to these practices.
You need to have testing on your mind from the very beginning, and plan your testing as part of the initial stages of designing your software—mapping your test procedures, infrastructure configuration and acceptance criteria. As you design and develop your app, you also need to think about how you’re going to test it. The more testing is on your mind, the more you’ll have a solid understanding of what you need to test, what’s the definition of “done,” and the faster, more streamlined, your overall process will be.
Let us review four questions every development or QA team should consider when planning their testing, as they try to balance quality with speed—managing and accelerating their testing, while ensuring a reasonable time-to-market.
How Much Testing is Enough?
The age-old question: How much testing do I really need to do? Unfortunately, there is no concrete answer. But with a bit of initial legwork, you can get a good feel for how much testing is necessary.
The amount of testing that is necessary is influenced largely by the industry you’re in. If you are testing mission-critical aircraft software, let’s hope it’s very thoroughly tested! On the other hand, if you’re building a game app, the amount of necessary testing may be lower. The truth is when you run tests, the success rate is usually not 100 percent. Therefore, the key is to define what, for your specific piece of software, can reasonably be considered “done.” Instead of arbitrarily deciding at some point that your release is ready to ship, make the decision before the fact: What are we going to accept? What level of pass/fail are we willing to roll out with? What are we comfortable with based on what we have done in the past?
The amount of testing required can sometimes be a fluid thing. A good way to measure your comfort level with your testing procedures and the amount of testing you’ve done, is to ask yourself, “If it was now 11 o’clock at night, and I needed to ship this code to my end users, am I sufficiently comfortable rolling this out, or am I not sure how it’s going to behave in the wild?” If you’re leaning toward the latter, than it’s probably an indication that you should have done some more testing. That’s a gut feeling and depending on the industry, the point at which you’ll fire off a release and not have any worries can vary greatly. And, obviously, for some regulated industries, you’ll need to prove compliance and ensure you’ve recorded that all tests were successful before you can even start thinking about releasing anything.
What Kind of Testing Should You Invest in?
Decide what kinds of things you really care about and what you need to verify and test for. What are the requirements that we expect from the application, and from the different stakeholders in this project?
Every piece of software needs to be tested differently. Because of this, you have to plan what the testing will be. You can’t just say, “I’m going to unit test the heck out of my code, each and every time, no matter what.” Sometimes, unit testing isn’t the most important test you need to be running.
The investment has to be in automation. No matter what type of tests you decide are most functional and relevant to your project—whether you’re doing a ton of unit testing or a bunch of functional stress tests—it needs to be automated.
Particularly, as you open up your services to the world, you need to put into place the relevant performance and reliability tests so that if suddenly your load increases by 200 percent, you have a plan for how to handle that. Are you going to autoscale? Can you autoscale? And finally, do you even need to auto-scale?
All of those questions need to be answered as part of your development cycle. Automating your tests will let you not only create tests for these scenarios, but it will let you run those tests over and over. By automating your tests, you can establish a baseline and see where you deviate.
It can also be helpful to think of testing in a different way. Don’t look at testing as a tool to validate technical requirements, but as a way to make your customers happy. You want to make sure your tests are directly contributing to your customer’s overall experience. So start with the highest-level tests that look at things such as overall business functionality—and make sure the software works at the business level.
How Can You Make Testing Faster?
Automate, automate, automate! That might seem obvious, but try to think outside the box. Find things you thought couldn’t be automated and wrangle them into shape. Remember that every minute that one of your developers is sitting there running a test that could be automated, it is costing you time (and money) that he or she could have spent on more productive things. In the long run, not automating can cost much more than making the commitment to doing it right.
Other simple things you can look at to increase the speed of your testing is to make sure your development environments are mirror images of your production environments, and that you have fidelity of configuration throughout the lower and higher environment.
It is also helpful to make sure you’re defining user requirements that are thin vertical slices, instead of trying to test huge cumbersome pieces of functionality. The ideal situation is if you can run only tests against components that have changed. Componentization of both your application and your testing procedures is critical because you do not want to have to rebuild and re-test everything all the time. Remember, also, that if you make incremental changes to your code and frequently release them to your users (Agile, anyone?), you will get just enough feedback to improve quickly and reliably. Be cognizant of not biting off more than you can chew.
Segregating the tests to several phases is a good way to accelerate your CI and test cycle—but make sure that both your application, build and test batteries are designed for that. With this approach, you are essentially running multiple phases of CI, starting with the fastest cycles that require the smallest build(s), with minimal dependencies, and move on to the outer/cascading cycles of tests. Following a submit, run a minimal build, so that you do not need to wait—sometimes many hours—to run an extensive set of all your tests with every single commit to see if it passes. Start with unit tests to minimize the build and provide fast feedback, then in various junctions incorporate additional tests, such as integration tests, installers and usability and performance tests.
Another way to streamline your testing process is to take a look at the entire scope of your build and test cycle and go after the most inefficient stage. After automating your manual processes, look at the bottlenecks and what takes the most time and figure out how you can optimize that specific process. Often the answer is parallelization, but other times it is hardware. Iterate on your process and figure out where you can gain the most ground!
Lastly, parallelization and distribution of the build and tests provide a huge time saver, so consider running tests across a cluster of CPUs to cut down the run time.
Where Do You Start if You Want to Make Sure All Your Builds Are Green?
Ask yourself, What are you trying to achieve? Is it a process where you can release code more quickly? Or where you can address bugs more readily? Then make sure you’re intimately aware of what you have in place. You need to know every step, task, process and tiny bit of your testing procedures and supporting infrastructure. Draw your entire end-to-end workflow on a whiteboard and find the bottlenecks that are slowing it down.
Once you’ve defined your objectives and are aware of your current state, you can put the relevant tools in place—from development environment to your testing frameworks all the way to post-dev tools such as your analytics tools—design the right testing pipelines and hire the relevant people with the relevant skills. Don’t insist on making all the mistakes yourself (tongue firmly in cheek), but find folks who have already made the mistakes and learned from them—and hire them!
Also, give your team a great system that gives feedback fast. Do frequent code reviews so there is accountability and a constant learning process.
Consider the following best practices for improving your testing processes:
Be sure to plan what your testing approach will be ahead of time if possible. When you’re designing your software, designing how you’re going to test it is important.
The amount of testing you need to do is dependent on what industry you’re in and your specific use case.
Consider what level of test pass/fail you’re comfortable with, based on what you’ve done in the past.
Consider pre-commit or pre-flight builds to ensure your tests are more likely to pass once checked to Mainline
Define your requirements for each test cycle
Segregate your tests to “fast” and “slow” and decide which battery of tests you need to run every time, and which processes you only need to run at certain junctions.
If you hire well and bring new skills to your team, your process will get better and faster.
Consider what problems your customers are facing and perform an analysis. Use whatever tools are best to test your software like a customer would to achieve the highest end user satisfaction.
Automate manual steps that you take for granted to accelerate cycle times and improve productivity. Speed comes second.
Iterate on the testing process, and not just on the code itself. You should always be looking for new ways to test.
Test your test processes. Hold regular meetings with your team to reassess if your testing strategy is working for you.
Enact code reviews so that there is accountability with the rest of the team.
As you map, design and optimize your tests, remember that your testing policies don’t need to be set in stone, and that you can—and should—adjust and tweak as you go.
Happy testing!
About the Author/Anders Wallgren
Anders Wallgren is chief technology officer at Electric Cloud. Anders has more than 25 years’ experience designing and building commercial software. Prior to joining Electric Cloud, he held executive positions at Aceva, Archistra, and Impresse and management positions at Macromedia (MACR), Common Ground Software, and Verity (VRTY), where he played critical technical leadership roles in delivering awardwinning technologies such as Macromedia’s Director 7. Anders holds a B.SC from MIT. He invites you to download the community edition of ElectricFlow to build your pipelines and deploy any application, to any environment for free.
Continuous integration and continuous delivery (CI/CD) are often cited as pillars of successful DevOps. To establish and optimize the CI/CD development model and reap the benefits, companies need to build an effective pipeline to automate their build, integration and testing processes. This requires architecting CI/CD with five key ingredients which, when combined effectively, operate as a successful cornerstone of DevOps and provide myriad bottom-line benefits.
Building the Pipeline
Let’s first take a look at the five essential ingredients needed for a healthy CI/CD architecture, following up with a discussion of their benefits to the business:
Micro-agility in the form of testing and committing code frequently: The development life cycle begins with a request to development, which then takes the request and designs a solution, codes it, tests it and commits it a code repository. From here, the code is tested by QA and then deployed to production. While each step is important, we begin here, as local testing occurs most frequently and bugs found at this stage of the process help avoid the large expense of bugs found in production.To achieve micro-agility, developers must have a clear understanding of where they are expected to commit their code after changes are made and they should be expected to test and update their code frequently—many times a day. This means development testing should be easy to run locally and is a subset of QA. The result is that code pushed by development is by definition QA-ready.
Automated QA: In my work I see many organizations whose QA team is one of the largest bottlenecks in the process. It’s not for a lack of skill, but too few staff who spend a good portion of their time waiting on the QA test environments they need to do their work. To avoid this situation, the QA process should be fully automated, from QA ready code coming in the door to one-click promotion on the other side. In between the two, QA should have a constant feedback loop with development where communication about—and tracking of—bugs is clear and easy to follow, allowing development to easily repair issues and recommit the code for further testing. Moreover, automated QA in a CI/CD environment should focus on microcycles consisting of small test batches rather than weeklong macro tests. Automation and communication are the overriding imperatives here, exhibited by reports to developers that are communicated immediately and code commits that are fail-rejected automatically.
Production-like environments: To increase efficiency and effectiveness, development and testing environments should mirror production as closely as possible. They should follow the PRQ rule, as I like to call it—production-like, repeatable and quick. Specifically, it should be fast and easy to set up new environments, to the point where developers can, with the push of a button, deploy new environments for their development and testing needs. Doing so allows developers to fail fast, breaking and re-creating environments without fear as they know they can easily recreate the same environment every time. I often recommend using tools such as Jenkins to help with this process; if developers and IT operations use the same tool, it makes it that much easier to streamline the CI/CD process and achieve the promise of DevOps.
One-click promotion: As I alluded to earlier, a well-architected CI/CD pipeline provides one-click promotion to production. This ensures changes are seamlessly moved to the overall production environment with the least amount of friction possible between development and operations. To that end, this reduction in friction should occur across all the steps of the process, with efficient code delivery pipelines that are designed to balance security and agility from the outset. Cloud infrastructure coupled with container technology and automation can easily optimize the process across teams and throughout the DevOps journey.
Fully automated deployments: The last factor of an efficient CI/CD pipeline is full automation of deployment. Any configuration changes made by development and accepted by QA should automatically be applied. Continuous integration in this way helps avoid system downtime and human errors are decreased greatly. Deployments also should be continuously monitored so that IT is the first to learn of potential problems. This helps nip issues in the bud, allowing for faster rollback and return of code to development to fix anything that may not be functioning properly. Catching issues before customers do helps ensure customer satisfaction and strength of reputation.
Benefits of the Pipeline
In addition to these benefits, when combined effectively, these ingredients minimize the time and cost of local test, QA and production processes; reduce the time it takes to identify problems and their causes; and accelerate delivery of production-ready code. We can see this throughout the CI/CD pipeline:
Developers have greater control over the process and can respond more quickly to market demands by quickly bringing up new environments, testing against them and easily starting over, if need be. Moreover, developers can quickly adjust to new environments—an approach that has proven to decrease failures and resolution time when they do occur.
Operations benefits as reduced friction in the CI/CD pipeline created by automation decreases repetitive, manual work and with it the opportunity for “fat finger” mistakes and the greater opportunity to introduce inefficiency into the process.
And, less manual, repetitive work provides everyone in the process with more time for strategic work, which provides direct bottom-line value to the organization.
CI/CD is a pillar of successful DevOps and according to research by CA, organizations that apply DevOps discover an 18 percent reduction in time to market of new applications and a 21 percent increase in delivering new software that they would not otherwise be able to complete. Teams that master an efficient CI/CD pipeline deliver faster, higher-quality applications to market bringing with it a 19 percent increase in revenue. Mixing these ingredients together results in an efficient, streamlined CI/CD pipeline that serves as a fundamental pillar of DevOps success that, when mixed together, delivers direct business benefits.
About the Author/Aater Suleman
Aater Suleman, CEO & co-founder at Flux7, is an industry veteran in performance optimization on servers and distributed systems. He earned his PhD at the University of Texas at Austin, where he also currently teaches computer systems design and architecture. As a faculty member at UT-Austin, Mr. Suleman focuses on optimizing DevOps architectures involving the AWS cloud. Along with the Flux7 consultant team, he works at every step of the dev flow, from creating local developer environments using Vagrant and Docker, to continuous integration and continuous delivery using Jenkins or RunDeck, to configuration management using Chef/Puppet/Ansible, to creating efficient systems for deploying new code from Git using Capistrano/Fabric/bash. His background is as an architect, and he has experience with both hardware and software. This combination brings a unique perspective and broad view to development and operations.
The unfortunate reality for today’s organizations is the fact that a security breach is bound to happen. Major breaches are happening with alarming frequency and fill the news headlines almost daily. And behind many of these major breach stories is a software vulnerability that has been exploited. There is a silver lining, however. Addressing security at the development stage can help organizations “get upstream” of certain issues, which goes a long way toward alleviating potential breaches.
Now there are always two sides to every coin, and this is where companies need to weigh the costs of not taking a proactive approach to making security a key part of their software development process. Is the upfront time and effort worth it when it comes to developing software and ensuring bugs and vulnerabilities are addressed before it is put into the wild? The simple answer is, yes! We are already seeing this in certain industries such as financial services, which learned the hard way that it is a critical component of software development—mitigating risks and significantly lowering the financial burden associated with a major breach.
In fact, security should have been more central to software development a long time ago. As software, and more broadly technology, continues to evolve and permeate our lives from more angles, it is crucial that security be a consideration—a “use case” as developers call it—from the earliest conceptual stages of a project. The costs of not doing so are simply too painful to imagine otherwise.
Shifting to Security-First
To build more secure software, organizations typically must implement a software security initiative. You can break it down into four steps:
Assess your current software assurance posture
Define the strategy your organization should take to ensure a viable posture
Formulate a road map of how to achieve that posture
Implement your strategy
The above steps are simple and when you think of the alternative to not doing this, adding security as a pivotal part of the software development process seems like a no-brainer. To better understand what I mean, you need only look at the cost the average breach has on an organization. According to the 10th annual “Cost of Data Breach Study” independently conducted by Ponemon Institute, the average consolidated total cost of a data breach in 2015 was $3.8 million, an increase of 23 percent from 2013. The study also reported that the cost incurred for each lost or stolen record containing sensitive and confidential information increased 6 percent, to a consolidated average of $154 from $145. Depending on how much data has been compromised, the financial fallout can be staggering, if not crippling, to an organization.
Conversely, the cost to repair a single vulnerability in an application during the design phase is less than $500, according to the IEEE Computer Society. The fact that the financial impact of not addressing security during the development process can quickly become cost prohibitive, is a clear indication that secure software development practices can no longer be overlooked. Breaches are now a boardroom issue and require a seat at the development table.
As we can see from the stats above, the economic fallout for organizations is obvious, but with the Internet of Things coming to a house near you, this issue is suddenly becoming more personal and painful for the average person. Whether an attacker takes over your thermostat from Russia or finds a way to monitor your home via your baby camera, vulnerabilities are making it easier for malicious actors to corrupt corporate, government and now individual networks like never before.
We are well beyond the “it will never happen to me” phase. Breach stories that used to happen to so called “other” people are more likely to happen to you if software defects continue to be ignored and are not fixed when they should be. No matter how you crunch the numbers, if security isn’t addressed at the software development level, the result will always end in a costly lesson learned.
About the Author/John Dickson
John B. Dickson is an internationally recognized security leader, entrepreneur and Principal at Denim Group, Ltd. He has nearly 20 years hands-on experience in intrusion detection, network security and application security in the commercial, public and military sectors.
Dickson is a popular speaker on security at industry venues including the RSA Security Conference, the SANS Institute, the Open Web Application Security Project (OWASP) and at other international conferences. He is a sought-after expert and regularly contributes to Dark Reading and other publications. A Distinguished Fellow of the International Systems Security Association, he has been a Certified Information Systems Security Professional (CISSP) since 1998.
As a Denim Group Principal, he helps executives and Chief Security Officers (CSOs) of Fortune 500 companies and government organizations launch and expand their critical application security initiatives.
DevOps and continuous delivery are the same thing, right? Far from it. We peer through the fog to understand the difference between the two related but distinct methodologies.
We live in an era of apps. More and more of our personal, social and work life is governed by apps—communicating with one another through social apps, booking cinema tickets through your mobile device or scheduling a business trip through the airline app. This digitization of services has arrived. Organizations that understand this sea change have a unique opportunity to jump ahead of their competitors, acquire customers more quickly and grow revenues.
The battleground for this service development is DevOps. It’s not a product, but rather a methodology for helping organizations build teams and build software. DevOps is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other IT professionals while automating the process of software delivery and infrastructure changes. It creates a culture and environment where building, testing and releasing software can happen rapidly, frequently and more reliably.
Organizations are marching with their feet. According to a recent study, market demand for DevOps skills is growing, and DevOps engineers are among the highest paid IT practitioners today. And it looks set to stay that way. Research in the U.K. shows that 30 percent of 300 organizations surveyed in the run up to the 2015 DevOps Summit had either started to implement a DevOps strategy or had already merged Dev and Ops, with another 13 percent planning to go down this route.
Two Related Children: DevOps and Continuous Delivery
So now we arrive at continuous delivery. If DevOps is about accelerating the delivery of new, high-quality services, surely continuous delivery is the same thing, isn’t it? In other words, ensuring a steady stream of new services? You might look at it as a manufacturing production line: DevOps is the machine that builds the service, while continuous delivery is the conveyer belt that rolls the services off the production line—one big unified service development cycle.
That’s not the case. Although the distinction between DevOps and continuous delivery can be a little grey and cause confusion, there is in reality clear water between the two. Continuous delivery has become an essential ingredient for teams doing iterative and incremental software delivery. It is an approach whereby teams ensure that every change to the system can be released, and that any version can be released at the push of a button. Think of it this way: Continuous delivery aims to make releases dull and reliable, so organizations can deliver frequently, at less risk, and get feedback faster from the end users until deployment becomes an integral part of the business process and competitiveness of the enterprise.
Continuous delivery and DevOps do share common traits, though. Both methodologies are aimed at agile and lean thinking: each delivers small and quick changes; each relies on tight business an IT collaboration; and each share the common goal of faster time to market for new services.
However, the goal of DevOps is merge dev and ops roles together—and the processes they manage—to achieve business goals. It’s all about a common, shared culture and enhanced collaboration. About clearly defined business processes.
Automation has an essential role to play in both DevOps and continuous delivery. Automating DevOps helps your organization manage and secure continuous delivery to the business and IT users. The result is a faster, low-risk migration to a new world of composite applications, faster business innovation and better support for rapidly changing business process needs.
If you ever wanted to learn about Rugged DevOps (some call it DevSecOps), sit down for a spell with Shannon Lietz, Ian Allison and Scott Kennedy from Intuit. We discussed a number of important topics including internal war games, culture hacking, gamification of Rugged DevOps and starting as a small team. There are 100 gold nuggets in this conversation for novices and experts alike.
Derek Weeks: I have some of the Intuit DevSecOps team here with me today. We’re going to talk to them a little bit about Rugged DevOps and how things work over at Intuit. Let’s start with some introductions.
Allison: I’m Ian Allison. I help run the Red Team at Intuit, which is, I guess you’d say, an interesting way of taking control of security at our company. We try to get ahead of the attackers by basically being the attackers. We’re essentially ethical hackers. We go after all of our own stuff to make sure we can find where the deficiencies lie in all of our software.
Lietz: I’m Shannon Lietz. I’ve been working at Intuit 3 1/2 years and helped to found the 24×7 DevSecOps capability at Intuit, leading the Red Team, our security operations capability, our cyber SOC, and what we also consider “blue teaming”: being able to hunt for defects.
The organization has really had to transform how we do software development because we’re a 30-year-old software company. We are now seeing the traditional way of putting together software really embracing DevOps. For us, it’s been exciting to really work in the industry with Rugged DevOps, trying to help build security into the DevOps movement.
Kennedy: I’m Scott Kennedy. I run the forensics and threat intelligence part of cyber work.
Weeks: Shannon (@devsecops), tell me a little bit about software supply chains and how that vision of software development has impacted the way you see things at Intuit.
Lietz: That’s really a great question. It was interesting when Josh Corman and I first talked; we had a lot in common. One of those things was the software supply chain. What I really love about the concept is being able to have processes driven a certain way so that you can reduce defects.
Having worked for Toyota in the past and understanding the supply chain mentality, you get a sense of how you could put something together better, incrementing on it, figuring out how to share that process, and then really figuring out what things are important. Having that notion of fewer, better suppliers was really a core concept.
I love the idea of transparency, building things a certain way, and really getting into continuous improvement. You need to look at things from an opportunities perspective—making sure you’re not just looking to make things perfect. You’re looking for those opportunities to improve over time.
https://www.youtube.com/watch?v=nRaURhaTH9o
Weeks: As we think about Rugged DevOps within your security team, how do you measure the success of what you’re doing? What kind of metrics are you looking at that matter to the business?
Lietz: We measure everything. For example, mean time to remediation (MTTR). Once somebody finds a defect, we analyze that defect from the time it got into the supply chain to when it actually gets resolved. We track everything from mean time to remediation to when the ticket was created, to being able to look at when the code actually got published, to when it actually got found, and then we work on those things over time. We really try to uplift.
____
Once somebody finds a defect, we analyze that defect from the time it got into the supply chain to when it actually gets resolved.
____
We leverage JIRA [Software] just like a software development team does. We register our defects and figure out how to get development teams to take responsibility for those ideas. It goes through their process of release and regression testing. As part of that, we look back to see where our opportunities are.
As an example, we started out where things may have taken weeks. We then reduced it down to days and ultimately got it down to hours. We’ve seen defect resolution where it’s now minutes. When it’s something we’ve discovered that was just a mistake by an engineer, we realize “mistakes do happen.” We found that our cycle times also help us to find fault stack vulnerabilities in real time because we get to do end-to-end testing more aggressively utilizing this method.
Weeks: How has consistency in your operations helped with Rugged DevOps and has it fragility within the organization?
Allison: One of the things we do is to utilize a golden image for all of the AMIs (Amazon Machine Image) we use, for all of our customers, and we require everybody to use these AMIs. We’ve also built some really interesting automation around scanning these AMIs. So one thing we realized quickly when we first started native U.S., when we try to do full vulnerability scans against another system, if it’s set up to autoscale, we all of a sudden have 50 systems. Right? We can’t … it’s really hard to do a full vulnerability scan right against the system, so we came up with a way to share back all of the AMIs with a special account. Then we bring those up and we scan them. Then we grade them.
Based upon the vulnerabilities that are found, you’ll get a letter grade, like A through F, based upon the system you have. While we always strive to have our base image be an A, people continue to run on older images. But they get graded, and those grades get pushed up, so everybody in their org structure gets to see what the grade is for their account. I think by being a little standardized, basically with these images, lets us know what’s in everything, and we have a grade for everyone. It helps everyone have a really good idea of where they stand when it comes to a security standpoint.
____
Based upon the vulnerabilities that are found, you’ll get a letter grade, like A through F … so everybody in their org structure gets to see what the grade is for their account.
____
Weeks: That’s not only a grading but a policy enforcement governance kind of role that grading plays. How rapid is the feedback loop in that grading system for the teams that you’re working with?
Lietz: It’s really quick, and we’ve discovered through some science that having component-based resources like AMIs provides us with an advantage when doing things like remediating vulnerabilities. Using AMI-based resources, we have seen that when there’s a defect in it, we can find and remediate all of the defective AMIs quickly. That improves everyone’s security across the company.
So if you’re just picking out really good components, keeping track of those components and adding security into them, then you’ll actually see a different effect across our pipeline. A single change can actually have a dramatic effect on reducing the problems within the pipeline.
Allison: It’s really interesting. This morning I got an email from somebody that said, “Why did our baseline AMI go from an A to a C today?”
We had just received notice of a new vulnerability. Our stuff caught it, we scanned it, and we pushed the grade out to our portal where all our customers go to look at the grades. Our customers were able to see that change quickly.
They could now say, “Wow, it changed from an A to a C in less than 12 hours.” I think the feedback is really important. The other important thing is that we have people going and looking. I wouldn’t be getting emails about why has this changed if people aren’t actually looking and wanting to make their grades better.
Weeks: You mentioned customers. Are these internal customers?
Allison: Internal.
Lietz: Yeah, for our development teams, we as a security team really have changed how we think about things. It used to be that the security team would go out and govern. Basically, you got the fear of the security team coming in, descending upon you.
We’ve really changed how that happens within our organization. We grade our resource components, and we grade the way in which our applications come together. That changes how developers want to operate because they really want to figure out how to get better grades in security. And it creates a learning dynamic that incentivizes somebody to improve continuously.
___
That changes how developers want to operate because they really want to figure out how to get better grades in security. And it creates a learning dynamic that incentivizes somebody to improve continuously.
___
Weeks: Does it create a competitiveness or gamification of who has better grades?
Lietz: Absolutely, which is why we did it in the first place. To your point there, gamification is something where when you start to grade components like that, you can actually start to leverage a leaderboard concept. We do have leaderboards as part of this. We have APIs where you can actually pull down your grades and include them in your automation. With these, you can make governance decisions.
If you sort of have that “game afoot,” right, your leaders can then ask for specific grades within their pipeline. That up-levels the system, and you just see a continuous improvement lifecycle come to bear. Ultimately, you see fewer defects, and ultimately, you get to the notion of Six Sigma in our way of thinking. DevOps is really about continuous improvement and embracing automation. Embracing that concept allows us to get to fewer defects faster.
___
DevOps is really about continuous improvement and embracing automation. Embracing that concept allows us to get to fewer defects faster.
___
Weeks: As you embraced continuous practices and DevOps practices, were there points when you realized certain old ways of doing things weren’t going to enable you to move forward?
Kennedy: In looking at the progression of what we’ve been doing, one of the decisions that was made in Intuit and one of the things that I saw was really unique was the way they decided we were going to migrate into AWS. Our idea was to have the chaos team be the first people out, and that’s the security team. So the security team was the one that was going out and finding out how to use each of the products that AWS has and creating the concept of whitelisting. Each product was rated as to whether or not it met security’s requirement.
Therefore, no team can go ahead and pull down this new cool tool that AWS released yesterday and use it in production because it’s not been “whitelisted.” That can go into their scoring. Their scoring is not only used by the development teams but also is useful when reporting to the board. When the board asks, “How are we doing as a company across the entire organization?” we can say that product A got a lower score than product B, and then they turn to the VP in charge of it and say, “Well, why?”
___
When the board asks, “How are we doing as a company across the entire organization?” we can say that product A got a lower score than product B, and then they turn to the VP in charge of it and say, “Well, why?”
___
We decided to not rush into the cloud but to take a careful, considered approach and migrate in a very intelligent and well-thought-out way. At the same time, we gave the chaos team the ability to make the mistakes and grow and learn, so they can immediately turn around and share the mistakes with everyone else. They could say, “Hey, these are the things that didn’t work for us. We came across a lot of problems, especially when you look at things like accounts and account roles.”
How do you control when you have thousands of accounts and you need to have some sort of administrative control?
You can either have a gigantic effort to force your namespace and your Active Directory to be the source of control. Or you can use the vendor-specific tools like IAM and have each account have their own separate islands, but with the concept of cross-account roles, you can then do remote administration from a centralized account. You have it locked down. You have the capability to have a restricted group and be able to remotely go in and do the necessary actions.
That also gives you an audit trail. That also gives you multifactor built in because the AWS products get those things added to them.
Lietz: I think when it comes down to it, I think culture-hacking your environment can have a profound effect, especially when you’re going through a DevOps transformation.
Weeks: What is culture hacking?
Lietz: That’s a great question. We use it when really trying to figure out how we as a security team can change and transform. A lot of the things that take place in a company are really based on traditional processes: What has worked before, and why would we change something that is working, right? If you’re really going to go into an innovative frame; if you’re really going to get into that next-generation innovation; if you are trying to figure out what’s going to work in that … it’s never going to be the thing that is working. It’s going to be the thing that you’re going to learn as you go to that next step.
Culture hacking is really about looking at the people who are operating right now and trying to figure out how you’re going to help them go from A to B, making that change. What is that the experience going to be like?
What we have done, to Scott’s point, is we’ve forced our security team to have empathy for the DevOps teams. We go through the process of developing something in the cloud, utilizing it as a method of taking their paranoia and trying to balance the notion of getting something done within a specific time frame. We try to really wrangle what it takes to do those things securely and safely but, ultimately, still be able to deliver for the business.
I think that culture hacking really comes into play when you’re trying to figure out how to move somebody from the rock they’re on to the rock you need them to be on and trying to figure out what those mechanisms are.
___
Culture hacking really comes into play when you’re figuring out how to move somebody from the rock they’re on to the rock you need them to be on.
___
Weeks: Part of your security practice is looking at open source and third-party components and your own binaries. Can you shed some light on how Intuit is using Sonatype solutions to better manage those vulnerabilities?
Lietz: Yeah, Sonatype Nexus is a fantastic platform. We love the Nexus repositories. We love how you guys put together a community. We learn a lot.
A lot of our DevOps practice is working together with it. We’ve put together our Nexus repositories to do code signing and figuring out how to really secure our pipelines a certain way. We are taking advantage of the fact that we can pick up components, track them and then scan them [for known vulnerabilities].
That’s allowed us to reduce the defect count that goes to production. Actually scanning and looking for vulnerabilities within our components and our open source libraries allows us to make better decisions about what we’re including in our software.
Weeks: When you govern what open source, third-party or proprietary components are being used by developers, is there any kind of feedback from the teams saying, “Hey, you’re restricting my behavior, not improving my innovation”?
Lietz: What we’ve found is that the notion of security approvals, exceptions and gates really doesn’t work. Quite often, you just create a culture where developers are going to go out and do it, and then you’re going to find out about it. When it comes to really partnering and being boundary-less about how you think about security in your business, it’s all about transparency. It’s all about benefits. It’s creating things like a security markdown file within your repository manager. It’s about taking responsibility and accountability for the things that you’re doing from a security perspective in your development process. It’s ultimately having an attacks.md file, keeping track of what’s out there, keeping track of your open source, understanding what components you’re leveraging, and why you made the decisions that you made to bring those things into your project.
___
It’s about taking responsibility and accountability for the things that you’re doing from a security perspective in your development process.
___
At a top level, all of those things work. But really having tools that can help the decisions that were made by some of the other open source programmers that you’re getting contributions from is really necessary. All of the things that they might be deciding are also part of your decision tree, and ultimately, you’re rolling all of that and bundling it together. The attack surface is not just the decisions that your team is making, but the ones that you share across the code base that you’ve got.
Weeks: Your practices are very mature. You’ve clearly developed them over a long time, and some people watching this might think, “Well, Intuit’s a huge organization,” and it may be daunting to them if they haven’t started down the path of Rugged DevOps. Can you be a small team and have success in these kind of practices?
Lietz: We’re not exactly a huge organization, but we are relatively large in size now. When we got started, I believe I was one of maybe three people that started this, only a couple years ago. We have hired into our group pretty extensively to help grow it, and some of the things that we’ve done have really allowed us to operate differently, to bring in people and have them immediately be successful. Our practices allow someone “day one” to be able to work with the environment, to be able to develop code, to be able to contribute code that week.
We do things like weekly demos, where we actually do video demos. A person has to come in, program something, secure something, operate it and create a demo, all within their first week. So having the right bar for those folks is really important, but more importantly, our Red Team leader here (points to Ian), he came in and just is amazing, has created a Red Team pretty much out of thin air. So is having somebody from forensics, who’s just done an incredible job to help us, to make it so that we have a life cycle where we can snapshot something and be able to learn from it when it’s actually offline.
___
A person has to come in, program something, secure something, operate it and create a demo, all within their first week.
___
Those are the types of practices where you start to extend yourself past the normal baseline practices of processes today and really think past that about how you’re going to support innovation. You get into it very quickly. You get a learning culture. You get people who know that making mistakes—and figuring out how to learn from them—is okay. That’s a really important part of that actual culture that you’re putting in place.
Allison: Yeah, I was going to say, it’s all about iteration, right? We started small, and we just continually iterate on what we’re doing to try to get better and be better at what we all do.
When I first started this journey, I was a security guy—a pen tester. It was always the developer’s fault. Developers always made the mistakes. I always had to clean up after them. But after six months of developing Ruby APIs and Ruby and working my butt off in code, the empathy was there.
I really understand what the developers are going through and why they make the choices they do. But I think by allowing us to help them, by creating tooling that allows them to self-serve, to understand it without making them … helps them make themselves more secure without them having to become a security professional. I think that’s kind of our ultimate goal.
Shannon: Being friendly hackers, right? Basically going out and attacking them so their applications don’t get attacked by external attackers is really part of that frame.
Kennedy: The Red Team shift at the company has been profound because you see how people react. When the Red Team started, it was not as well shared, and a lot of people suddenly were very upset that they were attacked by the Red Team. But when it was pointed out, “Well, what would you rather have happen? Would you rather have somebody in China do this to you, that didn’t work with you, didn’t sit next to you and help you fix the product, or would you like to have a friend who, by the way, their job is to attack?”
When we went through several drills and actually practiced the muscle of defending the company against an attack, people were upset. “Oh, I had to do all this work.”
My response to them: “Well, you did the right work.”
“You did the right thing. You saw something bad. You did it. You did good. You practiced the muscle. Now when it happens again and it’s not the Red Team, I know that you’ll know what to do. You know that the process works, and we can actually defend the company faster and more securely.”
___
You know that the process works, and we can actually defend the company faster and more securely.
___
Weeks: Yeah. That’s an incredible story. Thank you for sharing it.
My final question: If you could pick a superpower in dev, security or ops that you would have in the organization, what would it be?
Allison: To me, they’re all alike; they’re the same, right? That’s what we do, DevSecOps, right? We try not to actually separate them out because I think once you start to separate them out, you start to lose perspective.
Kennedy: Yep.
Allison: There’s a good thing about having them all be one thing, so I’d choose them all.
Kennedy: It’s been pretty consistent. DevSecOps is the answer. What was the question? (Laughter)
Lietz: I think the reason we went out and created DevSecOps was just simply to change how we thought about doing development and technology—and to really to get ahead of it, to realize that attackers weren’t setting up appointments or meetings to help you figure out how they were going to attack your software, and so then why were we? Why were we operating at a fragile level?
I think that the superpower that I would like to have is DevSecOps because I know that we are going through the process of creating a less-fragile capability in security that will allow us to get ahead of attackers, make it much harder for them to go after the software that gets built, and we’re seeing those improvements. That’s actually a great thing.
Weeks: It sounds really exciting, and it’s very cool, so thank you all very much. I really appreciate it.
All: Thank you.
If you loved this interview and are looking for more great stuff on Rugged DevOps, I invite you to download this awesome research paper from Amy DeMartine at Forrester, “The 7 Habits of Rugged DevOps.”
As Amy notes, “DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros’ expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps.”
Anders Wallgren is chief technology officer at 
Aater Suleman, CEO & co-founder at Flux7, is an industry veteran in performance optimization on servers and distributed systems. He earned his PhD at the University of Texas at Austin, where he also currently teaches computer systems design and architecture. As a faculty member at UT-Austin, Mr. Suleman focuses on optimizing DevOps architectures involving the AWS cloud. Along with the Flux7 consultant team, he works at every step of the dev flow, from creating local developer environments using Vagrant and Docker, to continuous integration and continuous delivery using Jenkins or RunDeck, to configuration management using Chef/Puppet/Ansible, to creating efficient systems for deploying new code from Git using Capistrano/Fabric/bash. His background is as an architect, and he has experience with both hardware and software. This combination brings a unique perspective and broad view to development and operations.
John B. Dickson is an internationally recognized security leader, entrepreneur and Principal at Denim Group, Ltd. He has nearly 20 years hands-on experience in intrusion detection, network security and application security in the commercial, public and military sectors.
