Author: Derek E. Weeks

  • State of the Software Supply Chain: Secure Coding Takes Spotlight

    State of the Software Supply Chain: Secure Coding Takes Spotlight

    After almost a year of research that involved studying 36,000 open source software projects, 12,000 enterprise development teams and 3.7 million open source releases, we at Sonatype are excited to share the “2019 State of the Software Supply Chain” report.

    This year, we worked with research partners Gene Kim, founder of IT Revolution, and Dr. Stephen Magill, principal scientist at Galois and CEO of Muse, to examine and empirically document objectively for the first time the attributes of exemplary development practices, especially in relation to secure coding practices. But, as in years past, we’ve also analyzed the rapidly expanding supply and continued exponential growth in demand for open source components.

    Not All Open Source Projects Are Created Equal

    For the past four years, we’ve studied the ins and outs of the software supply chain—what it’s comprised of; how vulnerabilities are getting in and how often; the growing regulations; and, most recently, a new trend in which adversaries are purposely attacking the supply chain with malicious components.

    For our fifth anniversary of the report, we wanted to look deeper. We wanted to understand exactly how enterprise development teams—and potentially even more importantly, how OSS projects—were thinking about and addressing the software supply chain security issues. We wanted to understand and identify the very best practices so we could share them with others.

    As a result of our research, we identified five common behavior patterns across 36,000 open source development teams. This includes identifying attributes of Large Exemplars and Small Exemplars who rest within the top 3%, or 1,229, OSS project development behaviors.

    To arrive at this list, we examined a large number of variables, including:

    • Do differences exist in how effectively OSS projects update their dependencies and fix vulnerabilities?
    • Are there exemplary teams that do this better than others?
    • Are components from exemplary teams more widely used than “non-exemplary” components?
    • What factors correlate with exemplary components?
    • What advice can be offered to producers of OSS components and the developers that consume them?

    The answers were quite striking—and the resulting data even more illuminating. While the report identifies Small Exemplars and Large Exemplars, we’ve also identified three additional groups of OSS projects: Laggards, Features First and Cautious.

    Secure coding: 2019 State of the Software Supply Chain

    Exemplary Commercial DevSecOps Practices Create Superior Software

    There are clear, competitive advantages for teams with exemplary DevSecOps practices.

    We’ve known for years that innovation is critical, speed is king and open source is at center stage. This research further underscores these accelerating trends throughout the software supply chain. It also shows that taming the supply chain is possible. By making better supplier choices, component selection and using automation, dev teams are seeing impressive rewards. In fact, for those development teams actively managing their software supply chains, the use of known vulnerable component releases was reduced by 55%.

    The report details 11 other behaviors and attributes of leading enterprise development teams, including their frequency of software releases, their use of repository managers and their reliance on a software bill of materials.

    Gene and Stephen helped shed new light on exemplary development and DevOps practices that I believe will help developers around the world better understand what secure coding means and how to start addressing it.

    — Derek E. Weeks

  • Reducing Risk in Applications Using Docker Containers

    Reducing Risk in Applications Using Docker Containers

    Docker. It seems like in this day and age you are either using Docker containers or you are going to use Docker containers. If you haven’t jumped on the bandwagon yet, check out a previous article, “Docker: The New Ordinary.” If you are on the wagon or are thinking about it but have concerns about their security, it’s time to read on.

    José Manuel Ortega (jmortega.github.io ) is a software engineer and security researcher in Spain. I recently watched his presentation online titled, “Testing Docker Images Security.” He gave an overview of typical Docker deployments, explained the attack surface and threats, presented how to detect vulnerabilities and outlined a couple of best practices. In short, his advice will help you learn how to better secure your Docker containers.

    (New to Docker? Read this paragraph; all others skip ahead.)  If you aren’t sure what Docker is, José offered this explanation: “Docker containers wrap a piece of software in a complete file system that contains everything it needs to run: code, runtime, system tools, system libraries—anything you can install on a server, regardless of the environment it is running in.”

    That is, containers are isolated but share an operating system and, where appropriate, binaries and libraries. Docker provides an additional layer of isolation, making your infrastructure safer by default. This makes the application life cycle faster and easier to configure, reducing risks in your applications.

    For starters, José laid out Docker’s default mechanisms for security:

    • Linux kernel namespaces
    • Linux Control Groups (cgroups)
    • The Docker daemon
    • Linux capabilities (libcap)
    • Linux security mechanisms like AppArmor or SELinux

     

    Screen Shot 2018-04-08 at 7.17.58 AM

    José walked through others tools, add-ons, best practices, etc., to increase Docker container security. I will cover most of them here.

    Docker Inspect Tool. The Docker Inspect Tool is built into Docker. It provides information about the host name, the ID of the image, etc. and it comes up when you start Docker.

    Docker Content Trust. It protects against untrusted images. It can enable signing checks on every managed host, guarantee integrity of your images when pulled and provide trust from publisher to consumer.

    Screen Shot 2018-04-08 at 7.19.01 AM

    Docker File Security. Docker files build Docker containers. They should not write secrets, such as users and passwords. You should remove unnecessary setuid and setgid permissions, download packages securely using GPG and certificates, and try to restrict an image or container to one service.

    Container Security. Docker security is about limiting and controlling the attack surface on the kernel. Don’t run your applications as root in containers, and create specific users for testing and policing the Docker image. Run filesystems as read-only so attackers can not overwrite data or save malicious scripts to the image.

    José provided a useful checklist to check the security of a Docker container, but it’s not a short one. Remember, if you are going to deploy hundreds or thousands of these containers, you’ll want to ensure consistent handling of security concerns to keep the hackers at bay:

      • Do not write secrets to Docker files
      • Create a user
      • Follow version pinning for base images, packages, etc.
      • Remove unnecessary setuid, setgid permissions
      • Do not write any kind of update instructions alone in a Docker file
      • Download packages securely
      • Do not download unnecessary packages
      • Use COPY instead of ADD
      • Use the HEALTHCHECK command
      • Use gosu instead of sudo whenever possible
      • Use no-cache (if applicable) when building
      • Enable Docker Content Trust
      • Ensure images are free from known vulnerabilities
      • Ensure images are scanned frequently throughout your DevOps pipeline
      • Ensure your images, packages are up to date
      • Use file monitoring solutions for image layers (if required)

    Auditing Docker Images. You can scan your images for known vulnerabilities with a wide variety of commercial and open source tools such as:

     

    All of these solutions can be integrated in one element of your CI/CD pipelines — some can be integrated in multiple places.

    José explored these solutions and best practices in more detail and offered up technical implementation tips in his full talk, available for free here. If you are working on or interested in Docker security, his talk is worth your time.

    Thirsty for more DevOps discussions led by practitioners (not vendors)?  You too can binge watch 99 other talks from the All Day DevOps conference here.

    — Derek E. Weeks

  • 200 Billion Downloads Can’t Be Wrong

    200 Billion Downloads Can’t Be Wrong

    Laurie Voss, COO and co-founder at npm (@seldo), tweeted recently that JavaScript packages downloaded from their repository has topped 4 billion. On an annual basis, that would be more than 200 billion downloads. Wow.

    Just as open source software turns 20 years old this year, these numbers are a testament to the incredible magic that happens when communities of developers openly share innovations.

    Laurie estimated that 12 million developers were responsible for the downloads. That means, on an annual basis, each JavaScript developer will download nearly 17,000 components. Imagine the time saved by not needing to code each of those functions from scratch simply by downloading a package in a millisecond from the internet.

    Software Supply Chains at Work

    What we’re witnessing here is a software supply chain in full effect. Open source projects contribute packaged code to the community, place it in public warehouses, and it’s then consumed by development teams around the globe to create new front-end, back-end or mobile applications for all of us. Every development team on the planet now utilizes a software supply chain that operates at insane speeds with massive throughput.

    In a second tweet, Laurie wrote, “Since we have ~12 million users, that means the average user installed the neatly round number of 1000 packages in 7 days. In reality, most individuals installed very few packages, and a bunch of CI build boxes installed 10s or 100s of thousands of packages each.”

    Essentially, there is an army of robots (automated CI tools) downloading billions of components from the web. While this type of automation is efficient, by itself it does not serve teams well.

    Repositories to the Rescue

    At Sonatype, we saw similar behavior by developers in the early days of Maven. Individual developers would download Java components directly from Maven Central. Maven users sitting right next to one another in the same room and on the same team would download the same versions of the same components needed for their builds, repeatedly. It was for this reason, among others, that our founders invented the Nexus repository manager.

    While repository managers have been used for quite some time in the Java community, Laurie’s post tells us that adoption of the technology still lags in younger, less mature, packaged code ecosystems.

    Companies looking for repository managers can look toward solutions like e-npm, an on-premises technology exclusively for JavaScript packages. For companies with polyglot development teams, universal repository managers provide full-function, (sometimes) free support for npm and 12 other component types, including Docker containers.

    Not All Components Are Created Equal

    I have often said that if you have 100 developers, you have 100 front doors open into your organization. If you have 1,000 developers? Then there are 1,000 front doors.

    If you dissect Laurie’s comment further, you will also realize that any developer can bring any package into your organization at any time. While this does make them more efficient, it also points to security and governance concerns. Every component was developed by someone else, donated out to the internet and consumed in the millions. But you have no true idea of the source or origin of many of the components.

    Research I have done in previous years at Sonatype demonstrated that 5.5 percent (1 in 18) of components downloaded by repository managers had known security vulnerabilities. While the percentage is not very large, keep in mind that a repository manager will only download a component once. Once cached, future downloads of that component are unnecessary and the components can be reused infinitely by that team.

    Sonatype analysis of more than 40,000 Nexus repositories reveals that the average repository holds more than 1,600 components. Deeper analysis of the 1,600 components housed in the average repository manager found 192 security vulnerabilities were present among the components (some components having more than one security vulnerability).

    Within software supply chains, repository managers and private container registries represent procurement gates into the development organization. The gates can be left wide open where component flows are not governed or they can represent opportunities for quality and security checkpoints that ensure defects are not passed downstream.

    What’s in Your Repository Manager is in Production

    Using data from more than 133,000 websites, researchers from Northeastern University showed that “37% include at least one JavaScript library with a known vulnerability. From a per-library perspective, at least 36.7% of jQuery, 40.1% of Angular, 86.6% of Handlebars, and 87.3% of YUI inclusions use a vulnerable version.

    “Alarmingly, many sites continue to rely on npm packages like YUI and SWFObject that are no longer maintained,” the researchers continued. “In fact, the median website in [NU’s] dataset is using a library version 1,177 days older than the newest release, which explains why so many vulnerable libraries tend to linger on the Web.”

    What is Wrong With 200 Billion Downloads?

    Nothing from an innovation standpoint. It’s awesome and we can all celebrate this achievement with Laurie.

    At the same time, it gives us pause to reflect upon other concerns of our day, including cybersecurity threats, technical debt and wasteful context switching.

    — Derek E. Weeks

  • Challenges in Leading a DevOps Team at a Fortune 100 Company

    Challenges in Leading a DevOps Team at a Fortune 100 Company

    A manager, an evangelist and a godfather all walk into a bar.

    Okay, this isn’t a bad joke, this is an article about how managers can implement DevOps in large organizations, especially where culture and organizational change hamper efforts. But, I promise, we will talk about managers, evangelists and godfathers.

    Uldis Karlovs-Karlovskis (@UldisKK) works at Accenture Latvia. Accenture is a global consulting giant, employing 400,000 with teams and clients across the globe. Uldis helped bring DevOps into the organization and shared his journey—the good and the bad—at theAllDayDevOps conference in his talk titled, “Lessons in Leading a Fortune 100 Team to a DevOps Philosophy.”

    Uldis joined Accenture in 2012 and faced three top challenges:

    1. How to maintain quality at such growth?
    2. How to enable continuous learning?
    3. How to stay fast and flexible at their size?

    His team was small, they held daily stand-ups that were productive, and they were flexible. Life seemed good, and then his team grew into a “spider.” They still held daily stand-ups, but they were useless because it was just people reporting and no one understood what is going on.

    What did Uldis do? Well, he didn’t freak out, if that is what you are wondering. Actually, he did. One morning he, “woke up, realized I had 25 people under me, freaked out and skipped the job for the day.” He walked around, thinking about it and wondering what to do. He thought about his three to four types of DevOps projects and all of the non-project activities that were monopolizing his time. He realized he focused on the urgent over the important tasks, working 10 to 12 hours per day.

    What is a manager to do? Have people help him, of course. Uldis decided to create an unofficial role that sounds interesting: the subject matter expert (SME). They would work on non-project activities, increase team engagement, improve satisfaction and assess DevOps maturity in projects. Initially, people were motivated to be SMEs, but they were always busy with other things. So the effort failed.

    Uldis had a new idea: Instead of him leading the SME team, he let them decide who wanted to do what, which SME team they wanted to join and who would be the lead SME for each team, and let them set their own goals and how they wanted to work, etc. This helped, but still didn’t solve the problems. Uldis was still burdened with the tyranny of the urgent.

    Back to the drawing board, and a new idea was born: evangelists. Uldis broke down 50 SMEs into seven subgroups and added a little hierarchy because the flat structure wasn’t working. Each evangelist led a guild—one of seven. Team members were to talk to their evangelist before coming to Uldis with questions.

    But, alas, Uldis was still just handling escalations and talks.

    So, what do you do to bring order? Bring in Godfathers.

    Now, team members could always go to the Godfather for help. Uldis decided that, generally, each person would have four topics they may want to talk about. For each topic, they have a person to talk to:

    • SME addresses how to contribute to the company in general
    • Godfather addresses escalations, such as not happy with the team lead
    • Evangelist addresses questions about the person’s skills
    • Team Lead addresses project tasks

    Uldis now realized the benefit of this approach: In a way, now, everyone is focused at some level on sales. As we all know, this is important yet very difficult in any organization—but especially when everyone is an engineer.

    While his approach has helped him and his team, he also acknowledged, “If I did the same thing today, I would fail, but maybe for different issues.”

    Todat, Ulids is looking at improving his culture. As we talk about at AllDayDevOps time and time again, good culture is key to successful DevOps. Uldis, looking at Accenture, noted it doesn’t fit into a typical organizational structure. Reading literature, industry buzz, etc., he realized something was missing from these models, but he wasn’t sure what. He knew they needed to be better.

    He discusses three dominant types or organizational cultures:

    • Pathological – Eight hours of work and your job is done
    • Bureaucratic – Two to 12 hours of work if the process requires it
    • Generative – Two to 12 hours of work and I’ll deliver it

    Uldis is trying to be at the third approach, and the key to achieving it is knowing that if you are having a bad day, you should leave your work for tomorrow when you will be more productive. Tomorrow it may take more than eight hours, but you will put your best into your work.

    His journey is ongoing, but he finished his talk covering a few things they do on his team:

    • Onboard new joiners through a continuous delivery pipeline
    • Meet weekly at lunch
    • Half-yearly performance assessments

    He also provided four takeaways:

    1. Everyone by default is good
    2. Intrinsic over extrinsic
    3. Engagement is your work responsibility
    4. Let them lead

    Uldis told an honest and open story about his journey with the hope it will help others. To hear it firsthand, watch it for free here. You can watch any of the 2017 AllDayDevOps sessions free of charge here.

    — Derek E. Weeks

  • DevSecOps: Don’t Invest In Hope

    DevSecOps: Don’t Invest In Hope

    A successful DevSecOps approach is rooted in action, not hope.

    There is a lot of investment in hope.

    I hope we won’t get breached.

    I hope our DevOps teams aren’t deploying thousands of vulnerable containers.

    I hope our developers aren’t downloading millions of vulnerable open source components.

    I hope our developers and security teams will figure out how to work together.

    I hope we won’t be fined under GDPR.

    I hope the hackers don’t notice.

    Yet, the fact remains that hope will not reduce breaches. Hope does not safeguard your containers or components. Hope cannot achieve collaboration. Hope won’t prevent a fine. Hope is not a strategy.

    Many organizations are considering their approach to DevSecOps. They are moving beyond hope to strategy. They are choosing action over indifference. They are taking the first step of their journey.

    Here are three ways to start your journey:

    Picture This

    We all learn from others. Here is a collection of 20 DevSecOps reference architectures. They reveal the choices and priorities others have made ahead of you. Look at what they did. Choose to do something similar, or create your own path from a mix of their ideas. If your canvas is blank, use these to draw something.

    Watch This

    The pioneers are the ones with the arrows in their backs. Another word for DevSecOps pioneers is practitioner. And many practitioners have navigated their journey successfully enough to share it. More than 15 practitioners shared the tale of their DevSecOps journey during All Day DevOps. Every session was recorded online. Every session is free. Start with this one from DJ Schleen at Aetna and then pick your next one.

    Start There

    In a panel discussion I sat on recently as a prelude to DevSecOps Days at this year’s RSA Conference, I heard some very practical advice (find the recording here). One of the other panelists, Stephanie Derdouri from Fannie Mae, advised our audience to start with one thing. Don’t pick a bunch of strategies and tactics. Just pick one painful, suboptimal or annoying practice tied to security and improve that element within your DevOps practice.

    Pick only one. Start there.

    Hope is not a strategy. Action is your only safe option. Take the first step.

    — Derek E. Weeks

  • Software Liability Goes Global

    Software Liability Goes Global

    This month, France turned up the conversation on software liability for manufacturers who place known defective software components in their products. But, they are not the first.

    Software Liability in France

    According to Lukasz Olejnik, lawmakers in France just suggested a desire to “put the security liability in hands of product suppliers. In other words, making companies responsible for the security of products they put on the market—for as long as the products are commercially available.”

    He continued: “The strategy then mentions that one of the solutions could be to release source code and documentation after an end of support date. The strategy itself mentions taking this discussion to the international level. This will be interesting.”

    You betcha, Lukasz.

    (If you can read French, you can find the 166-page proposal here).

    Software Liability in United Kingdom

    The need for improved cyberhygiene in the UK reached new heights in 2017 following large-scale ransomware attacks on its nation’s hospital system and an increased focus on software liability. According to the UK’s National Cyber Security Strategy 2016-2021 report, “Businesses and organizations decide on where and how to invest in cyber security based on a cost-benefit assessment, but they are ultimately liable for the security of their data and systems. Cyber attacks are not necessarily sophisticated or inevitable and are often the result of exploited–but easily rectifiable and, often, preventable–vulnerabilities. In most cases, it continues to be the vulnerability of the victim, rather than the ingenuity of the attacker, that is the deciding factor in the success of a cyber attack.”

    Shedding a spotlight on the cyberhygiene and software liability, Britain’s Information Commissioner’s Office (ICO)—the country’s data regulator, said a hacker exploited a well-known security flaw on a Gloucester City Council website months after the vulnerability had been widely reported on and updated component versions had been made available. The ICO fined the Gloucester City Council £100,000 in June 2017 for not preventing a cyberattack exploiting the OpenSSL Heartbleed vulnerability.

    Software Liability in the United States

    Last August, Sens. Mark R. Warner (D-VA) and Cory Gardner (R-CO), co-chairs of the Senate Cybersecurity Caucus, along with Sens. Ron Wyden (D-WA) and Steve Daines (R-MT), introduced bipartisan legislation called the Internet of Things Cybersecurity Improvement Act of 2017.

    According to a fact sheet released at the time, “While ‘Internet of Things’ (IoT) devices and the data they transmit present enormous benefits to consumers, the relative insecurity of many devices presents enormous challenges. This legislation is aimed at addressing the market failure by establishing minimum security requirements for federal procurements of connected devices.”

    The proposed legislation requires vendor commitments:

    • To ensure devices don’t contain known security vulnerabilities when shipped;
    • To ensure proper disclosure of new security vulnerabilities discovered within their devices; and
    • To prepare remediation plans for any IoT device where known vulnerabilities have been discovered.

    While the legislation is clearly aimed at consumer protections and privacy, it also focuses on quality, safety and regulatory standards applied to every other major manufacturing industry (thou shall not ship products with known defects).  The legislation specifically calls for vendors selling IoT devices “to provide written certification that the device does not contain, at the time of submitting the proposal, any hardware, software, or firmware component with any known security vulnerabilities or defects.”

    Software Liability in the European Union

    When the innovation race is being run without proper oversight, getting to the finish line safely will require greater (and faster) care. That’s set to be a major challenge for organizations developing software under the forthcoming EU General Data Protection Regulation (GDPR) in May 2018.

    Article 32 of the GDPR states that organizations must “implement appropriate technical and organizational measures” to “ensure the ongoing confidentiality, integrity, availability and resilience of processing systems and services.” When combined with Article 25, which mandates that data protection measures be implemented “by design and by default,” it’s clear that privacy and security must become ingrained in every element of IT infrastructure.

    If you fail to follow these rules and known software vulnerabilities end up inadvertently helping hackers steal sensitive consumer data, you could be on the hook for seriously big fines: up to €20 million, or 4 percent of global annual turnover, whichever is greater.

    Software Liability in Germany

    In July 2017, a consumer advocacy group in Germany has filed a law suit against a retailer in Cologne that sold an inexpensive smartphone made by Mobistel. The Mobistel model Cynus T6 was sold in Media Market stores for just 99 euros. Sounds like a great deal, right? Not so much. You see, the phone’s software came with 15 critical and known security vulnerabilities that were not disclosed to the consumer at the time of purchase.

    Instead, these security flaws were later identified by investigators from the Federal Office for Information Security (BSI).  Unfortunately for Mobistel and Media Market, consumer advocacy groups tend to fight back when manufacturers and retailers sell products to consumers without disclosing “essential information” such as known security defects in a smartphone.

    Although the complaint is at an early stage, it points to the possibility that companies manufacturing software applications could be held liable for selling defective products to consumers—in exactly the same way that auto makers have long been held liable for defective parts in cars.

    Think You Aren’t Shipping Vulnerable Software?

    Think again. According to Sonatype’s “2017 State of the Software Supply Chain Report,” 80 percent to 90 percent of a modern application is now assembled from open source and third-party components. Therefore, “It is incumbent upon development organizations to practice good hygiene when consuming open source components. Teams with suboptimal hygiene inevitably consume open source components with critical vulnerabilities. Last year, 5.5% (1 in 18) components downloaded from internet-based open source repositories contained known security vulnerabilities.”

    What does this mean for you? If you have not been paying attention, the chance that software your organization is building contains a known software vulnerability is nearly 100 percent.

    The Writing is on the Wall

    Organizations failing to manage software supply chains, who are unwittingly releasing vulnerable applications into production, will face increased liability due to gross negligence.

    The writing is on the wall. Are you prepared?

    — Derek E. Weeks

  • DevSecOps: Digging into Root Cause Analysis

    DevSecOps: Digging into Root Cause Analysis

    We have all been there in a postmortem when someone says, “Let’s get to the root of the problem.” And, we all know what that means: Who or what is to blame? We also all know that no one wants to play the blame game, yet we all do. But it isn’t our fault (no blame, see what I did there?). It has been the default system for solving problems in business for decades. It is called root cause analysis.

    We can change—for the better.

    I recently watched a presentation from Matthew Boeckman (@matthewboeckman) titled, “There is No Root Cause: Emergent Behavior in Complex System.” Matthew is a developer advocate with VictorOps and a technology strategist with Dryan.io. He grew up a systems guy and jokes that he has been in DevOps for 18 years, even though DevOps wasn’t around, because he has always been nice to developers.

    Digging in (pun intended), root cause analysis focuses on what went wrong, and how we can prevent it from happening again.

    null

    The core problems with root cause analysis for development is that it doesn’t provide for enough complexity and its natural focus is blame, which can undermine a positive DevOps culture.

    Root cause analysis was more applicable when waterfall was the development methodology because states stayed consistent for months or even years at a time. In the age of Agile, DevOps, CI/CD, microservices, etc., states of work are in a constant flux. Root cause analysis can’t provide solutions quickly enough. As Matthew notes, in root cause analysis, things are either good or bad, working or broken, uptime or failure. The reality is that our world is more nuanced.

    What Matthew recommends is to look at it through the principle of emergence because it, “separates judgment from the good and the bad binary approach to our system health, and instead focuses on behaviors and interactions, patterns and complexities of our system. With practice and effort we can manage them to more desirable states.”

    But what does this look like in practice?

    Getting back to the analogy of the tree and its roots, the answer is more of a forest than a tree. Trees are one living organism, forests are ecosystems.

    Matthew takes this philosophy and mental picture and gives us a better system: Cynefin. It is a Welsh word that means habitat, and was created by Dave Snowden (@snowded), originally for managing IBM’s intellectual capital. It draws on research in systems, complexity, network and learning theories.

    null

    Starting in the bottom right quadrant, working counter-clockwise, its goes from simple to more complex.

    Simple

    These are patterns or behaviors that don’t require a great deal of understanding. DevOps is increasingly setting up automated systems to respond to simple issues.

    Complicated

    These are known unknowns. You can imagine a set of realities where they can occur and they are probable, but not certain. For instance, a busy harbor might get a storm that causes damage to boats, docks, etc. It is hard for the harbor manager to manage and they need to think about it. This requires people to do some thinking, and it is difficult—if not impossible—to automate.

    Complex

    This is where we start to see emergent behaviors occur. We don’t have the metrics need to understand or manage these problems or you haven’t looked at that metric before. We start with probing, going into the system and exploring. Think of any collection of humans at any scale. Things are still in the scope of probable, but things change quickly. There are many moving parts that aren’t predictable and that we didn’t fully encounter in our test methodology.

    Chaotic

    This is, well, chaos. Matthews’ real-world example was when an entire region for AWS went down, causing other regions to be overloaded as system admins were moving services. In chaos, you act, then get a sense of where things are and then respond.

    Disorder

    In DevOps, this is where you have lack of communication and collaboration. Here teams need to: reduce: figure out what you agree on; analyze: build consensus; and, iterate: move to a quadrant and continue.

    Matthew notes that knowledge and practice move patterns toward more favorable quadrants. But, complacency erodes the process. Complex systems left poorly managed will create increasingly complex processes to manage.

    null

    How to adopt Cynefin

    • In the moment: Ask, what quadrant does this map to?
    • In the post incident report: How did we manage the pattern? Was it complicated, complex, simple? What can we do to change it?
    • In your sprint planning: Devote time to manage your patterns clockwise. What can we move with a little bit of work?

    null

    The reality is that root cause analysis is really only present after the fact. Cynefin calls us to action.

    Convinced that Cynefin might be just what your organization needs or want to dig a little deeper? Share and watch Matthew’s full talk here. You can watch any of the 2017 AllDayDevOps sessions free of charge here.

    — Derek E. Weeks

  • DevSecOps: If You Build It, They Will Come

    DevSecOps: If You Build It, They Will Come

    Spring training for Major League Baseball in the United States has begun. Millions of people share my love for baseball; however, the same can’t be said for security and compliance—well, at least not yet. Perhaps one day.

    Much like in the immortal baseball movie, “Field of Dreams,” if you build a friendly security and compliance system, the developers and operators will come. At least, that is the contention of Julie Tsai (@446688), senior director of security operations at Box.

    GettyImages-509936368

    I first met Julie at the RSA conference’s DevSecOps Days three years ago and have been following her musings since. I recently watched Julie’s presentation, “Build It and They Will Come-pliant: DevSecOps in the Real World,” from All Day DevOps. Here is a summary of what I learned:

    Good Architecture Sustains Sound Applications and Security

    Julie’s contention is that you can build a system that might actually bring a little joy to developers and operators, and it starts with realizing that, at the end of the day, we are all looking for good architecture. Good architecture sustains sound applications and security. It makes everyone’s life easier, so we all have a little time for baseball (or football or board games or even curling).

    The good news is that DevOps organizations are ahead of the game. Julie pointed out the old joke, “DevOps … isn’t that just where you give the keys to the developers?” Well, no, and it isn’t a specific tool or deployment. For security, it is about being lean on requirements, building compliance in from the start and integrating it across the development life cycle so you can build something that is secure and performant. Besides these being best practices for developers, they also allow you to use your code and policies to appease your auditors, as compliance is built-in. Bottom line: DevOps puts the rigor around security and compliance.

    It Takes a DevSecOps Village

    Julie pointed out a couple of takeaways from DevOps that help security and compliance:

    1. You can own a problem and be an individual leader. You can think globally but act locally by seeing across the silos and bringing a message of empowerment.
    2. It provides a path toward integration to internalize other groups’ values, bring them into your own words and ways and mutually reinforce and thrive. It also informs how people work together.

    Remove Friction to Scale DevSecOps

    The goal here is to ultimately get us something easier to scale and maintain and be compliant and secure. Julie outlines steps to get you closer to this idyllic system:

    1. Use configuration management because it leads to precision and more unified and verifiable work.
    2. Get rid of whatever you can to eliminate mistake factors. You will also get to a place to streamline workflow so there are fewer mistakes.
    3. Extend infrastructure as code. It can become configuration as policy and your audit trail can connect the intent to the execution.

    DevSecOps: Where Things Go Wrong

    With these in place, Julie contends you need to realize the world isn’t perfect, so you need to build a system that injects security and compliance where it is most effective. To find the areas in the development life cycle that it is important to inject, ask these questions:

    1. Where is someone intending to make a change?
    2. When you are releasing it live into your production stack?
    3. What is happening when something isn’t going as expected? Is it being changed back to where you need it to be, are you monitoring it, or are you doing nothing?

    DevSecOps: String Together Wins

    You also need to make it simple. She quoted Mark Burgess, “IT has a detail sickness,” noting that, “We are often burdened by complexity—we love it and dig right in, but it is important to understand the level of granularity you need. You need to look for the things that can make a critical uplift that gives you an incremental improvement. String together the wins.”

    In the end, Julie said to “try to reach a goal of being visible and streamlined and leverage the automation and technology in a way that is joyful in how we use it. It is not about a rigid process that is going to die soon; it is about what we are trying to bring to the whole world so that we work more efficiently and more technical.”

    The entirety of Julie’s presentation—including some live Q&A—is available here. If you missed any of the other 100 speakers at All Day DevOps, you can find their 30-minute presentations here.

    — Derek E. Weeks

  • Continuous Delivery: No Excuses

    Continuous Delivery: No Excuses

    Jez Humble’s (@jezhumble) career has spanned roles through coding, infrastructure, and product development across three continents and organizations of varying sizes. To say he knows a lot about continuous delivery is a total understatement. In 2010, he and Dave Farley literally wrote the book on continuous delivery—and if you have not yet read it, I would suggest you pick up a copy. It’s a fascinating read.

    I’ve heard Jez speak over the years at a number of DevOps conferences, and I was excited when he accepted our invitation to speak at All Day DevOps. At the conference, Jez presented “Continuous Delivery Sounds Great But It Won’t Work Here,” during which he addressed the four reasons he consistently hears from organizations on why continuous delivery won’t work in their organization:

    • We’re regulated
    • We’re not building websites
    • Too much legacy
    • Our people are too stupid

    Jez also outlined the real reasons behind the stated reasons: Their culture sucks and/or their architecture sucks.

    Jez spent his talk addressing each objection.

    We’re Regulated

    Jez noted it took Amazon four years to go from monolithic to service-oriented architecture (2001-2005) but the company did it. Do you think Amazon is the pillar of free enterprise, free of regulation? Not so. Being a publicly traded company, Amazon is heavily regulated; it is subject to Sarbanes-Oxley regulations; it handles personal and corporate financial and proprietary data; and it provides private cloud services to the U.S. government, including highly sensitive data for the Department of Defense.

     

    null

     

    Amazon Web Services’ GovCloud, in fact, provides the cloud platform for cloud.gov, the 18F project Jez worked on to address the authority to operate process within the government.

    The old process had screened every software application before it was allowed on a federal IT system. It stifled innovation because it duplicated efforts multiple times over. There was no cross-agency assumption that if one agency approved a software package, it was okay at another agency. Cloud.gov moved controls from operations into infrastructure as a service. Now, agencies can receive software through cloud.gov to integrate into their applications.

    null

    The bottom line is continuous delivery is excellent for heavily regulated environments because it gives assurance to regulators by removing opportunities for human error.

    We Aren’t Building Websites

    Jez’s favorite example is HP LaserJet firmware. HP had a 400-person team spanning three continents. Obviously, the firmware was critical for new printers, but the team was moving slowly. Only 5 percent of their time was spent on innovation. HP tried hiring, firing, insourcing and outsourcing. With business school tried-and-true solutions exhausted, and knowing it had to eliminate non-value adding work to increase innovation, the company looked to continuous delivery with two business goals in mind: 1) Get firmware off the critical path; 2) Achieve 10X increase in development productivity, measured by percent spent on innovation.

    HP had other goals:

    • Reduce hardware variation
    • Create a single package
    • Implement continuous integration
    • Implement comprehensive test automation
    • Create a simulator

     

    null

    Note: the chart above doesn’t show the 23 percent of time spent managing the automation tests. Time well-spent.

    In the end, between 2008 and 2011:

    • Overall development costs were reduced by ~40 percent
    • Programs under development increased by ~140 percent
    • Development costs per program decreased 78 percent
    • Resources now driving innovation increased by 8X

    Too Much Legacy

    Jez’s example here is Suncorp, one of Australia’s largest insurers (note: Also falls under the “We’re heavily regulated” excuse). Suncorp was burdened with loads of mainframes and legacy systems, but it built a set of test suites with open source tools to move into continuous delivery.

    Our People Are Too Stupid

    Jez looked outside of the software development industry into an old-line manufacturer: automotives. GM’s worst performing plant had been closed down. GM fired every one of the workers from the plant. The workers were notorious for problems ranging from sabotaging cars to drinking to gambling on the job.

    Remarkably, GM’s union leaders convinced Toyota to hire the old GM employees to learn the Toyota processes, and Toyota hired the GM employees to run their new NUMMI plant. Toyota and the old GM employees showed it wasn’t the people that was the problem; it was the leadership, the management system and the use of obstacles. At the GM plant, an employee had a job and a certain amount of time to do it. If it didn’t get done, the assembly line kept running. At the Toyota NUMMI plant, employees were empowered to ask managers for help—without retribution. The car had to be in a good state before proceeded to the next stage of the production line.

    In software development, this is continuous integration. Toyota’s NUMMI is the inspiration for Lean in the United States.

     

    null

    Jez’s take-home point for any organization is that results aren’t achieved when you adopt practices but don’t implement the culture. You need to:

    • Agree and communicate measurable business goals
    • Give teams support and resources to experiment
    • Talk to other teams
    • Achieve quick wins and share learning
    • Keep going

    Jez dives into more details in his talk, which you can watch here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.

    — Derek E. Weeks

  • DevOps Heals at NIH

    DevOps Heals at NIH

    Cancer sucks. But with folks like Sarah, DevOps is helping make a difference in the race to a cure.

    Sarah Elkins is not curing cancer herself, but she is employing DevOps practices to help those who are. Sarah supports the technology infrastructure for those who are trying to cure cancer at the National Institutes of Health (NIH).

    Screen Shot 2018-01-17 at 6.56.30 PM

    Sarah Elkins (@configures) configures technology solutions at the National Cancer Institute (NCI), where she has worked for nine years. NCI is a federal agency, part of the National Institutes of Health. It supports more than 700 websites, from basic HTML to content management systems to complex bioinformatics systems that have multiple tiers and thousands of servers. It uses multiple operating systems, containers and physical and virtual machines (both on-premises and some AWS). Developers range from lone scientists to large teams.

    Sarah and I have been following one another on Twitter for several years, but I recently caught the session she delivered in the “DevOps in Government” track at All Day DevOps. Sarah led a discussion about how the National Cancer Institute is automating its builds and deployments—showing it is possible even in a large bureaucracy.

    Her talk covered the processes and technologies that enable software to move from source code repositories all the way to production servers at nci.nih.gov, including the use of GitHub, Jenkins, Nexus and other technologies, with a variety of teams involved.

    We Need to Talk

    Sarah’s cure begins with engagement: development, infrastructure and security all working together, just as teams of specialists work with patients to help them beat cancer.

    To help speed development and approvals—a necessary evil that can grind development to a halt if not managed well—they agree that applications drawing on an existing technology catalog are approved as operational and security teams provide guidance to development on what is required for an Authority to Operate (ATO). In other words, they are trying to preapprove as much as possible and communicate earlier in the process.

    DevOps Practices are Maturing

    Source code is primarily kept on GitHub, including a public repository for non-proprietary source code, and NCI primarily uses Maven or Apache Ant for build scripts. Infrastructure teams provide XML templates to provide consistency.

    Most NCI software relies on build dependencies on open source software components. The organization uses artifacts during builds and utilizes Sonatype’s Nexus as its repository manager.

    All of this supports the automation of builds and deployments at NCI. For builds, most active projects are either on or migrating to Jenkins. Build artifacts may be .zip, .war/.ear or Docker images.

    For application deployments, NCI uses development, quality assurance, stage and production stages. Most teams use some form of automation, from simple (copy content, stop/start container) to more complex scripts. Developers are allowed to perform deployments for robust applications, and some manual orchestration is required—for instance, for database timing or related applications.

    As an organization, NCI is moving toward continuous integration (CI), with varying progress among teams.

    Building In Security

    NCI also now includes security early, often and automatically. The organization has deployed self-serve/on-demand Nessus scans, which allow developers to see, on their own, how they are doing as soon as the application is stable. Security teams run AppScan and Twistlock for Docker image scanning during Jenkins builds, just before deploying and frequently scanning the repository in between. For issues found, development and infrastructure teams work together to remediate security concerns.

    At the end of her talk, Sarah offered up three takeaways:

    1. DevOps at NCI is a work in progress (isn’t it everywhere?)
    2. The wide-ranging needs at NCI require flexibility, communication and teamwork
    3. There is no shortage of work

    Since it is a work in progress, what opportunities are ahead at NCI?

    • More automation for individual applications with Jenkins
      • Developers can perform container restarts on lower tiers
      • Database updates (some projects are using Liquibase already)
      • Some applications are still manual/batch scripted
    • More Docker and improving Jenkins/Docker instances
    • More orchestration automation and Puppet/Ansible integration
    • Security improvements through
      • Scan dependency artifacts before building
      • More integration and automation

    Sarah dives into more details in her talk, which you can watch here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. DevOps in Government is one of the five tracks.

    — Derek E. Weeks