Category: Enterprise DevOps

  • Early Automation: A Key Requirement for DevSecOps Success

    Early Automation: A Key Requirement for DevSecOps Success

    According to the “2017 DevSecOps Community Survey,” by Sonatype, almost 60 percent of the respondents consider security to be an inhibitor to DevOps agility, while more than 50 percent of developers say they do not have sufficient time to allocate to security.

    Hence arises the need for enhanced security automation in DevOps. Security automation helps in refining and improving application security by reducing the time taken for detecting an attack and responding to any identified issue.

    Tools for Security Automation

    Organizations looking to implement DevSecOps by integrating security into their DevOps pipelines have an increasing number of tools such as Tanium, InSpec, Splunk, Checkmarx, Metasploit, FireEye and Contrast Security for security analysis and testing throughout the software development life cycle (SDLC), right from source-code analysis to post-deployment monitoring.

    Early Automation, A Key Requirement for Success Through DevSecOpsFor security to become an integral part of the DevOps workflow, which includes continuous integration and continuous deployment (CI/CD), automation is essential. As most organizations are pushing tens of new versions of code into production for each application every day, it becomes necessary to embed security controls early in the development life cycle, which is possible only through automation.

    Around 40 percent of the 2,300 IT professionals surveyed have said that they have deviated from the waterfall development model, where automated security tests are run just ahead of production, by running them throughout the development life cycle.

    Using SAST Tools

    Static application security testing (SAST) tools help in scanning code and providing instant feedback to developers on security-related issues who, in turn, can remediate the potential vulnerabilities as part of the standard workflow.

    However, static analysis may not be alone sufficient for detecting all problems in the code.

    Automating DAST

    Automated dynamic application security testing (DAST) searches for vulnerabilities in real time while the application is running and is really a major improvement over static analysis that only looks for potential security issues in the code.

    Inclusion of automated security analysis helps in limiting the introduction of vulnerable code earlier in the development life cycle. The runtime analysis of the issues detected through automation enables developers to prioritize the code problems that need to be fixed.

    Red Teaming and Threat Modeling

    Organizations have begun to employ the concept of red teaming, a military terminology, by having separate teams to simultaneously take up attacking the code and performing security testing respectively. This ensures that any potential flaws are uncovered in the production stage itself and code fixes are pushed as soon as possible.

    Threat modeling is critical for the success of DevSecOps as it leads developers to look at software from an attacker’s perspective. While the threat modeling process cannot be automated, it helps in identifying flaws in the application design and architecture better than other security approaches.

    Are You Ready for Automation?

    Organizations can automate the provisioning and commissioning of DevSecOps environments to achieve a predictable, consistent and secure service delivery.

    While the goal of DevOps Services automation is to help development teams in faster deployment and monitoring of applications, DevSecOps adds security to the automation and contributes to enhancing the quality and efficiency of the software.

    DevSecOps adoption provides organizations with a strong foundation for mitigating risk in a proactive, efficient and streamlined manner.

    — Veritis

  • DevOps: It’s an Odyssey, not a Journey

    DevOps: It’s an Odyssey, not a Journey

    It has become somewhat of a standard practice to tell people, “DevOps is a journey, not a destination,” which attempts to convey the fact that there is no end point that is set in stone. But a journey is a well-planned, point-to-point trip, while DevOps transformation is a messy, figure-out-the-next-steps-as-you-go trip.

    So I propose we quit calling it a journey and start calling it an odyssey. This summer, when my mother takes her 10,000th trip back to Germany, it will be a journey. She’ll start in Cincinnati, Ohio, and travel through a couple of states before finally making her way to Hamburg, Germany. That’s a journey. She knows where she is going, and the discrete steps along the way.

    Her original flight from Germany? That was an odyssey. The trip went Berlin->Naples->Shanghai->Hawaii->California->New Jersey. Of all of that trip, only Berlin->Naples was a choice. Or, rather, the decision to go to Shanghai was a choice that required Berlin to Naples—very much in line with your DevOps choices making decisions for you along the way. And just like her family, you have no idea what your choices will be in a few iterations. They knew they had to go to Shanghai because that was the only port that would allow them in without visas, and they couldn’t get visas. After that? They knew they wanted to get to the United States or Australia (one assumes anywhere far from the insanity that was engulfing Europe and Asia during World War II), but how to get there was a total unknown. “First we start, later we finish,” was her father’s description of the goal.

    And that’s pretty much what DevOps is: First you start, and in the fullness of time you will discover things you hadn’t even considered when starting that just make sense and improve your overall position. There will be fits and starts, wrong turns and massive improvements. It is an odyssey, and one that is designed—just like my mothers’ family’s flight—to make things better in the long run.

    For example, the choice of Ansible or Puppet will make other choices easier. Giving teams more autonomy and increasing communications will make other choices more obvious. It will evolve into more than the first iteration’s goals would suggest, and that’s a good thing if increased agility and reduced manual intervention are the results.

    So keep on that odyssey, but don’t expect a simple end. More likely, the identification and acquisition cost of future improvements will cause more than the benefit they introduce, and you’ll find yourself in a better place, ready to breathe a bit and let the market find new ways to improve all things DevOps.

    Which is cool, because that should be your final goal: “We have improved so much that any improvements we could make right now would offer minimal incremental improvement for a significant cost.” At that point, your journey isn’t done, but you’re able to sit and take a rest, consolidate gains and assess future possibilities.

    And until you hit that point, be excited that every iteration of DevOps is improving responsiveness and taking you closer to your goal.

    Shameless plug: Read about my mothers’ odyssey here.

    — Don Macvittie

  • Challenges in Leading a DevOps Team at a Fortune 100 Company

    Challenges in Leading a DevOps Team at a Fortune 100 Company

    A manager, an evangelist and a godfather all walk into a bar.

    Okay, this isn’t a bad joke, this is an article about how managers can implement DevOps in large organizations, especially where culture and organizational change hamper efforts. But, I promise, we will talk about managers, evangelists and godfathers.

    Uldis Karlovs-Karlovskis (@UldisKK) works at Accenture Latvia. Accenture is a global consulting giant, employing 400,000 with teams and clients across the globe. Uldis helped bring DevOps into the organization and shared his journey—the good and the bad—at theAllDayDevOps conference in his talk titled, “Lessons in Leading a Fortune 100 Team to a DevOps Philosophy.”

    Uldis joined Accenture in 2012 and faced three top challenges:

    1. How to maintain quality at such growth?
    2. How to enable continuous learning?
    3. How to stay fast and flexible at their size?

    His team was small, they held daily stand-ups that were productive, and they were flexible. Life seemed good, and then his team grew into a “spider.” They still held daily stand-ups, but they were useless because it was just people reporting and no one understood what is going on.

    What did Uldis do? Well, he didn’t freak out, if that is what you are wondering. Actually, he did. One morning he, “woke up, realized I had 25 people under me, freaked out and skipped the job for the day.” He walked around, thinking about it and wondering what to do. He thought about his three to four types of DevOps projects and all of the non-project activities that were monopolizing his time. He realized he focused on the urgent over the important tasks, working 10 to 12 hours per day.

    What is a manager to do? Have people help him, of course. Uldis decided to create an unofficial role that sounds interesting: the subject matter expert (SME). They would work on non-project activities, increase team engagement, improve satisfaction and assess DevOps maturity in projects. Initially, people were motivated to be SMEs, but they were always busy with other things. So the effort failed.

    Uldis had a new idea: Instead of him leading the SME team, he let them decide who wanted to do what, which SME team they wanted to join and who would be the lead SME for each team, and let them set their own goals and how they wanted to work, etc. This helped, but still didn’t solve the problems. Uldis was still burdened with the tyranny of the urgent.

    Back to the drawing board, and a new idea was born: evangelists. Uldis broke down 50 SMEs into seven subgroups and added a little hierarchy because the flat structure wasn’t working. Each evangelist led a guild—one of seven. Team members were to talk to their evangelist before coming to Uldis with questions.

    But, alas, Uldis was still just handling escalations and talks.

    So, what do you do to bring order? Bring in Godfathers.

    Now, team members could always go to the Godfather for help. Uldis decided that, generally, each person would have four topics they may want to talk about. For each topic, they have a person to talk to:

    • SME addresses how to contribute to the company in general
    • Godfather addresses escalations, such as not happy with the team lead
    • Evangelist addresses questions about the person’s skills
    • Team Lead addresses project tasks

    Uldis now realized the benefit of this approach: In a way, now, everyone is focused at some level on sales. As we all know, this is important yet very difficult in any organization—but especially when everyone is an engineer.

    While his approach has helped him and his team, he also acknowledged, “If I did the same thing today, I would fail, but maybe for different issues.”

    Todat, Ulids is looking at improving his culture. As we talk about at AllDayDevOps time and time again, good culture is key to successful DevOps. Uldis, looking at Accenture, noted it doesn’t fit into a typical organizational structure. Reading literature, industry buzz, etc., he realized something was missing from these models, but he wasn’t sure what. He knew they needed to be better.

    He discusses three dominant types or organizational cultures:

    • Pathological – Eight hours of work and your job is done
    • Bureaucratic – Two to 12 hours of work if the process requires it
    • Generative – Two to 12 hours of work and I’ll deliver it

    Uldis is trying to be at the third approach, and the key to achieving it is knowing that if you are having a bad day, you should leave your work for tomorrow when you will be more productive. Tomorrow it may take more than eight hours, but you will put your best into your work.

    His journey is ongoing, but he finished his talk covering a few things they do on his team:

    • Onboard new joiners through a continuous delivery pipeline
    • Meet weekly at lunch
    • Half-yearly performance assessments

    He also provided four takeaways:

    1. Everyone by default is good
    2. Intrinsic over extrinsic
    3. Engagement is your work responsibility
    4. Let them lead

    Uldis told an honest and open story about his journey with the hope it will help others. To hear it firsthand, watch it for free here. You can watch any of the 2017 AllDayDevOps sessions free of charge here.

    — Derek E. Weeks

  • Transition to DevOps Increasing Across IT Firms

    Transition to DevOps Increasing Across IT Firms

    Demand for faster product development and quick market release created a need for increased collaboration between Development (Dev) and Operations (Ops) teams, which eventually led to “DevOps.” With this, transition to DevOps has become one of the key points of discussion across many IT firms. This article gives you an overview of the demand and trend for organizations to transition to DevOps.

    Rise in IT innovation and burgeoning digital revolution have led to the demand for faster development and quicker releases to stand out in the IT competitive scenario. This transition eventually led to a new culture that saw collaboration between two key IT teams, development and operations teams, which the industry calls DevOps.

    DevOps established a new style of IT functioning, easing different phases of software process from product development to testing and final market release, all in a rapid and reliable way.

    Why DevOps Transition?

    Rapid delivery of independent software components and continuous integration of the same with production systems forms the key principle behind DevOps functioning.

    DevOps is often confused with Agile methodology. Though they share similarities in providing improved IT performance through rapid development, DevOps stands apart from Agile because of its efficacy in the software deployment phase, ensuring a constant communication between the two teams. Thus, DevOps is often termed as a successor to Agile.

    A collaborative approach, key for information exchange across the line, is one of the major factors driving firms to undergo DevOps implementation.

    Though DevOps is well-known as a merger of development and operations teams, an effective DevOps model extends beyond that to other key functional areas of organizations.

    Breaking the old trend of limiting individual teams to their respective work domains, DevOps brought in a scope for knowledge-sharing among them, wherein developers can extend themselves to testing and deployment and operations team members can upgrade their skill set, learning new languages, architectures and tools.

    Increased Demand for DevOps Resource

    Transition to DevOps holds a key demand in the fast-growing IT market, wherein managing with the existing infrastructure or resource plays an almost equal role as coming up with something new.

    Greenfield projects aside, DevOps deployment is most often seen in brownfield projects that encourage building on the existing resources.

    With this situation in place, there is also a lot of demand for DevOps resources across firms. The “2017 Open Source Jobs Report” from the Linux Foundation shows 60 percent demand for DevOps human resources among more than 2,000 IT open source professionals and IT hiring managers.

    This clearly shows the level of demand that transition to DevOps is holding in the IT industry.

    “If there isn’t an existing DevOps role within your company, software developers should spend time sitting with the operations team or vice versa to understand how their processes work and how they can be improved through collaboration,” said Mike Durney, CEO of the job-hunting site Dice.

    The report found:

    • Nearly 50 percent of hiring managers are keen about those DevOps resources with professional credentials such as relevant certifications
    • More than 47 percent of the companies are ready to fund certification programs for their existing personnel
    • 42 percent of companies are keen on adding DevOps skills to their hiring portfolio
    • “DevOps Engineer” profile ranks second in Glassdoor’s Best 50 jobs in America

    Despite its advantages, however, DevOps has a few challenges to overcome related to application integration, test automation, budget and skill set, among others.

    — Veritis

  • DevSecOps: Don’t Invest In Hope

    DevSecOps: Don’t Invest In Hope

    A successful DevSecOps approach is rooted in action, not hope.

    There is a lot of investment in hope.

    I hope we won’t get breached.

    I hope our DevOps teams aren’t deploying thousands of vulnerable containers.

    I hope our developers aren’t downloading millions of vulnerable open source components.

    I hope our developers and security teams will figure out how to work together.

    I hope we won’t be fined under GDPR.

    I hope the hackers don’t notice.

    Yet, the fact remains that hope will not reduce breaches. Hope does not safeguard your containers or components. Hope cannot achieve collaboration. Hope won’t prevent a fine. Hope is not a strategy.

    Many organizations are considering their approach to DevSecOps. They are moving beyond hope to strategy. They are choosing action over indifference. They are taking the first step of their journey.

    Here are three ways to start your journey:

    Picture This

    We all learn from others. Here is a collection of 20 DevSecOps reference architectures. They reveal the choices and priorities others have made ahead of you. Look at what they did. Choose to do something similar, or create your own path from a mix of their ideas. If your canvas is blank, use these to draw something.

    Watch This

    The pioneers are the ones with the arrows in their backs. Another word for DevSecOps pioneers is practitioner. And many practitioners have navigated their journey successfully enough to share it. More than 15 practitioners shared the tale of their DevSecOps journey during All Day DevOps. Every session was recorded online. Every session is free. Start with this one from DJ Schleen at Aetna and then pick your next one.

    Start There

    In a panel discussion I sat on recently as a prelude to DevSecOps Days at this year’s RSA Conference, I heard some very practical advice (find the recording here). One of the other panelists, Stephanie Derdouri from Fannie Mae, advised our audience to start with one thing. Don’t pick a bunch of strategies and tactics. Just pick one painful, suboptimal or annoying practice tied to security and improve that element within your DevOps practice.

    Pick only one. Start there.

    Hope is not a strategy. Action is your only safe option. Take the first step.

    — Derek E. Weeks

  • Software Liability Goes Global

    Software Liability Goes Global

    This month, France turned up the conversation on software liability for manufacturers who place known defective software components in their products. But, they are not the first.

    Software Liability in France

    According to Lukasz Olejnik, lawmakers in France just suggested a desire to “put the security liability in hands of product suppliers. In other words, making companies responsible for the security of products they put on the market—for as long as the products are commercially available.”

    He continued: “The strategy then mentions that one of the solutions could be to release source code and documentation after an end of support date. The strategy itself mentions taking this discussion to the international level. This will be interesting.”

    You betcha, Lukasz.

    (If you can read French, you can find the 166-page proposal here).

    Software Liability in United Kingdom

    The need for improved cyberhygiene in the UK reached new heights in 2017 following large-scale ransomware attacks on its nation’s hospital system and an increased focus on software liability. According to the UK’s National Cyber Security Strategy 2016-2021 report, “Businesses and organizations decide on where and how to invest in cyber security based on a cost-benefit assessment, but they are ultimately liable for the security of their data and systems. Cyber attacks are not necessarily sophisticated or inevitable and are often the result of exploited–but easily rectifiable and, often, preventable–vulnerabilities. In most cases, it continues to be the vulnerability of the victim, rather than the ingenuity of the attacker, that is the deciding factor in the success of a cyber attack.”

    Shedding a spotlight on the cyberhygiene and software liability, Britain’s Information Commissioner’s Office (ICO)—the country’s data regulator, said a hacker exploited a well-known security flaw on a Gloucester City Council website months after the vulnerability had been widely reported on and updated component versions had been made available. The ICO fined the Gloucester City Council £100,000 in June 2017 for not preventing a cyberattack exploiting the OpenSSL Heartbleed vulnerability.

    Software Liability in the United States

    Last August, Sens. Mark R. Warner (D-VA) and Cory Gardner (R-CO), co-chairs of the Senate Cybersecurity Caucus, along with Sens. Ron Wyden (D-WA) and Steve Daines (R-MT), introduced bipartisan legislation called the Internet of Things Cybersecurity Improvement Act of 2017.

    According to a fact sheet released at the time, “While ‘Internet of Things’ (IoT) devices and the data they transmit present enormous benefits to consumers, the relative insecurity of many devices presents enormous challenges. This legislation is aimed at addressing the market failure by establishing minimum security requirements for federal procurements of connected devices.”

    The proposed legislation requires vendor commitments:

    • To ensure devices don’t contain known security vulnerabilities when shipped;
    • To ensure proper disclosure of new security vulnerabilities discovered within their devices; and
    • To prepare remediation plans for any IoT device where known vulnerabilities have been discovered.

    While the legislation is clearly aimed at consumer protections and privacy, it also focuses on quality, safety and regulatory standards applied to every other major manufacturing industry (thou shall not ship products with known defects).  The legislation specifically calls for vendors selling IoT devices “to provide written certification that the device does not contain, at the time of submitting the proposal, any hardware, software, or firmware component with any known security vulnerabilities or defects.”

    Software Liability in the European Union

    When the innovation race is being run without proper oversight, getting to the finish line safely will require greater (and faster) care. That’s set to be a major challenge for organizations developing software under the forthcoming EU General Data Protection Regulation (GDPR) in May 2018.

    Article 32 of the GDPR states that organizations must “implement appropriate technical and organizational measures” to “ensure the ongoing confidentiality, integrity, availability and resilience of processing systems and services.” When combined with Article 25, which mandates that data protection measures be implemented “by design and by default,” it’s clear that privacy and security must become ingrained in every element of IT infrastructure.

    If you fail to follow these rules and known software vulnerabilities end up inadvertently helping hackers steal sensitive consumer data, you could be on the hook for seriously big fines: up to €20 million, or 4 percent of global annual turnover, whichever is greater.

    Software Liability in Germany

    In July 2017, a consumer advocacy group in Germany has filed a law suit against a retailer in Cologne that sold an inexpensive smartphone made by Mobistel. The Mobistel model Cynus T6 was sold in Media Market stores for just 99 euros. Sounds like a great deal, right? Not so much. You see, the phone’s software came with 15 critical and known security vulnerabilities that were not disclosed to the consumer at the time of purchase.

    Instead, these security flaws were later identified by investigators from the Federal Office for Information Security (BSI).  Unfortunately for Mobistel and Media Market, consumer advocacy groups tend to fight back when manufacturers and retailers sell products to consumers without disclosing “essential information” such as known security defects in a smartphone.

    Although the complaint is at an early stage, it points to the possibility that companies manufacturing software applications could be held liable for selling defective products to consumers—in exactly the same way that auto makers have long been held liable for defective parts in cars.

    Think You Aren’t Shipping Vulnerable Software?

    Think again. According to Sonatype’s “2017 State of the Software Supply Chain Report,” 80 percent to 90 percent of a modern application is now assembled from open source and third-party components. Therefore, “It is incumbent upon development organizations to practice good hygiene when consuming open source components. Teams with suboptimal hygiene inevitably consume open source components with critical vulnerabilities. Last year, 5.5% (1 in 18) components downloaded from internet-based open source repositories contained known security vulnerabilities.”

    What does this mean for you? If you have not been paying attention, the chance that software your organization is building contains a known software vulnerability is nearly 100 percent.

    The Writing is on the Wall

    Organizations failing to manage software supply chains, who are unwittingly releasing vulnerable applications into production, will face increased liability due to gross negligence.

    The writing is on the wall. Are you prepared?

    — Derek E. Weeks

  • DevSecOps: Digging into Root Cause Analysis

    DevSecOps: Digging into Root Cause Analysis

    We have all been there in a postmortem when someone says, “Let’s get to the root of the problem.” And, we all know what that means: Who or what is to blame? We also all know that no one wants to play the blame game, yet we all do. But it isn’t our fault (no blame, see what I did there?). It has been the default system for solving problems in business for decades. It is called root cause analysis.

    We can change—for the better.

    I recently watched a presentation from Matthew Boeckman (@matthewboeckman) titled, “There is No Root Cause: Emergent Behavior in Complex System.” Matthew is a developer advocate with VictorOps and a technology strategist with Dryan.io. He grew up a systems guy and jokes that he has been in DevOps for 18 years, even though DevOps wasn’t around, because he has always been nice to developers.

    Digging in (pun intended), root cause analysis focuses on what went wrong, and how we can prevent it from happening again.

    null

    The core problems with root cause analysis for development is that it doesn’t provide for enough complexity and its natural focus is blame, which can undermine a positive DevOps culture.

    Root cause analysis was more applicable when waterfall was the development methodology because states stayed consistent for months or even years at a time. In the age of Agile, DevOps, CI/CD, microservices, etc., states of work are in a constant flux. Root cause analysis can’t provide solutions quickly enough. As Matthew notes, in root cause analysis, things are either good or bad, working or broken, uptime or failure. The reality is that our world is more nuanced.

    What Matthew recommends is to look at it through the principle of emergence because it, “separates judgment from the good and the bad binary approach to our system health, and instead focuses on behaviors and interactions, patterns and complexities of our system. With practice and effort we can manage them to more desirable states.”

    But what does this look like in practice?

    Getting back to the analogy of the tree and its roots, the answer is more of a forest than a tree. Trees are one living organism, forests are ecosystems.

    Matthew takes this philosophy and mental picture and gives us a better system: Cynefin. It is a Welsh word that means habitat, and was created by Dave Snowden (@snowded), originally for managing IBM’s intellectual capital. It draws on research in systems, complexity, network and learning theories.

    null

    Starting in the bottom right quadrant, working counter-clockwise, its goes from simple to more complex.

    Simple

    These are patterns or behaviors that don’t require a great deal of understanding. DevOps is increasingly setting up automated systems to respond to simple issues.

    Complicated

    These are known unknowns. You can imagine a set of realities where they can occur and they are probable, but not certain. For instance, a busy harbor might get a storm that causes damage to boats, docks, etc. It is hard for the harbor manager to manage and they need to think about it. This requires people to do some thinking, and it is difficult—if not impossible—to automate.

    Complex

    This is where we start to see emergent behaviors occur. We don’t have the metrics need to understand or manage these problems or you haven’t looked at that metric before. We start with probing, going into the system and exploring. Think of any collection of humans at any scale. Things are still in the scope of probable, but things change quickly. There are many moving parts that aren’t predictable and that we didn’t fully encounter in our test methodology.

    Chaotic

    This is, well, chaos. Matthews’ real-world example was when an entire region for AWS went down, causing other regions to be overloaded as system admins were moving services. In chaos, you act, then get a sense of where things are and then respond.

    Disorder

    In DevOps, this is where you have lack of communication and collaboration. Here teams need to: reduce: figure out what you agree on; analyze: build consensus; and, iterate: move to a quadrant and continue.

    Matthew notes that knowledge and practice move patterns toward more favorable quadrants. But, complacency erodes the process. Complex systems left poorly managed will create increasingly complex processes to manage.

    null

    How to adopt Cynefin

    • In the moment: Ask, what quadrant does this map to?
    • In the post incident report: How did we manage the pattern? Was it complicated, complex, simple? What can we do to change it?
    • In your sprint planning: Devote time to manage your patterns clockwise. What can we move with a little bit of work?

    null

    The reality is that root cause analysis is really only present after the fact. Cynefin calls us to action.

    Convinced that Cynefin might be just what your organization needs or want to dig a little deeper? Share and watch Matthew’s full talk here. You can watch any of the 2017 AllDayDevOps sessions free of charge here.

    — Derek E. Weeks

  • How to Get More Visibility Into Your Continuous Deployment Process

    How to Get More Visibility Into Your Continuous Deployment Process

    I know far too many developers who cringe at the thought of continuous deployment. Thoughts of broken code and an endless stream of customer support requests puts continuous deployment into the “nice to have, but not right now” pile.

    But, it might just be a case of clearing the fog on process so you get visibility into the deployment pipeline. If your team can get more visibility into releases, you will have more confidence your shipped code is quality. With more visibility, you’ll be deploying faster and earlier before you know it.

    What are the Results of an Effective Continuous Deployment Process?

    Shipping a big release is taxing on your QA and testing teams and the pressure to get every part of the build correct is huge. Continuous deployment allows you to deploy smaller chunks of code more frequently so you have more control over the release.

    Think “shipping responsibly.”

    With that in mind, it’s easy to see why a lack of visibility can hinder continuous deployment’s goal of shipping more code to a tight schedule.

    Why is Lack of Visibility a Problem?

    Best practices for your software development lifecycle (SDLC) demand that you know what’s going on in your code. And if you are responsible for your release process, you are probably held accountable for what customers receive.

    Production demands visibility into code quality; otherwise, you risk economic damage to your business in the form of bugs and crashes. With this in mind, understandably, many teams hesitate to continuously deploy thorough lack of confidence in the quality of shippable code through the build process. If more visibility means more confidence, let’s take a look at the stages in a build process and how to increase visibility at each stage.

    How to Get More Visibility Into Your Continuous Deployment Process

    Invest in modern architecture

    Continuous deployment will require resources and modern tooling initiated by your best people. Although budget often can be one of the biggest roadblocks to continuous deployment’s success, don’t withhold funds. Invest in the tools and people first, and the process will come. Modern tooling that is updated regularly will give you the best of what’s available, allowing you to effectively see what’s happening in each stage from source control to deployment and beyond.

    Allow for manual intervention

    You may not want your continuous deployment to be fully automatic. There are risks associated with continuous deployment, and to mitigate those risks you need a human to push the “deploy” button. It’s more about making your team feel accountable for their actions. The developer sees the change go live and is the first to test them.

    A viewable stage environment

    Do you have key stakeholders waiting for updates on feature releases? Rush boxes might be the answer. A rush box is a server used for work that is not ready for the QA process. This will give anyone to give feedback and authorize changes.

    Report

    Continuous deployment is certainly not without its risks. But, if you incorporate a culture of visibility into your processes, you can quickly identify where the gaps are so you can start to scale. Your toolkit will generate a huge amount of transactional data that will help to steer your company in the right direction.

    Monitor

    It amazes me how many companies still push out changes and rely on the end user to tell them whether the deployment was successful. Defects in production code have economic impacts on your company, and undiscovered bugs and errors can rack up into technical debt. But, you can’t change what you can’t see. Small changes are always going into production, so time for QA and regression tests naturally lessens. Therefore, you need to know what happened, and when, which is where a monitoring tool such as crash reporting can help. I’ve written before about the benefits of monitoring in continuous deployment, and it’s an integral step that is often missing. Monitoring can also help you make informed decisions about when to rollback a deployment.

    Rollbacks have an economic impact to your business, much in the way errors do. However, if a decision to roll back is data-driven, it’s an easy call for a senior developer to make.

    In Conclusion

    Don’t let the lack of visibility into your deployment pipeline affect your decision to implement a continuous deployment process. Invest in people and tooling first, and the process and scale will come.

    — Freyja Spaven

  • Sneckdowns In IT Operations

    Sneckdowns In IT Operations

    What IT operations professionals can learn from the actual paths people take through tools and processes, and the differences from the expected usage model.

    It has been a snowy winter across much of Europe, and although the snow is now beginning to melt in most places, there are still substantial sneckdowns to be seen across our cities.

    If you’re not familiar with the term “sneckdown,” it describes those areas of streets that are not actually used, as revealed by the unmarked snow that settles on them.

    Sneckdown example

    Transportation planners have even been known to use sneckdowns successfully as planning aids.

    The world of IT should learn from this example. We may not have anything as whimsical as snow to help us, but on the other hand, we do have enormous amounts of data. Let’s investigate some examples, starting with IT operations.

    Restate My Assumptions

    IT operations has a bunch of assumptions baked in, but just like with street layouts, some of them may be obsolete, leftovers from a previous generation of technology. My own hometown dates its founding back to Roman times-218 B.C., to be precise-but the sorts of design choices that Roman colonists might make to facilitate chariot and ox cart traffic have had to be updated significantly-first to accommodate motor vehicles, and then again to ensure that the motor vehicles did not crowd out other forms of transportation such as bicycles.

    Similarly, much of IT operations is based around the notion of a ticket, which describes and encompasses an atomic problem. This ticket has a single owner at any one time, and is processed sequentially through a series of steps. This is the operations equivalent to the waterfall model of development.

    These assumptions used to be reasonable in previous eras of IT technology, when a simple failure of a single piece of equipment was enough to trigger an incident. That incident ticket could easily be paired with the specific piece of equipment that was impacted, and from there a simple lookup would return the responsible group, who could then be assigned the ownership of the ticket.

    These days, that model is largely obsolete. Hardware, especially virtual hardware, is cheap enough to be made multiply redundant. Automated functions such as load balancing ensure that single failures are rarely enough to cause an actual incident. Instead, multiple failures have to occur in the right sequence—or the wrong one, depending on your point of view. Meanwhile, operators are drowning in a sea of red alerts, unable to find and focus on the few actionable ones because processes require them to respond to everything. Finally, when a real ticket does make it through the noise, it gets bounced around from team to team, as harried operators try to work out whether it’s their problem or somebody else’s.

    Operators are human, after all, and over time they develop their own paths through and around this rigid process, just as people walk and drive their own paths through the snow.

    Examples of IT Operations Sneckdowns

    Users Working Together

    One big IT operations sneckdown is collaboration. Instead of using the ticketing system and its rigid assumptions, operators work around it, using alternative methods to work together and share information with each other. The goal is to deliver full-stack operations for the entire complex business service that is being supported by the team.

    The problem with sneckdowns is that if there is no city planner watching and measuring, eventually the snow will melt and leave no visible trace of the paths that people take around their city. In the same way, the risk to the wider organization is that as people work and collaborate in alternative, unofficial tools—whether IRC or Slack—information gets stuck there and does not make it back into the official knowledge base, where it is available to be consulted and reused over time.

    Sneckdown Recommendation: Recognize that the nature of modern IT and network architectures means that there may be no single owner of a problem. The rapid rate of change, too, makes it almost certain that any owner will not be documented in a static database somewhere. Instead, modify processes to recognize this multiowner approach, and enable users to work easily with each other without having to play ticket ping-pong.

    The Right Tool for the Job

    This overlaps with another sneckdown: unofficial tool adoption. In the same way that operators might start using an unofficial collaboration tool to route around limitations in the official service desk process, they might look at Kanban to share task status, or start building libraries of automated diagnostic and remediation actions. The problem is that the rigid official policy often does not accommodate these extensions, and indeed has no mechanism to do so.

    Sneckdown Recommendation: Modern operations tools need to offer flexible interfaces to enable integrations and extensions to be done in a safe way. This way, if a particular integration does not work out or needs to be upgraded, that can be done without disruption to the wider process and to any other integrations that are working well.

    Look into the Shadows

    The final sneckdown—at least for this snow storm—is shadow IT. In the same way that much of our universe is theorized to be made out of dark matter, not visible to our current tools, a growing proportion of IT spending is happening outside the “official” IT budget. This also means that whatever is bought with that budget is outside the official IT operations and support process. That’s all very well on paper, but if the tool Sales or Marketing were relying on to do their jobs goes down, somehow, IT operations is still going to get the call to fix it.

    Sneckdown Recommendation: There isn’t an easy short-term fix for this one—no way to shovel it out of existence. Instead, IT operations professionals need to work on this issue from two angles. The first is to address the causes of shadow IT, those rigidities and frictions that make people look for outside solutions rather than engage with their in-house IT departments. When everything is properly aligned, the IT department has a lot to offer in terms of understanding of How Things Are Done inside the organization. The other angle is—yes—to make it easier to accommodate solutions that might have been procured originally as shadow IT. Figure out how to engage with support processes for SaaS solutions that people adopted on their own; if it’s working for them, it’s up to you to keep it working! The benefit is that this alignment will reduce the need for constant injections of energy by sysadmins and operators to keep things running: the Second Law of IT Ops.

    Conclusion

    Compared to city planners and transportation activists, we IT professionals actually have it pretty easy. We’re not reliant on particular weather conditions for our data, but rather, we have enormous amounts of data at our fingertips all the time. It’s up to us to look at the data with open minds and read the conclusions that are already there. The hard part, of course, is putting those conclusions into action, especially when that requires changes to widely accepted processes, but that is also an opportunity to revisit those processes and make sure they are still as fit for purpose as when they were first adopted.

     

    Game Day Chicago

    If you’re interested in what that process looks like, there is an interesting event series: Game Day ITSM & Incident Management Workshop. The next Game Day is in Chicago on March 21, but there is a whole series running, so there is probably an event near you soon. It’s a fun time, and also a great opportunity to rethink some assumptions and look at where some sneckdowns might be that could be incorporated into your planning.

    — Dominic Wellington

  • Defects: Pay More Attention to What, Less to Who

    Defects: Pay More Attention to What, Less to Who

    Software defects have been with us forever, and while DevOps and agile can reduce the number of defects generated, it is unlikely this number will ever get to zero for a given organization—and even if it does, it won’t likely stay there. Simply put, we’re developing increasingly complex systems on increasingly short timelines. The reduction possibilities that DevOps offers cannot completely combat this combination.

    But the quest to keep defects down is necessary. The goal should always be “Less, not more.”

    One of the things we have historically done is associated defects with developers. It is natural to want to know “who broke the entire payroll system,” but quite often “who” is the wrong question.

    A huge benefit of both agile and DevOps is that it allows us to look closely at where the process fails and determine why. While every developer in a shop knows “you should avoid modifying that module,”—and those are the easy ones to target for early bug reduction—there are other modules and source files that might be a hidden source of a certain percentage of defects. Using the defect tracking toolset to keep tabs on where bugs are occurring might show that the problem is a given developer, but more likely it will show that the problem is a given set of source.

    So if you are not already, it is time to start tracking defects closely and looking for patterns. There are some great documents out there about relatively common break points that might be useful. However, reviewing your own data is more useful, as it isn’t about common—it’s about actual problems in your source pool.

    This is all part of the process for agile and DevOps, I’m just saying it a different way than the usual “blameless” discussions because I think they fail to connect for some shops. Of course blameless is great, the point here is focus on problems—whether that is blameless or not depends upon which problem you find you have.

    Meanwhile, keep cranking it out. If you’re working on a viable product, then defects are an annoyance, but not something to gnash your teeth over. You are developing a viable product, after all.

    — Don Macvittie