Category: Blogs

  • Evolution Not Revolution – DevOps in the Wild

    Evolution Not Revolution – DevOps in the Wild

    Ever heard these questions:

    • What is DevOps?
    • How do I hire a DevOps Engineer?
    • Should we have a Director of DevOps?
    • Am I doing DevOps?
    • Why is DevOps failing in my organization?

    More and more frequently, I run into engineers, managers and business leaders describing failures in their organization due to “DevOps.”

    Oddly, digging a little deeper, I find that most of these cases are simply failure by substitution. The Operations team was failing, so they rebranded it DevOps. Alas, buzzword compliance was unable to change the trajectory of the team and the buzzword was blamed.

    DevOps today is–still–an ill-defined set of practices closely associated with automation and operations. If you accept this as fact, then DevOps can neither fail or succeed. If we want to measure the success of the DevOps movement in an organization, then we need a definition that leads to accountability.

    A Definition for DevOps

    The DevOps movement, unlike Agile, lacks a manifesto. To fill that gap, I assert that DevOps is a philosophy and that its base definition is:

    DevOps is the way in which a technology organization embeds itself in a business to the benefit of that business.

    The business in question need not be a technology company such at Docker or IBM. Indeed, all companies today are technology companies whether they like it or not. (1)

    In this and future articles I will write about philosophy, models and implementation. In short, a philosophy is what I reason from to create models of behavior. Models inform the actions taken in an implementation. Implementation without philosophy leads to failure. Jingoistic adherence to philosophy without accounting for new technology leads to failure. Balance must be acheived.

    As an example of this sort of failure, take a lesson from project management. Many shops adopt Jira because it can do anything. Since the tool can do anything, that means their workflow is capable of anything! Problem solved, QED. We all know tools are not process. When we look at our own track records of instituting change. too often the change stemmed from tool choice, not critical thinking about what was to be accomplished.

    Tools should reflect the philosophy they are implemented to support. Without that philosophy tools will fail. The more complex the problem or tool, the more spectacular the failure. In this example, Jira did not fail, leadership failed. Specifically the thought leaders entrusted with workflow management.

    Are you Doing IT DevOps Style

    DevOps is about evolution. If you are doing IT DevOps style, then you are asking questions like:

    • What should we change next?
    • How do we create a sandbox to measure the impact of the change?
    • How do we create demand for the change once vetted?
    • Who do we need to be patient zero?
    • What cultural paradigms are we shifting and how do we ease this for teams?

    While automate-all-the-things is often associated with the term DevOps, automation without cultural shift will result in broader failure, rather than success. Like any technology problem, at its core DevOps models account for their impact on people because businesses are composed primarily of people.

    Future Articles

    Over the course of the next few months. I will relate my experiences at several companies. Articles will describe:

    • What we changed.
    • Why we changed what we did.
    • What cultural issues we faced.
    • What technologies we used and why.

    I hope you will join me to learn how the practice of DevOps as a philosophy can lead to millions of dollars on the bottom line, happy bosses, fluffy bunnies and rainbows. If your lucky, you might even become a unicorn, though my own horn has yet to sprout.

    The first case study will describe a failing company with three months capital. An amazing team. Some amazing mistakes and, ultimately, some amazing success. Technology decisions driven by the needs of a sales team. Automation driven by a business model. A successful acquisition.

    References

    (1) The Phoenix Project – Gene Kim, Kevin Behr, George Spafford
    (2) Good to Great – Jim Collins

     

    About the Author/Boyd E Hemphill

    boyd-hempilBoyd E Hemphill is Director of Evangelism at StackEngine (link). , an enterprise grade, Docker operations management software company. DevOps raconteur and organizer of community events in Austin, find him on Twitter or LinkedIn.

  • DevOps – It’s all about continuous testing !

    DevOps – It’s all about continuous testing !

    I loved reading Steve Wilson’s blog post “Don’t Fear Testing Team. DevOps is Here!” https://devops.com/blogs/dont-fear-testing-team-devops-is-here/  The idea that DevOps  somehow reduces the need for testing  certainly puts fear in the hearts of QA and test teams who may envision the demise of product quality,  reduced test resources and even their very livelihood.  I understand the sentiment of the idea about testing going away, but this is misleading.

    Testing is important to DevOps. Prior to code integration to trunk, developers pre-flight test their code to make sure it doesn’t blow up the trunk. During the CI cycle and code commits from multiple developers integration testing is used to verify merges.  During the continuous testing phase where nightly and weekend regression runs are automated for trunk and release branches it is testing that does the job of finding problems before the software release.  During the development process the verdicts trends from testing are a primary measurement of progress. During the release deployment phase more testing verifies the release packages are ready for release.  Rest assured test team, you are needed.  What DevOps really does is bring testing into the mainstream of development processes, and avoids the problems created by having “big-bang” testing left to the end of the cycle, such as release delays, and quality issues.

    Testing maturity is a key differentiator of best practices DevOps infrastructures. My experiences with DevOps infrastructures over the years indicate that testing maturity is a key differentiator of overall DevOps infrastructure maturity. Many organizations can automate their integrations, builds and delivery processes but still have trouble with the subtleness of test orchestration and automation.  There is a vital role for testing architects and testing teams to offer their expertise for test design, test automation and test case development with DevOps.   Whether the organization is using a test driven development methodology, behavior based test creation, model-based testing or whatever they use, testing is a vital part of the overall DevOps process — not only to ensure verify code changes work and integrate well — but to ensure the changes do not blow up the product. So clearly testing is still an integral part of product development and delivery.

    At my company Spirent we think testing has a bright future in DevOps. You can read more about our views at http://www.spirent.com/solutions/devops

  • Metrics for DevOps

    Metrics for DevOps

    Your organization is now committed to DevOps methodology.  You’ve integrated development, operations, and QA, and you have moved drastically away from waterfall and into a fire hose of application development.  How do you find out how well it works?  How do you know if it’s working at all? Many net new DevOps organizations are surprised to find that it was easier to start then they thought, but hard to keep alive. This can be because of menacing habits, or just because the team is too far into the weeds. In a word, you need metrics

    (more…)

  • Change Management & Continuous Delivery: Clash or Harmony?

    Change Management & Continuous Delivery: Clash or Harmony?

    Introducing changes into running software systems is risky. There’s always a chance of failure, and that can have a negative impact on your business. It’s important to mitigate that risk, but there’s more than one way to do so.

    Change Management is all about ensuring that changes to existing systems and software in production are done in a controlled way. Change managers review the changes that are to be implemented, assess their impact, and perform due diligence, before giving or withholding their approval.

    Continuous Delivery (CD) is about improving the process of delivering software and increasing efficiency. It aims to cut down the time between ideation and realization. It’s about reducing the scope for error and increasing the speed to market to generate a competitive advantage.

    Is there an inevitable clash between CM and CD? Are they really compatible? At heart, the CD pipeline is intended to achieve the same thing as CM by enabling the smooth, error-free roll-out of changes, but today they are usually coming from very different angles.

    If CM is to have a future in the CD revolution, then it needs to change. Before we discuss how that might be achieved, let’s take a look at the current state of play 

    Is Change Management delivering?

    Ask yourself what impact change management is having on your organization. If change management was working perfectly then there wouldn’t be any production problems, and yet issues still crop up. There are a few things to consider:

    1. How many entries in the CM checklist could be automated? Is the change manager doing anything that really requires human input? Fact collection and verification could be a part of your CD pipeline.
    2. How much value do check boxes deliver in the first place? What is change management really bringing to the table?
    3. Is the process fit for purpose? Does it actually prevent problems, or is the change manager really just a fall guy, there to sign off on the release and take the blame if something goes wrong?

    Clashing with Continuous Delivery

    CM is intimately linked with the idea and practice of big complicated transformations, but with CD we’re not talking about enormous, disruptive changes. CD is about changing small parts of a system and doing it often. The risk to reward ratio is different when you think and work iteratively. Smaller change sets are less likely to cause problems and easier to rollback when they do.

    Does the process really require human input? Heavyweight validation will slow it down and speed is one of the key benefits Continuous Delivery offers. Streamlining and automating the checklist process makes perfect sense in many business contexts. The aims of CM and CD are in sync, but CD is better placed to deliver.

    A possible advantage of traditional CM is the presence of a human eye, offering intuition that a machine or a piece of software may not be capable of. By moving beyond the checklists and reimagining the CM role we can retain this advantage and push CD forward.

    From Gatekeepers to Masters of Continuous Delivery

    If the bulk of CM is really an administrative exercise in checking boxes then it could be an unnecessary overhead, but there is a way to retain that human oversight for organizations wary of full automation. Why not ask the former gatekeeper to own the CD process? For a CD pipeline to be truly effective someone has to guide the process, engage with all the teams involved and understand exactly why and how the process generates business value.

    A change manager, paired with a build or release engineer to handle the automation, could work to help organizations move towards CD. They are uniquely well placed to identify dependencies, map processes, and help construct the CD pipeline with minimal disruption. They can help sketch out what the pipeline should verify, what data and dependencies need to be taken into account, what automated tests are required, and how it fits with other components to automate that release procedure. They also have experience mediating between business and IT.

    The traditional CM approach of committee meetings and spreadsheet checklists is dated, and for many organizations it no longer makes sense. You can find much greater business value in focusing on an efficient rollback procedure that kicks in immediately and automatically when changes fail.

    Take a look at your CM and ask yourself – is it really adding anything to the process, or is it just another bottleneck? It may be time for a change.


    About the Author/Andrew Phillips

    andrew_PhillipsAndrew Phillips heads up product management at XebiaLabs. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.

  • There is No ‘I’ in Team: Transitioning Engineering Teams into a DevOps Model

    There is No ‘I’ in Team: Transitioning Engineering Teams into a DevOps Model

    As the pace of development and business continues to scale, teams need an agile and collaborative work environment to succeed. Moving to a DevOps model is a critical part of setting your engineering teams up to succeed, but making the transition can be challenging for many companies.

    What’s the deal? – Why you need a DevOps model

    DevOps is a movement in the software world  that focuses on collaboration between developers and operations, customer empathy and infrastructure automation. In traditional models, developers write code and then hand it to operations teams to deploy and and run in a production environment. This often leads to a lack of ownership between the two teams (“It worked on my local box!”) as well as a slower pace of development. In a DevOps model, by contrast, the two teams work closely with each other, and toward common, customer-facing goals – developers take ownership of their code even while it’s in production, and operations teams build tooling and processes that help developers write, test and ship code faster and more easily.

    By breaking through traditional developer-operations walls, development happens faster and with more customer empathy, ultimately resulting in a better customer experience and a happier team. Rather than seeing the other team as a gate or obstacle, if done right, developers and operations see each other as teammates working to deliver value to the customer and business.

    Unfortunately, introducing DevOps culture and practices to your company isn’t easy. Big changes to long-entrenched silos can generate resistance, and business leaders are often scared of the costs that come with change. To combat these concerns and hesitations, we’ve highlighted a few tips for transitioning to a DevOps model.

    Start with Collaboration – what’s working and what’s not?

    At the heart of DevOps is improved collaboration between your Operations and Development teams. You’re probably not starting from a vacuum, though – look at and talk about the best examples of how your operations and software development teams are collaborating today. Are there particular dev/ops groups that are already working tightly together, championing great communication or doing joint planning? Focusing on what is already working and how to make it work better, rather than what’s broken, keeps things positive and more likely to succeed.

    You could set up a group brainstorming session to discuss, or have one-on-one conversations with individual team members, depending on the format that you think will help people open up more easily. This article highlights opportunities for collaboration across the application delivery lifecycle, and could provide a useful framework for asking questions and identifying opportunities.

    In addition to the state of collaboration, you’ll also want to look for process successes and breakdowns in how you deploy and maintain applications, services and infrastructure. Identify the biggest opportunities for improvement – do your builds and test take hours? Are your 20 development teams all competing for the same three staging environments? Being able to quantify the impact of these pain points can help defend the transition to business stakeholders who may not have as much on-the-ground context about your teams’ pain points.

    It’s also important to build an understanding of the different perspectives operations and development teams may have. If your organization has traditionally been very siloed, thinking about other teams’ goals when planning and prioritizing may be a new mindset. Sharing each team’s objectives and what success looks like for them can help build this understanding and empathy. Focus on transparency (a framework like OKRs can help) to avoid conflict from mismatched assumptions about priorities.

    Create a plan  

    Once you’ve identified opportunities to improve the collaboration and workflow between developers and operations, it’s time to create a specific plan. Pick key opportunities and identify the first few steps to make a difference. Courtney Nash recommends starting with a group of people who are receptive to the vision and picking a small product or service as a test-bed. When implementing big changes, it helps to start small and scale from there after proving success.

    Engineers are not always known for their people skills, but transitioning to DevOps is as much about people as it is about process and tools. Throughout this whole process, you’ll want to be investing key stakeholders in your team and company in the transition. Change management sounds awfully corporate, but at it’s heart, it’s about making sure you build support for your projects and goals by collecting input from key stakeholders: gathering their feedback and goals, sharing and re-shaping your plan based on their input, and communicating your progress with them. It’s always easier to work in silos with people that already agree with you, but for lasting change, you’ll need to invest in building support across teams and throughout your organization.

    Finally, pick realistic timelines and objectives. It’s important to note that some effects of the change may take longer to realize as people ramp up on new tools and ways of working. Don’t discount or neglect the amount of time required for training, and try not to be fazed if the first project you pick doesn’t turn out to be a good one. As Courtney Nash mentions, Nordstrom.com realized their first project was not ultimately the right one to start with. Importantly, they didn’t give up, instead choosing a different, better-suited application.

    Measure and reflect

    While it’s rare for timelines in complex systems to happen exactly as planned, it’s important to timebox your work so that you can measure and reflect on how it went.

    After the specific milestone you’ve chosen, bring stakeholders together and run a retrospective. There are a lot of great resources out there to help with good retrospectives, but regardless of approach, make sure you make an intentional space to look critically at your results, celebrate success, and think about how you’d like to change or iterate the process for the future.

    When reflecting on progress, it’s important to be candid and honest while assuming the best intentions of your teammates. You are guaranteed to make mistakes – that’s inevitable with any new process – but remember (and help others remember) that you’re all working toward the same purpose and share a common set of values and goals.

    Don’t give up

    Adopting DevOps is a big change for an organization, and there will be bumps along the road. At the end of the day, though, the rewards of tighter collaboration among teams and a more empathy-driven engineering culture can be really powerful, and will show in product quality, velocity and team morale.

  • DevOps need to pay more attention to security

    DevOps need to pay more attention to security

    Ty Miller, founder of Threat Intelligence, suggests organisations adopting DevOps need to pay more attention to security.

    Over the last five years or so, the prevailing approach to IT security has changed – largely triggered by the activities of hacktivist groups such as Anonymous and LulzSec – and it is becoming an increasingly integrated part of the application lifecycle, he said.

    The less attention given to security in the earlier stages, the more likely it is that the application will have to be rearchitected and redeveloped before deployment, which can be 100 times more costly than building in security from the outset.

    Since DevOps is often adopted to allow more frequent code releases, it is important that security is built into the process from start to finish, as there is no longer time for extensive security testing between completion and release.

    Part of the problem is that developers tend not to be security experts, and because of their backgrounds security officers generally are not application security specialists. This can leave a security gap, he said.

    Miller’s recommendations include:

    Code libraries The reuse of known-good code instead of reinventing the wheel each time reduces the risk of building new vulnerabilities into software.

    Platform security While IT teams would not dream of omitting intrusion prevention systems and other security components from an on-premises system, “these sorts of things are quite commonly being left out” from cloud deployments possibly because they are thought to be insufficiently scalable – a symptom of the tendency to sacrifice security for flexibility. And organisations sometimes assume that SaaS products are secure, even though they can “become large scale data breaches.” Recent research suggests around 90 percent of SaaS offerings are not sufficiently secure for enterprise use, he said.

    Automated builds Automatically building systems based on pre-hardened templates helps reduce the risk of accidentally overlooking something and other human errors. If you can be confident that a new server will spin up in a secure configuration, development teams can concentrate on the code they are working on.

    Automated security testing Frequent releases are at odds with traditional security testing, such as engaging specialist penetration testers before a major release. “That’s thrown a spanner into the works,” Miller said. When do you do the security testing? Can you pinpoint everything that has changed since the last test, and focus testing efforts accordingly? “Automated security testing needs to be built into the DevOps process,” he advised.

    Fortify, AppScan and Burp Suite are among the more reputable tools, he said.

    HP’s Fortify http://www8.hp.com/us/en/software-solutions/application-security/ , a code review tool, “is very expensive [at up to $120,000 per year] but also very good,” according to Miller, even though it generates false positives as other tools do.

    IBM’s AppScan http://www-03.ibm.com/software/products/en/appscan/ helps with web and mobile app development, and costs around $30,000 per year, he said. While it has a good feature set and a convenient GUI, it does need a lot of setting up and maintenance when it comes to workflows.

    PortSwigger’s Burp Suite http://portswigger.net/burp/ is a semi-automated scanner for web applications. At $300 per seat it is relatively cheap, and “virtually every penetration tester around the world uses it,” said Miller. It might not be as intelligent as the big-name commercial products, but “it is a very accurate scanner.”

    But automated testing is not a panacea. Automated scans can find perhaps 20 percent of vulnerabilities, but that’s 20 percent less for humans to find. And the output of such tools tends to include many false positives, so some expertise is needed to interpret the results – an inexperienced user can waste a lot of time, Miller warned.

    For example, access control flaws can be hard to detect automatically. Does changing a number in a URL produce a different page in the same report, or a report for a different customer? Is any particular report public or private? A penetration tester understands the application and can easily decide whether or not there is a security issue, but the best an automated tool can manage is to warn there may be a problem here.

    An organisation that has a defined set of methodologies covering the entire process from development to production, implements security as a baseline feature in code and infrastructure builds, uses automated scanning and security testing, and integrates regular penetration tests and reviews “will be significantly ahead of other organisations,” Miller suggested.

    About the Author:

    stephenWithers

    Stephen Withers is one of Australia¹s most experienced IT journalists, covering everything from gadgets to enterprise systems. In previous lives he has been an academic, a systems programmer, an IT support manager, and an online services manager. Stephen holds an honours degree in Management Sciences, a PhD in Industrial and Business Studies, and is a senior member of the Australian Computer Society. You can see his Linkedin profile here: http://au.linkedin.com/pub/stephen-withers/0/222/687/

  • 15 must have Jenkins plugins to increase productivity

    15 must have Jenkins plugins to increase productivity

    Jenkins is a vastly used continuous integration tool for the IT industry which is built and used around plugins. It’s core is the Jenkins tool and there are 100’s of plugins available to enhance its power and usability.

    Let’s take a look at a few of the best plugins to increase the productivity.

    1. Global Build Stats Plugin–It’s essential to know your current capacity, usage and capability to prepare for capacity planning or system requirements, because the very first question is “what is your current capacity?”         global-build-stats

    You need to know how many builds are happening on a daily and weekly basis. How much time is taken by various builds, what is the waiting period. This plugin give you enough data to answer the question. This plugin can provide you data as graph which can be further consumed.

    1. Job GeneratorPlugin–In big or growing organizations, its a bit difficult to maintain the jobs for a project when developers are working on various branches and releases. You want to give access to developers to create their own job and at the same time you don’t want developers to create arbitrary jobs which may not fall under company standards. This plugin gives you flexibility to define templates and developers can create new jobs with the help of job generator template. Configuration access can be disabled via roll-based authorization plugin.
    2. Disable-failed-job –In rapid development environments failing jobs due to various unnecessary reasons are a pain, It’s difficult to keep track of such jobs and disable/delete them. This plugin can solve the problem by defining the upper number of consecutive failed builds and then disable automatically.
    3. Embeddable-build-status –This plugin can give you a link which can be pasted anywhere (example github project) to expose the status of the build and users can get the current state of the job while looking at project.
    4. Exclusion –This plugin enables you to handle conflicts between jobs. You can assign a resource (or lock) in multiple jobs, when build is executed, it will acquire the lock and other builds (if fired) will wait until lock is released.
    5. GitHub/GitLab Pull Request Builder-This is one of the best plugins I found to provide support to automate code review up to a certain extent in github/gitlab. This plugin does amazing job once defined for the project. For any new pull request this plugin does fox merge and runs the build on the code as well as gathers the necessary static analysis (if configured) or build results and comment back the status to pull request. This helps reviewers get an idea of the health of the code which is going to be merged. You can even define automatic merge if build passes.
    6. Hudson Extended Read Permission Plugin- During initial days of Jenkins setting configuration access was not available to any developer to make sure they do not change anything but developers used to ask to view configurations to know how job has been setup, they may want to see the build steps. Extended read plugin can provide option to developer to view the configuration without giving them write access.
    7. Post+build+task –There may be a need for performing some actions on the basis of the results of a build, for example if build passed you may want to upload artifact(ex debian) to some repo (apt) or perform some packaging part or similar. In case of failure you may want to roll back something (like release) . This plugin helps you to define the pass/fail criteria and let’s you decide what to do after that.
    8. JDK Parameter Plugin–This plugin is useful for an organization where many projects are using different versions of java. This plugin let’s you choose the java version during run time of the build.
    9. Job Configuration History Plugin-Ah! This is one of my favorite plugins. This plugin lets you keep track of config changes in each build including who did it. You can easily revert back to any previous config if you want.
    10. Multiple SCMs plugin–Default SCM section provides only one source control tool/URL option, what if you want to check out from more than 2 repos from multiple source control tools (like svn and git). This plugin will come in handy in such scenarios. This plugin will facilitate users to add any number of SCM URLs to checkout the code.
    11. Parameterized Trigger plugin– Another one of my favorite plugins. This plugin allows you to have user input as a variable and use on run time. This is the most used plugin in dynamic environments where you have lots of options and user-defined values to be used in the build which may keep changing.
    12. Pre SCM BuildStep Plugin- Just like post build task plugin, you may have requirements to perform some action even before checkout happens for the job, for example you may want to perform merging of the branch before the build and then checkout. This plugin can come in handy in various conditions and gives you flexibility when running the job.
    13. SCM Sync Configuration Plugin-backup, this is the most important task of any administrator. Without regular backup, the whole system cannot be reliable; this plugin provides you features to backup live Jenkins configs to any source control tool. It will keep committing the config files (including Jenkins and jobs) to SCM repository as soon as there is any change.
    14. Configuration Slicing Plugin – This plugin comes  in very handy when you want to make bulk changes in multiple jobs. This plugins allows you to change values of various fields like email, timer, shell script, configurations etc at one go.

    These are my favorites, but like I wrote, there are 100’s of plugins for Jenkins available.  What are some of your favorites?

  • Containers Are Designed For an Antiquated Application Architecture

    Containers Are Designed For an Antiquated Application Architecture

    In his most recent GigaOm article, “We’re finally headed towards autonomous, self-migrating containers for cloud-application automation“, David Linthicum discusses the value that containers, such as Docker and CoreOS Rocket, are bringing to DevOps and cloud systems management. For the record nothing Dave states in the article is incorrect with regard to the expressed value that containers provide for portability, however, there is an issue with the big picture view discussed for the role of containers in cloud application delivery.

    Containers are predicated on the goal of deploying and managing n-tier application designs. By their nature, containers manage n-tier application components, e.g. database servers, application servers, web servers, etc., at the operating system level. Indeed, portability is inherent because all operating system and application configuration dependencies are packaged and delivered inside a container to any other operating system platform. Containers are…

     

    –THIS STORY HAS BEEN MOVED TO CONTAINER JOURNAL–


  • Top Skills for Enterprise DevOps

    Top Skills for Enterprise DevOps

    There is one question I hear almost every time I talk with my large enterprise customers about DevOps:

    1. What skills do my people need to be successful with DevOps?

    Talk of DevOps almost immediately focuses on culture – like having empathy for fellow workers, being flexible and adaptable, seeking continuous improvement, building relationships, etc. However, while critically important in DevOps, culture is an outcome, not an input; and such attributes are mostly either innate or acquired slowly. Culture cannot easily be taught.

    At the other end of the spectrum, we often look at technologies – agile development, coding languages, virtualization/cloud platforms, management tools, etc. Again, these can be important, but while learning these skills might make you a better dev or op, they may not make you a DevOps practitioner.

    Like most transformations, DevOps will need new people, process and technology skills, so in this post I take a look at a handful of the most important, pragmatic, and trainable skills in all three areas for practitioners looking to adopt DevOps, especially in large and more complex organizations.

    People Skills

    • Communication skills – while this may not come easily to typically introverted IT staff, both written and verbal communications, within and across the organization, and even with customers, are essential to a strong DevOps approach. This skill is learnable from formal courses, groups like Toastmasters, direct mentoring, social activities, or informal sessions like ‘lunch and learn.’ For extra credit, jump on stage for open mic night at a local comedy club!
    • Business alignment – much of a dev or ops life is spent embedded in the technology of software delivery, rather than the business need. This can cause many issues, and is something DevOps tries to address. Understanding the business strategy, goals and priorities is essential to DevOps, helps drive and easily trainable. A ‘big picture view’ helps everyone collaborate on common objectives to deliver high-value business outcomes – a key tenet of DevOps
    • Workplace history – “an appreciation for history, both technical and organizational”* will help staff understand the critical antecedents of DevOps, so they can internalize its principles not just rote learn its actions. For example, understanding hierarchical and network structures, why we adopted open work environments (and why it hasn’t always worked), and the antecedents of DevOps in things like The Toyota Way will help staff not just to act predictably in known circumstances, but adapt appropriately to new ones. (*Thanks to @garethr for input on this!)

    Process Skills

    • Agile development – it may be possible to take a DevOps approach in a waterfall environment, but it is almost definitely less effective. DevOps arose out of the Agile movement, and they pair so well that it is sometimes referred to as ‘agile for infrastructure.’ Teaching on one or more specific agile approaches (Scrum, Kanban, Kaizen, etc.) will give staff an approach and mindset informed by agile practices to help teams work together to improve flow, delivery and quality
    • Lean approaches – ‘lean’ theories born in the manufacturing world (see ‘history’ above) give a pragmatic foundation for applied DevOps, not just new-age ideals. Concepts like Just-In-Time, Toyota Production System, Design for Manufacture, Theories of Constraint and Waste can be learned from original texts, courseware and many business courses. Some are also revised in modern IT-specific texts like The Phoenix Project or Implementing Lean Software Development
    • Process reengineering – DevOps is as much about refining and improving ‘flow’ in the SDLC as it is about encouraging a sharing culture, so classic learning about how to effect change in business processes is invaluable to early stage DevOps adoption. Understanding how and why to analyze needs, identify problems, and effect change in software processes, organizational structures, teaming etc. is essential to DevOps and a long-established body of knowledge

    Technology Skills

    • Coding/scripting – in its current iteration there is a lot of focus on helping operators to understand coding languages; and teaching developers to use script-driven operations tools. From JavaScript (eek!) to C# or .NET, any coding or scripting skills will help both dev and ops to have empathy for each other’s jobs, and help to develop the elusive ‘T-shaped people’ that are so well-suited to a DevOps approach
    • IT Automation – Enterprise DevOps especially will likely involve (require?) a raft of technologies in an automated tool chain. Because DevOps is concerned with end-to-end flow, such skills need to cover the whole SDLC. Look to provide training in Portfolio Management, Agile Project Management, Source Code Management, Version Control, Test Automation, Test Data Management, Service Virtualization, Continuous Integration, Release Automation, Cloud Provisioning, Configuration Management, Process Automation, Performance Monitoring and Event Management, whether through formal training or informal knowledge transfer
    • Workflow mapping – smaller organizations might get away with pen and paper, or even a whiteboard/flipchart, but to document, share, analyze and rationalize the SDLC for large, complex, enterprise applications, you will need to learn to use workflow mapping tools. This will allow team members across the org (and the world) to take ownership of the end-to-end process, and start to collaborate on continual improvement

    This is of course a (very) short list, but should give you a starting point for developing essential skills for enterprise DevOps practitioners. As always, please feel free to drop a comment and tell me what I missed, what is more important, or even what I got wrong. I am always interested in discussing ideas and learning new things!

  • The Top Metrics You Need to Track to Improve Operational Performance

    The Top Metrics You Need to Track to Improve Operational Performance

    Living in a data-rich world is a blessing and a curse. Flexible monitoring systems, open APIs, and easy data visualization resources make it simple to graph anything you want, but too much data quickly becomes noisy and un-actionable. At PagerDuty we’ve thought hard about what you should monitor and why from a systems perspective, but what about monitoring data on your operations performance? We’d like to share some specific metrics and guidelines that help teams measure and improve their operational performance.

     

    1. Raw Incident Count

    A spike or continuous upward trend in the number of incidents a team receives tells you two things: either that team’s infrastructure has a serious problem, or their monitoring tools are misconfigured and need adjustment.

    Incident counts may rise as an organization grows, but real incidents per responder should stay constant or move downward as the organization identifies and fixes low-quality alerts, builds runbooks, automates common fixes, and becomes more operationally mature.

    When looking at incidents, it’s important to break them down by team or service, and then drill into the underlying incidents to understand what is causing problems. Was that spike on Wednesday due to a failed deploy that caused issues across multiple teams, or just a flapping monitoring system on a low-severity service? Comparing incident counts across services and teams also helps to put your numbers in context, so you understand whether a particular incident load is better or worse than the organization average.

     

    1. Mean Time to Resolution (MTTR)

    Time to resolution is the gold standard for operational readiness. When an incident occurs, how long does it take your team to fix it? Downtime not only hurts your revenue but also customer loyalty, so it’s critical to make sure your team can react quickly to all incidents.

    While resolution time is important to track, it’s often hard to norm, and companies will see variances in TTR based on the complexity of their environment, the way teams and infrastructure responsibility are organized, industry, and other factors. However, standardized runbooks, infrastructure automation, reliable alerting and escalation policies will all help drive this number down.

     

    1. Time to Acknowledgement / Time to Response

    This is the metric most teams forget about– the time it takes a team to acknowledge and start work on an incident.

    While an incident responder may not always have control over the root cause of a particular incident, one factor they are 100% responsible for is their time to acknowledgement and response. Operationally mature teams have high expectations for their team members’ time to respond, and hold themselves accountable with internal targets on response time.

    If you’re using an incident management system like PagerDuty, an escalation timeout is a great way of enforcing a response time target. For example, if you decide that all incidents should be responded to within 5 minutes, then set your timeout to 5 minutes to make sure the next person in line is alerted. To gauge the team’s performance, and determine whether your target needs to be adjusted, you can track the number of incidents that are escalated.

     

    1. Escalations

    For most organizations using an incident management tool, an escalation is an exception – a sign that either a responder wasn’t able to get to an incident in time, or that he or she didn’t have the tools or skills to work on it. While escalation policies are a necessary and valuable part of incident management, teams should generally be trying to drive the number of escalations down over time.

    There are some situations in which an escalation will be part of standard operating practice. For example, you might have a NOC, first-tier support team or even auto-remediation tool that triages or escalates incoming incidents based on their content. In this case, you’ll want to track what types of alerts should be escalated, and what normal numbers should look like for those alerts.

    About the Author

    David Shackelford HeadshotDavid Shackelford is a product manager at PagerDuty, the leader in operations performance management. David works with teams across the company to plan, build, and ship features that improve operation teams’ quality of life, decrease time to incident resolution, and ultimately improve uptime for PagerDuty customers. Prior to PagerDuty, David worked in education technology, creating integrations between school information systems and digital content, and as a Teach for America corps member, teaching Mathematics in San Francisco public schools. You can follow David on LinkedIn: https://www.linkedin.com/in/dshackelford and Twitter: https://twitter.com/dshack