devopsy.kinsta.cloud

Category: DevOps Culture

  • How to Fix ‘Dark Debt’ in Your IT Organization’s Culture

    How to Fix ‘Dark Debt’ in Your IT Organization’s Culture

    Poor decisions, bad metrics, lack of communication can accrue dark debt in any company

    Buying a low-quality microwave meal from a gas station, kicking a sock under the bed before guests arrive, jaywalking … We all take shortcuts. But unlike enterprise software delivery, these decisions rarely cause any collateral damage. The ramifications of taking shortcuts in IT, however, are much more serious.

    These decisions—i.e. “technical debt,” the “rework” debt owed after opting for a convenient solution instead of a better approach that takes longer—has implications on your company’s ability to grow and adapt to change. Fortunately, since the 1990s, when the term was coined by Ward Cunningham, we’ve learned how to identify and manage most forms of technical debt, understanding how to limit its impact and manage the interest.

    Your organizational culture can accrue debt, too. For example, a company that finds it easier to hire a homogeneous team up front might discover the costly cultural trade-off of hiring diversity later. When it comes to this “cultural debt,” we are much less disciplined. We often make decisions in the dark, without understanding how they are going to play out over the long term. “Dark debt” builds up over time and creates a toxic work environment in which people are not set up for success.

    Common issues indicating dark debt include:

    • Failures lead to finger-pointing.
    • Metrics incentivize undesirable behavior.
    • Lack of candor.

    Ways to Combat Dark Debt

    Call Out Psychological Safety Indicators

    Failures are inevitable, whether we are discussing power outages, data breaches or false missile alerts. A “safe to fail” mentality promotes learning and support adaptability. If employees don’t feel safe telling the boss bad news, the latter might be the last person to know what’s going on—not a good situation for a company working to disrupt an industry, or one expecting to stay listed on the Dow Jones.

    A Net Promoter Score-like survey can help identify areas where employees experience dissatisfaction. Here are some sample statements for your NPS survey that are based on Westrum’s organizational culture model as introduced in the “2014 State of DevOps Report.”

    • On my team, failure causes inquiry and not blame.
    • Our leadership is open to hearing bad news.
    • In my organization, failures are learning opportunities and messengers are not punished.
    • People on our team trust one another.

    The survey results will shine a light on the forces shaping your culture. Instead of finger-pointing, you can learn to address issues earlier and ask questions such as, “Why did it make sense for someone to do that at that time?”

    Measure Thoughtfully

    Tell me how you’ll measure me and I’ll tell you how I’ll behave. Gaming metrics are fairly easy to do, especially when an employee is incentivized to hit an unrealistic target or avoid embarrassment. Take activity metrics, for example (lines of code, number of bugs): These metrics focus on keeping people busy. But busyness does not equal business value delivered. People can be 100 percent utilized sprinting from meeting to meeting while making scant progress on business goals.

    Instead, consider metrics that reveal progress toward important business objectives or ones that help you make better business decisions. Flow metrics such as flow load, flow time, flow efficiency, and flow velocity reveal trends that measure four key areas: workload, speed, waste and throughput (respectively). These metrics are helpful in numerous ways, such as:

    • When others imply that things take too long, teams work inefficiently or people aren’t fully utilized, it’s useful to test their opinion against the data.
    • Having a balanced set of metrics allows you to see how changes in one area impact other areas. If the amount of work-in-progress (flow load) goes up, things will take longer to do (because of context switching and conflicting priorities), which results in slower speeds (longer flow times).

    Think about the behaviors that occur in your organization by how metrics incentivize people. What we measure impacts people because people value what is measured. Think twice before automatically defaulting to free and easy metrics available in your existing toolset.

    Build Social Capital

    Ideas start out flawed and incomplete. High levels of social capital, debate and discussion are the means by which ideas flourish. Social capital is what enables people to collaborate more effectively. The absence of social capital makes it impossible for people to speak openly (due to the lack of trust), resulting in withheld feedback and invisible work.

    Not much happens without discussion. When we engage in debate, our capacity to see each other’s perspectives is realized. TED talk speaker Margaret Heffernan’s insight on social capital is compelling: “Time compounds social capital.” The longer teams work together, the more trust, helpfulness and candor they accrue. Longstanding teams, therefore, benefit from increased productivity and reduced risk.

    Unlike the costly and time-consuming transitions observed in project-managed work, where people disband after dumping their work on the operations team before starting another project, organizing teams around a product allows the people who developed, tested and delivered the functionality to not only stay in their area of expertise, but also grow and gain from social capital.

    — Dominica DeGrandis

  • A Vendor Guy Goes To Monitorama

    A Vendor Guy Goes To Monitorama

    I attended my very first Monitorama this year, in Amsterdam, and I have thoughts!
    Monitorama Amsterdam
    Monitorama Amsterdam

    First of all, it’s deeply weird speaking at Monitorama when you work for a vendor in the monitoring and observability space, but you’re not there as a sponsor. This would be pretty odd at the best of times, but the crowd at Monitorama are next-level. They are very comfortable with highly technical content, so they get irritated very quickly with (what they perceive as) marketing fluff. “Down in the weeds” doesn’t even begin to describe it; most people had never heard of the sorts of things that are part of life in enterprise IT (AIOps, Netcool, Patrol or any of our usual suspects), but, on the other hand, everyone I met could engage in a multi-hour argument over which monitoring tool is worst.

    In fact, I would say that one of the highlights of Monitorama was the conversations in the hallway track. This is where all the Twitter loudmouths—I mean, “influencers” and “thought leaders”—who drive this industry congregate. If you manage to get into a conversation with them over coffee or beer, both of which were flowing freely, there can be a real exchange of ideas. I know I learned probably as much from those small-group chats as from the presentations on the big stage. I hope I was able to share some perspective in return.

    Heck, as ridiculous as it sounds, even the dress code was a challenge. At the events I attend as a vendor, there are basically two speeds: full suit-and-tie battle dress or a T-shirt with my employer’s logo. Neither case requires much thought when packing. Monitorama, though? I’m no longer credible in my old Thinkgeek shirts from my sysadmin days, but I’ll be laughed out of the room if I show up in a tie.

    Preparation

    I had answered the Call for Papers with a pitch about AI for distributed systems. Here’s my abstract:

    The IT industry is in the middle of one of its regular swings between centralization and decentralization, driven by increased automation of the network fabric itself, as well as new use cases such as IoT. With more and more processing and autonomy devolved to the edges, old assumptions about how to manage and operate a network have to change. It no longer makes sense to try to forward all the edge alerts to a central location for analysis, but central visibility on the health of the network is more important than ever.

    New techniques in AI-enabled observability hold the promise of helping NOC teams deliver better experiences for users of their networks, without requiring excessive manual work or heavy impact on on Operations teams’ personal lives. Machine learning enables knowledge to be captured and made available in context when it is most useful, accelerating incident resolution and improving user experiences.

    My first hurdle was that, although I have public-speaking engagements at least once or twice a month—and have done for some years now—I normally appear under the flag of my employer. This fact has both pros and cons. On the positive side, I can usually rely on a substantial library of slides that I (and others) have created in the past. I have some known high notes that I will be aiming to hit, and the call to action at the end is pretty explicit (“try our stuff”). The other side of the coin is that, because the audience is aware of these factors at least at some level, I have to work that much harder to earn their attention, as they assume they are being sold to. As a vendor spokesperson, I am also somewhat constrained in how wacky and off-the-wall I can get—although I do always try to push the boundaries. At one speech last year, I talked about bicycles for several minutes, to the consternation of the organizers, until they worked out what I was building up to.

    So I slaved away for some time over a hot Keynote, watched videos from past Monitoramas, consulted people in the industry whose opinions I respect and rocked up in Amsterdam pretty pleased with myself—to the point that I stayed out at In de Wildeman perhaps later than I should have on the night before Monitorama kicked off.

    Not Enough Preparation

    As speaker after speaker took the stage on Tuesday, I sank deeper and deeper into my seat, realizing that I had completely missed the mark—and as if that were not bad enough, several key analogies in my presentation had already been used by the first-day speakers. I did go to the party at Delirium Café, but left early to sleep the restless sleep of the nervous speaker. In the wee hours of the morning, I gave it up as a bad job, and spent several hours basically rewriting my entire talk in the light of what I had seen. Goodness knows what my neighbors thought of me rehearsing my lines, although I tried to keep the noise down.

    Here is the result in Slideshare, or you can watch the video here. That link should take you to the correct time stamp, but if not, scroll to about 5h30′ in.

    Katja Budnikov was also doing amazing sketchnotes (live!) of all the presentations. Here is the one from my session:

    Relief

     

    In the end it seemed to go down pretty well. I did my level best (honest!) to call on my own now-obsolete experience as a practitioner and on conversations with people who are more current than I am with the nuts and bolts of modern IT, rather than relying on the crutch of vendor boilerplate, and I think I was mostly successful in that.

    If you are working for a vendor and considering submitting to the Call for Papers at this sort of event, whether it’s Monitorama, DevOpsDays or Config Management Camp, I would definitely suggest test-driving your content as widely as possible. You probably have a lot of subconscious reflexes as a speaker that are not appropriate for the sort of audience these events attract. Even when you are conscious of them, it’s hard to eradicate them completely, but do your best.

    In the other direction, while I have enormous respect for the technical level of Monitorama attendees, some of them seemed to be at risk of losing track of the big picture. I would not want Monitorama to change its character as a techie’s conference, but I am grateful that we had a few presentations outside that mold, and I hope that I was able to provide some context for how all the wonderful technology gets used out there in the big bad world.

    — Dominic Wellington

  • DevOps at Scale

    DevOps at Scale

    According to Accenture‘s APAC practice lead for Agile, DevOps and Test Automation Mirco Hering, there is general agreement that DevOps at scale is a good thing, but the problem is getting from where organizations are to where they want to be.

    Obstacles include having plenty of technology skills but little or no technology management (i.e., people and process) skills, or being driven heavily by technology vendors and ending up with a collection of tools that don’t all work well together.

    Hering has found value stream mapping to be a very successful technique for implementing DevOps at scale. It allows people from all parts of the business to participate in the development of a map of the software delivery process (and, thus, gets different parts of the organization talking to each other), it provides important context for any external consultants that may be engaged, and it provides a visual representation of the real situation (not a formalization that doesn’t reflect reality).

    It also provides an opportunity to explore what the participants think DevOps is really about. (Hering’s favorite definition is “building better solutions faster.”)

    Get the First Bite Right

    Once the map has been created and agreed, the next step is to determine what should be done first. “Find something that is meaningful but not impossible,” he recommends.

    In other words, avoid biting off more than you can chew, but also avoid projects that are so specific that they won’t seem relevant to other parts of the organization. And avoid those involving technology that’s likely to be retired soon, because there might not be time for the results of the project to be appreciated.

    It’s important to understand how you can show progress is being made, and organizations that understand the idea of rigorous continuous improvement are generally better at this.

    There is a particular problem that arises when senior leadership changes: it is important to be able to explain what’s going on and demonstrate the benefits, such as the actual savings made.

    Hering advocates the use of same rigor for applications supporting IT delivery as for business applications. For example, if software development systems or quality assurance systems go down, development and deployment come to a halt while the problem is rectified. The changes that are held up clearly have value to the organization (otherwise there’s no reason to make them), so any delay comes at a cost that can be minimized by applying the same types of backup and failover capabilities that are used to protect business systems. Other aspects of IT that can be applied to ‘doing IT’ include big data.

    Use IT to Support IT

    He points to the IT4IT reference architecture for managing the business of IT, saying “I think it is rarely implemented at this point, but at least someone’s thinking about it.”

    Doing DevOps and Agile at scale requires de-scaling the technology, said Hering. That is, where hundreds of people might once have worked on a single piece of software that was released in a ‘big bang,’ the adoption of microservices and other approaches means small teams work on different parts of the overall project and independently deploy the results.

    “That’s a huge jump,” he said. “A different capability is needed to do that.” Part of that capability involves resolving dependencies by using technology so they don’t need to be handled by people or processes.

    But practitioners sometimes miss the point when they hear about organizations deploying hundreds of releases a day, and then they make the assumption that the practices and technologies required are not relevant to them because they don’t want to deploy that frequently.

    Hering argues that it is really about having a deployment process that can support as many releases as you care to make, with “so little risk that it’s as natural as breathing.”

    Can You Really do Continuous Delivery?

    Many practitioners say they do continuous delivery, but Hering tests them by asking whether they could deploy a system at 4 a.m. the next day. If they are happy with doing that, they probably are on top of continuous delivery; if not, there’s probably something wrong, such as one or more manual steps that still haven’t been automated. Similar litmus tests can be created to cover other aspects of DevOps activity, he suggests.

    Processes that are robust enough to detect any problems before the altered code reaches the customer or user empower DevOps teams to make small changes as required (whether the triggers were user requests, a realization that performance could be improved, or to implement a new business or compliance requirement, for example) without putting the business at risk.

    Accenture has developed its own DevOps platform—a collection of open source based tooling—that provides a common reference point, and Hering recommends other organizations do something similar, warning that “it requires a difficult conversation about what is federated and what is centralized” because there needs to be a balance between the two extremes.

    Changes to DevOps capabilities should be treated as scientific experiments, Hering suggests. The hypothesis might be that increasing unit testing coverage will decrease the number of defects, so you increase the coverage in a particular project (experiment), measure the defects (results), and draw a conclusion. But “very few organizations are that mature,” he conceded.

    Training is Important

    Another danger is that a lot of the information that practitioners hear (eg, at conferences) and read (e.g., in blogs) has been curated by organizations’ marketing teams, and that makes it too easy to focus on specific products or technologies rather than focussing on the issues that are really important to the organization.

    So training is important to make sure everyone involved understands the concepts behind DevOps, and how to do unit testing and continuous integration. An increasing number of organizations say they want to adopt DevOps and Agile practices, but they don’t always understand the concepts and sometimes don’t really want to move out of their comfort zones.

    Other pieces of general advice from Hering include asking why the organization is looking into DevOps (i.e., what are the goals and expected outcomes), making sure someone is going to drive the project, running cross-team workshops with a neutral facilitator to break down existing silos, and taking advantage of your peers’ experience in other organizations rather than trying to re-invent the wheel.

    Very importantly, don’t try to move to DevOps at scale in one fell swoop because it is too much to swallow whole. Rather, take a piecemeal approach by focussing on a few aspects at a time and how they fit in with the next steps of your current project.

    Hering’s book, “DevOps for the Modern Enterprise: Winning Practices to Transform Legacy IT Organizations,” was published this year by IT Revolution and is available from Amazon. He blogs about software delivery, Agile and DevOps at Not A Factory Anymore.

    — Stephen Withers

  • Embracing DevOps into Your Company

    Embracing DevOps into Your Company

    Some companies are embarking on a new adventure in DevOps. As organizations start to consider the possibilities of implementing DevOps, they need to think about what problems it solves and what role it will play in their organization.

    Company Culture and DevOps

    DevOps is both a cultural and technological shift in the enterprise, offering a seamless path from idea to production. Because it involves development, operations, IT and the rest of the business, many organizations have seen the collapsing of their organizational silos.

    Company culture and values are central to DevOps adoption, with leadership encouraging more collaboration. As roles and responsibilities shift, worries over how these changes may impact jobs could result in some internal friction from employees. Throughout this process, communicating ongoing and upcoming changes is critical to keeping employees motivated. Thus, DevOps adoption will be easier if an organization with an open communication style and a philosophy of experimentation with new technologies.

    Security in DevOps

    Security benefits DevOps by reducing the possibility of risks early on, bringing greater agility to the development process. Given the average cost of a security breach—ranging between $2.2 to 6.9 million so far in 2018—and its impact on customers and brand reputation, having a DevSecOps strategy is critical.

    DevOps approaches emphasize the importance of security with speed. While traditional models made time for manual testing, there is no such time available in DevOps, where continuous deployment is the norm. Teams must integrate testing and resolution of security issues earlier (shifting left) in the development process to prevent bigger issues down the road.

    Tips for Best Cloud Platforms and Orchestration Tools

    Organizations will achieve the best results when they use the cloud and orchestration platforms that best serve their technology requirements. All of the major cloud platforms offer the basics, including compute and storage, as well as more niche and cloud-native solutions. Once you’ve chosen a platform, you’ll want to test and evaluate tooling and workflows.

    Here are some DevOps tools that you might consider:

    Version Control Continuous Integration Infrastructure as Code Continuous Testing Continuous Delivery Configuration Management
    GitHub

    GitLab

    Bitbucket

    		 Jenkins

    Bamboo

    TeamCity

    CircleCI

    Shippable

    Concourse CI

    		 Terraform

    AWS CloudFormation

    		 Selenium

    Cucumber

    Jasmine

    Karma

    Gauntlt

    Serverspec

    		 Spinnaker

    Octopus Deploy

    		 Chef

    Puppet

    Ansible

    SaltStack

     

    You’ll want to cherry pick the right set of tools and then create a cohesive and productive DevOps automation workflow that ultimately increases both the robustness and velocity of your releases. For example, in the following DevOps automation workflow we have chosen GitHub for version control, Bamboo for continuous integration and continuous deployment (CI/CD), Terraform for infrastructure as code, and Chef for configuration management:

    Recommendations for Getting Started

    We have five recommendations for getting started with DevOps in your organization:

    1. Start with a Process: The only way to understand whether a DevOps process is successful is by getting started. Make good use of the opportunity by engaging teams to work together to build innovation. Be prepared to experiment and fail. Eventually the right process will evolve.
    2. Define the Metrics. For transformation to work, you need to understand the key metrics that matter. Use these metrics as a starting point:
      1. Mean Time to Production: The length of time it takes for newly committed source code to get into production.
      2. Deployment Frequency: How often releases are deployed into production.
      3. Average Lead Time: How long it takes for a new feature to be developed, tested and deployed into production.
      4. Deployment Speed: How long it takes to deploy a new release into production.
      5. Production Failure Rate: The frequency of production failures.
      6. Mean Time to Recover (MTTR): The time required to recover from a failure.
    3. Visualize the Workflow: You won’t really know how effective DevOps will be until you put it into action. By creating a visual model, team members will have a better understanding of how their work impacts the entire DevOps process.
    4. Create Continuous Process: DevOps requires automation, and lots of it. High-velocity robust releasing can only be achieved through the implementation of CI and CD. In CI, members of a team integrate their work frequently to build software faster, while CD offers the opportunity to integrate automated testing into the release process.
    5. Improve Communication: With teams now integrated, the next step is to make sure they can communicate with one another. Start by establishing a unified language between business and engineering teams to express the desired acceptance criteria for both teams.

    As organizations embrace DevOps, culture will be as important as technology in making a successful transition. Make sure that your DevOps playbook addresses how your chosen DevOps tools and technologies will work together with people and processes.

    — Jeremy Cook

  • Education in the Age of Automation

    Education in the Age of Automation

    Automation is changing the nature of education and the ways people learn. More schools are using online learning to supplement in-class activities. In fact, some schools are completely online. Distributed computing on the internet and the automation that drives it makes this possible.

    Everything changes when the formal education process becomes embedded in the internet. Courseware delivery becomes asynchronous; people learn what they want, when they want to. Test administration becomes more about the efficient acquisition of data and evaluation thereof. Human-to-human interaction as a basis for a meaningful educational experience becomes optional. Today, it’s entirely possible to learn all you need to know to make a good living in the modern world without ever needing to sit in a classroom or interact with a human directly. Let’s face it: You can do a lot of learning from YouTube.

    As a result, more people are beginning to question the conventional wisdom of spending four years and thousands of dollars to get a bachelor’s degree—particularly now that companies including IBM, Google and Apple no longer require one to get hired.

    Coincidentally, as companies eliminate the college degree requirement for new hires, we’re seeing significant growth in technical bootcamps. One study reports that this year, 20,000 students will complete a course of study at a coding boot camp and be “job ready.” Considering that the number of students who graduated with a computer science degree from a typical college in 2017 numbered about 93,000, there’s a good case to be made that boot camps are siphoning off a number of students that otherwise would be headed to academia.

    Going to boot camp instead of college is an upward trend: The study projects that the number of boot camps are expected to grow 20 percent this year. The career path for a technical professional does not necessarily involve going college. Going to a boot camp for a year—if not less—will suffice.

    Is this a problem? I’m not sure. On the plus side, I am reminded of the scene in the film, “Good Will Hunting,” when Matt Damon’s character Will Hunting, a genius with a photographic memory who walked the halls of MIT working as a janitor, makes the profound remark:

    “… you dropped a hundred and fifty grand on a f****** education you coulda got for a dollar fifty in late charges at the public library.”

    There’s a good argument to be made that when it comes to getting a good paying job, the $15,000 median price tag for attending a coding boot camp has a better much better ROI than spending  ~$200,000, the cost of a four-year degree from MIT.

    But then, I am reminded of Aldous Huxley.

    In his book, “Brave New World,” people are grown in test tubes according to a specific formulas, the result of which is a biological caste system. Those in the Epsilon caste are grown to be 3 feet tall and abhor the outdoors. Their job is to work their way underneath the machines on the factory floor to do simple maintenance. Deltas, the next caste up in the hierarchy, do the menial chores—janitors, for example. Gammas are the bookkeepers and data entry clerks. Then you get to the higher levels, Alphas and Beta. Betas are technicians who do very sophisticated jobs, such as mixing dangerous chemical together. One mishap can result in an explosion, killing all. Finally, at the top of hierarchy are the Alphas. Alphas possess advanced analytical skills and a broad knowledge base. They are able to plan and strategize. They have a high degree of creativity. Their role is to run the society.

    Thus, I have a concern about the increasing role of technology and  automation education, and the emergence of tech-focused boot camps as the prime post-secondary educational experience for many.

    As more of those entering the workforce forgo a formal college education, we run the risk of creating a society in which most of the technical workforce is Betas. They’re able to do highly complex tasks, yet have no idea why they’re doing them.

    Let me be specific. Let’s talk about Kubernetes: It takes a lot of knowledge and experience to be good at getting Kubernetes to work. It’s a complex, evolving technology that you need to keep your eye on. Companies are spending a good deal of money making sure their IT staff are competent with Kubernetes—as well they should be. One little misstep can result in the digital equivalent of a chemical explosion. Yet, I wonder how many of those “working Kubernetes” are thinking about the bigger picture—for example, the purpose and value of the enterprise using Kubernetes. Are these engineers questioning if the technology they’re working is doing good or harm?

    Or, let’s turn the dials down a bit and ask a question that’s less loaded philosophically. How many of those working Kubernetes today wonder if an alternative orchestration technology such as Docker Swarm might be a better choice to meet the need at hand. Or are they so concerned with getting Kubernetes to run that they don’t have the bandwidth to think about anything else? Do they just defer such critical thinking to “the system architects”?

    Critical thinking, although an acquired skill, is not one that’s new on the cognitive landscape. It’s been taught for thousands of years, since the time Socrates sat under an olive tree asking simple questions that forced his students to transform their assumptions and allusions into thoughts based on reason and reflection.

    Developing the ability to think critically traditionally has been one the aims of a college education. The romantic ideal is that college is the place where students are exposed to a variety of ideas from a variety of viewpoints. The hope is that such exposure, along with instruction aimed at developing one’s ability to engage in higher-order thinking, will create graduates who will contribute to making a better-functioning, more prosperous and more humane society.

    But, that’s the traditional perception of college. The reality is changing. Maybe college is just the place you go to get the skills required to get a good job. If that’s the case, given that a college education is becoming less affordable to more people, maybe the boot camp approach is best. Yet, there is a risk.

    A good while back when I was considering enrolling in a doctoral program, I attended a required orientation at Harvard University. I sat in an auditorium composed of about 50 aspiring doctoral students. One of the department heads on stage said the following, “The objective of our doctoral programs is simple. We want to train you to run the world.”

    Yep, that’s right. And, I can say with a good deal of confidence that this objective is the same today—to create thinkers who have the skills and knowledge necessary to run the world. The Alphas, if you will.

    So, we’re at a crossroads. Automation is a liberating force in education in that it makes infinite knowledge available to anyone who wants it at a fraction of a cost of a traditional college education. All that’s required is the time and desire to take advantage of the opportunity at hand. However, if the result of such education is a workforce in which the concern of most is to ensure that the technology works and does not blow up, then the risk we run is that bigger decisions will be made by those who had the time and money for an increasingly elite education. In other words, as more people spend time focused on turning the dials and setting the levels to make the technology go, a very small minority actually will be determining the purpose and direction of the machine the technology supports.

    Still, the fact is, most of us need to make a buck to survive. This means being good at getting the tech to work. Yet, we will do well to understand that there is a Big Picture and if don’t pay attention to it, someone else who has been educated to run the world will.

    — Bob Reselman

  • Manual Processes and Silos Complicating DevOps Deployments

    Manual Processes and Silos Complicating DevOps Deployments

    The DevOps concept largely originated in 2009, as software and IT organizations were beginning to adopt Agile methodologies. It came on the heels of a movement to integrate lean manufacturing concepts and infrastructure as a service into IT organizations. Rapid development was the goal, but application and infrastructure teams were still working independently, creating barriers to progress. Flickr employees John Allspaw and Paul Hammond delivered a presentation at the O’Reilly Velocity Conference that outlined ideas for streamlining deployment through better collaboration between development and operations. They hoped to temper conflicts between team members and minimize waste. DevOps took off soon after, when Patrick Debois founded the DevOpsDays conference in Belgium.

    When DevOps is a mature practice in IT and software organizations, there are some powerful benefits:

    • Every change to the application stack triggers a new deployment.
    • Deployments become easy and not a fire drill.
    • Fixing errors in production become a simple redeployment without the need to perform a huge root cause analysis up front.

    The Long Road to DevOps

    While many organizations are adopting DevOps practices and tools today, we’re still a long way from reaching DevOps maturity. Our company recently conducted a survey on DevOps adoption, and the findings indicate that there are still too many manual processes and silos standing in the way. I will break them down here:

    Manual Infrastructure Management is Still Common 

    Nearly 40 percent of IT professionals surveyed say that they manage infrastructure manually. Yet nearly 60 percent report the use of infrastructure-as-code tools such as Terraform, configuration management and Kubernetes. This speaks to a disconnect between having the technology available to operate in an automated way and being able to execute change with those tools. Heavy manual processes lead to higher costs, since so much effort goes toward tinkering with configurations and fixing problems. Manual work also detracts from the ability to members to focus on the application and business needs.

    Managing Environments in Silos

    Another tenet of DevOps infrastructure management is environmental parity. This means that through infrastructure-as-a-code tools and processes, IT manages all development environments similarly through a common code base. According to the survey, only 25 percent of companies are managing development, test and production environments identically. This results in application stability issues. Without environmental parity, every change will eventually lead to an outage.

    Managing Teams in Silos

    From the start, DevOps has emphasized breaking down the walls between development, QA and IT operations. The goal is to enable continuous feedback, fewer errors and faster development cycles. Of survey respondents, 78 percent report that separate teams are still managing infrastructure/operations and application development. As well, just because you create a DevOps team consisting of various job roles doesn’t mean that people are working together and are committed to the team’s success.

    There must be incentives to encourage a developer to understand testing needs and for infrastructure engineers to get clear on a project’s requirements and objectives. There should be minimal tolerance for functional experts isolating themselves without making an effort to understand the bigger picture.

    When organizations are able to minimize manual processes and break down these technology and silos, they benefit from a totally different, more productive way of working:

    • Adoption of continuous integration/continuous deployment, whereby every code check-in triggers a testing cycle and a build.
    • Transparency between infrastructure people and developers, avoiding surprises resulting in issues or rework later.
    • Incorporation of success metrics, such as deployment frequency, mean time to detection (MTTD), mean time to recovery (MTTR) and change lead time.
    • Cost reduction, from fewer errors and labor-intensive manual processes.
    • Increased productivity, as developers get the infrastructure they need quickly to run new projects and QA and IT Ops people can focus on high quality, stability and user experience rather than repetitive tasks and troubleshooting.

    So, What’s Going On? 

    Despite the many benefits of DevOps, there seems to be a lack of urgency in some organizations to take the necessary steps for change. Progressive IT departments that applaud agility and business value should be ahead of the curve, but that’s not always the case.

    DevOps maturity comes down to setting the right culture for change, which must emanate from the top. The shift to DevOps takes time and investment, no doubt. There’s also need for patience. The transition temporarily slows down production as people adjust to new workflows and learn new skills. Replacing toolsets is just part of the strategy. Organizations must make a conscious effort to embrace DevOps with a well-thought out plan.

    — Lars Cromley

  • Revisiting My Bet on DevOps

    Revisiting My Bet on DevOps

    On Dec. 29, 2015, I bought some stock. It was a bet on DevOps that my wife kindly tolerated. Since then, the S&P 500 has risen 31 percent and my DevOps portfolio is up 113 percent. This blog is a retrospective on a bet that worked out.

    I was inspired by Nicole Forsgren’s talks and the “State of DevOps Report,” whose data showed better profitability, market share and stock price performance among companies claiming to do DevOps well. DevOps, after all, exists to help the business win, and as digital transformation sweeps industry after industry, technologists increasingly are in a position to make that difference.

    My process has been very, very simple: Companies whose talks at DevOps conferences convincingly convey a passion for DevOps are ones that I would consider buying. Would I do deep research on each one? Nah—whatever I could research aside from DevOps would already be priced in by people who understood that content better than me.

    At the one-year mark, I summarized the results. The crazy project was up 12 percent against an S&P that was up 8 percent—a nice little win. The first six months of 2018 have been tremendous, though: The S&P is up almost 2 percent and the portfolio is up 42 percent.

    A key disclaimer is that I am trying to read corporate culture from the outside. For insiders, I am sure I will have some of this comically wrong.

    Remixing the Strategy (and Portfolio)

    Over the past year and half, I’ve made a handful of trades based mostly on two triggers.

    The first trigger is a signal that a company is losing its commitment to DevOps. I did this with Nordstrom: Its DevOps leaders all moved on to different companies. For example, Courtney Kissler went to Starbucks and Suzanne Conniff was off to T-Mobile. About the same time at a meetup, a Nordstrom employee told me that DevOps had fizzled. I was out. In the last year, Nordstrom is up 7.5 percent while retailers I kept are up: Target 46 percent and Amazon 74 percent. That said, Target is a bit of a difficult case.

    The second trigger is silos. I worry about an either IT-centric view of DevOps or DevOps in just a small slice of the business. Before I invest in a new stock, I want to know the transformation is taking root enterprise-wide.

    Barclays for Bank of America is an example of chasing the all-in company that didn’t work out: negative 8 percent growth versus 25 percent at BoA. Overall though, shifting toward all-in and unicorns has been solid.

    Who’s in Today

    • Amazon: 121 percent
    • Barclays: -15 percent
    • Capital One: 26 percent
    • Cisco Systems: 52 percent
    • Etsy: 340 percent
    • Live Nation: 93 percent
    • Netflix: 235 percent
    • Nike: 20 percent (since December 2017)
    • Target: 0 percent

    Amazon: Amazon’s net gain at 121 percent looks a lot like the portfolio as a whole (113 percent). The shares I purchased in 2015 are up 150 percent. The company is a natural fit, given that its CEO wrote the “two pizza rule” and demands microservices. It is also a hedge, as the company directly competes with much of this list including Target, Netflix and Etsy. With cloud and DevOps often joined at the hip, Amazon’s cloud revenue is also a rough proxy for DevOps growing overall.

    Barclays: Did I invest in a British bank shortly after the Brexit vote? Yeah. It was a bit of a test of the “Everything except DevOps is already priced in” rule. This one hasn’t panned out, but from the outside Barclays is among the banks doing best at DevOps.

    Capital One: Has performed about average among banks that I looked at this morning. I adore its start-up attitude and how the company goes about getting talent and contributing to the community.

    Cisco Systems: Cisco has had some great talks and is undergoing a digital transformation of sorts of its own. The company is doing fine.

    Etsy: Holy cow, Etsy. Etsy is an interesting case in that it has been a DevOps darling for many years. With the stock flagging a bit, the CEO and the CTO (our DevOps hero, John Allspaw) were shown the door and the perception was the grown-up business types were taking over. I increased my bet based on the thought that technical foundations pointed ruthlessly at business problems might work out. When the stock price doubled, I cashed out my original investment.

    I’d be better off had I not sold any, but with as volatile as Etsy is, I was happy to route some of the winnings over to a bet on Nike.

    Live Nation: I saw a great talk from a Live Nation guy at DevOpsDays Rockies, and the company’s stock has paid off nicely since, nearly doubling. Live Nation gets it, and shows a passion for satisfying customers. Being Ticketmaster, it has had a risk of digital disruption knocking it down for years and has built up a strong innovation capacity—exactly what incumbents need to do.

    Netflix: DevOps skills? Check. Good at using data to delight customers? Check. Worth twice what I bought them for? Check. I wrote most of this analysis in early July. Since then, Netflix has had a bad earnings call. It’s still a massive winner so far.

    Nike: I loved Nike’s talk at DOES 2017. But what really struck me was DOES London. In the morning the company spoke about a one-team culture and engaging digitally—good stuff. In the evening a colleague complained his kid was spending $200 on sneakers. A week later, Nike’s stock spiked 10 percent. Yup.

    Target: Oh, Target. Target has done some great DevOps work, but word on the street is that too much of what it did at first lacked a connection back to the business. However, since the security fiasco with stolen credit cards, that has turned around. While its stock is right around where I bought it 2.5 years ago, the last year has shown strong momentum: Target is up 45 percent in the last 12 months.

    2018 Has Been Great

    Over the project’s first two years, I had a nagging worry: I was beating the market, but was I just picking riskier stocks and being rewarded in a bull market? So far the market has been flat in 2018, showing modest 1.7 percent growth through June 30. The DevOps Portfolio, however, is up 42.2 percent.

    Only the banks are trailing the S&P.

    So What Have We Learned?

    • It’s about customers and the business: There appears to be a moment of DevOps backlash when alignment to business outcomes isn’t quite strong enough. When the DevOps culture is strong but redirected back to catering to the business, things can go better. That’s one way to read Target and Etsy’s recent performance. When the transformation fizzles … see you, Nordstrom.
    • Banks may be different: This system really hasn’t worked out terribly well for me in the banking space. I suspect that is because most banks are undergoing similar transformations and may not be as able to differentiate as retailers do. At DOES London we heard similar stories from RBS and Barclays while Lloyd’s is winning DevOps awards at other conferences. I also lacked representation from the digital disruptors—PayPal would have been a good balance to CapOne or Barclays.
    • DORA should be a hedge fund: I hope they have a secret one.

    I’m thankful my that my wife was on board with this little project and I’m very happy that we can take our winnings here as we plot our second honeymoon.

    Of course, professionally I’ve been betting on DevOps a long time, working on UrbanCode continuous delivery tools the last 15 years. That worked out with an acquisition by IBM as it beefed up its DevOps solutions portfolio.

    — Eric Minick

  • A Right DevOps Approach: Assess Organizational Readiness

    A Right DevOps Approach: Assess Organizational Readiness

    Driving business in this fast-paced world is maddening as owners have to remember customer preferences, beat market competition and similarly cater to different things within the organization’s radius.

    How a business owner manages all this defines the business success, which is gauged through outcomes in the form of smart products, point-on services and almost-immediate feedback. Bank applications, the best smartphones and interactive technology are all proof of this.

    But what is it that stands at the heart of it all?

    The Right DevOps Approach

    When enough is done and businesses still find that there’s a missing piece that will complete “the successful business picture,” they go back to figuring out what and how they can fix the problem. They never come to realize that they will have to consider revamping their organizational structure to meet the growing demands of their influential business.

    How, then, do you assess your approach to DevOps?

    While most believe that it’s a technological approach that they should incorporate in their business, it’s more of a cultural approach. To implement DevOps into your existing organizational structure, you first must look at its current state of readiness.

    Doing this is important, as it helps decision-makers understand how they can embed the smart collaboration after a thorough consideration of the challenges that stand in the way of DevOps implementation. Once the initial assessment is done, business owners need to ask themselves three questions before assessing DevOps readiness:

    1. Is there a Management-Employee Agreement? A successful DevOps process not only relies on collaboration between Development and Operations, but also needs a top-down (management-employee) agreement. Once that’s in place, the company is already halfway through the success road.
    2. Is there a Plan in Place? Many firms just barge into technology like it’s all they need to reach their finish line. Stop! Wait! Think! Owners need to have a detailed strategy, equipped with process tools and goals clearly outlined and best practices chalked out. Once this is done, the culturally different teams need to collaborate and work ways to learn and equip themselves with the technologies and practices of each other’s processes. With all this in place, organizations will be ready to take the DevOps jump.
    3. Is the Organization Ready to Tread the Ocean? Now that the plan is in place, what’s next? Organizations, as mentioned earlier, are in hot pursuit to experiment with what they’ve learnt new. Anything new is fancy, but humbleness is the key! Starting small will set the new cultural shift to completely settle in its new mold. To run the boat in big waters, you first need to test it in small water. Similarly, to test the success of DevOps for a big business project, you first want to test its adaptability to a smaller one. Once you’ve cut off from the harbor, you are all set to wade the ocean.

    Conclusion

    Most business players underestimate the ability of DevOps with whatever industry rumors they hear and some of their own pre-conceived notions. Some other think that they know it all, but what matters is having a strategic approach to first considering DevOps and its tools, and then adapting it accordingly.

    Once the DevOps automation is in place, organizations will be busy reaping the benefits of its successful implementation.

    — Veritis

  • 9 Ways DevOps Success or Failure Is Like Baseball

    9 Ways DevOps Success or Failure Is Like Baseball

    The baseball season is in full swing. There are 30 major league teams competing for a playoff spot, but, as I write this, only five of them—or 16.67 percent—qualify as elite performers with a winning percentage over .600.

    The number is strikingly similar in DevOps, where, according to one survey, just 17 percent of organizations say they have fully embraced it. Most of the rest have only scattershot teams immersed in DevOps or only recently began adopting it company wide. Fifteen percent are saying, “Wait ’til next year” and have yet to undertake this agile methodology.

    In baseball, teams struggle because of weak hitting, lousy pitching, poor defense, inept managing or all of the above. But why do some DevOps initiatives fail, who is at fault and what can be done to right the ship?

    Baseball and DevOps: More Similarities Than Differences

    Let’s dissect the differences between a DevOps program that hits the ball out of the park with a truly collaborative effort to build, test and release software faster and more reliably, and one that swings and misses.

    Is It Understood There’s No “I” in Team?

    Most DevOps initiatives fail because they are undertaken by a single practitioner within a single team. This is a strong personality who bulldozes the team to DevOps success, either out of career growth ambitions or genuine belief in DevOps tenets, only to have the team revert to their old ways when the practitioner leaves.

    Ironically, many if not most successful DevOps initiatives also begin with a practitioner on a single team. The difference in the latter case is that the leader doesn’t operate in a vacuum but shares lessons and best practices across the organization. They understand that if you are going to do something transformational, you cannot do it in the shadows or within a silo. They consider the long term. They lead by example and demonstrate the way forward to others.

    Are the Top Leaders Onboard?

    Just as the best baseball teams tend to have smart, supportive, quality-minded owners, the most successful and sustainable DevOps initiatives have top-down leadership support or mandates.

    Clear, consistent direction from the C-suite is necessary to motivate and incentivize people across the company and move DevOps initiatives from a patchwork of projects to an organizationwide philosophy.

    “Without executive support,” a Forrester report warns, “agile transformation is limited to grassroots initiatives—a very slow and difficult road.”

    Are the Danger Signs Recognized?

    There are two telltale signs that a companywide program is floundering. One, success is occurring only in a single team. Two, no one outside the team knows much or anything about that success. DevOps can’t grow within an organization if it’s an isolated black box.

    Starting DevOps is kind of like an entrepreneurial act. Startups become something bigger when the entrepreneur is successfully able to extend their vision. DevOps is the same: You need to infect other teams with your success. Winning begets winning.

    Another indicator of failure is if a company claims to be adopting DevOps but clings to traditionally organized and siloed teams. You can’t have it both ways. DevOps personnel need to be aligned by product, outcome or value stream—not by specialized work function.

    Is Success Measured?

    How does a baseball team know it’s winning the game if no one is keeping score? Same in DevOps. If you can’t objectively measure success, you don’t know if your program is meeting its goal of accelerating the software life cycle.

    Figure out how to measure the value of what is being done and delivered now so you can demonstrate its impact objectively. Examples could be application deployment frequency and time, error rates and mean time to detection.

    Is Seeking Outside Help a No-No?

    The DevOps community isn’t especially fond of vendors and consultants, but outside experts can bring fresh perspectives. No one tries to cure an infection or solve marriage issues by themselves, and solving DevOps challenges should be no different. Don’t be afraid to ask for help and if you have the budget pay of it.

    Can the DevOps Advocates Speak Business Language?

    DevOps folks often stumble when pitching for leadership support because they are unable to articulate their proposals in the dollars and cents language that business leaders understand. But if they can talk that way, practitioners may find leaders are much more willing to play ball than they realize.

    Is the Blame Game in Fashion?

    The biggest mistake organizations can make is bearing the classic IT coat of arms: arms crossed and fingers pointing. Don’t fall into the blaming trap. DevOps, after all, espouses the concept of learning from failure. So instead of pointing fingers, evaluate, consider, learn and move on with the lessons learned.

    As legendary New York Yankess manager Casey Stengall said, “All blame is a waste of time. No matter how much fault you find with another, and regardless of how much you blame him, it will not change you.”

    Does Failure Breed Success?

    Not to sound harsh, but if you start an initiative and it fails, it is your fault. Sure, many company cultures aren’t ready for DevOps, but others are succeeding where you have failed. If your team lost the World Series, another team won it. Why? There is always something that can be done differently. Determine what went wrong, accept responsibility and learn from it.

    Do People Bounce Back?

    If your initiative wasn’t successful, don’t be discouraged. Pick yourself up and try again. Being a DevOps leader requires courage and persistence. You’ll get ‘em next time.

    “Every day is a new opportunity,” Hall of Fame pitcher Bob Feller said. “You can build on yesterday’s success or put its failures behind and start over again.”

    — Scott Willson

  • Successfully Integrating DevOps while Deploying Cloud Applications

    Successfully Integrating DevOps while Deploying Cloud Applications

    Transforming to a high-performance DevOps environment can provide competitive advantages, but it requires change—and that doesn’t come easily to most organizations. DevOps requires cultural changes that are inherent to automating previously manual tasks: They are perhaps intimidating to those involved and they require flexibility and enterprisewide buy-in to get right. Furthering these people-side transformational complications are cloud application deployments, creating a need for enterprises to deliver innovative technologies and services with greater speed and quality—without hits to reliability and security.

    DevOps Blended with AWS Proficiency

    One approach to this problem is blending DevOps knowledge and Amazon Web Services (AWS) proficiency to guide development teams to better align the people, processes and technology needed to facilitate successful cloud initiatives.

    In transitioning to a more modern, DevOps-driven approach, companies often encounter a number of challenges, on both a technological and cultural level. The most successfully integrated DevOps occurs as a function of value analysis and process overview, addressing the individual needs of an enterprise rather than fitting them into a preset mold.

    Implemented correctly, DevOps blended with AWS proficiency allows organizations to maintain quality and configuration of their infrastructure environments, allowing application development teams to retain control over their processes and toolsets. AWS also allows the enterprise to maintain security: Rather than protocols that are standalone, they’re built into IT processes from the beginning, allowing customization with ease.

    AWS understands what is important to its partners when building tools, successfully addressing needs while staying ahead of the demand and, in turn, saving time. They accelerate the ability to deliver and allow an enterprise to focus on what’s most important during their journey to the cloud: value analysis.

    Fundamentally Designed to Refine Business Processes

    It is very important to approach projects first from a top-level view: Where does this piece of work fit into my broader organization?

    DevOps at its core is fundamentally about incorporating different disciplines of a business to create better practices than what are already in place. When an enterprise chooses to implement automation, it typically gains a better view of its business as a whole, incrementally increasing awareness at a birds-eye level. When complex business functions and operations are laid out in more mathematical terms and formulas, it suddenly becomes clear where applying automation can reduce waste.

    Getting these changes implemented quickly and with as little effort as possible naturally leads to successful automation. Enterprises remove such tasks that traditionally required an excess in resources by asking, “What value doe this process add? Is there a more efficient alternative?” With automation, these useless business practices are eliminated, freeing resources for more important initiatives.

    Increase Employee Awareness, Increase Productivity   

    One of the most important parts of the entire DevOps integration process is ensuring that everyone understands each other and their role. Properly embraced, a DevOps culture will be characterized by increased collaboration, shared responsibility and autonomous teams—leading to improved quality, valued feedback and the end goal of increased automation.

    People are naturally resistant to change. Members need to be aware of the business and how they—and will continue to—fit in.

    Ultimately, DevOps is not something that can simply be bought. Success during implementation requires the application of technologies, tools and products that are actually a fit between new processes and existing systems.

    — Ashley Stovall