devopsy.kinsta.cloud

Author: Brian Dawson

SDM and the DevOps Delusion of Connected Organizations
How Software Delivery Management Clears Up the Confusion of Embracing and Implementing DevOps

Since the term was coined back in 2009, it wouldn’t be a stretch to describe DevOps as an impactful if not tectonic shift in software development and delivery in the past two decades. The move to implement DevOps has encouraged, among other things, better collaboration between siloed Dev and Ops teams, more reliance on automation, more focus on culture and better use of feedback loops to refine delivery processes.

In theory, DevOps has been a game-changer. But, in practice, its usefulness is more difficult to define. As DevOps enters its teenage years, the idea of a fully integrated software delivery function aligned toward a common goal of generating business value is more illusion than reality.

Are DevOps teams themselves as connected as they should be? The reality is that developers and operations personnel tend to have their own versions of “dev DevOps” and “ops DevOps,” working against each other inside supposedly integrated teams. Though they may have set up deployment pipelines, each side tends to use its own tools, access its own information and follow its own cadences, leading to occasional missed handoffs and delays in delivery schedules.

Looking across the enterprise, DevOps cultures fray further. A perfect DevOps practice envisions teams in different departments, business units and geographies all in sync, delivering value via software on a reliable cadence. In reality, enterprise DevOps is difficult to pull off because most organizations haven’t implemented shared languages, comment processes and best practices across all of their teams with management buy-in.

Then there’s the issue of connecting DevOps functions with the rest of the organization. Some definitions of DevOps extend the concept of collaboration to other stakeholders beyond conventional engineering functions, such as quality assurance, testing and security. But how extensively does a DevOps culture tie together the goals, needs, resources and processes of others in the organization that rely on software delivery, such as product marketing, sales and executive management?

Delivering Better Software

Gartner describes DevOps as being focused on “rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach.” In other words, delivering faster and in a more organized manner. But has DevOps helped software-first organizations deliver better software? Has it helped them institute the right processes to deliver the right software at the right pace to drive the most value possible to the organization’s bottom line? Paraphrasing the “If a tree falls in the forest …” proverb: If you deliver software and no one is there to consume it, have you truly delivered value?

Even if you’re doing DevOps relatively well, there’s a level of integration that’s missing from today’s software cultures. It’s the mission to tie together all functions of software delivery—not just those relating to what goes on in the labs. It’s the commitment to do more than just deliver better metrics. It’s the mentality to define everything relating to software delivery as a discipline all its own, like finance or sales—not just serving other disciplines.

A new technology category is emerging that aims to build on what DevOps has accomplished and extend those gains across the software delivery function. It’s called software delivery management (SDM). An SDM strategy is based on four foundational pillars that elevate software delivery to a core business priority, reflecting the level of importance software has achieved in today’s business environment:
- Common Data – All information within and around software delivery activity is captured and stored with a consistent domain model to enable and facilitate connections to processes, shared insights and collaboration.
- Connected Processes – Processes orchestrate software delivery and connect functions together to efficiently bring ideas to market with maximum value and adoption.
- All Functions Collaborating – All functions and teams within and around the software delivery organization work together to amplify value creation efforts.
- Universal Insights – Visibility and insights enable understanding and continuous learning from data across, up and down and throughout all functions in the organization.
Driving Improvements Across Organizations

Adopting an SDM strategy on top of an existing DevOps culture can help an organization in a number of ways.

Having disconnected tools, disconnected data and a lack of common language make it difficult for one side of the organization to know what the other wants. Even in mature DevOps implementations, software delivery ends up being a patchwork of different processes for different teams.

It’s also difficult to determine if our teams are delivering the right end product if there’s no shared visibility and insight. SDM helps by establishing common data and common processes, giving visibility and insights across teams of different maturities, different tools and different technologies so you get all functions to collaborate. That way you can be sure you’re not just deploying more—you’re actually delivering continuous value.

You can’t manage what you can’t measure. What SDM offers is a model to understand how you’re performing with DevOps by and large, what teams are performing well and what teams aren’t, and how you can leverage those insights to improve.

Tips for Implementing SDM

Here are five ways SDM build on the successes of DevOps:
- Integrate your tools: Disconnected toolchains make it difficult, if not impossible, to orchestrate a delivery process end to end. Try to get DevOps tools onto a common platform where they can share information across teams and organizational functions.
- Capture data: Ensure you’re establishing an audit trail and capturing data on events and actions that are taken across that end-to-end process. Then collect those data points in a common data store as normalized objects. Making common data accessible across functions allows you to, for example, reconcile a particular support request to the right user story and commit to the right change that is being deployed out to a server.
- Don’t fail to communicate: Once you’ve created your data store, establish a strong relationship and regular communication with the business side of the organization. Continuous delivery isn’t just for developers anymore; software delivery functions have to address business needs. That can’t be done without high levels of communication.
- Establish KPIs: You should have a taxonomy or hierarchy of business objectives you’re trying to achieve with your DevOps implementation. Establish the hypotheses that can achieve those objectives and then integrate the features/user stories or implementation-level entities that will drive you toward them.
- Leverage observability practices: This needs to be done so you can measure those performances against KPIs. If you believe that updating your user interface will increase the amount of time users spend in your SaaS application, you need to establish what that target KPI is. Then, as those changes are delivered, measure, analyze and adjust to continue to drive those toward the hypothesis.
Conclusion

DevOps has taken software delivery a long way and implementing an SDM strategy can take it the rest of the way. It won’t change DevOps, but it could reinvigorate the practice. Deploying both strategies in concert, software delivery management and DevOps can focus less on automation and tools and more on the outcomes and on the people and steps involved in driving those outcomes.
July 2, 2020
Why DevOps is Like Football
When it’s done right, DevOps is an impressive feat. You have groups of professionals all melding together as one large team working toward a common goal. The groups–and the individuals themselves–perform their roles and hand off tasks to each other at just the right times. They follow pre-designed plans, but they constantly tweak those plans, in real time, to deal with variables that threaten to undermine the whole mission.

This is DevOps in action. But take a closer look at some of the descriptors above and ask yourself if it sounds like something else that’s starting to creep into a lot of conversations this time of year. If you’re a sports fan like me, this sounds just like football.

Let’s face it, football parallels have been attached to all kinds of institutions and practices–everything from the military to business to marriage. The emphasis on discipline, teamwork and execution can apply to most initiatives in everyday life.

But I’d argue that football’s very structure aligns especially well with what we do in DevOps.

In football, the goal is to move the ball from the line of scrimmage to the end zone. In DevOps, we have our own starting and ending points. Think of our line of scrimmage as the point where features get added to the application and our end zone as the point where an application is successfully delivered to a customer. Football has its individual units (offense, defense, special teams) and we have ours (development, quality assurance and operations).

In football, the offense players have different specialized skills and all work together to move the ball forward to reach the end zone. The players start with a set of instructions (plays), which they all understand. This is like automation in a DevOps environment. A DevOps team execute its own plays, collaborating and improvising as needed to achieve their shared goal.

Picture what this would look like in a pro football game:
- The quarterback passes the ball: Think of a software developer handing off the application to a test engineer.
- Linemen block for the quarterback: This is like quality assurance continuously protecting the application from bugs that threaten to sack the whole project.
- A receiver runs a route and catches the ball: This is similar to an operations engineer preparing a server and receiving the application.
- A running back runs the ball into the end zone for a touchdown: The application is deployed. Goal!
The team has worked together and achieved the goal–in this case, a touchdown. They will be able to repeat this success time and time again because they have repeatable plays and they collaborate.

Without a DevOps approach imagine what a football game would look like:

No Automation: The team has no plays. Nobody on the field has any idea what his assignment is on each snap, so players basically do their own thing. All the speedy, skilled players yell “Throw it to me!” This kind of freelancing might work for youngsters playing pick-up football, but it won’t work in a more advanced game.

No Collaboration: The positions don’t work together. Even if team members have plays and know them, if the linemen don’t block as a unit, a play will fail. If a running back zigs when linemen are opening up a hole for him to zag, the defense will stop him.

No Shared Objectives: If everybody knows the play but they’re only concerned about executing their parts of it, the offense won’t get far. Imagine:
- The quarterback being concerned with just getting the ball in the air, not where it goes or even if it is caught. The receiver can worry about that. End result: Lots of failed passes–and probably a few interceptions.
- Linemen will block, but they’re not worried about who to block or when. End result: Lots of quarterback sacks.
- Receivers will run routes, but they’ll choose where they want to go. They might run beautifully, and even get open periodically. But if the quarterback doesn’t know where the receivers are, the team won’t complete many passes.
Without a DevOps approach, the chances of getting a ball to the end zone would be very low. And when it did get there, the process would be messy and unrepeatable.

When football is done right, teams defend well, score a lot of points and rack up victories. When DevOps is done right, organizations deliver high-quality software to customers in a rapid, repeatable and reliable way. DevOps, like football, codifies a winning process that serves a vast group of stakeholders both inside and outside of the organizations playing the game.

— Brian Dawson
September 17, 2019
‘DevOoops’ Moves: Deploying Code Out of Step
‘Everything as code’ can create opportunities for misadventure for DevOps teams

DevOps’ ultimate aim is to create efficiencies in the software delivery process. But it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. Following is a discussion submitted by Director of Engineering Laura Frank Tacho about deploying code out of step.

Scenario:

Running in parallel during CI/CD can speed up your automated testing and shorten your feedback cycles. Just make sure deployment is not executing as a parallel step. That misconfiguration will cause your code to deploy at the same time the tests are running, regardless of their exit codes, pretty much defeating the purpose of automated testing before a deploy.

This scenario touches on a theme we’ve seen throughout most of the “DevOoops” stories our colleagues have come up with. The theme is how the emerging concept of “everything as code” can create opportunities for DevOps teams to slip on proverbial banana peels.

In this case, we’re talking about sequencing problems in a process we can call pipeline as code.

To be successful, teams need to extend a quality-first approach in automated testing throughout their continuous delivery life cycle—from code to commit to test, as well as through the various tiers of the development life cycle. And with a quality-first approach, you need to apply testing and quality management to your environment configurations and infrastructure as code.

Laura’s example highlights how teams need to treat their CD pipelines as applications themselves. They need to check for and eliminate bugs and errors in CD pipeline code to ensure that code is operating as expected. Failure to properly manage the creation and evolution of complex CD delivery pipelines will result in significant cost or risk.

So, if you don’t have proper test coverage on application code, you could end up releasing an application that hasn’t been tested at all. If the app doesn’t pass through a security gate, as it’s supposed to, you could release unintended preproduction data.

While there are always risks with untested code, we have to give unique consideration to untested or unmanaged CD pipelines. We need to consider the risks of what missed gates could mean for our developers, our organization and our customers.

Teams should do the following as they release their CD pipelines:
- Build: Start from the left-hand side of process and work through the defining and coding of CD pipeline left to right in the direction your apps and development artifacts flow: from dev to test to deployment. While we may not be able to define an ideal automation solution completely on the first pass, going left to right encourages teams to focus on what the life cycle of an artifact looks like.
- Manage: Define your CD pipeline using an everything-as-code or pipeline-as-code approach. For this to add value, you must manage development of your CD pipeline as you would your application code and apply software best practices. Check your CD pipeline code into a source code repository and leverage language-specific technologies that can check your syntax. (If you’re using Groovy-based code or Jenkins-based code, there are versions of Lint that can perform these functions.)
- Create staging environments: You should establish staging environments or test pipelines to validate your CD process. These pipelines should go through all checks and gates against your real tools, using reference code that is representative of the applications your pipeline would be servicing but stops short of production into environments.
This will give you an opportunity to iterate changes against your application code, do some initial scans and checks to collaborate to define your CD pipeline in code-managing SCM. Execute initial validation on scans as pipeline as code and then validate expected operations in a staging environment before you deploy it as the basis of the pipeline for your production of business-critical applications.

Key Takeaways
- This story highlights how crucial it is to ensure that your application code and environments pass through certain gates and checks before being deployed out to the customer.
- You need to construct your CD pipelines so your parallel testing is managed as a gate to production, ensuring that properly tested and validated changes are not automatically deployed.
- Deployment happens at the end of all the processes within a pipeline, and all required tests, validations, checks and approvals have been done before any change is deployed to the customer.
To read our previous “DevOoops Moves” post, click here.

— Brian Dawson
May 7, 2019
‘DevOoops’ Moves: Unforeseen Lock-Outs
DevOps’ ultimate aim is to create efficiencies in the software delivery process. But it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. Following is a discussion submitted by Carlos Sanchez, principal software engineer, about unforeseen lock-outs.

Scenario:

When you’re automating deployments using configuration as code, make sure you have the right protections in place (i.e., validation of configuration). Otherwise, it’s easy to lock yourself out, forcing yourself to manually log in to each machine to fix it when a bad change is pushed and deployed to all the machines.

Having a strong technology solution that securely and centrally stores credentials for the names of people who access those machines is critical to maintaining a successful DevOps program. It’s a no-brainer. When you have to service accounts, you’re much more likely to be able to leverage a fast-acting script to log in and reconcile logouts if credentials are centrally stored than if you had eight individually managed service account-based credentials.

How do you reconcile lock-outs? How do you avoid the kind of DevOoops situation that can derail all of the good progress you’re making? The process you adopt will depend on what your software development and deployment life cycles look like. Here are several potential approaches:
- You can use multiple staging and preproduction environments that share common configurations with the production environment. This sets up a process where, before you try to deploy an app or a new configuration to your production environment, you know you have successfully deployed it to environments that are as close as possible to production, multiple times.
- You can have post-deployment validation scripts run after every deployment and have them validate things such as, Can account X log into environment Y? The scripts should include common or leading indicators of success or failure of a deployment. Once artifacts have been deployed and the system starts, every deployment will immediately run these scripts as part of the deployment process. So, every part of the deployment is automatically validated as success or failure.
When taking an approach such as rolling deployments, you also need to ensure that broken configurations are identified before they’re propagated across multiple servers or server clusters. This means you need to validate your deployment and your configuration changes before you go to production, validate deployment and configuration changes on first deployment and then validate, validate, validate at each subsequent step.

By doing all of this, you avoid surprises. You extend a quality-first, continuous validation approach beyond just your application code and apply that to your environment. Avoiding unnecessary lock-outs will help you mitigate risk, eliminate wasted time and avoid bad customer experiences.

Takeaways
- As you bake quality into the processes and shift automation left, you need to make sure you validate all changes early and often.
- Organizations need to implement proper processes, not only for application code changes but also for infrastructure as code changes that manage environments.
- In automating in pursuit of DevOps, it’s necessary to have a strategy for credentials and secrets management of permissions for your DevOps and production systems.
— Brian Dawson
April 30, 2019
‘DevOoops’ Moves: SCM Conflicts
DevOps’ ultimate aim is to create efficiencies in the software delivery process. But it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call a “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. Following is a discussion submitted by solution architect Will Refvem about siloed departments.

Scenario:

One of the biggest challenges in DevOps is SCM. Early in my career, being given access to Git repositories was a bit like handing a belt-fed machine gun to a drunken toddler. My biggest DevOoops moment came when I was trying to master the basics of Git in a project that used Git submodules—don’t, just don’t—and had a series of bash scripts that implemented a primitive CI/CD workflow by running in the environments we were deploying to (seriously). I can’t tell you how many merge conflicts and detached HEADs I encountered, or how I got out of them. Fortunately, we were using the fork-PR model, so I did no real damage.

Software configuration management (SCM) is key not only for versioning, but also for providing a shared view of code and other assets, from development through to production. Giving proper consideration to your SCM structure, branching and merging model, access controls, etc., will support collaboration, reliability and traceability—all key for DevOps success.

In the 2019 DevOps and Jenkins Community Report, 66 percent of organizations respondents reported the use of modern solutions SCMs, specifically GIT-based solutions GitHub or BitBucket. But the older SCM solutions aren’t going away completely—45 percent of all organizations still use pre-GIT, pre-DVCS (distributed version control systems) such as Perforce, Subversion and CVS. While it’s possible to apply DevOps practices using these legacy solutions, it is important (and more challenging) to design a branching/merging model and surrounding workflow, which supports the key aspects of DevOps such as peer review, collaboration and short-lived branches.

Modern SCMs such as GitHub have features that inherently support collaboration, including the game-changing Pull Request, making it much easier to implement processes that best support CD and DevOps.

Poorly designed CD workflows or inadequate CD systems create friction multiple ways. If the interactions are too cumbersome, users will not commit code to the pipeline frequently enough. If it’s difficult to access the pipeline and create visibility into the process, it can lead to unresolved conflicts.

You need to pick the right tool. If you don’t have the option to get the modern tool, you need to take the time to design a legacy tool.

It’s also important to consider two other issues in relation to this DevOoops example: branching and acquiring the appropriate level of DevOps skills.
- Branching: A proper branching and merging model is required to ensure you are feeding the appropriate inputs into your DevOps pipeline, such that you are able to continuously integrate, test and ultimately deploy changes. A simple guide here is encapsulated in the KISS (keep it simple, stupid) principle. If you need a full-wall whiteboard or a spreadsheet fit for an accountant to list your active branches, you need to rethink things. Get as close to the ideal of mainline trunk development as possible, with short-lived feature branches, minimal application versions, etc.
- Skills: As organizations seek to implement DevOps, most likely new tools and practices will be implemented simultaneously. It’s important to understand and accommodate the fact that there will be bumps and bruises as you learn to use these new tools and practices. For instance, shifting from centralized version control such as Apache Subversion (SVN) to distributed version control such as Git is a paradigm shift that can be difficult to fully grasp and result in destructive mistakes, such as deletion of nodes from the change history, overwriting changes or misconfiguration access control. Consider hiring for these new skills and/or account for time and training.
Takeaways
- A proper branching and merging model are required to ensure that you are feeding the appropriate inputs into your DevOps pipeline to let you continuously integrate, test and ultimately deploy changes.
- If you are using a pre-Git legacy SCM solution, develop a branching and merging strategy within the legacy solution to achieve appropriate velocity in your DevOps model.
- Consider that you will need to develop new skills to adopt new SCM tools and practice and account for this in your plan.
— Brian Dawson
March 11, 2019
‘DevOoops’ Moves: Poorly Defined KPIs
DevOps’ ultimate aim is to create efficiencies in the software delivery process. But, it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call a “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. The following is a discussion submitted by product marketer Juni Mukherjee about poorly defined key performance indicators (KPIs).

Scenario:

How we define KPIs can change the game. For example, one quality engineering team defined the number of tests executed per sprint as a success metric. Humans are driven by incentives and the natural tendency of the team was to add more and more tests without even considering archival of outdated ones. Think of this: We could have a larger impact with fewer tests and, more importantly, fewer tests lower the test cycle time. So, instead of sheer quantity, we should focus on the coverage and effectiveness of tests. Similarly, a release engineering team defined the number of releases per sprint as a success metric. The number of releases reflects on velocity for sure. However, releases move bits from point A to point B without assessing the value added to business. It is critical to tie a business KPI (such as the number of new customers acquired, percentage of revenue increase, etc.) to velocity, such that we know we are investing in valuable speed and not suicidal speed.

Management consultant Peter Drucker boiled his whole philosophy down to the following few words: “If you can’t measure it, you can’t improve it.”

Successful DevOps leaders take this saying to heart. To understand where they’re going, how to get there and ultimately how far they’ve come, these leaders know how critical it is to establish KPIs that fit the overall objectives of their DevOps transformations. KPIs provide appropriate and continuous insight—and they plot a route to help you get to your destination.

The big question is: Are you measuring the right things? If not, you could be spinning their wheels.

To avoid this “DevOoops” scenario, you need to start by establishing KPIs that not only align to your objectives but can also be established so that they create a continuous feedback loop. As the project progresses, as you mature and learn more, you will likely need to maintain, monitor and possibly adjust and update these KPIs.

What would be an appropriate test coverage goal? Teams all want to improve quality. But by how much? Seventy percent? Eighty percent? One hundred percent? This is an example of a KPI that needs to be considered in the context of your application as well as measured against your real objective.

You can have 70 percent test or code coverage, but if that’s just treated as a measure to test as much code as possible, you may miss the mark. If you are covering 70 percent of the wrong code and the 30 percent that’s missing is critical code, you’ve put in significant effort with the risk of little or no return on quality level.

In terms of your goals, increasing and monitoring test coverage may be an indicator of quality, but it may not measure your actual quality of work. In this case you want a KPI that’s measuring the outcome rather than the activity. Look at what is severity 1 and severity 2 defect rate in production. You can see code coverage get to 90 percent, but if you’re not seeing a decrease in severity 2 defects, you won’t really be seeing an improvement in our code coverage metric.

Takeaways:
- You’re not implementing DevOps just to say you’re doing it; you need to develop goals and objectives for your initiative and align to them.
- Once goals are defined, you need to align KPIs and success metrics to those objectives. Properly defining these success metrics and KPIs will dictate what gets prioritized when building pipelines.
— Brian Dawson
February 22, 2019
DevOoops Moves: Failure to Address Organizational Blockers
DevOps’ aim is to create efficiencies in the software delivery process. But, it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call a “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. Following is a discussion submitted by consultant Viktor Farcic about siloed departments.

Scenario:

A real obstacle is cultural and caused by having siloed departments. When we start looking at a system as a whole and detecting areas of improvements, the answer tends to be, “This is not our department; we don’t know them, they won’t speak with us.” Those siloes are the main argument in favor of DevOps; it tries to remove them. But, in reality, most companies fake it by creating yet another silo called the “DevOps Department.” DevOps in practice directly contradicts DevOps ideas.

More than a half century ago, while working on compilers, Melvin Conway nailed what is now the crux of the problem we are seeking to solve with modern software development and DevOps practices: “Organizations which design systems,” he stated, “are constrained to produce designs which are copies of the communication structures of these organizations.”

This concept, known as Conway’s Law, suggests how failure to break down silos and organize cross-functional, collaborative teams lingers today as one of the most overlooked steps to implementing DevOps and, despite all best intentions, results in dysfunctional software.

Most companies are organized by function—with business, project management, project owners, developers, quality assurance, app deployment, infrastructure and security groups all focusing on their own areas of expertise.

DevOps initiatives attempt to break down these silos. But it’s easier said than done. Most organizations embarking on DevOps journeys have to deal with an “us versus them” mentality.

Organizational issues create tension, resulting in difficult, friction-laden handoffs between teams. The problems often show up in the software delivery process and in the software itself in the form of quality gaps, inconsistent user experiences and other issues.

Picture this: Input to the application is coming from multiple nonaligned sources, each with its own particular variant on the user experience. If a business analyst and a developer interpret a customer requirement for the customer differently, the end product won’t meet expectations. If you have one team responsible for login and authentication and another responsible for the primary landing page, you can have a login identification that looks and feels different than the core landing page does.

Organizations need to acknowledge that culture will have a direct bearing on DevOps success and commit to taking on those challenges. It won’t be easy. It won’t happen right away. But, with some work, it can get done.

Here are three strategies for removing organizational blockers and avoiding this particular DevOoops.
- Alignment – Think about how to best align real or virtual teams organized around products, product functionality or features instead of skills sets or functional groups. For example, construct a small agile team around middleware that includes representatives from the business, developers, quality and operations input.
- Communication – Continually encourage communication, trust and transparency. This can be done by using real-time communication tools such as Slack and by centralizing and sharing data using tools such as Jira. Teams need to break down the lingering feelings of distrust. It’s OK to not be perfect. Organizations should set up reporting mechanisms that include successes and failures—all in the name of creating a better final product.
- Inclusion – Involve key stakeholders early and continuously. Once you have built these teams, as you embark on your DevOps journey, make sure all stakeholders in the software have a voice. You need to create a common understanding of what their individual needs are and align those around team needs, which sets the stage for ongoing collaboration and communication.
Takeaways
- Organize your teams around subjects such as product and functionality and include all the stakeholders in the software development process in these product or feature teams.
- Culture in DevOps is just as important as process, practices, tools and technology, yet it’s one of the most difficult things to change.
— Brian Dawson
February 7, 2019
A Certifiable Reason to Attend Training at DevOps World | Jenkins World 2018

There are many factors that make DevOps World | Jenkins World the essential DevOps event to attend each year. Whether it is the keynotes and speaker sessions from industry-leading DevOps experts, the expo where you can check out the latest and greatest from our friends in the software industry or the themed after-party, there are many reasons why year over year the conference grows. An added benefit that contributes to DevOps World | Jenkins World being the DevOps event of the year is the wealth of informative preconference training and certification. Think of them as the workout before the big game, providing you the tools and knowledge to make the most of the conference and beyond.

The growth of attendance and variety of attendees of the DevOps World | Jenkins World conference is one of the indicators that the industry is constantly innovating and evolving. Last year, Forrester declared 2018 as the “Year of DevOps,” with 50 percent of survey respondents reporting they had already implemented DevOps with plans to expand. For DevOps practitioners to keep up with the continuous pace of modern software development, there also has to be continuous learning. Training and certification at DevOps World | Jenkins World will cover relevant topics in software development, such as authorization and security, build management, instance administration and interfacing with cloud services, in addition to Jenkins and CloudBees Core fundamentals.

Not only are all training sessions available to add on to your full conference sessions pass, but you also can receive a free Jenkins certification exam voucher—a $150 value—for the Certified Jenkins Engineer (CJE) and Certified CloudBees Jenkins Platform Engineer (CCJPE) exams. You can choose to sign up for an onsite exam at DevOps World | Jenkins World (first-come, first-serve) or use it at a later date. There are also several half-, one- and two-day courses that can be added during the event registration process.

In addition to our recommendations below, check out the Training and Certification page to see what courses can help strengthen your software delivery pipeline skills.

DevOps Leader Certification Training

Successful implementation of DevOps requires leadership. Whether you are in a tactical or strategic role, if you help design, influence, implement or motivate the cultural transformation in DevOps adoption, this course will take your DevOps leadership to the next level.

Value Stream Mapping for DevOps

If you’re looking to better understand DevOps, you are in the right place! Join Mike Orzen, a recognized and prominent Lean thought leader and author, to learn the ins and outs of DevOps value stream mapping, helping you identify inefficiencies in your current process, and optimize it to deliver more value to your customers.

Jenkins Fundamentals

It wouldn’t be Jenkins World without Jenkins! To prepare yourself for both the Certified Jenkins Engineer (CJE) or Certified CloudBees Jenkins Platform Engineer (CCJPE) certifications, Jenkins Fundamentals introduces you to the best practices for leveraging Jenkins to drive your continuous development life cycle with Jenkins.

CloudBees Core Fundamentals

As the foundation of CloudBees, once you’ve mastered Jenkins Fundamentals, CloudBees Core Fundamentals is the obvious next step. In this training, attendees will gain the skills needed to leverage the CloudBees Core product suite to successfully scale CD and DevOps in their organizations. The training includes an overview of the key proprietary features provided with CloudBees Core to enhance the capabilities of Jenkins.

CloudBees Core on Kubernetes

Kubernetes is seeing widespread adoption as the operating system for cloud native applications and the industry’s go-to container orchestration system. This course will provide attendees with the tools needs to run CloudBees Core on Kubernetes, and leverage public and private clouds to establish an elastic, scalable CD and DevOps platform. The course includes hands-on installation of CloudBees Core on Kubernetes.

Go to https://www.cloudbees.com/devops-world to learn more and to register. Additionally, use this special code just for devopsy.kinsta.cloud readers—JWDEVOPSCUST—at registration, and you’ll get a 20 percent discount.

— Brian Dawson

August 20, 2018
7 Signs You’re Mastering Continuous Integration
A majority of organizations think they are doing continuous integration (CI), yet when asked if they are following key practices of CI such as running tests to validate every build, many of them aren’t. Ask yourself the following: “Are we building our application with every change, or at least multiple times a day? Do we avoid long running feature branches? Do we prioritize fixing broken builds over committing new changes?” If your answer to any of these questions is “No,” then guess what? You are not truly doing CI—and you’re not alone.

Continuous delivery (CD), a foundation of DevOps, is built on the practices of continuous integration. Accordingly, you have to implement CI properly before you can truly achieve success with CD and DevOps. In other words, you have to crawl before you walk.

Steps to Mastering Continuous Integration

To reach business goals, here are the seven practices organizations must do to master CI:

Automate the Build—the Entire Build—on a Central Server

This seems like an obvious one—and it is. Sure, prior to implementing continuous integration, organizations either build locally on a developer’s desktop, have a build engineer manually integrate code or have to work through a series of fragile, manual steps that are prone to error. But, surprisingly, many organizations that claim to practice CI still get this one wrong.

Simply put, your continuous integration build should run a central shared server, be able to run completely unattended and be automated such that the same input will produce the same, same output every time.

Build as Frequently as Possible

Teams practicing CI build multiple times per day, ideally triggering the build per change on a pull request, merge or commit. Building per commit enables errors to be found as soon as they are introduced and are more easily fixed.

The word “continuous” in continuous integration should be taken as literally as possible. Often times when I ask organizations how frequently they build, the answer is one of the following:
1. We do “push-button” builds, X times per-week
2. We build both every night, and once per week
3. We build once per-day at X time
As you have likely deduced, these are not “continuous” builds. They may be central builds, automated builds or scheduled builds. But those don’t provide the benefit of continuous integration. If a build breaks and you have two days of 30 commits by 10 different developers, you may have to sort through all of that work to figure out and correct the cause of the build. Productivity killed.

Commit or Merge to Mainline Frequently

Teams doing CI are likely committing/merging to trunk or main with every change or at least once a day. If you are maintaining long-running feature branches, for instance, you are not getting the benefit of validating integrated changes frequently. If you branch for a long period, when it is merged, the integration has been delayed and you may be left with conflicts or bugs to fix.

Build Quickly

If you are practicing continuous integration, your build likely completes within 5-10 minutes. If builds take longer (some organizations have five-hour-plus CI builds), not only are you delaying integration and, therefore, valuable feedback, but developers are less likely to commit changes because of the cost to their productivity—or, even worse, they continue to commit against broken code.

Take time to optimize your build and validation time. This may require you to decompose your CI build process to align with components or teams and then validate the entire integrated application as a separate step in the overall development process.

Validate the Build

CI includes validating the changes and resulting built. This kind of testing should be a part of the overall build process. If your only validation is based solely on confirming the software compiles and links, then you are not truly practicing CI. Most teams practicing CI include code scans and unit tests in the CI build.

Validate in a Production-like Environment

Mature CI practitioners ensure consistency between the dev, test and pre-production environments. The most mature are using VMs or containers, service virtualization and other techniques to ensure they are performing validation in a production-like environment and reducing the risk of discovering errors downstream in pre-production or production, well after changes were integrated.

Fix Broken Builds Immediately

As mentioned, it’s critical that teams find and fix issues quickly so that they don’t move downstream. With CI, processes are set up where builds are validated and committed continuously, so if something is problematic, it’s easier to fix. If broken builds are ignored, the issue will be more difficult to find and fix, but worse is the culture it encourages. CI is based on a quality-first approach. If the build breaks,“Stop the line” and fix the build. Everybody will be better for it in the long run.

Don’t Forget About Culture

If an organization can look at these seven signs and say it does all of these things, then it is really doing CI. These may seem simple, but when these principles are combined, the outcome is remarkable.

Organizations that can’t say they follow each of these principles likely will have problems delivering functional builds regularly. The initiative will fail and team members will become frustrated and revert back to their old ways. Not only will there be a process issue, but also a cultural issue—and this is where many organizations get stuck.

Teams must change how they work together and they must also make sure they are aligned in their mindset. CI has many benefits for organizations, but don’t forget that it’s actually designed to embrace failure, not eliminate it. Continuous integration creates a system where developers can fail fast, so they can find and fix errors early and quickly.

Continuous integration is one of the foundational elements to a real DevOps transformation, which everyone knows is a top priority for many businesses today. Getting it right—truly getting it right—is essential to keeping up with modern business needs.

— Brian Dawson
July 18, 2018
Tales of DevOps Discovery, Part 3: Q&A and Explanations
This is part three of the series, Tales of DevOps Discovery. To catch up, Read part one and part two.

In preparing for a DevOps transformation a critical first step is to perform upfront discovery, often asking even the most fundamental, seemingly obvious, questions of stakeholders. The major portion of such discovery should be done in a group setting. Asking stakeholders these fundamental questions in a group setting can provide key insights into differing perspectives and even differing objectives regarding a planned DevOps implementation. Sharing these different perspectives during the discovery phase sets the stage for mutual empathy and enables stakeholders to identify and align, an important step to aligning on shared objectives. In this article we will outline common questions asked, share real-world answers and offer guidance on interpreting and applying the results.

Discovery Question: What do you want to achieve with this transformation/DevOps?

One of the first things to consider is, “Why are we doing this?” as this establishes the goals and objectives. Accordingly, I usually begin by asking the stakeholders during discovery what they expect to achieve through the transformation to DevOps. The answers I receive to this question often differ distinctly, depending on the role of of the person answering. Understandably, they are heavily influenced by the unique perspective of that role.

Answers:

CIO: “We want to increase our ability to “scale-up” delivery as we shift to a primarily SaaS (software-as-a-service) model and see a rapid increase in business. These means we have to increase developer productivity, and respond to customer requests faster, all while reducing our number of Severity #1 issues.”

Operations:
“We want to reduce technical debt, to minimize firefighting and increase our ability to adopt modern best practices such as Infrastructure-as-code. We have to be able to keep the lights on while delivering applications to production.”

Development:
“We would like to minimize customer-specific one-off work, which is resulting in painful complexity in development, QA (quality assurance) and deployment, and is an overall source of inefficiencies. We would like to be able to spend more time improving user experience, performance and increasing valuable functionality. But, instead, we are chasing bugs and reworking misunderstood requirements.”

Guidance:

Not surprisingly, executives often respond with a focus on business-oriented goals such as increased productivity and customer response, while Operations focuses on stability and minimizing firefights and Development wants to innovate. But within these differences we see some commonality that can be harnessed, such as productivity. For example, one point of commonality is productivity:

CIO: “We have to increase developer productivity.”

Operations: “… minimize firefighting and increase our ability to adopt modern best practices …”

Development: ” … spend more time improving user experience, performance and increasing valuable functionality.”

Another point of commonality in the CIO, Development and Operations answers is the focus on quality:

CIO: “ … reducing our number of Severity #1 issues.”

Operations: “We want to reduce technical debt … keep the lights on …”

Development: “… instead, we are chasing bugs … ”

These points of commonality can be used to establish common goals with cross-functional buy-in, such as:
- Our effort will result in increased productivity and minimize overtime and firefights (across teams)
- Our efforts will result in increased application quality and uptime
They then can be used not just as driving principles that inform decisions such as process or tool choices, but also to establish key performance indicators (KPI), which will be used to quantifiably measure success of the transformation. Example KPIs may be:
- Reduction in cycle time, from story acceptance to user acceptance (or release)
- Number of Severity 1, or Severity 2 production issues per release
- Defects per build
- Defects per release
- Hours worked by Development? Operations?
Discovery Question: Do you practice agile today? If so, how do you define done for a story or sprint?

Continuous delivery (CD) and lean practices are foundational to a successful, sustainable DevOps transformation (though not required to meet the some definitions of DevOps.) If we look at CD and lean practices as a pipeline through which we need to manage a continuous flow of changes represented by work, then it is important to consider what work is sent into the pipeline, how the work is initiated and how the changes are managed. Accordingly, it is important during discovery to ask customers high-level questions about development methodologies used and overall practices for definition, planning and coding.

The answers given vary between different levels of, “Yes, we are agile,” “No, we are waterfall,” and, “We practice both.” Most Most often, organizations’ answers describe adherence to a form of agile with room for improvement.

Answers:

“We practice agile and/or scrum. We work in two-week iterations, planning the work for a sprint at the end of the prior sprint.
We consider a story done once the code has successfully been committed and merged to a team branch, and is successfully built. We consider a sprint complete when all stories for that sprint are committed and successfully built.

“Roughly every three months we integrate code from team branches to trunk and perform a mix of automated and manual regression and acceptance tests. Once the build passes QA we manually deploy to preproduction environment, validate and schedule a manual deployment to production.”

Guidance:

This is a classic story of agile adoption in traditionally waterfall organizations. The organization adopts agile within Development without involving other key functional teams such as QA or Operations. Usually, this is because it simply is easier. There are a number of obstacles to overcome in extending agile practices across organizational, technical and cultural boundaries, so the obstacles are avoided by limited the agile adoption to these boundaries.

This is a case of confining the adoption to “agile upstream” while practicing waterfall or legacy “downstream.” Studies show that less than 33 percent of organizations have adopted “agile upstream” and exercise agile or iterative development practices for the planning, coding and team builds, yet only 13 percent of organizations have adopted agile downstream, and follow practices such as continuous integration (CI), CD, automated provisioning testing and deployment.

The negative effects of limiting agile adoption to agile upstream are multifold, but can be summarized as, “You never really know when you are done-done.” As a result, you cannot accurately measure progress, introduce deferred risks or disrupt the development plan to address issues found during the downstream portion of the process, such as poorly understood requirements surfacing during acceptance testing or runtime issues discovered during deployment to a production-like environment or, in the worst case, issues found in production by the end user.

A seminal book on DevOps is, “The Phoenix Project,” by Gene Kim, which presents three ways of DevOps thinking. The first way, “Systems Thinking,” addresses to this gap between agile upstream and downstream, Dev and Ops, business and customer by taking a systems view across the traditional boundaries and establishing “flow” of the system through the boundaries.

Discovery Question: Do you practice CI today? What types of validation do you perform?

As CD is foundational to DevOps, CI is foundational to CD. CI can be scene seen as the connection point for agile upstream to agile downstream, and the pump that fills the CD pipe. It is both the practice most organizations say they follow and the practice that most often is implemented incorrectly. So when surveying a customer’s current state during discovery, it is important to know whether CI is currently part of the picture.

Answers:

“Yes, we practice continuous integration. Each team has a Jenkins CI server. Developers commit their changes multiple times per week. Jenkins then builds the changes as part of a nightly build.”

“We do some unit testing, but we do not track code coverage. If the nightly build fails we may continue to commit, but we usually fix the issues within a few days.”

Guidance:

This is a common answer and an example of how CI can be improperly implemented. CI is more than having a shared Jenkins CI server and running scheduled builds. CI can be defined as:

“A a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” — Martin Fowler

The benefit of practicing CI includes identifying integration and quality issues close to the source of the issue, thus reducing the time to correct and enabling higher-quality software to be developed more rapidly.

Essential elements of properly implemented CI are :
- Integrating changes frequently, ideally multiple times per day or even per change.
- Performing consistent validation of quality through component level testing, code scans, etc.
- Always maintaining a working build, suspending development when a build fails, and not resuming development until the issues is corrected or reverted.
Teams that allow a broken build for “a few days,” deferring commit or builds to just “multiple time per week” or that do not validate consistently through testing practices are not practicing CI and will not yield the benefits of CI.
July 26, 2016