devopsy.kinsta.cloud

Category: Doin’ DevOps

Do You Continuously Deliver Your CD Pipeline?
If your job is to build and maintain the continuous delivery (CD) pipeline for your organization, do you feel you can’t keep up with your development counterparts in innovation? Everyone is raving about the new cool features in your company’s product. Who deserves the credit? People who wrote the code, right? It’s not that simple. Churning out new features requires a sound process and solid infrastructure. A well-designed CD pipeline is crucial to the developers’ success. It mitigates the inherent risks in a fast-paced, frequent iteration regime now prevalent in agile software shops. On the flip side, as software gets more and more complex and the appetite for a faster time to market grows, opportunities for innovation are in abundance for DevOps teams.

Multi-speed IT

So you’ve got two tactical teams with significantly different risk-reward equations, working at a different pace due to distinct objectives but marching toward the same strategic goal. You’re finding yourself in a multispeed IT environment. Development is focused on innovation and quick frequent iteration cycles. DevOps’ primary focus is to ensure the stability of the CD pipeline and keep a precise track record of what takes place inside the pipeline every step of the way. How do you make sure to stay ahead of the technological curve while minimizing risk?

Based on my experience on both sides of the DevOps isle, I recommend the following four steps to successfully build and maintain a continuous delivery pipeline:

1. Automate, Automate, Automate!

My personal motivations for rolling up my sleeves and scripting a task are:
- It became an annoying, mindless routine;
- It’s no longer a one-time thing and isn’t going away;
- My time can be spent on more important things; or
- I make mistakes and then waste time fixing them.
As with any type of code, organize the scripts into a sound logical structure easy to navigate and version-control it. You can always revise the code or completely rewrite it as the needs evolve.

2. Treat Automation Code as a First-Class Citizen

That’s right! As a practitioner, I learned the hard way that automation starts innocently small and simple. Then the requirements for more automated functionality begin to pile up rather quickly and things quickly become really messy. So write high-quality code from the get-go and reassess the design and implementation every step of the way. Remember that the visibility of your code is very high.

Apply the same standards to your code as your peer development teams do to theirs. Write tests—unit, integration, functional, end-to-end—as appropriate. Use the same CD pipeline to automatically build, package, deploy and test your automation code. This will give your pipeline another round of validation, your code will be of a much better quality, and your team’s credibility and reputation will soar.

3. Containerize the CD Pipeline Components

In the past, because the CD pipeline components such as build servers were manually provisioned, they often became the single points of failure, which made it hard to plan and roll out the modifications to homegrown, open source and commercial tools. It wasn’t unusual for some components to fall behind the latest release by a full year or more. In the meantime, a year’s worth of updates and patches could address a litany of issues, but the need for a maintenance window, usually on a weekend, and the uncertainty of the outcome often would outweigh the sought-after benefits. Containerizing the CD pipeline component can be the solution. Running a build server in a container, for example, will make it a lot easier to create an exact copy of the production version in a test environment and iron out the upgrade process more thoroughly ahead of the upgrade.

4. Test Commercial Tools

Commercial and open source projects test their software, of course. However, it could make a whole lot of sense for your organization to create your own integration, end-to-end and performance tests for the software you don’t write—especially if the software has a plug-in architecture like Jenkins or UrbanCode Deploy and you use community or homegrown plugins. Commercial tools typically don’t provide any guarantees that those plugins will work after the upgrade. Running your own automated tests that exercise the functionality of the tool in your environment and on your specific data gives the opportunity to validate updates as they become available and identify any regressions proactively.

The objectives of DevOps are not only to identify and eliminate the existing bottleneck in the organization’s continuous delivery process, but also to look out for any future potential bottlenecks and proactively address them before they become real issues.

So your team is DevOps. Your product is a CD pipeline. How do you continuously improve the CD pipeline for your product?

If want to learn more about how to effectively build and maintain your continuous delivery pipeline, please check out the newly released 2nd edition of the “Application Deployment and Release for Dummies” eBook.

About the author / Alex Seriy

Alex is an accomplished technologist and DevOps evangelist with over 25 years of experience under the belt. He is an automation enthusiast and usability guru, passionate about uncovering new use cases and innovating applications for platforms, frameworks and tools. At IBM, Alex is a subject matter expert in Continuous Integration, Delivery and Testing. He currently focuses on integration testing in the API Economy through service virtualization. Prior to joining IBM, Alex served as the Senior DevOps Architect at TIAA-CREF for 4 years. Before that, he was a consultant specializing in SCM, Release Engineering, QA and DevOps for over 10 years.

Connect with Alex on LinkedIn / Twitter
June 14, 2016
What Happens to Test in a DevOps World?
I am always surprised by the urban legends around DevOps. One of my favorites is, If you implement a DevOps methodology, you can stop doing test. People famously quote that Netflix only tests in production as evidence of this. This is perhaps more like of game of Telephone, where much of the story gets lost in translation. It is true that Netflix created a tool called “Chaos Monkey” which generates failures in production as a way to test the fault tolerance of its software in large-scale production use. However, this is not Netflix’s only test; it is an additional type of testing at scale that allows the company to test in new and different ways. Netflix still tests in software development.

Okay, now that we are past the notion that DevOps is about eliminating testing, let’s look at what really happens with test in the DevOps world. In fact, the goal is for testing to happen everywhere and all the time. Let’s look at the two directions test can shift: left or right.

Shift Left

DevOps methodology encourages automation of activities across the entire development process. This can be realized by automating the link between development and test. Often implemented with an automated build tool such as Jenkins, a set of automated tests are triggered whenever a developer completes a build of new code. Of course, this means that both development and testing processes must be automated. Once you reach this goal, then testing can happen all the time. Every time developers want to check in new code, they can trigger a small set of “smoke tests.” If those succeed, they can check in code and when the build happens, they can trigger a larger set of regression tests. The result is a left shift of testing into development.

Benefits of left-shifting test include:
- Move testers into scrum teams to work on stories together with developers.
- Design tests in parallel with development.
- Give developers and testers the flexibility to help each other to reduce bottlenecks.
- Early detection of bugs, shortening the critical metric for time between creating a bug and fixing it.
- Increased transparency of code dependency issues because they are raised during the nightly (or more often) regression testing.
- Virtual elimination of the test bottleneck that occurs after development is completed.
Shift Right

The urban legend about Netflix may be wrong, but that doesn’t mean that there isn’t merit to right-shifting test into production. Automated testing in production is effective for tests that would be impossible to perform outside of production, including scale testing and resiliency testing. Production is also a great place to introduce A/B testing by offering different interfaces to some customers and measuring the differences in business results.

Benefits of testing in production include:
- Better understanding of the real production environment and workloads.
- Early feedback from customers on new features and user experience.
- Better resiliency.
- Overall better user experience for customers.
No Shift

Wait, if we shift left and shift right, is there any testing left to be performed in test? There are still a range of tests that need to be performed by the test team. With full automation, these tests can be performed at a variety of times and don’t need to be a bottleneck to release. These tests include:
- Performance testing
- End-to-end use-case testing
- Security testing (don’t think about doing this in production!)
- Manual testing – often for user interface or mobile testing that is most difficult to automate
DevOps methodology is not promoting the end of testing, but in fact a widening of testing to encompass all of the steps in the DevOps cycle. You know what they say: Test early and often! And next time you are sitting around the campfire discussing urban legends, let’s leave Netflix out.

About the Author/Joan Wrabetz

Joan Wrabetz is the CTO for QualiSystems. Most recently, she was the VP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.
June 13, 2016
Where’s the Heat? Talking DevOps, Use Cases and More

Transitioning your organization from a traditional IT infrastructure into the world of DevOps is certainly a journey. Sometimes a long one, with many obstacles, but undoubtedly a voyage with a worthwhile prize at the end. Many organizations just embarking on their DevOps journey find themselves wondering, what use cases should they attack first? And who is driving these initiatives?

Recently, Amy DeMartine, senior analyst at research leader Forrester, and Wesley Pullen, general manager and vice president of Deployment Solutions at Electric Cloud, participated in a discussion to answer these questions and discuss how to orchestrate the DevOps tool chain, how to approach application release automation, and how DevOps fits in the greater scheme of compliance and auditability. For folks just beginning their DevOps journey—and IT pros who have been walking the walk for years—there were some great takeaways from their discussion.

DeMartine started off the discussion with something so basic that many folks often forget to consider it before starting their journey. But it is one of the most fundamental and important steps before beginning to explore DevOps; that is, answering the question, “What is DevOps?”

Here’s a great definition used by Forrester:

DevOps provides a set of practices and cultural changes—supported by complementary tools—that automates the software delivery pipeline, enabling organizations to win, serve, and retain consumers better and faster than ever before.

Part of this is the idea of taking the huge monolithic releases of the past and breaking them up into smaller and smaller pieces—at least on the agile development side of things. Then infrastructure and operations folks get involved and complete the continuous delivery life cycle by automating builds and testing and making sure releases are consistent and fast. Finally, there is collaboration among development, operations and security to make the product even more successful than before—and in all of this, you are creating feedback loops and building quality as well as consistency.

So now that you’ve got an established idea of the fundamentals of DevOps, let’s dive right into use cases. One of the most often-cited use cases is the idea of governance. People venturing into DevOps love the idea of moving faster and delivering more quickly than ever before, but find themselves asking, “How do you do that safely?”

It’s helpful to look at this using the visual of an upside-down triangle, with governance at the top. It applies to all companies and gives them the ability to control their releases all the way from development to production. Part of the benefit of using DevOps is that all of these tools you are using are creating log files, which essentially are breadcrumbs that give you an easy way to follow an audit trail. If you’re using a standard technology stack across a single application and apply it to multiple applications throughout your organization, you have the ability to look at governance across all of these various applications through the same lens.

On the next level of the triangle is compliance, which, depending on your industry, can mean a variety of things, but they are usually predefined and concrete. Finally, the bottom of the triangle is the audit aspect, which is where the best practices of DevOps become the most important.

DevOps Drivers

Ultimately, there are three different drivers of use cases that we see in DevOps, each with unique needs and perspectives. The first of these drivers are folks involved with infrastructure and operations. We often see projects and initiatives coming down from CEOs, CIOs and other senior executives in which the operations staff is driving the initiative and ultimately is responsible, so the initiative gets the budget. This is very hot in financial services.

Another driver are members of IT ops, DevOps and enterprise architects. These members often are focused on cross-team, cross-project initiatives. These are the senior architects, developers and operations staff that all are working together on a single solution. They already have the tools in place, but want to drive and orchestrate an end-to-end solution. Whether they are doing a build or need to make a quick change to provision a server, their goal is to consolidate and have a seamless pipeline.

Finally, the development-driven initiatives are coming from senior architects, development leads and QA who are pushing all kinds of different automation aspects, including continuous integration, continuous testing and continuous delivery.

Diving deeper into these heat zones and specific drivers, it’s helpful to look at what these groups are really focusing on. The top automation initiatives within the subgroups of infrastructure and operations folks start with the need to do deployments faster. They need to drive deployment automation faster. They also are interested in release management, or coordination across a release stream for different teams. When an organization has a series of activities across multiple teams, there is a distinct need to release everything at once to make sure it works. This group also is focusing on compliance and audit—particularly in highly regulated industries such as health care, financial services and insurance—and is always driving initiatives in which it’s trying to build out auditing packages.

In cross-team-driven initiatives, members include IT ops, DevOps and enterprise architects who are focused on cross-teams, and the cross-project initiatives primarily focus on end-to-end orchestration. They also are concerned with cloud resource management. When teams are paying for tools, they want to make sure they are maximizing their value and getting the most use possible out of those resources. That can mean scaling up or down in strategic ways to get the biggest “bang for their buck.” Finally, this group also focuses on configuration management orchestration.

Lastly, developer team-driven initiatives are focused not only on items such as continuous delivery, continuous integration and continuous testing, but also on container orchestration. As there is a greater awareness of potential security holes in Docker, it is necessary to place an increased emphasis on container orchestration, particularly with more and more individuals moving to containers. There also is an increased focus on QA lab provisioning. With the incredible growth in the realms of the Internet of Things (IoT) and embedded, these areas are correspondingly heating up for developer team-driven initiatives. One example put forth in the discussion between DeMartine and Pullen was a recent issue around one of the most well-known IoT devices on the planet—the Nest smart thermostat. In January, Nest went down, leaving many people without heat. This is a great example of why development teams want to focus on IoT use cases: DevOps enables rapid fixes for the massive demands of the IoT world.

Ultimately, you can see from these use cases and initiative drivers that DevOps offers an incredibly dynamic opportunity, ideal for meeting the needs and perspectives of organizations from top to bottom—and the incredible growth of DevOps, as the organizational structure for IT organizations across multiple industries proves that out. In fact, DeMartine and Pullen polled the audience during their discussion and found that 61 percent of attendees planned to adopt DevOps practices in the next 12 months, while 83 percent of attendees planned to adopt deployment automation/ARA in the next 12 months.

If you aren’t already considering DevOps for your organization, whether you are and CEO or a developer on the front line, it is worth investigating to see how it can answer your most pressing challenges and use cases.

The full conversation between Amy and Wesley is available here.

About the Author / Sam Fell

Sam Fell is the Head of Marketing at Electric Cloud, responsible for driving awareness and demand for our DevOps Release Automation and Acceleration solutions. For 20 years, Sam has helped companies achieve success by marketing, selling, and implementing DevOps Release Automation, IT Change Management and Application Performance Management solutions. Prior to joining Electric Cloud, Sam held leadership roles at BMC, CA, Marimba and Nimsoft.

Sam holds a Bachelor of Arts degree from Illinois Wesleyan University.

June 10, 2016
How DevOps is Killing QA
If you work in software quality assurance (QA), it’s time to find a new job.

The overall software development process has always consisted of the following core activities:
- Creating a requirements analysis;
- Producing a specification of the software artifact;
- Constructing the software;
- Investigating the quality of the product or service;
- Finding and fixing defects; then
- Deploying to production.
In most organizations, this process generally has occurred within one or more methodologies such as waterfall or agile. In waterfall, the testing process occurs relatively late. The software is produced, bugs are found by the QA team and then the software is fixed. These last two practices repeat in a cycle until management is comfortable that it can be released to the public.

In agile, individuals and teams—including QA—work closely together to release and then update software on an ongoing basis rather than deploy an entire platform all together and all at once. DevOps, as we have written in our “What is DevOps?” guide, is the next generation of agile development. Agile is a way of thinking in software development; DevOps goes one step further and is a concrete cultural change that implements the philosophy within an organization.

And that implementation eliminates the need for QA as a separate entity in the organization and disperses the responsibility for quality across different teams, even though many have argued that the discipline is still needed in one form or another.

Past Interpretations of QA in DevOps

One common argument is that DevOps sits at the center of a Venn diagram of dev, ops, and QA:

On one side, QA works together with development in trying to push more of their tests into the continuous integration system. Tests must have zero human intervention and generate their own test data.

On the other side, QA works with operations to collaborate on monitoring tools; perhaps to continuously run smoke tests in production. Chances are that operations is developing scripts for backup and restore of production systems, rollback of deploy, or scripts for disaster recovery.

Tim Hinds at NeoTYS looks at the issue differently and says that “DevOps QA” is meant to prevent defects rather than find them:

QA takes a critical role in this organizational structure because they have the visibility and the directive to push code out when it is working, and roll it back when it is not. This is a very different mindset from QA organizations of 10 years ago, whose primary responsibilities involved finding bugs. Today QA groups are charged with preventing defects from reaching the public site.

With all due respect, both are wrong.

Why DevOps Does Not Need QA

DevOps often uses continuous integration (CI) and continuous delivery (CD). In CI, developers use various continuous integration tools to integrate code into a shared repository multiple times each day, and DevOps relies on automation to determine the quality of the version. You cannot have any human interaction if you want to run CD, which ensures that any version of a batch of code in the repository can be released at any given time.

Essentially, the traditional QA cannot work in a full CI/CD environment. In older structures, the responsibility for software quality was in the hands of QA. Today, it’s part of the DevOps culture and methodology—the developers now own the responsibility rather than a separate entity within the organization.

Specifically, DevOps entails the use of vendors and tools such as BUGtrack, JIRA and GitHub to aggregate and report software bugs and defects. Test automation tools including Selenium, Cucumber, JUnit, TestNG and JMeter manage, execute and measure functional tests and cycles.

In the end, if there is a layer of people in the middle between development and operations, then you cannot perform CI and CD seamlessly and, therefore, cannot do DevOps. To run a proper DevOps operation, you cannot have QA at all.

The Future of QA

So, what will happen to all of the people who work in QA? One of the happiest jobs in the United States might not be happy for long as more and more organizations move to DevOps and they become more and more redundant.

According to the U.S. Bureau of Labor Statistics (BLS), software quality engineering is one of the high-tech professions whose growth rate is projected to be slower than average:

However, the BLS numbers are likely too generous because the bureau does not yet recognize “DevOps” as a separate profession at all:

As evidence, just look at how the relative number of Google searches in Google Trends for “sqa jobs” is slowly declining while the number for “devops jobs” is rapidly increasing:

For QA, the trend definitely does not look good.
June 7, 2016
DevOps: Shift Left to Reduce Failure

DevOps is all about increasing the feedback loop from idea to customer and back again. Tremendous gains have been achieved in automation and tools to drastically reduce the amount of manual labor involved in progressing down the release pipeline. But mission-critical applications in a highly competitive market cannot afford to go down. Complex applications, environments and infrastructure increase risk, which can cause failure.

The failure rate of production deployments is still too large. Studies show that the average cost of a production outage is roughly $5,000 per minute, while the average cost of a critical application failure per hour is $500,000 to $1 million.

Why so much failure? All too often we’re still seeing a lack of consistency across the release pipeline. Many times the provisioning and deployment processes are very different at the opposite ends of the pipeline. But by shifting left the operations concerns, you can achieve much more success in production.

Understanding How to Shift Left

The term “shift left” refers to a practice in software development in which teams focus on quality, work on problem prevention instead of detection, and begin testing earlier than ever before. The goal is to increase quality, shorten long test cycles and reduce the possibility of unpleasant surprises at the end of the development cycle—or, worse, in production.

Shifting left requires two key DevOps practices: continuous testing and continuous deployment. Continuous testing involves automating tests and running those tests as early and often as possible, along with service virtualization to mimic unavailable systems. Continuous deployment automates the provisioning and deployment of new builds, enabling continuous testing to happen quickly and efficiently.

The first and most obvious way to shift left operations is to work side by side with development in creating the deployment and testing processes. Failures observed in production often are not seen earlier in the life cycle. Many times these failures can be attributed directly to differences in deployment procedures. Development may create its own deployment procedures that are very different from those used by operations for production. Sometimes production procedures are much more manual and may even use different tooling. Operations and development need to take ownership in building standard deployment procedures. The deployment process is then practiced potentially hundreds of times in test environments before reaching production. You then have much more confidence that the production deployment will be successful.

Another way to reduce the failure rate is to make all environments in the pipeline look as much like production as possible. This is almost impossible using traditional provisioning processes. Production environments usually are much heavier and detailed than are needed by development. But by using cloud and pattern capabilities, you can get much closer.

The ability to use patterns to define consistent environments eliminates the failures that occur simply by configuration inconsistencies. This again requires operations and development to work together in creating the provisioning process so developers create their test environments the same way they are created in production. Using IBM UrbanCode as an example, this can be done by connecting UrbanCode Deploy to an automated self-service provisioning system such as OpenStack HEAT or IBM Cloud Orchestrator connected to OpenStack.

These standard patterns can be used to create environments for all stages of the pipeline. And by using cloud capabilities you can deprovision these environments as fast as you can create them. This also allows the enterprise to have confidence in their environment patterns. They are tested on a daily basis and, therefore, operations can feel confident that testing is done on production-like environments.

No one wants to experience production failures—and it is in everyone’s best interest to do all they can to prevent them. At the root of DevOps is the removal of those silos between development and operations. By applying these shift left operations principles you are well on your way to reducing your failure rate.

The ingredients to make shift left are real and available: automated testing and deployment tools, service virtualization technology, agile practices and cloud environments. The main obstacle that remains is the mindset of delivery organizations. Learn more from the newly updated Application Deployment and Release for Dummies ebook to find out more.

About the author / Dr. Darrell R. Schrag

Dr. Darrell R. Schrag (@devopsdrs) is an experienced software lifecycle consultant specializing in the IBM cloud solution, helping customers establish their cloud strategy across the enterprise. He’s focused on implementing continuous delivery of software applications through the delivery pipeline incorporating cloud and provisioning solutions. He’s experienced in helping clients transition to “born on the cloud” applications utilizing microservices to take advantage of a flexible and nimble cloud platform.

June 2, 2016
Learning in the Continuous Customer Economy
We live in a continuous customer economy. Buyers in the 21st century are continuously discovering, adopting and discarding products and services online, telling us—very loudly and unequivocally—that value isn’t perceived in the actual product itself, but rather in our ability to change a product continuously. So, it’s now about the options to change the product, not the product options themselves.

This utter lack and loss of brand loyalty puts companies at an insane disadvantage. They’re living with a “see you later” digital mindset, where it’s easy for consumers to say goodbye to something they know online for an opportunity with the newest, next best thing.

Take this to the next logical business level and you see why continuous transformation of your organization is so critical. If you can’t continuously transform the way you deploy your people, processes, skills and resources, so everything is aligned with the consumer’s “see you later” mindset, there’s no way you can keep up with customers, who are continuously discovering, adopting and discarding products at what seems like warp speed.

Obviously, this means learning and acquiring new knowledge fast.

The same old knowledge, the knowledge of the past, just doesn’t work in the world of now. But, more profoundly, to continuously transform your organization and meet customers where they are today, you need to learn how to learn.

Yes, sure, you have to learn from customers and gain an understanding of what they want and need. But every organization does that today. It’s no longer enough to simply grasp that insight and intelligence and run with it. No, you have to continuously improve how you learn from customers in this fluid and fast-changing world.

Why? Because, at the end of the day, continuously improving how you learn is the only way to continuously transform your organization. And, thus, continuously improving how you learn is the only competitive advantage you have.

Continuously Improving What—and How—You Ship

Assume, for a moment, that you’re shipping an update of 1,000 lines of software code each week.

That’s great.

You’re likely using continuous delivery.

But if you can’t continuously improve how you’re learning from customers and continuously improve the teams inside your organization, you can’t ship continuously improved products. So, you need to improve what you ship and how you ship to continuously transform your organization.

Here’s an example of how this works.

Let’s say your software development team senses the customer wants an improved red button that’s square instead of round. Okay, development of an improved red button proceeds and is completed in one week. But, at the same time, the team creates an automated test for the red button as well as clear metrics for determining if the red button is actually the problem. Remember, the idea that the button needed changing is a hypothesis until it has been tested with customers.

So, in addition to shipping the new square button to some customers, the team keeps some subset of customers using the round button. The customer test is then simply whether the click rate of the square and round button differs.

Let’s imagine that the processes are not currently in place to enable the development team to receive these metrics, as many teams are kept in splendid isolation from the results of their work. Then the establishment of this relationship, perhaps with the web operations team, is also going to be required.

This is as much a requirement of successful deployment of these features as the features themselves. The focus on the delivery of function, as well as developing internal capability, is a core principle of continuous transformation.

The Two-Product Principle

This is the Two-Product Principle.

Think of the product we ship to the customer as “Product 1.”

Meanwhile, the internal capabilities we wish to develop—our view of what we need to improve while we build—is “Product 2.”

To recap two critical points:
- You always ship twice—you ship an improved product externally and you ship improved capabilities internally.
- All of our product decisions, what to build and what to deliver, are theory until the customer actually uses the deployed product and we learn from that interaction.
If you take these two critical points to heart, then your organization will do all it can to shorten the time to deployment and action to shorten the time to gain new learning in the market. This limits the length of the costly business guessing game, the cash-draining theory time.

Continuously Learning at High Velocity

The way you can do this is by continuously shipping, continuously delivering, continuously learning how to improve how you learn and deliver—continuously improving your internal capabilities and continuously transforming.

Unfortunately, many organizations that want to increase velocity go faster without improving their ability to learn at this new speed.

The implications of this short-sightedness are profound: When you’re leaning and looking backwards, you’re not improving how you learn from your customers in real time and, instead, you’re adding to your risk going forward because you’re so out of step, and out of touch, with what the market wants and needs.

Every Act of Design is an Act of Feedback

Too many companies fail to obtain crucial feedback. They throw a ball up in the air and walk away before it actually lands. We see this in software development when an organization releases a new quarterly product or feature and returns four months later; sadly, this approach doesn’t allow for much effectiveness tracking, or much learning, or much improvement.

Which is why every single action in the continuous transformation process needs to be an act of design. The act of design is an act of feedback. And, in continuous transformation, you’re continuously getting feedback—whether you’re shipping product externally to a customer or you’re making changes inside your organization.

The bottom line is that you need to continuously test your assumptions about the world, as well as your effectiveness in responding to those assumptions, if you want to keep up with continuously changing customers in the continuous economy of the 21st century.
May 30, 2016
Tales of DevOps Discovery Part 2: Definitions Matter
Definitions Matter: Step Two in a DevOps Transformation

The first post in this series on DevOps Discovery discussed the importance of asking fundamental questions of stakeholders to facilitate alignment on priorities and establish a mission statement when beginning a DevOps transformation. Part 2, below, reviews definitions of common DevOps-related terms and provides examples of how failure to agree on such terms can hinder your transformation.

The industry is full of terms such as continuous integration, agile, continuous delivery, DevOps, etc. The practices described by these terms often are defined differently by various industry “experts,” resulting in stakeholders in the software delivery process having different understandings of what constitutes these practices. Some will state they are already following the practices; others will state they do not. Some will believe the objectives unachievable, while others will be bullish on the prospects; differing interpretations blocking agreement on the organization’s current and target states. If stakeholders cannot agree on the current state (or the start point) and the target state (or the destination), it is impossible to “map a route” for the journey to DevOps.

For example, let’s look at how some define continuous delivery:

Continuous Delivery – Definition 1:

A software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing and releasing software faster and more frequently.

The approach helps reduce the cost, time and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

Source: Wikipedia

Continuous Delivery – Definition 2:

Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.

[Continuous Delivery Blog Post by Martin Fowler]

You may read the above and initially think there is no difference in the definitions but, note that the first definition addresses deployment of software with the statement: “A straightforward and repeatable deployment process is important for continuous delivery,” whereas the second definition addresses deployment of software with the statement: “Build software in such a way that the software can be released to production at any time.”

This distinction may seem trivial, but it is the often the source of confusion between the practices of continuous delivery and the practice that is commonly called continuous deployment. In contrast to continuous delivery, which states that every software build “could be released,” continuous deployment is loosely defined as “the practice of releasing every good build to users.”

When establishing your target state for the DevOps transformation a seemly slight confusion such as this can result in pushback from stakeholders concerned that continuous deployment is unachievable or too disruptive. These sentiments include:
- Business: “We cannot gather, spec and track our requirements at that rate.”
- Operations: “There is no way we can support daily delivery into production, ensure security, uptimes and perform our day-to-day work. It takes us an entire weekend and multiple attempts to deploy today.”
- Customers: “Our compliance team needs three months to review and approve every new version of our accounting system.”
The confusion and potential pushback exemplify the need for establishing baseline definitions and alignment at the start of a DevOps transformation, and doing so in the specific context of your organization. The importance of points such as how far it deploys varies with the organization—some organizations can support continuous deployment to production multiple times a week or even multiple times a day, such as Netflix does. Other organizations, such as providers of critical financial systems, may never pursue this cadence of deployment. Be sure to apply these industry practices pragmatically as fit for your organization, not dogmatically based on generic industry definitions.

Let’s look at a couple of definitions of DevOps:

DevOps – Definition 1:

“DevOps (development and operations) is an enterprise software development phrase used to mean a type of agile relationship between Development and IT Operations. The goal of DevOps is to change and improve the relationship by advocating better communication and collaboration between the two business units.”

[Webopedia]

DevOps Definition 2:

“DevOps (a clipped compound of “development” and “operations”) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably.”

[Wikipedia]

Like the example with continuous delivery, these two definitions also are very close. Yet on detailed observation, it is clear the first definition describes DevOps as being focused on relationships (i.e., culture and collaboration). The second definition extends this to include the “automating the process of software delivery and infrastructure changes,” giving consideration to tools and technical practices such as continuous delivery.

Does this matter? Yes. Often if the executive sponsors and key stakeholders do not account for the need to establish tools and technical practices and only focus on relationships, there is a high chance for unrealistic expectations for timeline, budgets and results. Conversely, if the need for changes in culture and process are not considered and the DevOps transformation focuses only on tools and technical practices, the efforts are also unlikely to yield the expected results. A DevOps transformation must account for the trinity of “People, Process and Tools.”

When considering DevOps, it objectives and the objective of its component practices it is useful to establish a clear idea of the relationships between theses practices. It is often useful to visualize them as below:

The DevOps Transformation Triangle

DevOps is the “What,” the target state as defined by a set of guiding principles. Agile development, continuous integration and continuous delivery are the “How,” a collection of process, practices and tools that together build the foundation for a sustainable DevOps implementation.

In the next section, we will review some of the initial discovery questions asked when beginning on a DevOps transformation, provide real-world examples of answers from various stakeholders in real organizations, and analyze those answer to better understand the journey to DevOps.
May 27, 2016
PTC: Using DevOps to Transform Applications
Large companies and monolithic applications are not usually associated with DevOps. But I recently discovered that some of the best DevOps operations have come from the largest organizations and most complex applications, and PTC is the latest example.

If you are not familiar with PTC, the company has a wide range of offerings for enterprises looking to improve their operations. Its solutions range from enterprise resource planning (ERP) to product lifecycle management (PLM) and Internet of Things (IoT) platforms as a service (PaaS). Its existing products started as on-premises solutions, and have expanded to modern solutions including IoT. But now there is a large demand to convert them to cloud-based offerings. And as PTC goes down this path, it needs to have a modern delivery chain to support it.

Enter Cloud Services

So PTC created its Cloud Services group to deliver cloud-based versions of its existing platforms to customers. Its cloud-based offerings are subject to all the same demands of any other modern software-as-a-service (SaaS) and PaaS offering—faster releases, more frequent updates and higher quality. After a few years of building the Cloud Services group, the company has proven it can automate and improve delivery of its traditionally on-prem platforms in the same fashion as poster-child DevOps environments.

The journey was not easy, and it did not start with DevOps as a goal. The goal was simply to deliver solutions to its customers. DevOps was the natural outcome following the challenges it faced. Some of the challenges were common for all development operations, but others were very unique:
1. Lots of variables: PTC has to support a wide range of customers across a wide range of deployments. PTC works with its customers on their varying needs of platform versions and also understands their varying requirements around software updates. And each potential variation has a unique snowflake configuration, which, unlike most cloud platforms that offer one production version to all users, its entire ecosystem of releases is actually a wide range of configurations and versions.
2. Data integrity: Data is important no matter what. But PTC works with customers that have had data in their on-premises solution for a long time, and that same data needs to make it to the cloud, securely, without a single flaw. That data is part of business-critical applications and ongoing business processes.
3. Educating users: As customers continue on their journey to the cloud, PTC is there to support the changes and enable them through customer success, continuous knowledge transfers and an endless online platform of PTC educational services.
So not only does it face technical challenges, it also is facing huge change control and people challenges. Before any software delivery can begin, there is a specific part of the Cloud Services group that is responsible for the necessary development to SaaS-ify the existing applications and build a gold master for each release.

The initial delivery chain was hand-to-hand combat: one solution, one customer, one release at a time. This required a lot of warm bodies to manage manual release processes and customers.

“We could not keep on throwing people at the problem,” notes Tameem Hourani, senior director of Cloud Services. Even though PTC had established workflows via ServiceNow, each step was manual and simply not scalable. But the customers and releases were increasing at an ever-climbing rate. It simply would not be feasible to on-board new team members to handle the load.

So it turned to good practices and tools to build a delivery chain that can take care of a brand-new deployment in less than two hours, instead of days, and upgrades in less than 10 minutes. Here is what the resulting setup looks like:

PTC was not new to automation. The company had been leveraging ITIL practices for some time. But what ITIL lacked was exposure to the customer. So with the ITIL backbone and ServiceNow in place, PTC added pipeline-level orchestration called Foreman.

In this configuration, both ServiceNow and Foreman essentially act as a system of record for the entire environment. ServiceNow is the system of record for all states and workflows. It also is the base entry point for any new update or deployment. And Foreman is the system of record for infrastructure and configurations across all customers and all deployments. But it also serves as the “checklist” of all the aspects of orchestration.

To manage the infrastructure orchestration and create the templates for configurations, PTC turned to Saltstack. It originally used another tool, but the benefit of Salt is that it allows for more sequential orchestration, as opposed to another tool with a stateful-only option.

PTC uses another unique tool called Intigua. Intigua peaked my interest because I have previously talked about the overhead in both resources and management of server agents. And because so much of what is done for headless monitoring and control of servers these days is done by agents, this actually becomes a serious management problem.

“The only way to grow a modern delivery chain is tools. And most new tooling is driven by agents,” Hourani notes. “It is easy to forget that the new tooling and processes also come at a cost. Usually the cost is less than the benefit they provide, but it is there nonetheless. Management of agents seems simple enough, but [when] actually doing this at scale, things can easily get out of hand.”

PTC had four-plus unique agents across a wide range of configurations. One of Intigua’s value points is the ability to “self-heal” and make sure the agents are always running correctly.

Once it is handed an IP, Intigua installs and configures agents based on tags, policies and AWS regions. When Intigua is running, it will make sure agents are up to date, and that self-heal agents go to work should something go wrong. The ability to use Intigua as a single pane of glass in a highly variable grid of configurations is a huge benefit.

Triggered by ServiceNow, configured by Foreman, Salt and Intigua, the infrastructure and code are deployed. But to learn and maintain all environments for the customer, PTC needed some strong production monitoring tools. It uses SumoLogic, deployed as an agent via Intigua, for log analysis. For most applications, using a single instance of Sumo is fine. But PTC has to segregate an instance per customer, and the process must be fully automated. There is some overlap with another monitoring tool, Zabbix, which has been set up to attempt self-healing upon a triggered alert.

And then there’s Jenkins. While not used for the entire deployment process, it is used to drive testing and manage states during the testing process. Upgrades to production releases are done incrementally, and complete deployments are done full-stack.

Services Layers Are All the Rage

Cloud services and shared services are not necessarily new. I recently did a write-up on Wix, which has a similar approach. But the complexities associated with SaaS-ifying on-premises solutions, along with the number of variables, adds not only another layer of complexity, but also another layer of value to a repeatable delivery chain.

I am humbled by case studies like these, because it proves that those who say their environments are too complex for DevOps have yet to see what’s possible.
May 25, 2016
Five Harsh Truths About DevOps

DevOps can provide tremendous productivity improvements, but it’s not a panacea. As much as we’d want it to be the solution for all of our software development problems, here are five truths you should consider.

1. DevOps may lead to juggling job roles and coming out of your comfort zone

With DevOps, the development team is responsible for writing the automated scripts that deploy the application. That used to be manual work done by someone in operations. Just as in manufacturing, automation will eliminate jobs. Typically, that system administrator can be retrained to work in development writing those deployment scripts, but programming is a different skill from restoring databases. If you can’t adapt, it’s time to update that resume.

2. Not everything should be automated

Yes, DevOps can be made to work on any platform, including the mainframe. No, it’s not confined to projects using agile. You don’t have to use Ruby or PHP; it’ll work for Fortran applications too. But be realistic. Transforming an application to use DevOps is expensive. You should not invest in something that is slated for sunset, or only has one release per year to fix some minor bugs. Legacy systems are considered legacy for a reason. Focus on the top 20 percent of your portfolio instead.

3. Developers will need to get their hands dirty

Some developers like to do one thing, and one thing only: write code. Grudgingly, they’ll test; they’ll attend team meetings if they have to; and, if tortured, they’ll write documentation. But forget about getting them to create the installation scripts.

That mindset is not going to fly in a DevOps world. Developers need to think about how the application will run as a nonfunctional requirement, and treat it with the same level of importance as security, performance and data privacy. Unless they feel ownership of the installation process, monitoring the environment and collecting end-user feedback, you have not really implemented DevOps.

4. Operations no longer can be ticket-driven

Usually, the operations team doesn’t get involved until someone from the development side opens a ticket for deployment. That’s too late for any kind of meaningful integration. The answer is not opening a ticket three months earlier; tickets are meant to be assigned, worked and closed within a short period of time. The type of collaboration needed between development and operations is far more complicated, and should be treated like it’s a project. That’s really difficult for most operations organizations. It’s not part of their culture, and they’re not staffed for that.

5. Your process is really, really bad

As soon as you start automating your software delivery process, you’ll find out exactly how inefficient it really is: duplicate reviews, inconsistent forms, competing standards … When you take away the people layer, you can’t hide the crazy anymore. You don’t want to take what you have today and force it into an automation tool. You will need to standardize before you optimize, and optimize before you automate. That means a lot of different towers will need to cooperate—something that won’t happen unless there is an outside party driving the cooperation with a big stick.

To learn more, check out the hangout with industry experts: What will DevOps look like in three to five years?

What aspect of DevOps do you think people are uncomfortable to talk about? Leave a comment below or contact me @baspluim

About the author / Jan S (Bas) Pluim

Bas Pluim is a Cloud/ DevOps solution architect in IBM’s innovation practice. He works with clients on infrastructure solutions, process re-engineering, automation tools and transforming the overall software delivery process. Bas is also a member of the IT Specialist certification board, a gaming enthusiast and an advocate of social media tools

May 24, 2016
‘DevOps is a Lifestyle, Not a Title’

“DevOps is a lifestyle, not a title.” This is a quote from Ron Melanson, of Amazon, and I hope he won’t mind me sharing it here. It struck a chord with me, and this blog explains why.

I have spoken about DevOps on a few occasions. Some audiences are not from a development background, so I explain the “why” of DevOps to them to help set context.

Chances are you are past the “why” stage and into the “how” stage. Even so, it is always useful to think about the “why” part, or even to revisit it—it is important to make sure you really are getting what you want from the DevOps journey.

In my DevOps presentations I vary between the transactional aspects of the “why” and the emotive aspects of the “why.” The transactional aspects are what the tooling can help with, and so that matters to a degree. However, since emotions influence us the most, especially from a lifestyle perspective, it is the emotional aspects that I want to focus on here.

The picture below captures, for me, the essence of why people care about DevOps: they want to be happy at work, and to sleep soundly at night. If you are not where you want to be with respect to those things, then you need to change the way you lead your life.

The DevOps lifestyle is a virtuous cycle

I highlight the “change the way you lead your life” aspect to illustrate contrasts among the differing approaches to DevOps that I have witnessed.

Now that DevOps has grabbed the popular imagination, it is quite common to see job advertisements for people to be part of a DevOps team.

One ex-colleague of mine, hired to lead a DevOps team, expressed some concern about whether that notion actually made sense. He saw his team off in a corner focused on tooling and processes, i.e. the transactional, while everyone else in the organization seemed to be doing what they always did.

When his team suggested a change to a process, or tried to explain how a tool could be adopted, they often were at a disadvantage. The people in development and operations, whose working practices would be affected, had not actually bought into the whole idea in practice, and/or simply did not have the bandwidth to think about it.

Consequently, with little or no time to pay attention to what the DevOps team were doing, there was no engagement from the people for whose supposed benefit the DevOps team was laboring. Thus, there was no driver for changes in behavior. They were all missing the emotive “why” part.

The challenge my colleague faced is common. Hiring someone to do DevOps does allow you to focus on getting people with the skills and experience for the transactional and technical aspects of DevOps. That is necessary, but it is far from sufficient.

If the benefits are to accrue to the people working in development and operations—and, indeed, the wider company and business they serve—then it is those people who have to own the idea, top to bottom.

Owning the idea means rethinking what you are doing, and why—in other words, changing your lifestyle. At least at work, anyway.

May 17, 2016