devopsy.kinsta.cloud

Tag: continuous integration

How DevOps is Killing QA
If you work in software quality assurance (QA), it’s time to find a new job.

The overall software development process has always consisted of the following core activities:
- Creating a requirements analysis;
- Producing a specification of the software artifact;
- Constructing the software;
- Investigating the quality of the product or service;
- Finding and fixing defects; then
- Deploying to production.
In most organizations, this process generally has occurred within one or more methodologies such as waterfall or agile. In waterfall, the testing process occurs relatively late. The software is produced, bugs are found by the QA team and then the software is fixed. These last two practices repeat in a cycle until management is comfortable that it can be released to the public.

In agile, individuals and teams—including QA—work closely together to release and then update software on an ongoing basis rather than deploy an entire platform all together and all at once. DevOps, as we have written in our “What is DevOps?” guide, is the next generation of agile development. Agile is a way of thinking in software development; DevOps goes one step further and is a concrete cultural change that implements the philosophy within an organization.

And that implementation eliminates the need for QA as a separate entity in the organization and disperses the responsibility for quality across different teams, even though many have argued that the discipline is still needed in one form or another.

Past Interpretations of QA in DevOps

One common argument is that DevOps sits at the center of a Venn diagram of dev, ops, and QA:

On one side, QA works together with development in trying to push more of their tests into the continuous integration system. Tests must have zero human intervention and generate their own test data.

On the other side, QA works with operations to collaborate on monitoring tools; perhaps to continuously run smoke tests in production. Chances are that operations is developing scripts for backup and restore of production systems, rollback of deploy, or scripts for disaster recovery.

Tim Hinds at NeoTYS looks at the issue differently and says that “DevOps QA” is meant to prevent defects rather than find them:

QA takes a critical role in this organizational structure because they have the visibility and the directive to push code out when it is working, and roll it back when it is not. This is a very different mindset from QA organizations of 10 years ago, whose primary responsibilities involved finding bugs. Today QA groups are charged with preventing defects from reaching the public site.

With all due respect, both are wrong.

Why DevOps Does Not Need QA

DevOps often uses continuous integration (CI) and continuous delivery (CD). In CI, developers use various continuous integration tools to integrate code into a shared repository multiple times each day, and DevOps relies on automation to determine the quality of the version. You cannot have any human interaction if you want to run CD, which ensures that any version of a batch of code in the repository can be released at any given time.

Essentially, the traditional QA cannot work in a full CI/CD environment. In older structures, the responsibility for software quality was in the hands of QA. Today, it’s part of the DevOps culture and methodology—the developers now own the responsibility rather than a separate entity within the organization.

Specifically, DevOps entails the use of vendors and tools such as BUGtrack, JIRA and GitHub to aggregate and report software bugs and defects. Test automation tools including Selenium, Cucumber, JUnit, TestNG and JMeter manage, execute and measure functional tests and cycles.

In the end, if there is a layer of people in the middle between development and operations, then you cannot perform CI and CD seamlessly and, therefore, cannot do DevOps. To run a proper DevOps operation, you cannot have QA at all.

The Future of QA

So, what will happen to all of the people who work in QA? One of the happiest jobs in the United States might not be happy for long as more and more organizations move to DevOps and they become more and more redundant.

According to the U.S. Bureau of Labor Statistics (BLS), software quality engineering is one of the high-tech professions whose growth rate is projected to be slower than average:

However, the BLS numbers are likely too generous because the bureau does not yet recognize “DevOps” as a separate profession at all:

As evidence, just look at how the relative number of Google searches in Google Trends for “sqa jobs” is slowly declining while the number for “devops jobs” is rapidly increasing:

For QA, the trend definitely does not look good.
June 7, 2016
Q&A: CircleCI’s Kevin Bell on Continuous Delivery

There’s been a lot of talk and momentum lately when it comes to continuous delivery. And it seems every enterprise has moved part, or all, of its development efforts to continuous delivery—or is at least talking about doing so. But there are many challenges such as security, dealing with legacy systems and internal politics associated with the move. With that in mind, we wanted to get a sense of where continuous delivery stands in the market today. And for that we turned to Kevin Bell of continuous integration and delivery platform maker CircleCI.

Kevin Bell has been a both developer and product manager of cloud-based software products, and is currently developer evangelist at CircleCI, where he gets to play with almost every deployment-related technology out there, hack a little Clojure and absorb the wisdom of very smart colleagues and customers. We thought it’d be a good idea to get his perspective on continuous delivery in the marketplace.

devopsy.kinsta.cloud: I wanted to get a sense from you where your customers are in the range of continuous integration and continuous delivery? What are you observing in the market?

Bell: I would say continuous integration is a hard prerequisite to continuous delivery. You need to have continuous integration first, and once you have that down, you can expand from that into continuous delivery. Maybe there are people that automate deployments with a minimum of testing in the middle, but you really want to have continuous integration set up.

Generally, I would say of our customers, on the order of half of the active customers, are deep into continuous delivery. There are certainly people that just use us to test and then do something else to deploy, and then there are those who aggressively, many times a day, deploy to production directly from CircleCI in a totally automated fashion. So we definitely have both extremes and then people that are experimenting in the middle on trying to figure out how to make that jump.

devopsy.kinsta.cloud: Have you seen many of these changes in the past year? And are many of the changes to continuous integration and deployment happening outside of Internet and SaaS (software-as-a-service) companies?

Bell: Lots of companies want to do it. So one change I’ve seen in the past year is that, not necessarily a change in the market, but a change in CircleCI’s world is that we have rolled out an enterprise version, which basically means a version of CircleCI that you can deploy behind your firewall. We are seeing a lot of interest in this among bigger companies. And we’re talking to a lot of different types of companies that are not SaaS companies, but are just any kind of company from apparel to finance institutions—just all manner of companies that take their internal software development practice very seriously, and very much want to do continuous delivery and have a modern software delivery methodology.

It’s kind of just becoming table stakes that you’re going to follow these kinds of practices today, even if you’re not a SaaS startup. It’s not just about software deployment, either. It’s also necessary to attract talent.

devopsy.kinsta.cloud: I would imagine, because we’re starting to see pilots and success stories, enterprises are seeing that it’s not just speed when it comes to feature release, but once the continuous delivery pipeline is up and running, it’s a more effective way to create and maintain software.

Bell: That’s definitely a big part of the promise of continuous delivery, and I think it delivers. Pardon the pun. It does let you get software features out faster. If you do end up releasing a bug, it lets you fix it faster. But I think some of the most emotional stories of success that I hear from people talking about the dark old days when they deployed at once every two weeks, or once every month, is just how risky that felt. And when they clicked the big deployment button in those days, how much was at stake and how many things could go wrong because it was such a big blob of software being deployed all at once. A big part of the idea of continuous delivery is that you don’t do those big bang software deployments. You’re always just putting out a little bit at a time, and keep everything in an equilibrium throughout the process.

devopsy.kinsta.cloud: These enterprises that you were talking to, what are some of the challenges or hurdles they have moving from legacy development to continuous deployment? What are the barriers they need to overcome to get to where they want to be?

Bell: There’s certainly various barriers, and it depends on the organization. Some organizations have a momentum built around a certain process. Maybe there’s political issues in an organization, maybe they have a new CIO that wants to change how they do things, but they have a lot of legacy software and processes that still operate in the old-mold paradigm that they need to update. So there are a lot of systematic challenges organizations run into. But there are definitely technical challenges, too. And I think just broadly to compare, for example, continuous integration and continuous delivery, I would say that continuous integration has been around longer now and has kind of matured a bit.

And a lot of the challenges have been uncovered within continuous integration when it relates to complicated and offbeat projects, whether that’s using things like mobile emulators like you can use on CircleCI to do iOS testing, or mobile device clouds, or browser testing, or all these things to kind of fill in the gaps, and test suites for big complicated software has sort of been explored and addressed to a large extent. But in continuous delivery there remain complicated, nontrivial systems that they’re deploying, not just little apps that go to Heroku or something. There’s this whole world of different policies and structures of software and systems that need to be continuously delivered. I think the industry and toolset makers are still exploring how to provide that.

devopsy.kinsta.cloud: Earlier you mentioned that if an organization wants to attract talent, they need to have these modern development processes in place. So I would imagine developers don’t want to get bogged down, in their view, in old methodologies and want to get aligned with companies that have these more modern practices as part of their career development?

Bell: That’s come up often at big companies. I would say that category of Fortune 500 companies, and not just software companies, they need to assure potential hires that they have modern software practices, and that even though they’re not a software company per se, that they value developers, and that they’re going to be a good place to work.

June 7, 2016
Tales of DevOps Discovery Part 2: Definitions Matter
Definitions Matter: Step Two in a DevOps Transformation

The first post in this series on DevOps Discovery discussed the importance of asking fundamental questions of stakeholders to facilitate alignment on priorities and establish a mission statement when beginning a DevOps transformation. Part 2, below, reviews definitions of common DevOps-related terms and provides examples of how failure to agree on such terms can hinder your transformation.

The industry is full of terms such as continuous integration, agile, continuous delivery, DevOps, etc. The practices described by these terms often are defined differently by various industry “experts,” resulting in stakeholders in the software delivery process having different understandings of what constitutes these practices. Some will state they are already following the practices; others will state they do not. Some will believe the objectives unachievable, while others will be bullish on the prospects; differing interpretations blocking agreement on the organization’s current and target states. If stakeholders cannot agree on the current state (or the start point) and the target state (or the destination), it is impossible to “map a route” for the journey to DevOps.

For example, let’s look at how some define continuous delivery:

Continuous Delivery – Definition 1:

A software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing and releasing software faster and more frequently.

The approach helps reduce the cost, time and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

Source: Wikipedia

Continuous Delivery – Definition 2:

Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.

[Continuous Delivery Blog Post by Martin Fowler]

You may read the above and initially think there is no difference in the definitions but, note that the first definition addresses deployment of software with the statement: “A straightforward and repeatable deployment process is important for continuous delivery,” whereas the second definition addresses deployment of software with the statement: “Build software in such a way that the software can be released to production at any time.”

This distinction may seem trivial, but it is the often the source of confusion between the practices of continuous delivery and the practice that is commonly called continuous deployment. In contrast to continuous delivery, which states that every software build “could be released,” continuous deployment is loosely defined as “the practice of releasing every good build to users.”

When establishing your target state for the DevOps transformation a seemly slight confusion such as this can result in pushback from stakeholders concerned that continuous deployment is unachievable or too disruptive. These sentiments include:
- Business: “We cannot gather, spec and track our requirements at that rate.”
- Operations: “There is no way we can support daily delivery into production, ensure security, uptimes and perform our day-to-day work. It takes us an entire weekend and multiple attempts to deploy today.”
- Customers: “Our compliance team needs three months to review and approve every new version of our accounting system.”
The confusion and potential pushback exemplify the need for establishing baseline definitions and alignment at the start of a DevOps transformation, and doing so in the specific context of your organization. The importance of points such as how far it deploys varies with the organization—some organizations can support continuous deployment to production multiple times a week or even multiple times a day, such as Netflix does. Other organizations, such as providers of critical financial systems, may never pursue this cadence of deployment. Be sure to apply these industry practices pragmatically as fit for your organization, not dogmatically based on generic industry definitions.

Let’s look at a couple of definitions of DevOps:

DevOps – Definition 1:

“DevOps (development and operations) is an enterprise software development phrase used to mean a type of agile relationship between Development and IT Operations. The goal of DevOps is to change and improve the relationship by advocating better communication and collaboration between the two business units.”

[Webopedia]

DevOps Definition 2:

“DevOps (a clipped compound of “development” and “operations”) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably.”

[Wikipedia]

Like the example with continuous delivery, these two definitions also are very close. Yet on detailed observation, it is clear the first definition describes DevOps as being focused on relationships (i.e., culture and collaboration). The second definition extends this to include the “automating the process of software delivery and infrastructure changes,” giving consideration to tools and technical practices such as continuous delivery.

Does this matter? Yes. Often if the executive sponsors and key stakeholders do not account for the need to establish tools and technical practices and only focus on relationships, there is a high chance for unrealistic expectations for timeline, budgets and results. Conversely, if the need for changes in culture and process are not considered and the DevOps transformation focuses only on tools and technical practices, the efforts are also unlikely to yield the expected results. A DevOps transformation must account for the trinity of “People, Process and Tools.”

When considering DevOps, it objectives and the objective of its component practices it is useful to establish a clear idea of the relationships between theses practices. It is often useful to visualize them as below:

The DevOps Transformation Triangle

DevOps is the “What,” the target state as defined by a set of guiding principles. Agile development, continuous integration and continuous delivery are the “How,” a collection of process, practices and tools that together build the foundation for a sustainable DevOps implementation.

In the next section, we will review some of the initial discovery questions asked when beginning on a DevOps transformation, provide real-world examples of answers from various stakeholders in real organizations, and analyze those answer to better understand the journey to DevOps.
May 27, 2016
Data Protection for Operational Reporting Platforms
Making real-time business decisions based on data from social, mobile and cloud platforms is a requirement in today’s world. That’s why an ecosystem of next-generation operational reporting platforms such as Spark, Apache Cassandra, Kafka, Docker, Mesos, Marathon and more have come to life. Enterprises are using these technologies to re-architect their data storage and processing frameworks to achieve lower costs, scalability and faster response times. However, as enterprises onboard business-critical applications on this next-generation technology stack, they need to prepare for failures or errors that may disrupt customer-facing applications.

One of the key benefits that operational reporting platforms provide is real-time visibility, which allows businesses to make strategic decisions based on actionable insights. However, the new applications generate a large amount of data and archaic operational reporting platforms (those based on traditional data warehouses, data marts and ETL logic) are unable to catch up and, as a result:
- Business decisions are based on stale data
- Fragmented data increases the risk of errors
- Multiple copies of data increase storage costs
These issues have driven many progressive, data-centric enterprises to re-architect their operational reporting platforms. In this new architecture, Kafka serves as a persistent message bus that ingests data from relational stores and provides failure resiliency and message buffering. Spark or Spark Streaming may be used for transformation, aggregation and other lightweight stream processing. These processing nodes may be deployed in Docker containers. Finally, Apache Cassandra serves as a distributed data storage layer that provides linear scalability and high-availability. Kafka also enables the ability to add multiple data consumers such as Hadoop/HBase through Flafka.

However, as enterprises adopt operational reporting platforms, they want to ensure that bad data feeds and human error do not result in data loss. Apache Cassandra natively replicates data but that does not provide the capability to go back in time to recover data efficiently and at scale.

As enterprises onboard their critical applications to achieve development agility, scalability and lower operating costs, they must also explore data protection solutions that are specifically dedicated for scale-out databases.
May 18, 2016
Faster, Smarter DevOps

Whether you call it DevOps or not, if you are concerned about releasing more code faster and at a higher quality, the resulting software delivery chain and process will look and smell like DevOps. But for existing development teams, no matter what the velocity objective is, getting from here to there is not something that can be done without a plan.

Moving your release cadence from months to weeks is not just about learning agile practices and getting some automation tools. It involves people, tooling and a transition plan. I will discuss some of the benefits and approaches to getting there.

Waterfall to agile, agile to continuous integration, continuous integration to continuous deployment. Whatever your processes are, the theme is the same: Find a way to get code to users faster without sacrificing quality. But speed and quality are sometimes at opposition to each other. Going faster means things can break faster, and when we only think about DevOps as releases, it’s easy to fall into this trap.

Established development shops cannot just jump from one flow to another. Unless you start out net new, the goal is to introduce new processes without delaying releases for three months or more to do transition in lump. This often is done using a pincer approach that addresses both bottom-up tactics and top-down oversight and culture at the same time.

However, because adopting DevOps tools is so easy, the trend is to focus on tactics only and adopt from the bottom up without consideration of the entire pipeline. The outcome is release automation tools that dictate your delivery chain for you, and not the other way around.

Structured Automation

DevOps requires automation. But what is often not considered is automation that sustains and fits into the entire delivery chain. Governance, artifacts organization and inventory, metrics and security must be considered. If an organization establishes a vetting process for all new automation and how it fits into the pipeline’s orchestration, then new automation will support what exists today and in the future.

For example, many organizations driving down the DevOps path have encountered challenges when trying to incorporate practices from security or governance teams. Historically, these teams have resided outside of the Dev and Ops echo chamber and their processes were asynchronously aligned to the work being done. The challenge for many organizations is to determine the best ways to bring the people, processes and technology supporting these initiatives into the fold without slowing things down. The best organizations are finding new ways to automate policies from security and governance teams by shifting away from artisanal, asynchronous approaches to synchronous processes earlier in the life cycle.

Let’s take a look at an example of application security. A number of technology vendors in the application security arena are touting “automation” as key value point for their solutions to better fit them into a DevOps tool chain. In some instances, automation means that machines are now fully responsible for monitoring, analyzing and fixing security vulnerabilities for those applications at wire-speed. In other instances, automation implies facilitation of human-centric workflows that might represent hours or days of asynchronous analysis not fit for continuous operations. In both cases, the technologies may accomplish similar ends, but their approaches could be dramatically different.

Also, one solution might be built to support asynchronous investigations by a security professional, while the other might provide synchronous support to a developer at the design and build stages of the software development life cycle. Establishing a vetting process can help determine if the automation levels required by a team or process can truly be delivered before investments are made. It is also worth noting that layers of obscurity frequently exist within words such as “automation,” “integration,” “configuration,” and “continuous.”

There are other areas that also tend to get overlooked in building a DevOps environment. The complete story can be found here.

May 12, 2016
Trust, You Must: 8 Tips for Adopting CI/CD

We have done continuous deployment (CD) for four years, and enable many of our customers to do continuous deployment with quality assurance (QA). We’ve helped tons of companies adopt and optimize their CD setups. We’ve found that continuous deployment is all about trust — augmented with processes and infrastructure.

To that point, here are some common tips for adopting continuous integration (CI)/CD that I find myself sharing frequently:

Use a CI Tool

This sounds obvious, but you’d be surprised at how many companies try and roll their own. Don’t. There’s no upside for your company—and infinite downside. We love Circle, but there are plenty of other great options. Most are at least somewhat prescriptive when it comes to workflow, which is important when you’re getting started.

Do Unit Testing

It’s always worth reminding everyone that continuous deployment is simply continuous integration applied to deployments. To do continuous integration, you need good unit test coverage. If you haven’t got a solid base of unit testing and continuous integration already, you’re not ready for continuous deployment.

Have Solid Production Monitoring

Bugs and regressions will get shipped; it’s inevitable. Put a system in place, by looking at the data in production, to tell you when a regression or bug was shipped. Tie this into automated rollbacks, so that if a significant performance or key performance indicator (KPI) regression (i.e., signups per minute) appears, your CD system rolls back to the most recent stable version while your team investigates.

Trust Your Team

We talked about the infrastructure requirements, but if you don’t have trust, then continuous deployment isn’t going to work for you. Trusting your team is about giving your developers the freedom to make mistakes, deploy as they see fit and check each others’ code. It is possible to combine continuous deployment with a culture of hierarchical permissions, but we have yet to see that truly work.

Remove Red Tape from the Code Review Process

Your team should review code changes, typically through a pull request, which goes hand in hand with trusting your team. Developers should be trusted to choose the most qualified and insightful person to review their code. One of the most common mistakes among teams trying to adopt continuous deployment is that they only let one or two “senior” developers review the rest of the team’s code, “bottlenecking” all releases. Don’t make this mistake.

Involve Developers Closely in Production Operations

The most common mistake we notice among companies that don’t make a successful transition to continuous deployment is that their development organizations are siloed. Dev and Ops should be joined at the hip. Have your developers deeply involved in the setup and definition of your CD infrastructure, and half the battle is already won.

Shift Left: Test Early and Often

Your team needs constant feedback. Think of your QA objective as getting the right feedback in the right hands at the right time. When it comes to deployments, this means knowing what’s working and what’s broken as the breaking change is made. The sooner you uncover the bug after it’s created, the easier it is to fix. The best companies doing continuous deployment have comprehensive unit test and integration test coverage that runs before any code hits the staging server.

Treat Staging as Your Holy Place

The final pre-production environment is usually known as ‘staging.’ A common mistake is to have developers ship new code direct to staging, while figuring out if the new code is ready to go live. This is a terrible idea. As soon as more than one developer is trying to ship their changes, your Slack channel will blow up with cussing and recriminations. Isolate the pre-production environments to each project, and enforce the “staging as ready to go live” philosophy. This way, integration testing can happen at a self-defined pace, without anyone else’s release being blocked.

About the Author / Fred Stevens-Smith

Fred Stevens-Smith is Co-founder and CEO of Rainforest QA, a quality assurance-as-a-service startup backed by Storm Ventures, Rincon, Andreessen Horowitz, Morado and Crosslink.

May 5, 2016
Optimizing an Effective CI/CD Pipeline
Continuous integration and continuous delivery (CI/CD) are often cited as pillars of successful DevOps. To establish and optimize the CI/CD development model and reap the benefits, companies need to build an effective pipeline to automate their build, integration and testing processes. This requires architecting CI/CD with five key ingredients which, when combined effectively, operate as a successful cornerstone of DevOps and provide myriad bottom-line benefits.

Building the Pipeline

Let’s first take a look at the five essential ingredients needed for a healthy CI/CD architecture, following up with a discussion of their benefits to the business:
- Micro-agility in the form of testing and committing code frequently: The development life cycle begins with a request to development, which then takes the request and designs a solution, codes it, tests it and commits it a code repository. From here, the code is tested by QA and then deployed to production. While each step is important, we begin here, as local testing occurs most frequently and bugs found at this stage of the process help avoid the large expense of bugs found in production.To achieve micro-agility, developers must have a clear understanding of where they are expected to commit their code after changes are made and they should be expected to test and update their code frequently—many times a day. This means development testing should be easy to run locally and is a subset of QA. The result is that code pushed by development is by definition QA-ready.
- Automated QA: In my work I see many organizations whose QA team is one of the largest bottlenecks in the process. It’s not for a lack of skill, but too few staff who spend a good portion of their time waiting on the QA test environments they need to do their work. To avoid this situation, the QA process should be fully automated, from QA ready code coming in the door to one-click promotion on the other side. In between the two, QA should have a constant feedback loop with development where communication about—and tracking of—bugs is clear and easy to follow, allowing development to easily repair issues and recommit the code for further testing. Moreover, automated QA in a CI/CD environment should focus on microcycles consisting of small test batches rather than weeklong macro tests. Automation and communication are the overriding imperatives here, exhibited by reports to developers that are communicated immediately and code commits that are fail-rejected automatically.
- Production-like environments: To increase efficiency and effectiveness, development and testing environments should mirror production as closely as possible. They should follow the PRQ rule, as I like to call it—production-like, repeatable and quick. Specifically, it should be fast and easy to set up new environments, to the point where developers can, with the push of a button, deploy new environments for their development and testing needs. Doing so allows developers to fail fast, breaking and re-creating environments without fear as they know they can easily recreate the same environment every time. I often recommend using tools such as Jenkins to help with this process; if developers and IT operations use the same tool, it makes it that much easier to streamline the CI/CD process and achieve the promise of DevOps.
- One-click promotion: As I alluded to earlier, a well-architected CI/CD pipeline provides one-click promotion to production. This ensures changes are seamlessly moved to the overall production environment with the least amount of friction possible between development and operations. To that end, this reduction in friction should occur across all the steps of the process, with efficient code delivery pipelines that are designed to balance security and agility from the outset. Cloud infrastructure coupled with container technology and automation can easily optimize the process across teams and throughout the DevOps journey.
- Fully automated deployments: The last factor of an efficient CI/CD pipeline is full automation of deployment. Any configuration changes made by development and accepted by QA should automatically be applied. Continuous integration in this way helps avoid system downtime and human errors are decreased greatly. Deployments also should be continuously monitored so that IT is the first to learn of potential problems. This helps nip issues in the bud, allowing for faster rollback and return of code to development to fix anything that may not be functioning properly. Catching issues before customers do helps ensure customer satisfaction and strength of reputation.
Benefits of the Pipeline

In addition to these benefits, when combined effectively, these ingredients minimize the time and cost of local test, QA and production processes; reduce the time it takes to identify problems and their causes; and accelerate delivery of production-ready code. We can see this throughout the CI/CD pipeline:
- Developers have greater control over the process and can respond more quickly to market demands by quickly bringing up new environments, testing against them and easily starting over, if need be. Moreover, developers can quickly adjust to new environments—an approach that has proven to decrease failures and resolution time when they do occur.
- Operations benefits as reduced friction in the CI/CD pipeline created by automation decreases repetitive, manual work and with it the opportunity for “fat finger” mistakes and the greater opportunity to introduce inefficiency into the process.
- And, less manual, repetitive work provides everyone in the process with more time for strategic work, which provides direct bottom-line value to the organization.
CI/CD is a pillar of successful DevOps and according to research by CA, organizations that apply DevOps discover an 18 percent reduction in time to market of new applications and a 21 percent increase in delivering new software that they would not otherwise be able to complete. Teams that master an efficient CI/CD pipeline deliver faster, higher-quality applications to market bringing with it a 19 percent increase in revenue. Mixing these ingredients together results in an efficient, streamlined CI/CD pipeline that serves as a fundamental pillar of DevOps success that, when mixed together, delivers direct business benefits.

About the Author/Aater Suleman

Aater Suleman, CEO & co-founder at Flux7, is an industry veteran in performance optimization on servers and distributed systems. He earned his PhD at the University of Texas at Austin, where he also currently teaches computer systems design and architecture. As a faculty member at UT-Austin, Mr. Suleman focuses on optimizing DevOps architectures involving the AWS cloud. Along with the Flux7 consultant team, he works at every step of the dev flow, from creating local developer environments using Vagrant and Docker, to continuous integration and continuous delivery using Jenkins or RunDeck, to configuration management using Chef/Puppet/Ansible, to creating efficient systems for deploying new code from Git using Capistrano/Fabric/bash. His background is as an architect, and he has experience with both hardware and software. This combination brings a unique perspective and broad view to development and operations.
April 15, 2016
Continuous Testing Live Network DevOps
My earlier blog, “From Laboratories to DevOps: Factories for Continuous Testing,” discussed the importance of and suggested practices for a fully automated, orchestrated software DevOps laboratory infrastructure, akin to a modern factory staffed by robots churning out products at a dizzy pace. The blog applies to development of software for networks, and the practices increasingly are important as the network becomes more software-based thanks to software-defined networking (SDN), network functions virtualization infrastructure (NFVI) and cloud-based networking. The edges of mobile networks, even the radio access networks, are becoming more software-dependent due to the need to position computing resources closer to mobile customers to resolve latency, bandwidth and spectrum concerns.

However, the need for highly automated software processes does not stop within the development lab.

At a recent DevOps Networking event in Santa Clara, California, Google Network Operations Manager Anees Shaikh indicated that Google’s live network has many thousands of nodes and management elements, supported by millions of lines of software-based configuration files that are updated tens of thousands of times per month. The sheer volume and speed of software updates for large networks is not feasible using manual methods. Automation tools for live network software updates are vital.

Typical live network DevOps environments use DevOps continuous tool chains such as Git for software version management, YAML for defining abstract configuration definitions, Jinja2 to translate the configurations into concrete templates that can be consumed by real devices and Ansible for automating the distribution and activation of config files to the network nodes. But what about testing? How can a network operator be confident that the many updates will perform as expected once deployed across the live network? This requires much more than simple syntax and semantics checking of the config file entries.

Recommended Practices for Continuous Testing of Live Network DevOps Environments

For continuous testing of live network DevOps environments to work, here are some best practices to consider:
1. Lab-to-live transition: Prior to deployment of any updates to the live network software updates should be tested in a lab environment using production-equivalent equipment and topologies.
2. What to test? The actual tests to be executed depend on the risk of changes that are being deployed. If a small, low-risk configuration change is all that is being deployed, then a simple configuration check may be sufficient. On the other hand, if a change has the potential to disrupt the network performance significantly, then a much more comprehensive performance test is needed.
3. Test orchestration tools: The test environment prior to deployment will vary depending on the tests that are selected. Test environment orchestration tools are needed to set up network test topologies that match the tests rapidly—and release them just as rapidly so the test resources can be used for the next tests.
4. Elastic on-demand topologies: The ability to scale test resources horizontally and vertically, on-demand, is essential to be able to match the speed of testing required for short duration deployment cycles.
5. Automate live network tests during deployment: To verify software updates do not interrupt or degrade network performance, functional and performance tests between pairs of nodes across the network mesh are required as the updates are deployed in stages across the live network nodes, clusters, centers and regions.
6. Automate live network testing after deployment: After verifying that the deployments are successful, network tests need to be scheduled periodically to ensure latent problems with the new software do not impact the network.
This is just a partial list of suggestions for continuous testing essentials for live network DevOps.
April 11, 2016
Does DevOps Improve or Undermine Quality?

In my earlier blog, “From QA to Continuous Testing,” I explained that “several industry developments converged to make continuous testing the winning method for software engineering quality assurance (QA).” The “combination of improved test methods, agile processes, continuous integration speed afforded by automation and orchestration tools and low-cost test dynamically configurable infrastructures afforded by virtualization has resulted in DevOps.”

However, a recent survey by Parasoft indicated that some important concerns such as non-functional requirements (NFRs) (examples are: reliability, testability, availability, resiliency) are being overlooked in many organizations that are operating fast-paced Agile and DevOps environments.

Some results gleaned from Parasoft’s survey run counter to popular expectations:
1. Agile teams are less likely to monitor NFRs than non-agile teams.
2. Ownership of quality, especially for agile teams, tends to be QA team members.
3. Business members think NFRs are being measured but technical folks do not.
4. Confusion and disagreements exist regarding who owns NFR objectives.

The report implies that quick Agile and DevOps processing cycles steal time from broader QA concerns.

The Question

So what is the truth? Do agile, DevOps and continuous testing improve QA, or do they undermine quality?

My own observation is that DevOps can improve or undermine QA. The outcome depends on whether an organization explicitly takes a holistic view of quality that assures all of the “ilities” are taken care of during agile, DevOps and continuous testing.

A complete DevOps environment requires a more sophisticated view of continuous testing than simply “shift left,” “fail early,” “fail often” and “test fast,” which are four popular tenets espoused by the DevOps community. A fifth tenet, “relevance,” must also be fully considered. With “relevance” as a tenet of DevOps QA, the entire DevOps and continuous testing pipeline has woven into it, from end to end, the concept that all of the quality aspects relevant to customers are emphasized in continuous testing and analysis of test results. When relevance is fully considered as an equal tenet to the others, then NFRs and “ilities” are not forgotten and the tools, goals and tests explicitly take into account all aspects relevant to customers’ perception of quality. What is more, when relevance is applied to continuous test selection and results analysis, the system not only produces better quality but it also is more effective and efficient.

At Spirent we think that a holistic approach to DevOps and continuous testing, including “relevance,” is critical for successful DevOps. When continuous testing is implemented with relevance as an equal tenet, quality is improved.

You can read more about our views at Spirent.com/solutions/DevOps

What are your experiences and suggestions?

April 4, 2016
Evolving to Continuous Testing
As agile development practices mature and DevOps principles infiltrate our corporate cultures, organizations are realizing the distinct opportunity to accelerate software delivery. However, when you speed up any process, immature practice areas such as testing and roadblocks become much more pronounced. It’s the difference between driving over a speed bump at 5 mph vs. 50 mph … at 50 mph, that speed bump is going to be quite jarring.

Accelerating any business process will expose systemic constraints that shackle the entire organization to its slowest moving component. In the case of the accelerated software development life cycle (SDLC), testing has become the most significant barrier to taking full advantage of more iterative approaches to software development. For organizations to leverage these transformative development strategies, they must shift from test automation to continuous testing.

Drawing a distinction between test automation and continuous testing may seem like an exercise in semantics, but the gap between automating functional tests and executing a continuous testing process is substantial. This gap will be bridged over time as the process of delivering software matures. Both internal and external influences will drive the evolution of continuous testing. Internally, agile, DevOps and lean process initiatives will be the main drivers that generate the demand for change. Externally, the expense and overhead of auditing government and industry-based compliance programs will be the primary impetus for change.

A Roadmap for Testing

Any true change initiative requires the alignment of people, process and technology—with technology being an enabler and not the silver bullet. Yet there are some basic technology themes we must explore as we migrate to a true quality assurance process. In general, we must shift from a sole focus on test automation to automating the process of measuring risk. To begin this journey, we must consider the following:

Driven by business objectives, organizations must shift to more automated methods of quality assurance and away from the tactical task of testing software from the bottom up.

From Causal Observations to Probabilistic Risk Assessment

With quality assurance (QA) traditionally executing manual or automated tests, the feedback from the testing effort is focused on the event of a test passing or failing—this is not enough. Tests are causal, meaning that tests are constructed to validate a very specific scope of functionality and are evaluated as isolated data points. Although these standalone data points are critical, we must also use them as inputs to an expanded equation for statistically identifying application hot spots.

The SDLC produces a significant amount of data that is rather simple to correlate. Monitoring process patterns can produce very actionable results. For example, a code review should be triggered if an application component experiences all of the following issues in a given continuous integration build:
- Regression failures greater than the average
- Static analysis defect density greater than the average
- Cyclomatic complexity greater than a prescribed threshold
From Defect Documentation to Simulated Replay

The ping-pong between testers and developers over the reproducibility of a reported defect has become legendary. It’s harder to return a defect to development than it is to send back an entrée from a world-renowned chef. Given the aggressive goal to accelerate software release cycles, most organizations will save a significant amount of time by just eliminating this back and forth.

By leveraging Service Virtualization for simulating a test environment and/or virtual machine record and playback technologies for observing how a program executed, testers should be able to ship development a very specific test and environment instance in a simple containerized package. This package should isolate a defect by encapsulating it with a test, as well as give developers the framework required to verify the fix.

From Structured Data to Structured and Unstructured

The current tools and infrastructure systems used to manage the SDLC have made significant improvements in the generation and integration of structured data (e.g., how CI engines import and present test results). This data is valuable and must be leveraged much more effectively (as we stated above in the “From Causal Observations to Probabilistic” section.

The wealth of unstructured quality data scattered across both internal and publicly-accessible applications often holds the secrets that make the difference between happy end users and unhappy prospects using a competitor’s product. For example, developers of a mobile application would want constant feedback on trends from end user comments on:
- iTunes app store
- Android app store
- Stackoverflow
- Twitter
- Facebook
- The company’s release announcements
- Competitors’ release announcements
This data is considered unstructured since the critical findings are not presented in a canonical format: parsing and secondary analysis are required to extract the valuable information. Although these inputs might be monitored by product marketers or managers, providing these data points directly to development and testing teams—in terms that practitioners can take action on—is imperative.

From Dashboards to Business Policies

In a Continuous Everything world, quality gates will enable a release candidate to be promoted through the delivery pipeline. Anything that requires human validation clogs the pipeline. Dashboards require human interpretation—delaying the process.

Dashboards are very convenient for aggregating data, providing historical perspectives on repetitive data, and visualizing information. However, they are too cumbersome for real-time decision making because they do not offer actionable intelligence.

Business policies help organizations evolve from dashboards to automated decision making. By defining and automatically monitoring policies that determine whether the release candidate is satisfying business expectations, quality gates will stop high-risk candidates from reaching the end user. This is key for mitigating the risks inherent in rapid and fully-automated delivery processes such as Continuous Delivery.

From Tool Dependent to SDLC Sensors

Let’s face it—it’s cheap to run tools. And with the availability of process intelligence engines, the more data observations we can collect across the SDLC, the more opportunities will emerge to discover defect prevention patterns.

Given the benefit of a large and diverse tool set, we need to shift focus from depending on a single “suite” of tools from a specific vendor (with a specific set of strengths and weaknesses) to having a broad array of SDLC sensors scattered across the software development life cycle. And to optimize both the accuracy and value of these sensors, it’s critical to stop allowing tools to be applied in the ad hoc manner that is still extremely common today. Rather, we need to ensure they are applied consistently and that their observations are funneled into a process intelligence engine, where they can be correlated with other observations across tools, across test runs and over time. This will not only increase the likelihood of identifying application hot spots, but will also decrease the risk of false negatives.

About the Author/Wayne Ariola

Wayne Ariola, Chief Strategy Officer, leads the development and execution of Parasoft’s long-term strategy. He leverages customer input and fosters partnerships with industry leaders to ensure that Parasoft solutions continuously evolve to support the ever-changing complexities of real-world business processes and systems. Ariola has contributed to the design of core Parasoft technologies and has been awarded several patents for his inventions. A recognized leader on topics such as service virtualization, SOA and API quality, quality policy governance, and application security, Ariola is a frequent contributor to publications including software test & performance, SOA World, and ebizQ—as well as a sought-after speaker at key industry events. Ariola brings more than 20 years strategic consulting experience within the technology and software development industries.
March 29, 2016